From pnorvig at google.com Thu Apr 1 00:00:55 2004 From: pnorvig at google.com (Peter Norvig) Date: Thu Apr 1 00:00:59 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon Message-ID: <42B02151.420DCBB8@mail.google.com> I have to say I favor the "last before colon" approach, but if it has to be before the def, then I think it should have a keyword, and if you don't want to introduce a new keyword, then it looks like "is" is the only reasonable candidate. And if you do have a keyword, you don't need the square brackets. So you have is: classmethod def f(x): pass is: author("Guido"), signature(int, result=None) def g(x): pass From jack at performancedrivers.com Thu Apr 1 00:27:55 2004 From: jack at performancedrivers.com (Jack Diederich) Date: Thu Apr 1 00:28:01 2004 Subject: [Python-Dev] PEP 318: Decorators last before colon In-Reply-To: <1080790860.21196.23.camel@geddy.wooz.org> References: <1080674306.12643.76.camel@localhost.localdomain> <200403302121.i2ULLIM09840@guido.python.org> <1080790860.21196.23.camel@geddy.wooz.org> Message-ID: <20040401052755.GA18083@performancedrivers.com> On Wed, Mar 31, 2004 at 10:41:00PM -0500, Barry Warsaw wrote: > On Tue, 2004-03-30 at 16:21, Guido van Rossum wrote: > > > Another possibility that has been suggested is > > > > > > [decorator] > > > def func(arg, arg): > > > > And one that I currently favor. I'm out of bandwidth to participate > > on a msg-by-msg basis, but perhaps folks can see if they can come to > > terms with this solution? > > I don't like it. It already has a meaning (albeit fairly useless) and > it doesn't seem obvious from just looking at it that the decorator is > connected to the following method. It doesn't taste Pythonic to me. > Me too, but no one contrasted my actual use cases with any in any other format, so I must be missing something fundamental. I support an all Guido ticket now and forever, but I don't get the pulling-the-rabbit-out-of-a-hat decorator syntax anymore than the at-sign assignment. Something in the water at PyCon? -jackdied From guido at python.org Thu Apr 1 00:44:44 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 00:44:53 2004 Subject: [Python-Dev] PEP 318 bake-off? In-Reply-To: Your message of "Wed, 31 Mar 2004 21:00:55 PST." <42B02151.420DCBB8@mail.google.com> References: <42B02151.420DCBB8@mail.google.com> Message-ID: <200404010544.i315iiB15938@guido.python.org> Despite some positive responses, my proposal (which was originally due to Jim Hugunin, who probably has more C# experience than all of us arguing together) is getting some tough opposition. I'd like to have a bake-off, where we view a serious amount of code using decorators to the hilt with each of three proposed syntaxes: 1) Last-before-colon: def foo(cls, arg1, arg2) [funcattrs(foo=42), deprecated, overrides, classmethod]: pass 2) Prefix list: [funcattrs(foo=42), deprecated, overrides, classmethod] def foo(cls, arg1, arg2): pass 2a) Prefix list with leading *: *[funcattrs(foo=42), deprecated, overrides, classmethod] def foo(cls, arg1, arg2): pass 3) Prefix suite (could use a different keyword than 'decorate'): decorate: funcattrs(foo=42) deprecated overrides classmethod def foo(cls, arg1, arg2): pass None of the other positions between def and arglist are acceptable any more; a keyword after the argument list isn't acceptable; I don't think any syntax that puts the decorators inside the body (between the colon and the docstring) will work. So this is the field. What I'm asking (especially of Phillip) is to collect a set of realistic method declarations using decorators; we can then collectively format these using any of the possible syntaxes, and see how they look. We can also review different ways of spreading multiple decorators across several lines, e.g. [funcattrs(foo=42), deprecated, overrides, classmethod] def foo(cls, arg1, arg2): pass --Guido van Rossum (home page: http://www.python.org/~guido/) From tanzer at swing.co.at Thu Apr 1 01:46:47 2004 From: tanzer at swing.co.at (Christian Tanzer) Date: Thu Apr 1 01:48:44 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Wed, 31 Mar 2004 20:42:39 MDT." <16491.33183.102141.327504@montanaro.dyndns.org> Message-ID: > On the advisability of > > [decorator] > def func(): > pass > > vs. > > def func() [decorator]: > pass > > I'm beginning to think Guido has staged an elaborate April Fool's joke. > > +1 -- Christian Tanzer http://www.c-tanzer.at/ From paul at prescod.net Thu Apr 1 02:11:20 2004 From: paul at prescod.net (Paul Prescod) Date: Thu Apr 1 02:13:37 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200403312332.i2VNWpe15013@guido.python.org> References: <1080674306.12643.76.camel@localhost.localdomain> <200403302121.i2ULLIM09840@guido.python.org> <2CF5B513-82EB-11D8-BD7A-0003934AD54A@chello.se> <200403311542.i2VFgWf13002@guido.python.org> <406B4FD2.1040506@prescod.net> <200403312332.i2VNWpe15013@guido.python.org> Message-ID: <406BC098.7040209@prescod.net> Guido van Rossum wrote: >>>Why does <...> look better than [...]? To me, <...> just reminds me >>>of XML, which is totally the wrong association. >> >>I vote for << >>. The brackets and parens have too many meanings in >>Python already. <> looks more like French >>than XML. ;) > > > <<...>> has the same practical problems as <...>: no automatic line > breaking, >> is ambiguous. Sorry. I forgot that ">>" is also an operator. ;) Anyhow, I just meant that if you find a pair of characters that are illegal today then many of the objections about abusing existing syntax would go away. Paul Prescod From pyth at devel.trillke.net Thu Apr 1 03:10:02 2004 From: pyth at devel.trillke.net (Holger Krekel) Date: Thu Apr 1 03:10:39 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <8CF77A2E-8356-11D8-8201-000A95686CD8@redivi.com> References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331203701.GC6361@solar.trillke> <8CF77A2E-8356-11D8-8201-000A95686CD8@redivi.com> Message-ID: <20040401081002.GD6361@solar.trillke> Bob Ippolito wrote: > On Mar 31, 2004, at 3:37 PM, Holger Krekel wrote: > >Specifically, in PyPy we use something like > > > > def add_List_List(self, w_list1, w_list2): > > ... > > def add_Int_Int(self, w_list1, w_list2): > > ... > > > >and later on just one: > > > > register_all(vars(), W_ListType) > > > >(which will register the methods to some multimethod dispatcher). It's > >a matter of taste i guess if using decorators - whatever syntax - at > >each > >function definition would be more readable. > > Would you have even considered writing something as crazy as the > aforementioned example if decorators were already in Python? I doubt > it. Yes, we might have have used some decorator syntax. But then again, we also might use full Macros if they were available. The question (to me) is if there are enough use cases to warrant a new special decorator syntax. At least consider that Python's battery library only has ~10 occurences of 'classmethod' and no 'staticmethod' at all. And let me add that IMO decorators generally make things harder to read because you have to wonder what magic takes place in those decorators. IOW, i am not sure that seeing decorators occurring everywhere (because they become so convenient to write) will improve readability and debuggability of a python program. > >for example you could try to use the first line of the docstring > >for it - maybe building on the typical way to document C-defined python > >functions. e.g.: > > > > "(some_type_signature) -> (some_return_type)" > > > >I am not saying straight away this is better but i doubt that the > >only good solution to the above problem is to add new syntax. > > Mangling *doc* strings is really stupid and is not even an option, > IMHO. The only reason people do this is because we don't have a > decorator syntax. Well, the above is (supposed to be) used for documenting the signature of C-functions. Read "Documentation strings" in http://www.python.org/peps/pep-0007.html Anyway, I just happen to think that adding decorator syntax is not the only viable solution to all problems reported so far. > >>If decorators do not make > >>Python 2.4, that is another major release cycle that extensions such > >>as > >>PyObjC and ctypes will be hampered by lack of syntax... to the point > >>where I'd be inclined to just fork Python (or Stackless, more likely) > >>in order to get the syntax. > > > >Nah, just work with us on PyPy to allow it to add decorator syntax > >at runtime :-) > > That'd be an option, if I could wait a few years until it's capable > enough to do everything it needs to do. I am not sure it will take that long but i see your point :-) cheers, holger From mwh at python.net Thu Apr 1 07:13:40 2004 From: mwh at python.net (Michael Hudson) Date: Thu Apr 1 07:13:47 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <20040331234312.GA14993@mems-exchange.org> (Neil Schemenauer's message of "Wed, 31 Mar 2004 18:43:12 -0500") References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> Message-ID: <2mekr7hp57.fsf@starship.python.net> Neil Schemenauer writes: > On Wed, Mar 31, 2004 at 03:04:18PM -0500, Bob Ippolito wrote: >> On Mar 31, 2004, at 1:59 PM, Michel Pelletier wrote: >> >Please don't add any decorator syntax to Python, at least not yet. All >> >of the proposals I have seen so far are, to be blunt, and in my opinion >> >of course, ugly and are getting uglier as the discussion ensues. > > I agree with Michel. The decorator syntax being discussed looks > ugly. I think it would be okay if the set of valid decorations were > limited to 'classmethod', 'staticmethod' and maybe a few others. > Allowing more general expressions seems to asking for abuse. I humbly submit that that's a terrible idea: it knackers Bob's PyObjC example, for one thing. >> Decorators solve a *huge* problem with the current syntax: >> >> def someObjectiveCSelector_yesTheyCanBeThisLong_sometimesLonger_(takes, >> some, args, here): >> pass >> someObjectiveCSelector_yesTheyCanBeThisLong_sometimesLonger_ = >> objc.selector(someObjectiveCSelector_yesTheyCanBeThisLong_sometimesLonge >> r_, signature='some type signature') > > I would be happer if there was an easier way for you to do what you > want _without_ introducing new syntax to that language. For > example, what if '_' was bound to the last defined function? You > could then do something like this: > > def someObjectiveCSelector_itsReallyLong_(takes, some, args, here): > pass > objc.selector(_, signature='some type signature') > > That looks pretty nice and is even shorter to type than the proposed > syntax. And doesn't work. Oh well! >> Please understand that just because you haven't need them yet doesn't >> make them worthless, ugly, etc. > > I don't think Michel is saying they are worthless. However, the > proposed syntax is highly contentious. It would be good if there > was a short term solution that wouldn't require new syntax. That > would give Guido and the Python community time to figure out the > best syntax. We've been discussing this off and on for OVER A YEAR! If 'the best syntax' hasn't been figured out yet after N thousand emails on the subject, I see no reason to believe enlightenment is going to arrive soon (or ever). Cheers, mwh -- If trees could scream, would we be so cavalier about cutting them down? We might, if they screamed all the time, for no good reason. -- Jack Handey From FBatista at uniFON.com.ar Thu Apr 1 08:24:51 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 1 08:26:53 2004 Subject: [Python-Dev] Expert floats Message-ID: Greg Ewing wrote: #- > Decimal floating-point has almost all the pitfalls of binary #- > floating-point, yet I do not see anyone arguing against decimal #- > floating-point on the basis that it makes the pitfalls less #- > apparent. #- #- But they're not the pitfalls at issue here. The pitfalls at #- issue are the ones due to binary floating point behaving #- *differently* from decimal floating point. #- #- Most people's mental model of arithmetic, including floating #- point, works in decimal. They can reason about it based on This is a problem for future, outside of this world, portability. When an alien, with, say, twelve fingers in each hand, uses Python? The standard fp may be will still be the binary one, so they can make a Twodecimal.py (I damn my english). :) . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040401/e1118d43/attachment.html From skip at pobox.com Thu Apr 1 08:43:22 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Apr 1 08:43:36 2004 Subject: [Python-Dev] PEP 318: Decorators last before colon In-Reply-To: <1080790860.21196.23.camel@geddy.wooz.org> References: <1080674306.12643.76.camel@localhost.localdomain> <200403302121.i2ULLIM09840@guido.python.org> <1080790860.21196.23.camel@geddy.wooz.org> Message-ID: <16492.7290.766907.377725@montanaro.dyndns.org> >> > [decorator] >> > def func(arg, arg): Barry> I don't like it. It already has a meaning (albeit fairly Barry> useless) and it doesn't seem obvious from just looking at it that Barry> the decorator is connected to the following method. It doesn't Barry> taste Pythonic to me. I'm with Barry. It seems magic to me. If I write def f(): "doc" [47] return 3 today, although a bit weird, [47] doesn't affect the following statement in any way. Now the proposal (still, assuming this isn't an elaborate AFJ) on the table means to change the semantics of an unassigned list expression in one special case. I suggest: Special cases aren't special enough to break the rules. though I know someone will follow with Although practicality beats purity. IN this case there are other practical proposals on the table, not the least of which is the status quo. Skip From skip at pobox.com Thu Apr 1 08:46:27 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Apr 1 08:46:48 2004 Subject: [Python-Dev] PEP 318: Decorators last before colon In-Reply-To: <1080794825.22892.113.camel@localhost.localdomain> References: <200404010430.i314UGCj017063@cosc353.cosc.canterbury.ac.nz> <1080794825.22892.113.camel@localhost.localdomain> Message-ID: <16492.7475.436062.172933@montanaro.dyndns.org> Jeremy> PS Unspeakably horrible is a minor objection on the Internet. Jeremy> It's not really bad unless its untype-ably bad. I take it the test for "untype-ably bad" is you check to see if you have to wash your keyboard (or buy a new one) after typing such a construct. Skip From bob at redivi.com Thu Apr 1 09:53:20 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Apr 1 09:49:18 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <20040401081002.GD6361@solar.trillke> References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331203701.GC6361@solar.trillke> <8CF77A2E-8356-11D8-8201-000A95686CD8@redivi.com> <20040401081002.GD6361@solar.trillke> Message-ID: <50F44CDC-83EC-11D8-8201-000A95686CD8@redivi.com> On Apr 1, 2004, at 3:10 AM, Holger Krekel wrote: > Bob Ippolito wrote: >> On Mar 31, 2004, at 3:37 PM, Holger Krekel wrote: >>> Specifically, in PyPy we use something like >>> >>> def add_List_List(self, w_list1, w_list2): >>> ... >>> def add_Int_Int(self, w_list1, w_list2): >>> ... >>> >>> and later on just one: >>> >>> register_all(vars(), W_ListType) >>> >>> (which will register the methods to some multimethod dispatcher). >>> It's >>> a matter of taste i guess if using decorators - whatever syntax - at >>> each >>> function definition would be more readable. >> >> Would you have even considered writing something as crazy as the >> aforementioned example if decorators were already in Python? I doubt >> it. > > Yes, we might have have used some decorator syntax. But then again, > we also might use full Macros if they were available. The question > (to me) is if there are enough use cases to warrant a new special > decorator syntax. Well, I would be happy with full macros too.. but I'm not crazy enough to propose adding them :) > At least consider that Python's battery library only has ~10 > occurences of 'classmethod' and no 'staticmethod' at all. That's a bogus argument, using decorators is currently such a PITA that people avoid them, and not much of the standard library was written after the release of 2.2. > And let me add that IMO decorators generally make things harder to > read because you have to wonder what magic takes place in those > decorators. IOW, i am not sure that seeing decorators occurring > everywhere (because they become so convenient to write) will improve > readability and debuggability of a python program. It's not really about readability and debuggability, this is a case where the information is absolutely necessary for the program to work at all.. the information needs to be there, right up against that def, or else it's not readable and hard to debug. Putting it in a doc string and having a metaclass understand how to parse it is not more readable and is harder to debug. >>> for example you could try to use the first line of the docstring >>> for it - maybe building on the typical way to document C-defined >>> python >>> functions. e.g.: >>> >>> "(some_type_signature) -> (some_return_type)" >>> >>> I am not saying straight away this is better but i doubt that the >>> only good solution to the above problem is to add new syntax. >> >> Mangling *doc* strings is really stupid and is not even an option, >> IMHO. The only reason people do this is because we don't have a >> decorator syntax. > > Well, the above is (supposed to be) used for documenting the signature > of C-functions. Read "Documentation strings" in I am *not talking about documentation*! This is *code*! It tells the bridge how to mangle the input and output of the method when it is crossing the Objective C <-> Python bridge! Without these selectors, the code does not work at all, because it needs to convert objects into C types (int, float, struct, etc.) and vice versa in many cases.. sometimes even worse (pointers, yay!). -bob From barry at python.org Thu Apr 1 10:19:53 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 1 10:19:59 2004 Subject: [Python-Dev] PEP 318: Decorators last before colon In-Reply-To: <16492.7290.766907.377725@montanaro.dyndns.org> References: <1080674306.12643.76.camel@localhost.localdomain> <200403302121.i2ULLIM09840@guido.python.org> <1080790860.21196.23.camel@geddy.wooz.org> <16492.7290.766907.377725@montanaro.dyndns.org> Message-ID: <1080832792.1021.82.camel@anthem.wooz.org> On Thu, 2004-04-01 at 08:43, Skip Montanaro wrote: > IN this case there are other practical proposals on the table, not the least > of which is the status quo. So I hacked up python-mode a bit to support the syntax coloring of Guido's previously MFS (most favored syntax). I wanted to see if the concerns about visual obscurity were real or not. Then I rewrote a few methods of mine that would benefit from decorators (with some elaboration). A screen shot of my XEmacs buffer is here: http://barry.warsaw.us/xemacs.png Now, I'm sure most of you will hate my color scheme, but I'm used to it so for me, the decorator stands out perfectly fine. I'd have no problem supporting decorator-before-colon and vastly prefer it to decorator-before-def. You can play with this yourself by checking out the latest version of python-mode.el (from sf.net/projects/python-mode). Note that you won't get the nice colorization on multiline decorators on the fly because XEmacs doesn't handle as-you-type font-locking across newlines well. Re-fontifying the buffer will pick up the right colors though. -Barry From jeremy at alum.mit.edu Thu Apr 1 10:24:02 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu Apr 1 10:27:59 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <2mekr7hp57.fsf@starship.python.net> References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> Message-ID: <1080833042.22892.122.camel@localhost.localdomain> On Thu, 2004-04-01 at 07:13, Michael Hudson wrote: > > I don't think Michel is saying they are worthless. However, the > > proposed syntax is highly contentious. It would be good if there > > was a short term solution that wouldn't require new syntax. That > > would give Guido and the Python community time to figure out the > > best syntax. > > We've been discussing this off and on for OVER A YEAR! If 'the best > syntax' hasn't been figured out yet after N thousand emails on the > subject, I see no reason to believe enlightenment is going to arrive > soon (or ever). There's no particular reason to believe that effort alone will arrive at an elegant solution. On the other hand, maybe there isn't a good syntax for arbitrary decorators. Jeremy From guido at python.org Thu Apr 1 10:44:39 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 10:44:45 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Wed, 31 Mar 2004 23:11:20 PST." <406BC098.7040209@prescod.net> References: <1080674306.12643.76.camel@localhost.localdomain> <200403302121.i2ULLIM09840@guido.python.org> <2CF5B513-82EB-11D8-BD7A-0003934AD54A@chello.se> <200403311542.i2VFgWf13002@guido.python.org> <406B4FD2.1040506@prescod.net> <200403312332.i2VNWpe15013@guido.python.org> <406BC098.7040209@prescod.net> Message-ID: <200404011544.i31Fiea17172@guido.python.org> > Anyhow, I just meant that if you find a pair of characters that are > illegal today then many of the objections about abusing existing syntax > would go away. So how about *[...]* ? Or perhaps [*...*]? Or [|...|]? --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Thu Apr 1 10:56:40 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 1 10:56:49 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404011544.i31Fiea17172@guido.python.org> References: <1080674306.12643.76.camel@localhost.localdomain> <200403302121.i2ULLIM09840@guido.python.org> <2CF5B513-82EB-11D8-BD7A-0003934AD54A@chello.se> <200403311542.i2VFgWf13002@guido.python.org> <406B4FD2.1040506@prescod.net> <200403312332.i2VNWpe15013@guido.python.org> <406BC098.7040209@prescod.net> <200404011544.i31Fiea17172@guido.python.org> Message-ID: <1080834999.1021.119.camel@anthem.wooz.org> On Thu, 2004-04-01 at 10:44, Guido van Rossum wrote: > > Anyhow, I just meant that if you find a pair of characters that are > > illegal today then many of the objections about abusing existing syntax > > would go away. > > So how about *[...]* ? Or perhaps [*...*]? Or [|...|]? I'm -0 on whether it's necessary, but of these choices the latter seems okay. -Barry From tjreedy at udel.edu Thu Apr 1 11:16:31 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Thu Apr 1 11:16:30 2004 Subject: [Python-Dev] Re: Re: PEP 318: Decorators last before colon References: <42B02151.420DCBB8@mail.google.com> Message-ID: "Peter Norvig" wrote in message news:42B02151.420DCBB8@mail.google.com... > I have to say I favor the "last before colon" approach, The way I see it today, we are trying to select syntax to define something like a 'function metatype' (FMT) in order to get non-standard function-like objects. The idea and result strike me as similar (but *not* identical) to running the pieces of a would-be class thru a metaclass. If FMTs were required to be defined and named before use, then [FMTidentifier] would be analogous to __metaclass__ = MCidentifier after a class statement. While a metaclass can be defined in-place, over multiple lines, this does not work nearly so well for an anonymous FMT. (Has any consideration been given to an actual metafunc mechanism more directly analogous to metaclasses, that would be given the *pieces* of a would-be function (name, param names, default args, code body, etc), so that there would not necessarily ever be a standard function object?) > but if it has to be before the def, then I think it should have a keyword, Yes. My support for 'as' was based on misremembering its quasi status. > and if > you don't want to introduce a new keyword, then it looks like "is" is > the only reasonable candidate. And if you do have a keyword, you > don't need the square brackets. Since the sequence is not going to be mutated, I would see them as a positive distraction. > So you have > > is: classmethod > def f(x): > pass > > > is: author("Guido"), signature(int, result=None) > def g(x): > pass For this usage, 'is' is semantically better than 'as' anyway. Terry J. Reedy From s.percivall at chello.se Thu Apr 1 11:24:25 2004 From: s.percivall at chello.se (Simon Percivall) Date: Thu Apr 1 11:24:30 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404011544.i31Fiea17172@guido.python.org> References: <1080674306.12643.76.camel@localhost.localdomain> <200403302121.i2ULLIM09840@guido.python.org> <2CF5B513-82EB-11D8-BD7A-0003934AD54A@chello.se> <200403311542.i2VFgWf13002@guido.python.org> <406B4FD2.1040506@prescod.net> <200403312332.i2VNWpe15013@guido.python.org> <406BC098.7040209@prescod.net> <200404011544.i31Fiea17172@guido.python.org> Message-ID: <0A6B20B6-83F9-11D8-AF78-0003934AD54A@chello.se> On 2004-04-01, at 17.44, Guido van Rossum wrote: >> Anyhow, I just meant that if you find a pair of characters that are >> illegal today then many of the objections about abusing existing >> syntax >> would go away. > > So how about *[...]* ? Or perhaps [*...*]? Or [|...|]? Also: Is it definite that it should be square brackets instead of parentheses? --- (sigh ... mail problems) From pje at telecommunity.com Thu Apr 1 11:26:00 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Apr 1 11:26:07 2004 Subject: [Python-Dev] PEP 318 bake-off? In-Reply-To: <200404010544.i315iiB15938@guido.python.org> References: <42B02151.420DCBB8@mail.google.com> Message-ID: <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> At 09:44 PM 3/31/04 -0800, Guido van Rossum wrote: >What I'm asking (especially of Phillip) is to collect a set of >realistic method declarations using decorators; we can then >collectively format these using any of the possible syntaxes, and see >how they look. I'd be happy to scrounge up some samples from existing code using 'property' and 'classmethod' as well as some of PEAK's decorators, and I definitely think that Jack Diedrich and Bob Ippolito's samples should be included as well. Important question, though: do we include code bodies, or just use 'pass' for the bodies? If we include the body, how much of the body? Should we include entire classes, especially if the class itself needs a decorator, and multiple methods have decorators? Next, does anybody have any actual use cases for attribute decoration today? We're probably not going to get a lot of that from current code samples. I can make up some examples that throw in every possible option that PEAK provides in order to get some "lots of decoration" samples, but they wouldn't be "real" uses in that case. But I guess that maybe Bob's examples might be wordy enough. From guido at python.org Thu Apr 1 11:36:15 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 11:36:19 2004 Subject: [Python-Dev] Re: Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Thu, 01 Apr 2004 11:16:31 EST." References: <42B02151.420DCBB8@mail.google.com> Message-ID: <200404011636.i31GaFg17456@guido.python.org> > (Has any consideration been given to an actual metafunc mechanism > more directly analogous to metaclasses, that would be given the > *pieces* of a would-be function (name, param names, default args, > code body, etc), so that there would not necessarily ever be a > standard function object?) Deconstructing a function like that is too invasive -- I don't want to touch the calling sequence, for example, because it's so performance critical. None of the people arguing for decorators has shown a use case for that either. However, if you really want to do that, you *can* take the function apart and construct a new one using the 'new' module. --Guido van Rossum (home page: http://www.python.org/~guido/) From evan at 4-am.com Thu Apr 1 11:47:21 2004 From: evan at 4-am.com (Evan Simpson) Date: Thu Apr 1 11:48:49 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: <200404010544.i315iiB15938@guido.python.org> References: <42B02151.420DCBB8@mail.google.com> <200404010544.i315iiB15938@guido.python.org> Message-ID: <406C4799.6030502@4-am.com> Guido van Rossum wrote: > 2a) Prefix list with leading *: > > *[funcattrs(foo=42), deprecated, overrides, classmethod] > def foo(cls, arg1, arg2): > pass How about adding: 2b) Prefix list with repeated keyword: def [funcattrs(foo=42), deprecated, overrides, classmethod] def foo(cls, arg1, arg2): pass class [singleton] class foo: pass 3a) Prefix suite with repeated keyword: def: funcattrs(foo=42) deprecated, overrides, classmethod def foo(cls, arg1, arg2): pass Cheers, Evan @ 4-am From edloper at gradient.cis.upenn.edu Thu Apr 1 11:56:39 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Thu Apr 1 11:55:12 2004 Subject: [Python-Dev] PEP 318 -- a couple use cases Message-ID: <406C49C7.80304@gradient.cis.upenn.edu> I saw both of the following use cases mentioned on the list, and they seemed interesting, so I went ahead and wrote up implementations: def property_getter(func): """ A decorator that can be used to incrementally construct properties. For example, the following code constructs a property 'x' from an fget, an fset, and an fdel function: >>> class A: ... def x(self) [property_getter]: ... return self.__x ... def x(self, val) [property_setter]: ... self.__x = val ... def x(self) [property_deleter]: ... del self.__x In particular, this decorator checks if a property named 'func' is defined in the enclosing frame. If so, then it updates that property's fget to be 'func'. If not, then it creates a new property whose fget is 'func'. The property's docstring is taken from the first decorated property function to define a docstring. """ def generic(*type_signature): """ A decorator-generator that can be used to incrementally construct a generic function that delegates to individual functions based on the type signature of the arguments. For example, the following code defines a generic function that uses two different actual functions, depending on whether its argument is a string or an int: >>> def f(x) [generic(int)]: ... print x, 'is an int' >>> def f(x) [generic(str)]: ... print x, 'is a string' In particular, the decorator returned by this function checks if a Generic function-delegation object with the same name as the decorated function is defined in the enclosing frame. If so, then it registers the function with that generic object, under 'type_signature'. If not, then it creates a new Generic function-delegation object, and registers the function. """ Full code is at . But I still don't feel like I have a good handle on what "acceptable" uses of decorators are.. So, for each of these 2 use cases, is it... - an excellent example that should be included in the stdlib - perfectly acceptable, and belongs in the python cookbook - somewhat hackish, but ok under the right circumstances - an abhorition of nature (If one of the first 3, then maybe they should be added to the pep?) Thanks for the feedback. -Edward From mike at nospam.com Thu Apr 1 12:10:04 2004 From: mike at nospam.com (Mike Rovner) Date: Thu Apr 1 12:10:14 2004 Subject: [Python-Dev] Re: ANNOUNCE: 'goto' for Python References: Message-ID: > 'goto' example: breaking out from a deeply nested loop: > goto .end mark that space--^ Shall I file a bug about python lexer which allows space between class and attribute? Or that's a fiature? Regards, Mike From michel at dialnetwork.com Thu Apr 1 12:08:28 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Thu Apr 1 12:27:14 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: References: Message-ID: <1080839290.2036.29.camel@heinlein> I apologize for not changing my digest subjects in earlier messages. > Message: 4 > Date: Wed, 31 Mar 2004 19:16:06 -0500 > From: "Phillip J. Eby" > Subject: Re: [Python-Dev] Re: PEP 318: Decorators last before colon > To: Neil Schemenauer , python-dev@python.org > Message-ID: <5.1.1.6.0.20040331184627.01e86a10@telecommunity.com> > Content-Type: text/plain; charset="us-ascii"; format=flowed > > Ugly or not, *all* of the proposed syntaxes that don't involve creating an > extra suite have the practical effect of improving the semantic readability > of decorator use over today's Python. In addition, they also have the > practical effect of making decorator use more convenient, since the two > extra repetitions of the function or class name go away. Agreed. > I suspect this is part of why there is such disagreement on the subject of > decorators: people who make heavy use of them today know exactly what > problems today's syntax has from both the readability and writability > viewpoints. Whereas people who do not use them, don't get why limiting > their use to fixed subsets, or single decorators, or any number of other > ideas just negate the usefulness of having a syntax in the first > place. Or, like Michel, they don't see a point to having a syntax at > all. I'd like to clarify that I'm not against a new syntax (I'd be more than a bit hypocritical having written two PEPs proposing new syntax!). My original comment was in fact that the line-preceding-def was ugly, not pointless. My second comment is that I wonder if decoration of any syntax has been thought out in the light of other objects being decorated, like classes, functions, or interfaces. I just feel like there's more thought to be applied here on decoration syntax in general. When we take decoration to the next level, will this syntax suffice? Will it be abused for reasons not considered, like run-time type checking? I'm not positive but it seems this is what Bob I. wants it for. And have we considered all the options? I'm not making any proposals, but perhaps decoration can be bundled into a separate object that a class refers to, similar to interfaces. This could allow a lot more flexibility and still look like Python. Or perhaps this separate object could be mixed in (blech). But you see my point, the upshot is you could apply different decorations at different times, decorators would be separately managed and it wouldn't look like a shark fin duct taped to a VW bug. But having said that, I concede: I know that some people make a lot of redundant finger movements working around the lack of decorations. Phillip is right that I don't use decorators (I have used classmethod quite a bit, but that's all) and that he does, so his needs outweigh mine. So I cave. See below for my vote. Later, Barry said: > So I hacked up python-mode a bit to support the syntax coloring of > Guido's previously MFS (most favored syntax). I wanted to see if the > concerns about visual obscurity were real or not. Then I rewrote a > few > methods of mine that would benefit from decorators (with some > elaboration). A screen shot of my XEmacs buffer is here: > > http://barry.warsaw.us/xemacs.png Mmm.. I like your colors, it reminds me of a scheme Ken M. shared with me once. I've long lost it, along with his handy pop-to-shell script. Later, Guido said: > 3) Prefix suite (could use a different keyword than 'decorate'): > > decorate: > funcattrs(foo=42) > deprecated > overrides > classmethod > def foo(cls, arg1, arg2): > pass I don't really like it, but it's the only one that looks like Python to me. Otherwise, I vote last-before-colon. Line-preceding-def would be my last choice. -Michel From barry at python.org Thu Apr 1 12:30:53 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 1 12:31:05 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1080839290.2036.29.camel@heinlein> References: <1080839290.2036.29.camel@heinlein> Message-ID: <1080840653.3054.8.camel@anthem.wooz.org> On Thu, 2004-04-01 at 12:08, Michel Pelletier wrote: > Mmm.. I like your colors, it reminds me of a scheme Ken M. shared with > me once. I've long lost it, along with his handy pop-to-shell script. That probably shouldn't be surprising. :) -Barry From ark-mlist at att.net Thu Apr 1 12:38:21 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Thu Apr 1 12:38:18 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404011544.i31Fiea17172@guido.python.org> Message-ID: <00f001c41810$20afc4f0$6402a8c0@arkdesktop> > > Anyhow, I just meant that if you find a pair of characters that are > > illegal today then many of the objections about abusing existing syntax > > would go away. > > So how about *[...]* ? Or perhaps [*...*]? Or [|...|]? How about INDENT[...]:DEDENT? <0.5 serious> From bob at redivi.com Thu Apr 1 12:48:56 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Apr 1 12:44:51 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1080839290.2036.29.camel@heinlein> References: <1080839290.2036.29.camel@heinlein> Message-ID: On Apr 1, 2004, at 12:08 PM, Michel Pelletier wrote: > My second comment is that I wonder if decoration of any syntax has been > thought out in the light of other objects being decorated, like > classes, > functions, or interfaces. I just feel like there's more thought to be > applied here on decoration syntax in general. When we take decoration > to the next level, will this syntax suffice? Will it be abused for > reasons not considered, like run-time type checking? I'm not positive > but it seems this is what Bob I. wants it for. It's not run-time type *checking*.. it's run-time type *bridge* between the Python and Objective C runtimes. Without these decorators (and other FFI machinery), the bridge does not work, because a PyObject* is not a struct _NSRect or a NSObject*, etc. It's essentially the same kind of thing as ctypes, except the Objective C runtime has a lot more RTTI available than the regular 'ol C runtime and it also uses reference counting so it's more safe and sane. ctypes probably needs decorator syntax more than PyObjC does, but the brave few have hobbled along without it :) > And have we considered all the options? I'm not making any proposals, > but perhaps decoration can be bundled into a separate object that a > class refers to, similar to interfaces. This could allow a lot more > flexibility and still look like Python. Or perhaps this separate > object could be mixed in (blech). But you see my point, the upshot is > you could apply different decorations at different times, decorators > would be separately managed and it wouldn't look like a shark fin duct > taped to a VW bug. In my use case it doesn't really make sense to keep them separate, as the functions are broken without their decorator because they can not be called from the Objective C side of the bridge. The only time you need to use these decorators are when you are creating a class that complies with an informal protocol, whose selectors have non-object type signatures (returning void or BOOL is exceedingly common, as is taking integers, and sometimes nasty pointer stuff). In any case, it's actually really common with AppKit and Foundation because most of it is event driven and these informal protocols are part of the delegate callback machinery for everything from URL loading to printing. -bob From pje at telecommunity.com Thu Apr 1 12:47:11 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Apr 1 12:47:20 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1080839290.2036.29.camel@heinlein> References: Message-ID: <5.1.1.6.0.20040401124254.02c2acd0@telecommunity.com> At 09:08 AM 4/1/04 -0800, Michel Pelletier wrote: >Later, Guido said: > > > 3) Prefix suite (could use a different keyword than 'decorate'): > > > > decorate: > > funcattrs(foo=42) > > deprecated > > overrides > > classmethod > > def foo(cls, arg1, arg2): > > pass > > >I don't really like it, but it's the only one that looks like Python to >me. Otherwise, I vote last-before-colon. Line-preceding-def would be >my last choice. There is another option: as [classmethod] def blah(cls): ... as [singleton] class SignalManager: ... as [implementorOf(IFoo, forTypes=[Bar])] class FooToBarAdapter: ... and it could also be on the same line, e.g.: as [classmethod] def blah(cls): ... It reads a bit better with function definitions than with classes, but it's not too bad. It's more of a replacement for the *[decorators] syntax than the 'decorate:' syntax, but it's an English word rather than a symbol, which brings back a bit more of the "Python look and feel". From skip at pobox.com Thu Apr 1 13:05:59 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Apr 1 13:06:29 2004 Subject: [Python-Dev] Re: Re: PEP 318: Decorators last before colon In-Reply-To: <200404011636.i31GaFg17456@guido.python.org> References: <42B02151.420DCBB8@mail.google.com> <200404011636.i31GaFg17456@guido.python.org> Message-ID: <16492.23047.867494.68059@montanaro.dyndns.org> >> (Has any consideration been given to an actual metafunc mechanism >> more directly analogous to metaclasses, that would be given the >> *pieces* of a would-be function (name, param names, default args, >> code body, etc), so that there would not necessarily ever be a >> standard function object?) Guido> Deconstructing a function like that is too invasive -- I don't Guido> want to touch the calling sequence, for example, because it's so Guido> performance critical. None of the people arguing for decorators Guido> has shown a use case for that either. However, if you really Guido> want to do that, you *can* take the function apart and construct Guido> a new one using the 'new' module. One thing that occurred to me is that a function's func_name attribute might be made read-write so that decorator functions can easily make that aspect of a function wrapper behave the same as the original function. Calling new.function() shouldn't be required in such simple cases. Skip From michel at dialnetwork.com Thu Apr 1 13:07:35 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Thu Apr 1 13:23:24 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <5.1.1.6.0.20040401124254.02c2acd0@telecommunity.com> References: <5.1.1.6.0.20040401124254.02c2acd0@telecommunity.com> Message-ID: <1080842227.2804.7.camel@heinlein> On Thu, 2004-04-01 at 09:47, Phillip J. Eby wrote: > as [implementorOf(IFoo, forTypes=[Bar])] > class FooToBarAdapter: > ... > > and it could also be on the same line, e.g.: > > as [classmethod] def blah(cls): > ... > > It reads a bit better with function definitions than with classes, but it's > not too bad. Agreed, I missed this one. It's not too bad with classes if you can warp your mind to think of "to class" as a verb like "to define". > It's more of a replacement for the *[decorators] syntax than > the 'decorate:' syntax, but it's an English word rather than a symbol, > which brings back a bit more of the "Python look and feel". One minor nit, why use the brackets at all? as sychronized, sharkfin(size=10) def VWBug(...): that looks better to me than: as [sychronized, sharkfin(size=10)] def VWBug(...): without brackets it looks more to me like Python, you'd never see: from foo import [bar, baz] For longer lists of decorators use the familiar tuple rule, wrap in parens: as (sychronized, sharkfin(size=10), deprecated) def VWBug(...): -Michel From jim.jewett at eds.com Thu Apr 1 13:42:34 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Thu Apr 1 13:43:06 2004 Subject: [Python-Dev] PEP 318: Decorators last before colon/bakeoff Message-ID: >> Typically the suite (things indented after the colon) are all >> of the same type. Josiah Carlson: > Funny, but this isn't the case with Python anywhere else; I can have > functions, class and variable definitions inside any other suite. I phrased that badly. The statements following a suite are all interchangable from the *suite's* perspective. If not, then a second header clause (e.g. "else:") is added. If a particular expression is special, it comes before the ":" within the header. (x, y in "for x in y:") Docstrings are arguably an exception, but they get away with it because they evaluate to themselves, without a side-effect. If you remove their special status, programs don't change their behavior. (Which is why optimized code can just remove them.) Decorators behave differently from other statements, and this should be clear. Everything in Guido's bakeoff meets this requirement. Option 2 *might* cause confusion with normal lists, which is why he also listed 2.a. (2) Individually fine, but the tie between the list and def is week. [d1, d2, d3] def func (args): pass (2a) Also warns you that the list is strange, but doesn't say how. *[d1, d2, d3] def func (args): pass (2b) Explicitly relates the list to the def, but does add a keyword. On the other hand, adding this keyword makes it slightly less difficult to extend the syntax again, if that ever becomes desirable. [d1, d2, d3] of def func (args): pass -jJ From goodger at python.org Thu Apr 1 13:54:30 2004 From: goodger at python.org (David Goodger) Date: Thu Apr 1 13:54:35 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> References: <42B02151.420DCBB8@mail.google.com> <200404010544.i315iiB15938@guido.python.org> <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> Message-ID: <406C6566.3040104@python.org> Phillip J. Eby wrote: > Next, does anybody have any actual use cases for attribute decoration > today? If you mean function attributes, Docutils uses them a lot. They're described in and used in all the modules in that directory, such as . -- David Goodger From guido at python.org Thu Apr 1 14:08:26 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 14:08:33 2004 Subject: [Python-Dev] PEP 318 bake-off? In-Reply-To: Your message of "Thu, 01 Apr 2004 11:26:00 EST." <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> References: <42B02151.420DCBB8@mail.google.com> <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> Message-ID: <200404011908.i31J8Qu17815@guido.python.org> > >What I'm asking (especially of Phillip) is to collect a set of > >realistic method declarations using decorators; we can then > >collectively format these using any of the possible syntaxes, and see > >how they look. > > I'd be happy to scrounge up some samples from existing code using > 'property' and 'classmethod' as well as some of PEAK's decorators, and I > definitely think that Jack Diedrich and Bob Ippolito's samples should be > included as well. > > Important question, though: do we include code bodies, or just use 'pass' > for the bodies? If we include the body, how much of the body? Should we > include entire classes, especially if the class itself needs a decorator, > and multiple methods have decorators? Why not provide the bodies, for added realism? (I still think class decorators are a separate case, and much weaker -- you can do this by having a single 'decoratable' metaclass and setting __decorators__ = [...] in the class body.) > Next, does anybody have any actual use cases for attribute decoration > today? We're probably not going to get a lot of that from current code > samples. I can make up some examples that throw in every possible option > that PEAK provides in order to get some "lots of decoration" samples, but > they wouldn't be "real" uses in that case. But I guess that maybe Bob's > examples might be wordy enough. I think that SPARK syntax and everything else that people have traditionally added to docstring markup that isn't strictly speaking documentation (even some extreme cases of doctest usage) ought to be considered as candidates for attribute-ification. --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Apr 1 14:10:39 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Apr 1 14:10:46 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: <406C6566.3040104@python.org> References: <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> <42B02151.420DCBB8@mail.google.com> <200404010544.i315iiB15938@guido.python.org> <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> Message-ID: <5.1.1.6.0.20040401140508.02d80090@telecommunity.com> At 01:54 PM 4/1/04 -0500, David Goodger wrote: >Phillip J. Eby wrote: >>Next, does anybody have any actual use cases for attribute decoration today? > >If you mean function attributes, Docutils uses them a lot. >They're described in > >and used in all the modules in that directory, such as >. Thanks! So for example, this: def admonition(*args): return make_admonition(nodes.admonition, *args) admonition.arguments = (1, 0, 1) admonition.options = {'class': directives.class_option} admonition.content = 1 def attention(*args): return make_admonition(nodes.attention, *args) attention.content = 1 might be rephrased as (say): as [rst_directive( arguments=(1,0,1), options={'class': directives.class_option}, content=1 ) ] def admonition(*args): return make_admonition(nodes.admonition, *args) as [rst_directive(content=1)] def attention(*args): return make_admonition(nodes.attention, *args) From guido at python.org Thu Apr 1 14:16:33 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 14:16:40 2004 Subject: [Python-Dev] PEP 318 -- a couple use cases In-Reply-To: Your message of "Thu, 01 Apr 2004 11:56:39 EST." <406C49C7.80304@gradient.cis.upenn.edu> References: <406C49C7.80304@gradient.cis.upenn.edu> Message-ID: <200404011916.i31JGXm17863@guido.python.org> > I saw both of the following use cases mentioned on the list, and they > seemed interesting, so I went ahead and wrote up implementations: > > def property_getter(func): [...] > def generic(*type_signature): [...] > Full code is at . > > But I still don't feel like I have a good handle on what "acceptable" > uses of decorators are.. So, for each of these 2 use cases, is it... > - an excellent example that should be included in the stdlib > - perfectly acceptable, and belongs in the python cookbook > - somewhat hackish, but ok under the right circumstances > - an abhorition of nature I'm wavering between 2 (perfectly acceptable) and 3 (somewhat hackish), only because I consider anything that uses sys._getframe() to be a danger to society unless proven innocent. I would probably prefer to do both of these using explicit, different names. In particular for properties, I *like* the fact that I can also explicitly call the getter or setter functions, so I'd probably continue to write those like this: class C(object): def getX(self): return self.__x def setX(self, x): self.__x = x x = property(getX, setX) I'm not sufficiently comfortable with generic functions to quite know what feels best there. But I like the notation you proposed (I would like it better with the decorators up front though). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 1 14:21:12 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 14:21:20 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Thu, 01 Apr 2004 12:38:21 EST." <00f001c41810$20afc4f0$6402a8c0@arkdesktop> References: <00f001c41810$20afc4f0$6402a8c0@arkdesktop> Message-ID: <200404011921.i31JLCF17896@guido.python.org> > How about INDENT[...]:DEDENT? <0.5 serious> Alas, won't work -- if the decoration is the first thing in a class body, you'd have something starting with double indent, but the lexer never gives you that. class C: [staticmethod] def foo(): pass This would be tokenized as 'class' 'C' ':' NEWLINE INDENT '[' 'staticmethod' ']' NEWLINE --Guido van Rossum (home page: http://www.python.org/~guido/) From bkc at murkworks.com Thu Apr 1 14:28:28 2004 From: bkc at murkworks.com (Brad Clements) Date: Thu Apr 1 14:27:03 2004 Subject: [Python-Dev] PEP 318 bake-off? In-Reply-To: <200404011908.i31J8Qu17815@guido.python.org> References: Your message of "Thu, 01 Apr 2004 11:26:00 EST." <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> Message-ID: <406C26A3.28912.6817562C@localhost> On 1 Apr 2004 at 11:08, Guido van Rossum wrote: > I think that SPARK syntax and everything else that people have > traditionally added to docstring markup that isn't strictly speaking > documentation (even some extreme cases of doctest usage) ought to be > considered as candidates for attribute-ification. Where do method attribute type signatures and DBC fit in? As a decorator, or in the docstring? I'm concerned that the funcattrs(a="xyz" .. ) sample tossed around here will be rather ugly for specifying DBC strings. DBC will also need class invariants, so a funcattrs work-alike at the class level will be needed. Finally, I don't have a need to access DBC annotations at runtime once my module is distributed. I would not want to pay the memory cost overhead of loading DBC information or attribute type signatures at runtime. However another person at PyCon poo-poo'd my concern over bloating .pyc files and subsequent memory use. As a compromise I suggested that "annotation" information could go into the .pyc, but be loaded "on demand" at runtime. For example, a traceback handler might be able to use DBC info to suggest the call level that may have caused the problem. To do "on demand loading", I suggested giving .pyc files a "resource section" that could hold this meta information. Normal imports would not load the resource section, but resource information could be loaded later using another mechanism. I thought that putting meta data into a different file was more complicated, and that using -OO to "strip out" annotation was too heavy handed. The standard library might benefit from the addition of method attribute type information. However I think it would be better to not load this information at runtime unless needed. The same could be said for docstrings. If docstrings were "packed" in a resource section of the .pyc file, they might be loaded "on-demand", thereby saving some memory overhead. -- Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax http://www.wecanstopspam.org/ AOL-IM: BKClements From bob at redivi.com Thu Apr 1 14:33:24 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Apr 1 14:29:19 2004 Subject: [Python-Dev] PEP 318 bake-off? In-Reply-To: <200404011908.i31J8Qu17815@guido.python.org> References: <42B02151.420DCBB8@mail.google.com> <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> <200404011908.i31J8Qu17815@guido.python.org> Message-ID: <712CE402-8413-11D8-8B4D-000A95686CD8@redivi.com> On Apr 1, 2004, at 2:08 PM, Guido van Rossum wrote: >>> What I'm asking (especially of Phillip) is to collect a set of >>> realistic method declarations using decorators; we can then >>> collectively format these using any of the possible syntaxes, and see >>> how they look. >> >> I'd be happy to scrounge up some samples from existing code using >> 'property' and 'classmethod' as well as some of PEAK's decorators, >> and I >> definitely think that Jack Diedrich and Bob Ippolito's samples should >> be >> included as well. >> >> Important question, though: do we include code bodies, or just use >> 'pass' >> for the bodies? If we include the body, how much of the body? >> Should we >> include entire classes, especially if the class itself needs a >> decorator, >> and multiple methods have decorators? > > Why not provide the bodies, for added realism? > > (I still think class decorators are a separate case, and much weaker > -- you can do this by having a single 'decoratable' metaclass and > setting __decorators__ = [...] in the class body.) Here's something I wrote today.. it's a delegate for the exception handling mechanism so you can listen in on (any of) the exceptions that ObjC throws whether or not they are caught by something else.. the Python->PyObjC exceptions are logged as tracebacks and the ObjC exceptions are thrown over to atos so they turn into human-readable stack traces. As you can see, it's rather ugly with regular old Python syntax. class PyObjCDebuggingDelegate(NSObject): def exceptionHandler_shouldLogException_mask_(self, sender, exception, aMask): try: if isPythonException(exception): if self.verbosity() & LOGSTACKTRACE: nsLogObjCException(exception) return nsLogPythonException(exception) elif self.verbosity() & LOGSTACKTRACE: return nsLogObjCException(exception) else: return False except: print >>sys.stderr, "*** Exception occurred during exception handler ***" traceback.print_exc(sys.stderr) return True exceptionHandler_shouldLogException_mask_ = objc.selector(exceptionHandler_shouldLogException_mask_, signature='c@:@@I') def exceptionHandler_shouldHandleException_mask_(self, sender, exception, aMask): return False exceptionHandler_shouldHandleException_mask_ = objc.selector(exceptionHandler_shouldHandleException_mask_, signature='c@:@@I') the objc.selector signatures say that they return a char (a BOOL, actually), the following @: represents "self" and the selector (the "method name"), the next two @@ say that the sender and exception arguments are both ObjC objects, and the trailing I means that aMask is an unsigned int. -bob From aahz at pythoncraft.com Thu Apr 1 14:47:08 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Apr 1 14:47:22 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404011544.i31Fiea17172@guido.python.org> References: <1080674306.12643.76.camel@localhost.localdomain> <200403302121.i2ULLIM09840@guido.python.org> <2CF5B513-82EB-11D8-BD7A-0003934AD54A@chello.se> <200403311542.i2VFgWf13002@guido.python.org> <406B4FD2.1040506@prescod.net> <200403312332.i2VNWpe15013@guido.python.org> <406BC098.7040209@prescod.net> <200404011544.i31Fiea17172@guido.python.org> Message-ID: <20040401194707.GB3825@panix.com> On Thu, Apr 01, 2004, Guido van Rossum wrote: > > So how about *[...]* ? Or perhaps [*...*]? Or [|...|]? -0, +1, +0 (That last is mainly 'cause it's four pinky keystrokes, or I'd be +1 for it, too.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From edloper at gradient.cis.upenn.edu Thu Apr 1 14:57:09 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Thu Apr 1 14:55:40 2004 Subject: [Python-Dev] PEP 318 -- a couple use cases In-Reply-To: <200404011916.i31JGXm17863@guido.python.org> References: <406C49C7.80304@gradient.cis.upenn.edu> <200404011916.i31JGXm17863@guido.python.org> Message-ID: <406C7415.5000005@gradient.cis.upenn.edu> > I'm wavering between 2 (perfectly acceptable) and 3 (somewhat > hackish), only because I consider anything that uses sys._getframe() > to be a danger to society unless proven innocent. Would these be preferable:? class C(object): [property] def x(self): return self.__x [property_setter(x)] def x(self, val): self.__x = val [property_deleter(x)] def x(self): del self.__x and: [generic(None, (int,int))] def f(x,y): print 'two ints' [generic(f, (float,float))] def f(x,y): print 'two floats' I.e., explicitly pass the old value in, rather than grabbing it from sys._getframe. Advantages: - no sys._getframe magic Disadvantages: - you have to type the property/function name twice - you have to do something special for the first def - you can't easily rearrange the defs -Edward From python at rcn.com Thu Apr 1 15:47:26 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Apr 1 15:47:44 2004 Subject: [Python-Dev] Vote No on 318 In-Reply-To: <1080833042.22892.122.camel@localhost.localdomain> Message-ID: <002a01c4182a$8b057420$7329cb97@oemcomputer> [Michael] > > We've been discussing this off and on for OVER A YEAR! If 'the best > > syntax' hasn't been figured out yet after N thousand emails on the > > subject, I see no reason to believe enlightenment is going to arrive > > soon (or ever). [Jeremy] > There's no particular reason to believe that effort alone will arrive at > an elegant solution. On the other hand, maybe there isn't a good syntax > for arbitrary decorators. IMHO, the best syntax is what we have now. Its only weakness is being distant from the first line in the definition. Other than that, it is understandable, unambiguous, simple, backward compatible, and flexible. A failing common to all of the proposals is that they turn action words (like classmethod) into adjectives with implicit actions. By making them implicit, it becomes less obvious what is occurring, when it is occurring, and, for multiple decorators, which order they occur. That will make the wrapped functions harder to debug. A second failing is introducing a second way to do it. The current approach will always be there for some purposes, properties, metaclasses, existing code, or dynamic wrapping. This is a prime example where two ways to do something is especially bad; if you do not see [staticmethod] in the definition, can you presume that it is not a staticmethod -- the answer is clearly no. So, why create optical illusions. A third failing is clashing with a design preference for functions/expressions over statements/keywords/syntax. This concept was at the foundation for discussions about having a print() function, using sorted() in an expression, or implementing if-then-else expressions. My understanding of language design history is that of regrets for not using functions wherever possible. Also, some proposed uses were strange enough that I wonder whether the syntax encourages people to do weird things with their code. How much complexity has to be introduced just to bring decorations closer to the first line in a definition? Raymond Hettinger Chairman, Citizens Against 318 ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From guido at python.org Thu Apr 1 16:56:37 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 16:56:46 2004 Subject: [Python-Dev] PEP 318 -- a couple use cases In-Reply-To: Your message of "Thu, 01 Apr 2004 14:57:09 EST." <406C7415.5000005@gradient.cis.upenn.edu> References: <406C49C7.80304@gradient.cis.upenn.edu> <200404011916.i31JGXm17863@guido.python.org> <406C7415.5000005@gradient.cis.upenn.edu> Message-ID: <200404012156.i31Lubc18438@guido.python.org> > > I'm wavering between 2 (perfectly acceptable) and 3 (somewhat > > hackish), only because I consider anything that uses sys._getframe() > > to be a danger to society unless proven innocent. > > Would these be preferable:? > > class C(object): > [property] > def x(self): return self.__x > > [property_setter(x)] > def x(self, val): self.__x = val > > [property_deleter(x)] > def x(self): del self.__x > and: > [generic(None, (int,int))] > def f(x,y): print 'two ints' > > [generic(f, (float,float))] > def f(x,y): print 'two floats' > > I.e., explicitly pass the old value in, rather than grabbing it from > sys._getframe. > > Advantages: > - no sys._getframe magic > Disadvantages: > - you have to type the property/function name twice > - you have to do something special for the first def > - you can't easily rearrange the defs I see no big advantage to this solution; the requirement to type the name an extra time seems arbitrary. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave at boost-consulting.com Thu Apr 1 17:25:20 2004 From: dave at boost-consulting.com (David Abrahams) Date: Thu Apr 1 17:40:26 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> <1080833042.22892.122.camel@localhost.localdomain> Message-ID: Jeremy Hylton writes: > On Thu, 2004-04-01 at 07:13, Michael Hudson wrote: >> > I don't think Michel is saying they are worthless. However, the >> > proposed syntax is highly contentious. It would be good if there >> > was a short term solution that wouldn't require new syntax. That >> > would give Guido and the Python community time to figure out the >> > best syntax. >> >> We've been discussing this off and on for OVER A YEAR! If 'the best >> syntax' hasn't been figured out yet after N thousand emails on the >> subject, I see no reason to believe enlightenment is going to arrive >> soon (or ever). > > There's no particular reason to believe that effort alone will arrive at > an elegant solution. On the other hand, maybe there isn't a good syntax > for arbitrary decorators. Has something along these lines been discussed? with [staticmethod, classmethod]: def foo(x): pass def bar(x): pass IIUC, the PyObjC application needs whole swathes of functions with the same decoration, but this syntax isn't much worse for one function than for many. -- Dave Abrahams Boost Consulting www.boost-consulting.com From barry at python.org Thu Apr 1 17:56:16 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 1 17:56:23 2004 Subject: [Python-Dev] Vote No on 318 In-Reply-To: <002a01c4182a$8b057420$7329cb97@oemcomputer> References: <002a01c4182a$8b057420$7329cb97@oemcomputer> Message-ID: <1080860175.5481.19.camel@anthem.wooz.org> On Thu, 2004-04-01 at 15:47, Raymond Hettinger wrote: > IMHO, the best syntax is what we have now. Its only weakness is being > distant from the first line in the definition. A significant drawback. Plus the fact that you have to type everything twice. > A failing common to all of the proposals is that they turn action words > (like classmethod) into adjectives with implicit actions. I've never thought of "classmethod" as an action, i.e. verb. To me, and in my code, the things inside the decorator syntax have always been nouns. That's why I favored decorator-after-def (although I've given up on that now) because it read like English: "define a class method named foo with arguments bar and baz". I'm becoming more dejected that we're just not going to reach agreement on this for Python 2.4. In the hopes of avoiding hanging our Chad, I'm throwing my endorsement to Les "Decorator Before Colon" Decidawredy. plus-python-mode-already-supports-that-syntax-so-what-more-do-you-need?-ly y'rs, -Barry From bob at redivi.com Thu Apr 1 18:02:44 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Apr 1 17:58:41 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> <1080833042.22892.122.camel@localhost.localdomain> Message-ID: On Apr 1, 2004, at 5:25 PM, David Abrahams wrote: > Jeremy Hylton writes: > >> On Thu, 2004-04-01 at 07:13, Michael Hudson wrote: >>>> I don't think Michel is saying they are worthless. However, the >>>> proposed syntax is highly contentious. It would be good if there >>>> was a short term solution that wouldn't require new syntax. That >>>> would give Guido and the Python community time to figure out the >>>> best syntax. >>> >>> We've been discussing this off and on for OVER A YEAR! If 'the best >>> syntax' hasn't been figured out yet after N thousand emails on the >>> subject, I see no reason to believe enlightenment is going to arrive >>> soon (or ever). >> >> There's no particular reason to believe that effort alone will arrive >> at >> an elegant solution. On the other hand, maybe there isn't a good >> syntax >> for arbitrary decorators. > > Has something along these lines been discussed? > > with [staticmethod, classmethod]: > > def foo(x): > pass > > def bar(x): > pass > > IIUC, the PyObjC application needs whole swathes of functions with > the same decoration, but this syntax isn't much worse for one > function than for many. No, that's a misunderstanding.. it was by coincidence alone that the last example I gave had two callbacks with an identical type signature. Using a "with" block for decorators is as nonsensical as this: with args(x): def foo: pass def bar: pass -bob From pje at telecommunity.com Thu Apr 1 18:18:47 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Apr 1 18:19:03 2004 Subject: [Python-Dev] PEP 318 bake-off? In-Reply-To: <200404011908.i31J8Qu17815@guido.python.org> References: <42B02151.420DCBB8@mail.google.com> <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> Message-ID: <5.1.1.6.0.20040401175455.0271c6c0@telecommunity.com> At 11:08 AM 4/1/04 -0800, Guido van Rossum wrote: > > >What I'm asking (especially of Phillip) is to collect a set of > > >realistic method declarations using decorators; we can then > > >collectively format these using any of the possible syntaxes, and see > > >how they look. > > > > I'd be happy to scrounge up some samples from existing code using > > 'property' and 'classmethod' as well as some of PEAK's decorators, and I > > definitely think that Jack Diedrich and Bob Ippolito's samples should be > > included as well. > > > > Important question, though: do we include code bodies, or just use 'pass' > > for the bodies? If we include the body, how much of the body? Should we > > include entire classes, especially if the class itself needs a decorator, > > and multiple methods have decorators? > >Why not provide the bodies, for added realism? Okay, here's one using two decorators, in today's syntax. It's excerpted from an I/O scheduling component in 'peak.events'. The component manages a set of read/write/error file handles, and allows pseudothreads waiting on those handles to resume when I/O is possible. The 'select()' operation is itself performed by a pseudothread called 'monitor'. Each instance of the component should have exactly one such pseudothread, which should begin running as soon as the component is "assembled" (attached to an application). Two decorators control this: 'events.taskFactory', which accepts a generator function and returns a function that returns a new pseudothread each time it's invoked. That is, it's roughly equivalent to: def taskFactory(func): return lambda *args,**kw: Task(func(*args,**kw)) except that there is some extra magic so that introspecting the returned function still shows the same argument signature. (Which is important for documentation tools like pydoc and epydoc). The second decorator is 'binding.Make', which takes a 1-argument callable and returns a descriptor that will invoke the callable only once: when the attribute is first accessed for a given instance. The result of the callable is cached in the object's instance dictionary, where it will be retrieved on any subsequent access. So, applying the two decorators (i.e. [events.taskFactory, binding.Make]) to a 1-argument function results in an attribute that will be automatically initialized when first used. By applying an extra keyword argument to 'binding.Make' in the current implementation, we can tell the descriptor to automatically initialize itself when the componet is assembled. (Note: this is not the same as __init__ time; a PEAK component is considered "assembled" when it is made the child of a component that knows its "root" component, and thus can be certain of its entire configuration environment.) So, I would probably render this example with these decorators: [events.taskFactory, binding.Make(uponAssembly=True)] in order to specify that the function is a generator that should be run as a task (pseudothread), it should be run at most once, and it should exist as soon as the component knows its configuration environment. (Note: the 'uponAssembly' bit works today only with old-style 'foo=binding.Make(foo,...)' syntax, not decorator syntax.) Anyway, here's the example: def monitor(self) [events.taskFactory, binding.Make(uponAssembly=True)]: r,w,e = self.rwe count = self.count sleep = self.scheduler.sleep() time_available = self.scheduler.time_available select = self.select error = self._error while True: yield count; resume() # wait until there are selectables yield sleep; resume() # ensure we are in top-level loop delay = time_available() if delay is None: delay = self.checkInterval try: rwe = self.select(r.keys(),w.keys(),e.keys(),delay) except error, v: if v.args[0]==EINTR: continue # signal received during select, try again else: raise for fired,events in zip(rwe,self.rwe): for stream in fired: events[stream]().send(True) >(I still think class decorators are a separate case, and much weaker >-- you can do this by having a single 'decoratable' metaclass and >setting __decorators__ = [...] in the class body.) Or you can use the "class advisors" mechanism I implemented for PyProtocols and Zope 3, which is clean and convenient in today's syntax. Its only pitfall is that you absolutely *must* specify __metaclass__ first if you are specifying one, or your class advisors won't work right. Actually, the other pitfall for anybody taking this approach is that a correct implementation of class advisors is *hard*, because it depends on a correct re-implementation of much of Python's metaclass validation logic. But, at least there is one working implementation available to steal from. :) >I think that SPARK syntax and everything else that people have >traditionally added to docstring markup that isn't strictly speaking >documentation (even some extreme cases of doctest usage) ought to be >considered as candidates for attribute-ification. David Goodger mentioned docutils, so I mocked up a couple of 'rst_directive' examples in a seperate message. From pje at telecommunity.com Thu Apr 1 18:21:47 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Apr 1 18:21:53 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> <1080833042.22892.122.camel@localhost.localdomain> Message-ID: <5.1.1.6.0.20040401181954.0271fec0@telecommunity.com> At 06:02 PM 4/1/04 -0500, Bob Ippolito wrote: >No, that's a misunderstanding.. it was by coincidence alone that the last >example I gave had two callbacks with an identical type signature. Using >a "with" block for decorators is as nonsensical as this: > >with args(x): > > def foo: > pass > > def bar: > pass I'm not sure I'd go that far. For 'classmethod' it can actually be attractive to group a bunch of them together. On the other hand, that's the only use case that comes to mind for grouping a bunch of functions under the same decorator list. From goodger at python.org Thu Apr 1 18:43:07 2004 From: goodger at python.org (David Goodger) Date: Thu Apr 1 18:43:13 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: <5.1.1.6.0.20040401140508.02d80090@telecommunity.com> References: <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> <42B02151.420DCBB8@mail.google.com> <200404010544.i315iiB15938@guido.python.org> <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> <406C6566.3040104@python.org> <5.1.1.6.0.20040401140508.02d80090@telecommunity.com> Message-ID: <406CA90B.5020302@python.org> Phillip J. Eby wrote: > Thanks! So for example, this: > > def admonition(*args): > return make_admonition(nodes.admonition, *args) > > admonition.arguments = (1, 0, 1) > admonition.options = {'class': directives.class_option} > admonition.content = 1 ... > > might be rephrased as (say): > > as [rst_directive( > arguments=(1,0,1), > options={'class': directives.class_option}, > content=1 > ) > ] def admonition(*args): > return make_admonition(nodes.admonition, *args) I suppose so, but a generic function attribute decorator would do just as well. IOW, it doesn't have to be "rst_directive", just "attributes" would do fine. -- David Goodger From dave at boost-consulting.com Thu Apr 1 18:45:26 2004 From: dave at boost-consulting.com (David Abrahams) Date: Thu Apr 1 18:46:13 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> <1080833042.22892.122.camel@localhost.localdomain> <5.1.1.6.0.20040401181954.0271fec0@telecommunity.com> Message-ID: "Phillip J. Eby" writes: > I'm not sure I'd go that far. For 'classmethod' it can actually be > attractive to group a bunch of them together. On the other hand, > that's the only use case that comes to mind for grouping a bunch of > functions under the same decorator list. I really suggested it because I think it solves some of the syntactic problems of other approaches. -- Dave Abrahams Boost Consulting www.boost-consulting.com From guido at python.org Thu Apr 1 18:47:29 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 18:47:35 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: Your message of "Thu, 01 Apr 2004 18:43:07 EST." <406CA90B.5020302@python.org> References: <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> <42B02151.420DCBB8@mail.google.com> <200404010544.i315iiB15938@guido.python.org> <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> <406C6566.3040104@python.org> <5.1.1.6.0.20040401140508.02d80090@telecommunity.com> <406CA90B.5020302@python.org> Message-ID: <200404012347.i31NlTH18844@guido.python.org> > > as [rst_directive( > > arguments=(1,0,1), > > options={'class': directives.class_option}, > > content=1 > > ) > > ] def admonition(*args): > > return make_admonition(nodes.admonition, *args) > > I suppose so, but a generic function attribute decorator > would do just as well. IOW, it doesn't have to be > "rst_directive", just "attributes" would do fine. I like rst_directive(), because it's more specific -- it would catch mistakes earlier. E.g. if you misspelled argumnets, attributes() would have no clue about it; but rst_directive() should have specific keywords. --Guido van Rossum (home page: http://www.python.org/~guido/) From shane at zope.com Thu Apr 1 18:54:11 2004 From: shane at zope.com (Shane Hathaway) Date: Thu Apr 1 18:54:49 2004 Subject: [Python-Dev] PEP 318 bake-off? In-Reply-To: <200404010544.i315iiB15938@guido.python.org> References: Your message of "Wed, 31 Mar 2004 21:00:55 PST." <42B02151.420DCBB8@mail.google.com> <200404010544.i315iiB15938@guido.python.org> Message-ID: <406CABA3.7080805@zope.com> Guido van Rossum wrote: > 1) Last-before-colon: > > def foo(cls, arg1, arg2) [funcattrs(foo=42), deprecated, > overrides, classmethod]: > pass > > 2) Prefix list: > > [funcattrs(foo=42), deprecated, overrides, classmethod] > def foo(cls, arg1, arg2): > pass I bet most of us who prefer #2 would be just as happy with #1. Am I right? Shane From greg at cosc.canterbury.ac.nz Thu Apr 1 19:05:39 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Apr 1 19:05:51 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <200403312049.i2VKnEi14444@guido.python.org> Message-ID: <200404020005.i3205dhf018441@cosc353.cosc.canterbury.ac.nz> I think most of the worry about long, complicated decorator lists with baroque parameters is FUD. What are the realistic use cases for such things? I don't remember seeing any proposed. The ones I have seen proposed (e.g. PyObjC) use only one or two decorators at a time, and if there are any parameters, they're not lengthly. Furthermore, if in some circumstance someone finds themselves writing decorator lists so long that they become unreadable, well, there are ways of dealing with that. For example, foo_decorators = decorator_list( # Big long-winded # list of # decorators ) def foo(args) [foo_decorators]: ... which is not much different from Guido's preceding-list version, with the advantage that it's explicit about the fact that you need to look above to get the full picture. Please let's not allow unfounded speculation to get in the way of coming up with a nice syntax that works well in the majority of cases we expect to actually see. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Thu Apr 1 19:10:03 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 1 19:10:21 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: Your message of "Thu, 01 Apr 2004 10:47:21 CST." <406C4799.6030502@4-am.com> References: <42B02151.420DCBB8@mail.google.com> <200404010544.i315iiB15938@guido.python.org> <406C4799.6030502@4-am.com> Message-ID: <200404020010.i320A3X18910@guido.python.org> > How about adding: > > 2b) Prefix list with repeated keyword: > > def [funcattrs(foo=42), deprecated, overrides, classmethod] > def foo(cls, arg1, arg2): > pass > > class [singleton] > class foo: > pass > > 3a) Prefix suite with repeated keyword: > > def: > funcattrs(foo=42) > deprecated, overrides, classmethod > def foo(cls, arg1, arg2): > pass You know, that one occurred to me in the shower, because it should parse easily. But I immediately rejected it as too weird. The stutter doesn't have any semantic connotation to it and is bound to confuse source-scanning tools. (The nice thing about plain prefix list syntax is that tools which look for 'def' but don't process other statements are just as oblivious to it as they are to current decorator syntax.) --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Apr 1 19:12:35 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Apr 1 19:12:41 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: <200404012347.i31NlTH18844@guido.python.org> References: <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> <42B02151.420DCBB8@mail.google.com> <200404010544.i315iiB15938@guido.python.org> <5.1.1.6.0.20040401111202.02d2a530@telecommunity.com> <406C6566.3040104@python.org> <5.1.1.6.0.20040401140508.02d80090@telecommunity.com> <406CA90B.5020302@python.org> Message-ID: <5.1.1.6.0.20040401190831.028443c0@telecommunity.com> At 03:47 PM 4/1/04 -0800, Guido van Rossum wrote: > > > as [rst_directive( > > > arguments=(1,0,1), > > > options={'class': directives.class_option}, > > > content=1 > > > ) > > > ] def admonition(*args): > > > return make_admonition(nodes.admonition, *args) > > > > I suppose so, but a generic function attribute decorator > > would do just as well. IOW, it doesn't have to be > > "rst_directive", just "attributes" would do fine. > >I like rst_directive(), because it's more specific -- it would catch >mistakes earlier. E.g. if you misspelled argumnets, attributes() >would have no clue about it; but rst_directive() should have specific >keywords. Yeah, that's why I wrote the example that way. Once you have an 'attributes()' decorator, it's easy to create task-specific versions of it, e.g.: def my_decorator(foo,bar,baz): return attributes(**locals()) And of course you can have defaults, and do validation of the attributes before the return statement. And there's an additional reason to use task-specific decorators: if you later decide that function attributes aren't as useful, or you need to change what the attributes are named, or you decide to stick all the options into an object and use only one attribute, you need only change the task-specific decorator, not all the things that call it. From shane at zope.com Thu Apr 1 19:24:18 2004 From: shane at zope.com (Shane Hathaway) Date: Thu Apr 1 19:24:22 2004 Subject: [Python-Dev] Vote No on 318 In-Reply-To: <002a01c4182a$8b057420$7329cb97@oemcomputer> References: <1080833042.22892.122.camel@localhost.localdomain> <002a01c4182a$8b057420$7329cb97@oemcomputer> Message-ID: <406CB2B2.1060209@zope.com> Raymond Hettinger wrote: > A failing common to all of the proposals is that they turn action words > (like classmethod) into adjectives with implicit actions. By making > them implicit, it becomes less obvious what is occurring, when it is > occurring, and, for multiple decorators, which order they occur. That > will make the wrapped functions harder to debug. "foo = classmethod(foo)" also uses a noun as a verb. We're no worse than before. I agree that the meaning of 'y' in "def foo(x) [y]" is not obvious, but it is easy to explain: "Python creates function foo, which accepts one parameter named x, then applies the y decorator to the function." > A second failing is introducing a second way to do it. The current > approach will always be there for some purposes, properties, > metaclasses, existing code, or dynamic wrapping. This is a prime > example where two ways to do something is especially bad; if you do not > see [staticmethod] in the definition, can you presume that it is not a > staticmethod -- the answer is clearly no. So, why create optical > illusions. In all places I've seen it used, the current approach sacrifices a lot of clarity. Python should advise against the current approach. > A third failing is clashing with a design preference for > functions/expressions over statements/keywords/syntax. This concept was > at the foundation for discussions about having a print() function, using > sorted() in an expression, or implementing if-then-else expressions. My > understanding of language design history is that of regrets for not > using functions wherever possible. That's an excellent pattern, but Python is much clearer than Lisp because it does not try to take that pattern to its ultimate extreme. > Also, some proposed uses were strange enough that I wonder whether the > syntax encourages people to do weird things with their code. The code many of are working with is already weird, and we're grasping for some way to clean it up. Shane From jack at performancedrivers.com Thu Apr 1 20:11:41 2004 From: jack at performancedrivers.com (Jack Diederich) Date: Thu Apr 1 20:11:47 2004 Subject: [Python-Dev] Vote No on 318 In-Reply-To: <002a01c4182a$8b057420$7329cb97@oemcomputer> References: <1080833042.22892.122.camel@localhost.localdomain> <002a01c4182a$8b057420$7329cb97@oemcomputer> Message-ID: <20040402011141.GB26450@performancedrivers.com> On Thu, Apr 01, 2004 at 03:47:26PM -0500, Raymond Hettinger wrote: > [Jeremy] > > There's no particular reason to believe that effort alone will arrive > > at > > an elegant solution. On the other hand, maybe there isn't a good > > syntax > > for arbitrary decorators. > > IMHO, the best syntax is what we have now. Its only weakness is being > distant from the first line in the definition. Other than that, it is > understandable, unambiguous, simple, backward compatible, and flexible. > > A failing common to all of the proposals is that they turn action words > (like classmethod) into adjectives with implicit actions. By making > them implicit, it becomes less obvious what is occurring, when it is > occurring, and, for multiple decorators, which order they occur. That > will make the wrapped functions harder to debug. I think of them as annotations instead of adjectives. I sympathize for the 'when it is occuring' but I much prefer it included in the func def than very far away. If it is way after the func/class def it is obvious the action happens afterwards. The trick is noticing it happens at all. As for implicit, def foo(cls, arg1, arg2): pass # implicitly a classmethod def foo(cls, arg1, arg2) [classmethod]: pass # unambiguously classmethod > A second failing is introducing a second way to do it. The current > approach will always be there for some purposes, properties, > metaclasses, existing code, or dynamic wrapping. This is a prime > example where two ways to do something is especially bad; if you do not > see [staticmethod] in the definition, can you presume that it is not a > staticmethod -- the answer is clearly no. So, why create optical > illusions. Decorators will be unifying, there is currently more than one equal way to do it (metaclasses, frame hacks, inspection, eval'ed docstrings). 'import this' says there should be one _obvious_ way to do it. Decorators (especially the before-colon kind) are _obvious_. I think 'classmethod' when I've typed 'def foo(cls' *next thought classmethod* I don't think *foo = classmethod(foo)* when I've typed 'return' > A third failing is clashing with a design preference for > functions/expressions over statements/keywords/syntax. This concept was > at the foundation for discussions about having a print() function, using > sorted() in an expression, or implementing if-then-else expressions. My > understanding of language design history is that of regrets for not > using functions wherever possible. 'readability counts', this helps readability for the places where you need it. I'll use decorators more often than genexps, but I like not having to import itertools and calling a function to make an iterator. genexps read like intention, uneccessary function calls read like a recipe. Java is a recipe language, we've all seen comments like "I'm about to do the boilerplate for X, page down for the important stuff" in Java (if you haven't been exposed to Java, feel lucky). def foo(cls): # see modifiers below a = 1 + 1 b = a * 10 return true foo = classmethod(foo) # boilerplate foo.__doc__ = "This function returns true" # boilerplate (which was rectified) > Also, some proposed uses were strange enough that I wonder whether the > syntax encourages people to do weird things with their code. As above, we're cutting out the weird ways people do things now (eval'ing docstrings, examining stackframes, abusing metaclasses) to achieve decorator functionality by making decorators standard. They already had plenty of rope, let's make it all the same color. -jackdied From jim.jewett at eds.com Thu Apr 1 20:37:04 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Thu Apr 1 20:37:04 2004 Subject: [Python-Dev] Re: PEP 318: "as", barelist Message-ID: Phillip J. Eby: > 'as' is not currently a keyword, and making it so would break > any programs that use it as a name. It has been documented as a pseudo-keyword for some time, with the warning that it will may become a real keyword later. > Really, of the current "before the def" proposals, I think I > like Samuele's: *[classmethod] def func(*args): pass approach the best. The '*' seems to say "special note, pay attention". :) I agree that it says special note; it just doesn't say *why* the list is special. We happen to know that it secretly means "After you evaluate this list, keep a reference, then call each member on the function or class I'm about to define." but a keyword (even as) makes it more explicit why this list is special. Of course, as: d1, d2, d2 def func(args): pass also works; if you put a keyword at the front, then [] becomes redundant. And that was one of the bakeoff options, except that Guido used "decorate:" instead of "as:" in his example. -jJ From jack at performancedrivers.com Thu Apr 1 20:38:36 2004 From: jack at performancedrivers.com (Jack Diederich) Date: Thu Apr 1 20:38:42 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <200404020005.i3205dhf018441@cosc353.cosc.canterbury.ac.nz> References: <200403312049.i2VKnEi14444@guido.python.org> <200404020005.i3205dhf018441@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040402013836.GD26450@performancedrivers.com> On Fri, Apr 02, 2004 at 12:05:39PM +1200, Greg Ewing wrote: > I think most of the worry about long, complicated > decorator lists with baroque parameters is FUD. > What are the realistic use cases for such things? > I don't remember seeing any proposed. > > The ones I have seen proposed (e.g. PyObjC) use > only one or two decorators at a time, and if there > are any parameters, they're not lengthly. > > Furthermore, if in some circumstance someone finds > themselves writing decorator lists so long that they > become unreadable, well, there are ways of dealing > with that. For example, > > foo_decorators = decorator_list( > # Big long-winded > # list of > # decorators > ) > > def foo(args) [foo_decorators]: > ... > I was actually about post the same in the bake-off thread, but with the super-obvious see_above = decorator_list( decorator1, .. decoratorN, ) def foo(args) [see_above]: ... You can't beat that for obviousness, even newbies know _something_ is going on and it happens just above. no_joy = decorator_list( decorator1, .. decoratorN, ) [no_joy] def foo(args): # modified by no_joy So the best case is a comment that will vary by style. Do newbies read comments? dunno. -jackdied From greg at cosc.canterbury.ac.nz Thu Apr 1 21:31:06 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Apr 1 21:31:22 2004 Subject: [Python-Dev] Re: ANNOUNCE: 'goto' for Python In-Reply-To: Message-ID: <200404020231.i322V6iC018697@cosc353.cosc.canterbury.ac.nz> Mike Rovner : > Shall I file a bug about python lexer which allows space between class and > attribute? Or that's a fiature? It's certainly not a bug. An identifier and a dot following it are separate lexical tokens, so whitespace should be allowed between them in the interests of consistency, even if it's not all that useful. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu Apr 1 22:13:59 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Apr 1 22:14:12 2004 Subject: [Python-Dev] Vote No on 318 In-Reply-To: <002a01c4182a$8b057420$7329cb97@oemcomputer> Message-ID: <200404020313.i323DxL7024193@cosc353.cosc.canterbury.ac.nz> Raymond Hettinger : > A failing common to all of the proposals is that they turn action > words (like classmethod) into adjectives with implicit actions. Action words? I would have said "classmethod" is a noun, and as such it's a perfectly good name for a decorator. The fact that an action occurs at function definition time is an implementation detail. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From perky at i18n.org Thu Apr 1 22:24:24 2004 From: perky at i18n.org (Hye-Shik Chang) Date: Thu Apr 1 22:24:36 2004 Subject: [Python-Dev] Re: ANNOUNCE: 'goto' for Python In-Reply-To: <200404020231.i322V6iC018697@cosc353.cosc.canterbury.ac.nz> References: <200404020231.i322V6iC018697@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040402032424.GA96150@i18n.org> On Fri, Apr 02, 2004 at 02:31:06PM +1200, Greg Ewing wrote: > Mike Rovner : > > > Shall I file a bug about python lexer which allows space between class and > > attribute? Or that's a fiature? > > It's certainly not a bug. An identifier and a dot following it > are separate lexical tokens, so whitespace should be allowed > between them in the interests of consistency, even if it's not > all that useful. > Agreed. At least it has one usage: >>> 1 .__add__ >>> 1.__add__ File "", line 1 1.__add__ ^ SyntaxError: invalid syntax >>> addone = 1 .__add__ >>> addone(5) 6 Even though it's not so beautiful. :) Hye-Shik From greg at cosc.canterbury.ac.nz Thu Apr 1 22:34:10 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Apr 1 22:34:33 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <200404020005.i3205dhf018441@cosc353.cosc.canterbury.ac.nz> Message-ID: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> Another thought about: [decorator] def foo(args): ... What are newcomers going to make of this? It looks like a thrown-away list followed by a def, and there's no clue that there's any connection between the two. And even if the newcomer suspects that there is some kind of connection, what are they going to look up in the manual? At least with def foo(args) [decorator]: ... it's clear that (1) something different is going on, and (2) it's somehow a part of the 'def' statement. Everywhere else in Python where you can say something someotherthing it means "do something and then do someotherthing", and the combined result can be deduced from knowing what something and someotherthing each do on their own. This breaks that. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From DavidA at ActiveState.com Thu Apr 1 23:09:24 2004 From: DavidA at ActiveState.com (David Ascher) Date: Thu Apr 1 23:10:49 2004 Subject: [Python-Dev] Re: PEP 318: "as", barelist In-Reply-To: References: Message-ID: <406CE774.60400@ActiveState.com> Jewett, Jim J wrote: >>Really, of the current "before the def" proposals, I think I >>like Samuele's: > > > *[classmethod] > def func(*args): > pass > > approach the best. The '*' seems to say "special note, pay > attention". :) I know -- just like Perl6 uses unicode characters (see e.g. http://www.perl.com/pub/a/2004/02/27/exegesis7.html?page=3), we could use the unicode DAGGER character (?). --david From bob at redivi.com Thu Apr 1 23:28:59 2004 From: bob at redivi.com (Bob Ippolito) Date: Thu Apr 1 23:24:53 2004 Subject: [Python-Dev] Re: PEP 318: "as", barelist In-Reply-To: <406CE774.60400@ActiveState.com> References: <406CE774.60400@ActiveState.com> Message-ID: <433D680A-845E-11D8-8B4D-000A95686CD8@redivi.com> On Apr 1, 2004, at 11:09 PM, David Ascher wrote: > Jewett, Jim J wrote: > >>> Really, of the current "before the def" proposals, I think I >>> like Samuele's: >> *[classmethod] >> def func(*args): >> pass >> approach the best. The '*' seems to say "special note, pay >> attention". :) > > I know -- just like Perl6 uses unicode characters (see e.g. > http://www.perl.com/pub/a/2004/02/27/exegesis7.html?page=3), we could > use the unicode DAGGER character (?). if datetime.date.today().isoformat().endswith('04-01'): or we could just ?start using chevrons?. They're particularly easy to type on the Mac (opt-\ and shift-opt-\), and some people are already familiar with them from lowlevel applescript hackery :) -bob From jeremy at alum.mit.edu Thu Apr 1 23:28:50 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu Apr 1 23:30:49 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> Message-ID: <1080880130.22892.158.camel@localhost.localdomain> On Thu, 2004-04-01 at 22:34, Greg Ewing wrote: > Another thought about: > > [decorator] > def foo(args): > ... > > What are newcomers going to make of this? > > It looks like a thrown-away list followed by a def, and > there's no clue that there's any connection between the > two. I think it's obvious that there's a connection between the two. A bare list by itself would be nonsense and the actual list would probably say something like [classmethod] which suggests it's saying something about a method. > And even if the newcomer suspects that there is some > kind of connection, what are they going to look up > in the manual? Maybe they'll lookup functions in the reference manual? I assume they'd be just as puzzled by: def f(x, y, *args, **kwargs): Who came up with that bizarre syntax? Jeremy From python at rcn.com Thu Apr 1 23:40:45 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Apr 1 23:40:59 2004 Subject: [Python-Dev] Vote No on 318 In-Reply-To: <200404020313.i323DxL7024193@cosc353.cosc.canterbury.ac.nz> Message-ID: <008201c4186c$a9c96820$7329cb97@oemcomputer> > > A failing common to all of the proposals is that they turn action > > words (like classmethod) into adjectives with implicit actions. [Greg Ewing] > Action words? I would have said "classmethod" is a noun, and as such > it's a perfectly good name for a decorator. Sure, the current name happens to be a noun instead of "classmethodify", but what it does is action oriented. So, the real issue is declarative versus imperative. Both are great, but don't mix well. > The fact that an action occurs at function definition time is an > implementation detail. It's a somewhat important detail. Raymond From tim.one at comcast.net Fri Apr 2 00:07:33 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri Apr 2 00:07:35 2004 Subject: [Python-Dev] Expert floats In-Reply-To: <007801c41739$cf5b2c10$6402a8c0@arkdesktop> Message-ID: [Andrew Koenig, on the move-to-wider-platform thing] > ... > OK, but either that objection is relevant for Python or it isn't. It's not important to me. It may be important to some Python users now; I don't know. It may be more or less important in the future, because things do change. > ... > On the other hand, maybe ?!ng is right about the desirable properties > of display for people being different from those for > marshalling/unmarshalling. There's no necessary connection between the two, as I've granted for all the years this debate has raged . OTOH, there's no fixed choice for displaying floats that will make everyone happy all the time, or, I think, even most people happy most of the time. You gave a list of 4 different behaviors you sometimes want. I have my own list, including some not on yours: + Display numbers with a comma every third digit. + Display numbers with just a few significant digits, but in "engineering" notation (== exponent always a multiple of 3). Because of this, arguments about repr() and str() are futile, IMO. The flexibility most people want some of the time can't be gotten by changing one fixed behavior for another. This isn't limited to floats, either. ... >> Python would love to grab their I/O implementation then <0.8 wink>. > http://www.research.att.com/sw/tools/sfio/ > > I think the licensing terms are compatible with Python, if you're > serious. It was considered before, but its portablility to non-Unixish system looks poor, and in particular its Windows story appears to require commercial users to buy a UWIN license. Guido started on something different in the Python CVS sandbox/sio. I don't think he wrote a design doc, but the thrust is more to make composable filtering streams (like streams that deal with Unicode gracefully, and that hide differences among the different line end conventions (it's increasingly common to have to deal with a variety of file formats on a single box)). From barry at python.org Fri Apr 2 08:24:43 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 2 08:24:55 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <1080880130.22892.158.camel@localhost.localdomain> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> Message-ID: <1080912282.5481.158.camel@anthem.wooz.org> On Thu, 2004-04-01 at 23:28, Jeremy Hylton wrote: > On Thu, 2004-04-01 at 22:34, Greg Ewing wrote: > > Another thought about: > > > > [decorator] > > def foo(args): > > ... > > > > What are newcomers going to make of this? > > > > It looks like a thrown-away list followed by a def, and > > there's no clue that there's any connection between the > > two. > > I think it's obvious that there's a connection between the two. A bare > list by itself would be nonsense and the actual list would probably say > something like [classmethod] which suggests it's saying something about > a method. It may be nonsense, but it means something today. So it can't be obvious that they're connected because today, they aren't. If tomorrow this same code means something different, users looking at the code will have to know what version of Python they're using, and make sure it's the right one ("uh, how do I do that?"). If they were to use decorator-before-def code in an older version of Python, the program would be accepted but silently do the wrong thing. At least with decorator-before-colon trying to run the code in older interpreters will barf loudly. -Barry From ark-mlist at att.net Fri Apr 2 09:40:08 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Fri Apr 2 09:40:07 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon -- another new syntax idea In-Reply-To: Message-ID: <007501c418c0$655eb6f0$6402a8c0@arkdesktop> > Has something along these lines been discussed? > > with [staticmethod, classmethod]: > > def foo(x): > pass > > def bar(x): > pass > > IIUC, the PyObjC application needs whole swathes of functions with > the same decoration, but this syntax isn't much worse for one > function than for many. If I understand it correctly, the motivation for this idea is to try to avoid cluttering the definitions with large numbers of decorators. In which case, how about this? sc = [staticmethod, classmethod] def foo(x) sc: pass def bar(x) sc: pass In other words, if the decorators are too voluminous, do what you always do with a complicated expression: Bind a variable to it and use the variable instead. From casey at zope.com Fri Apr 2 10:55:35 2004 From: casey at zope.com (Casey Duncan) Date: Fri Apr 2 10:58:46 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> <1080833042.22892.122.camel@localhost.localdomain> Message-ID: <20040402105535.0526c27d.casey@zope.com> On Thu, 01 Apr 2004 17:25:20 -0500 David Abrahams wrote: > Jeremy Hylton writes: > > > On Thu, 2004-04-01 at 07:13, Michael Hudson wrote: > >> > I don't think Michel is saying they are worthless. However, the > >> > proposed syntax is highly contentious. It would be good if there > >> > was a short term solution that wouldn't require new syntax. That > >> > would give Guido and the Python community time to figure out the > >> > best syntax. > >> > >> We've been discussing this off and on for OVER A YEAR! If 'the > >best> syntax' hasn't been figured out yet after N thousand emails on > >the> subject, I see no reason to believe enlightenment is going to > >arrive> soon (or ever). > > > > There's no particular reason to believe that effort alone will > > arrive at an elegant solution. On the other hand, maybe there isn't > > a good syntax for arbitrary decorators. > > Has something along these lines been discussed? > > with [staticmethod, classmethod]: > > def foo(x): > pass > > def bar(x): > pass > > IIUC, the PyObjC application needs whole swathes of functions with > the same decoration, but this syntax isn't much worse for one > function than for many. What if you dropped the keyword? [classmethod]: def splat(cls): pass def baz(cls): pass Or how about as? as classmethod: def jinkies(cls): pass Which seems nice in the face of other declarations: as protected(some_permission): def zoinks(self, scooby, snack): pass def jinkies(self): pass or [protected_by(some_permission)]: def dangerous(self, risk): pass At least then the ambiguity is gone wrt unassigned lists before defs (and I won't have to rewrite all that code where I'm using list comprehensions as docs before method defs ;^). Also the grouping seems useful. OTOH, madatory bracketing seems unpythonic to me, which makes as/with option (without mandatory brackets) seem compelling. -Casey From mwh at python.net Fri Apr 2 11:34:02 2004 From: mwh at python.net (Michael Hudson) Date: Fri Apr 2 11:34:06 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <1080912282.5481.158.camel@anthem.wooz.org> (Barry Warsaw's message of "Fri, 02 Apr 2004 08:24:43 -0500") References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> Message-ID: <2m1xn6gwzp.fsf@starship.python.net> Barry Warsaw writes: > If tomorrow this same code means something different, users looking at > the code will have to know what version of Python they're using, and > make sure it's the right one ("uh, how do I do that?"). If they were to > use decorator-before-def code in an older version of Python, the program > would be accepted but silently do the wrong thing. > > At least with decorator-before-colon trying to run the code in older > interpreters will barf loudly. I think this is a good point that hadn't occurred to me: silent failures suck. Cheers, mwh -- Like most people, I don't always agree with the BDFL (especially when he wants to change things I've just written about in very large books), ... -- Mark Lutz, http://python.oreilly.com/news/python_0501.html From jim.jewett at eds.com Fri Apr 2 11:44:17 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Fri Apr 2 11:45:12 2004 Subject: [Python-Dev] PEP 318 bake-off? Message-ID: Brad Clements: > Guido van Rossum wrote: >> I think that SPARK syntax and everything else that people have >> traditionally added to docstring markup that isn't strictly speaking >> documentation (even some extreme cases of doctest usage) ought to be >> considered as candidates for attribute-ification. > Where do method attribute type signatures and DBC fit in? > As a decorator, or in the docstring? While it will still be *possible* to abuse the docstring, it *should* go as a decorator. Not every machine-readable class invariant is useful documentation to someone who isn't already debugging that code in particular. > I'm concerned that the funcattrs(a="xyz" .. ) sample tossed > around here will be rather ugly for specifying DBC strings. I agree that it would be better to create a DBC class. If there is a DBC attribute on a code object, then DBC-aware tools will look to that object for the DBC conditions. Whether you express them as strings or predicate functions or ... your choice. As a specific case, a debugger could use object.__DBC__.__call__ instead of object.__call__. (If you wanted to enforce the DBC checks at all times, then your decorator could just return a new object that checks and delegates, rather than an annotated version of the original.) > Finally, I don't have a need to access DBC annotations at > runtime once my module is distributed. I would not want to > pay the memory cost overhead of loading DBC information or > attribute type signatures at runtime. Then define a release-DBC class whose constructor is pass, and whose decoration is to return the object unchanged (without a DBC attribute). Use that in your released code. Whether to strip DBC info entirely or just throw it away on loading is up to you. > However another person at PyCon poo-poo'd my concern over > bloating .pyc files and subsequent memory use. As a compromise > I suggested that "annotation" information could go into the > .pyc, but be loaded "on demand" at runtime. Changing the format of .pyc is beyond the scope of PEP318. If you want a special case for DBC, you can write it. Make your DBC class save the annotations for example.py to example.dbc, and retrieve them on demand. You may have to go through a rewrite/reload step to get them out of the .pyc without removing them from the .py, but you can do it. If on-demand is not required, you could probably change the compiler to ignore any attribute registered as ignorable during optimization, instead of just __doc__. -jJ From fdrake at acm.org Fri Apr 2 11:59:47 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Apr 2 12:01:02 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <2m1xn6gwzp.fsf@starship.python.net> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080912282.5481.158.camel@anthem.wooz.org> <2m1xn6gwzp.fsf@starship.python.net> Message-ID: <200404021159.47419.fdrake@acm.org> Barry Warsaw writes: > At least with decorator-before-colon trying to run the code in older > interpreters will barf loudly. On Friday 02 April 2004 11:34 am, Michael Hudson wrote: > I think this is a good point that hadn't occurred to me: silent > failures suck. There's also the little matter that using syntax only, without new keywords, allows us to avoid __future__ imports and all the implementation cruft that entails. Decorator-before-colon wins out big time for me. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jeremy at alum.mit.edu Fri Apr 2 12:05:13 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Apr 2 12:07:16 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <1080912282.5481.158.camel@anthem.wooz.org> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> Message-ID: <1080925513.22892.173.camel@localhost.localdomain> On Fri, 2004-04-02 at 08:24, Barry Warsaw wrote: > It may be nonsense, but it means something today. So it can't be > obvious that they're connected because today, they aren't. Your original complaint was: "What are newcomers going to make of this?" Newcomers aren't going to be worried about the small change in semantics that decorator-before would entail, because they won't know the old semantics. It's only a problem for people who already know the language, but one would hope they read the "What's New in Python 2.4" document. I expect a newcomer to read the code in the most natural way. Only a language lawyer would worry about whether the expression "[classmethod]" was evaluated and thrown away rather than referring to fromFoo. Language lawyers should be clever enough to figure out the new rules :-). class Quux: def __init__(self): ... [classmethod] def fromFoo(cls, arg): """Create a Quux instance from a Foo.""" ... > If tomorrow this same code means something different, users looking at > the code will have to know what version of Python they're using, and > make sure it's the right one ("uh, how do I do that?"). If they were to > use decorator-before-def code in an older version of Python, the program > would be accepted but silently do the wrong thing. I agree there's a risk here, but we've faced that kind of risk before. We used future statements for nested scopes, but only one version. If you're looking at code with free variables, you need to know whether it was written to run with or without nested scopes. Code written for one version will fail at runtime on the other. (It may or may not fail silently.) Jeremy From theller at python.net Fri Apr 2 12:11:42 2004 From: theller at python.net (Thomas Heller) Date: Fri Apr 2 12:26:04 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <2m1xn6gwzp.fsf@starship.python.net> (Michael Hudson's message of "Fri, 02 Apr 2004 17:34:02 +0100") References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <2m1xn6gwzp.fsf@starship.python.net> Message-ID: Michael Hudson writes: > Barry Warsaw writes: > >> If tomorrow this same code means something different, users looking at >> the code will have to know what version of Python they're using, and >> make sure it's the right one ("uh, how do I do that?"). If they were to >> use decorator-before-def code in an older version of Python, the program >> would be accepted but silently do the wrong thing. >> >> At least with decorator-before-colon trying to run the code in older >> interpreters will barf loudly. > > I think this is a good point that hadn't occurred to me: silent > failures suck. Wouldn't the decorator-before-def require a 'from __future__ import decorators' (Although I'm still on favor of the decorator-before-colon version)? Thomas From barry at python.org Fri Apr 2 14:08:56 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 2 14:09:09 2004 Subject: [Python-Dev] PEP 318: Let's not give in to FUD In-Reply-To: <1080925513.22892.173.camel@localhost.localdomain> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> Message-ID: <1080932935.5481.282.camel@anthem.wooz.org> On Fri, 2004-04-02 at 12:05, Jeremy Hylton wrote: > On Fri, 2004-04-02 at 08:24, Barry Warsaw wrote: > > It may be nonsense, but it means something today. So it can't be > > obvious that they're connected because today, they aren't. > > Your original complaint was: "What are newcomers going to make of > this?" Newcomers aren't going to be worried about the small change in > semantics that decorator-before would entail, because they won't know > the old semantics. They already know the old semantics. If they were to encounter that syntax today, they'd know that Python throws away the results. If /I/ saw that construct in today's Python code, I'd start looking for side effects. > > If tomorrow this same code means something different, users looking at > > the code will have to know what version of Python they're using, and > > make sure it's the right one ("uh, how do I do that?"). If they were to > > use decorator-before-def code in an older version of Python, the program > > would be accepted but silently do the wrong thing. > > I agree there's a risk here, but we've faced that kind of risk before. > We used future statements for nested scopes, but only one version. If > you're looking at code with free variables, you need to know whether it > was written to run with or without nested scopes. Code written for one > version will fail at runtime on the other. (It may or may not fail > silently.) Adding a future statement would make decorator-before-def slightly more acceptable. Adding a keyword to introduce decorator-before-def would make it slightly more acceptable. All-in-all, I still think decorator-before-colon is plenty readable and the more obvious of the two choices. -Barry From guido at python.org Fri Apr 2 14:50:35 2004 From: guido at python.org (Guido van Rossum) Date: Fri Apr 2 14:50:42 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: Your message of "Fri, 02 Apr 2004 14:08:56 EST." <1080932935.5481.282.camel@anthem.wooz.org> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> Message-ID: <200404021950.i32JoZN21741@guido.python.org> While I ponder the decorator syntax, let's propose some built-in decorators. We've already got classmethod and staticmethod. I propose this one to set function attributes: class func_attrs(objects): def __init__(self, **kwds): self.attrs = kwds def __call__(self, funcobj): funcobj.__dict__.update(self.attrs) Why a class? So you can subclass it like this: class rst_attrs(func_attrs): def __init__(self, arguments, options, content): func_attrs.__init__(self, arguments=arguments, options=options, content=content) Why would you want to do that? So that mistakes in the attribute names would be caught early, and perhaps default values could be provided. We could also add a standard implementation of synchronized. Or perhaps that should be imported from threading. (But is that really a good thing to copy from Java?) Other possibilities (all these are pretty much thinking aloud and very much up for votes; and not original, I've seen these proposed before!): deprecated(comment_string) -- mostly for documentation, but would set the deprecated attribute. The comment_string argument is a string explaining why it is deprecated. Maybe this should also have arguments specifying the first Python release in which it was considered deprecated and the last Python release (if known) where it will be available. overrides -- indicates that this overrides a base class method. Maybe the default metaclass could check that if this is used there actually is a corresponding base class method, and we might have a "strict" metaclass that checks this is set for all overriding methods. doctest_script(multi_line_string) -- specifies a doctest script, for use by the doctest module (if this is absent, it will continue to look in the doc string). I like to separate the doctest script from the actual documentation string because, when used for rigorous unit testing (as in Jim&Tim's talk at PyCON), the doctest script is too long to double as reasonable documentation. Maybe this should be imported from the doctest module. decorator_chain(*args) -- takes any number of decorator arguments and applies them sequentially. Implementation could be: def decorator_chain(*args): def decorate(func): for arg in args: func = arg(func) return func return decorate (I don't see a reason to do this with a class.) I'm not a fan of using function attributes for specifying author, version, copyright etc.; those things usually work on a larger granularity than methods anyway, and belong in the module doc string. I'm still torn whether to promote defining properties this way: [propget] def x(self): "Doc string for x" return self.__x [propset] def x(self, newx): self.__x = newx [propdel] def x(self): del self.__x but if people like this (whatever the decorator syntax :) we might as well make this the recommended way to define properties. Should there be a a separate module from which all those decorators are imported, or should we make them built-ins, following the trend set by classmethod etc.? Not counting the ones that I already mark as to be imported from a special place, like synchronized (from threading.py) and doctest_script (from doctest.py). Proposals for other standard decorators are welcome! --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs at theopalgroup.com Fri Apr 2 15:03:15 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Fri Apr 2 15:03:18 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: <200404021950.i32JoZN21741@guido.python.org> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> Message-ID: <406DC703.8050708@theopalgroup.com> Guido van Rossum wrote: >While I ponder the decorator syntax, let's propose some built-in >decorators. > >We've already got classmethod and staticmethod. > >I propose this one to set function attributes: > > class func_attrs(objects): > > def __init__(self, **kwds): > self.attrs = kwds > > def __call__(self, funcobj): > funcobj.__dict__.update(self.attrs) > > > Did you leave out the 'return funcobj' from the end of __call__? I thought that decorators were supposed to be inherently cooperative, and should return their modified funcobj, or a new func-like-obj. Or maybe I haven't been paying close enough attention... -Kevin From guido at python.org Fri Apr 2 15:08:03 2004 From: guido at python.org (Guido van Rossum) Date: Fri Apr 2 15:08:14 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: Your message of "Fri, 02 Apr 2004 15:03:15 EST." <406DC703.8050708@theopalgroup.com> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> <406DC703.8050708@theopalgroup.com> Message-ID: <200404022008.i32K83F21842@guido.python.org> > > class func_attrs(objects): > > > > def __init__(self, **kwds): > > self.attrs = kwds > > > > def __call__(self, funcobj): > > funcobj.__dict__.update(self.attrs) > > > Did you leave out the 'return funcobj' from the end of __call__? I > thought that decorators were supposed to be inherently cooperative, > and should return their modified funcobj, or a new func-like-obj. Sorry, you're right. (I've been thinking of interpreting a None result as "keep the input object" but the code generation would be too messy. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs at theopalgroup.com Fri Apr 2 15:15:20 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Fri Apr 2 15:15:25 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: <200404022008.i32K83F21842@guido.python.org> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> <406DC703.8050708@theopalgroup.com> <200404022008.i32K83F21842@guido.python.org> Message-ID: <406DC9D8.8040206@theopalgroup.com> Guido van Rossum wrote: >>> class func_attrs(objects): >>> >>> def __init__(self, **kwds): >>> self.attrs = kwds >>> >>> def __call__(self, funcobj): >>> funcobj.__dict__.update(self.attrs) >>> >>> >>> >>Did you leave out the 'return funcobj' from the end of __call__? I >>thought that decorators were supposed to be inherently cooperative, >>and should return their modified funcobj, or a new func-like-obj. >> >> > >Sorry, you're right. (I've been thinking of interpreting a None >result as "keep the input object" but the code generation would be too >messy. > > Cool. And now that I have my pedantic hat on, I may as well go all out. First, why call it func_attrs, when staticmethod and classmethod are underscoreless? Second, I know it is effectively the same, but shouldn't the .update line use vars(funcobj) instead of funcobj.__dict__? This is something that I am asked (often!) by my Python students. I use vars(obj) since it looks less magical. -Kevin From guido at python.org Fri Apr 2 15:32:16 2004 From: guido at python.org (Guido van Rossum) Date: Fri Apr 2 15:32:27 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: Your message of "Fri, 02 Apr 2004 15:15:20 EST." <406DC9D8.8040206@theopalgroup.com> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> <406DC703.8050708@theopalgroup.com> <200404022008.i32K83F21842@guido.python.org> <406DC9D8.8040206@theopalgroup.com> Message-ID: <200404022032.i32KWG821938@guido.python.org> > Cool. And now that I have my pedantic hat on, I may as well go all > out. First, why call it func_attrs, when staticmethod and > classmethod are underscoreless? Just for variety. :-) We had a discussion about naming conventions recently, and I believe the outcome was that most folks like underscores in their names. OTOH if I was writing this just for *me*, I would indeed make them underscoreless. > Second, I know it is effectively the same, but shouldn't the .update > line use vars(funcobj) instead of funcobj.__dict__? This is > something that I am asked (often!) by my Python students. I use > vars(obj) since it looks less magical. No, vars() might return a copy. __dict__ is the real thing (unless it isn't -- try updating the __dict__ of a new-style class :-). But perhaps the real implementation should use setattr() anyway -- some function attributes are not stored in the __dict__ (like __doc__) and these should still be settable this way (if they are settable at all). And if they are *not* settable, this should raise an error rather than silently creating an inaccessible entry in __dict__. So let's rephrase that class as: class funcattrs(object): def __init__(self, **kwds): self.attrs = kwds def __call__(self, func): for name, value in self.attrs.iteritems(): setattr(func, name, value) return func --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Fri Apr 2 15:33:17 2004 From: aahz at pythoncraft.com (Aahz) Date: Fri Apr 2 15:33:25 2004 Subject: [Python-Dev] PEP 327: Decimal data type In-Reply-To: References: Message-ID: <20040402203317.GA8514@panix.com> On Wed, Mar 31, 2004, Batista, Facundo wrote: > > This PEP is ready for your review. Excellent work! Only one nitpick: > Syntax for All Cases > '''''''''''''''''''' > > :: > > Decimal(value1) > Decimal.from_float(value2, [decimal_digits]) > > where ``value1`` can be int, long, string, tuple or Decimal, > ``value1`` can be only float, and ``decimal_digits`` is an optional > int. You mean value2 in the second case, right? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From pje at telecommunity.com Fri Apr 2 15:35:26 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Apr 2 15:35:49 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: <406DC9D8.8040206@theopalgroup.com> References: <200404022008.i32K83F21842@guido.python.org> <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> <406DC703.8050708@theopalgroup.com> <200404022008.i32K83F21842@guido.python.org> Message-ID: <5.1.1.6.0.20040402153139.02d704a0@telecommunity.com> At 03:15 PM 4/2/04 -0500, Kevin Jacobs wrote: >Guido van Rossum wrote: > >>>>class func_attrs(objects): >>>> >>>> def __init__(self, **kwds): >>>> self.attrs = kwds >>>> >>>> def __call__(self, funcobj): >>>> funcobj.__dict__.update(self.attrs) >>>> >>>> >>>Did you leave out the 'return funcobj' from the end of __call__? I >>>thought that decorators were supposed to be inherently cooperative, >>>and should return their modified funcobj, or a new func-like-obj. >>> >> >>Sorry, you're right. (I've been thinking of interpreting a None >>result as "keep the input object" but the code generation would be too >>messy. >> > >Cool. And now that I have my pedantic hat on, I may as well go all >out. First, >why call it func_attrs, when staticmethod and classmethod are underscoreless? >Second, I know it is effectively the same, but shouldn't the .update line use >vars(funcobj) instead of funcobj.__dict__? This is something that I am asked >(often!) by my Python students. I use vars(obj) since it looks less magical. For that matter, updating the dictionary is nice as a quick trick, but what if the object doesn't *have* a dictionary? Using setattr would be safer, if this is intended to be subclassed. Suppose, for example, that 'synchronized' actually returns a callable wrapper written in C, rather than a function object, and that wrapper then delegates setattr to the thing it wraps? So, I think using setattr would be safer, and __dict__.update() is premature optimization, given that function object creation and decorator invocation isn't likely to be a performance bottleneck. From barry at python.org Fri Apr 2 15:40:42 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 2 15:41:00 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: <200404022032.i32KWG821938@guido.python.org> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> <406DC703.8050708@theopalgroup.com> <200404022008.i32K83F21842@guido.python.org> <406DC9D8.8040206@theopalgroup.com> <200404022032.i32KWG821938@guido.python.org> Message-ID: <1080938441.5481.306.camel@anthem.wooz.org> On Fri, 2004-04-02 at 15:32, Guido van Rossum wrote: > We had a discussion about naming conventions recently, and I believe > the outcome was that most folks like underscores in their names. OTOH > if I was writing this just for *me*, I would indeed make them > underscoreless. Actually, for multi-word method and function names, we preferred underscores to camelCasing. But I still wish has_key() were haskey() and since neither "func" nor "attr" are actual words, funcattrs() is just fine. :) I'm torn on whether new decorators should go in as built-ins or not. It's darn convenient to stick them there, but I worry that the documentation for the builtin module is getting pretty hefty now (and no, I really don't want to do something like an os module split -- man, I hate those pages ;). -Barry From dave at boost-consulting.com Fri Apr 2 15:45:12 2004 From: dave at boost-consulting.com (David Abrahams) Date: Fri Apr 2 15:46:23 2004 Subject: [Python-Dev] Re: Vote No on 318 References: <200404020313.i323DxL7024193@cosc353.cosc.canterbury.ac.nz> <008201c4186c$a9c96820$7329cb97@oemcomputer> Message-ID: "Raymond Hettinger" writes: >> > A failing common to all of the proposals is that they turn action >> > words (like classmethod) into adjectives with implicit actions. > > [Greg Ewing] >> Action words? I would have said "classmethod" is a noun, and as such >> it's a perfectly good name for a decorator. > > Sure, the current name happens to be a noun instead of "classmethodify", > but what it does is action oriented. So, the real issue is declarative > versus imperative. Both are great, but don't mix well. I disagree, especially where Python is concerned. Declarations in Python, if that even has meaning, are imperative by nature. Try running this: def f(): g() f() def g(): pass The highest level languages are often declarative. The fact that the declarations may be executed is, as Greg says, an implementation detail. -- Dave Abrahams Boost Consulting www.boost-consulting.com From tim.one at comcast.net Fri Apr 2 15:48:37 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri Apr 2 15:48:42 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: <200404021950.i32JoZN21741@guido.python.org> Message-ID: [Guido] > ... > doctest_script(multi_line_string) -- specifies a doctest script, for > use by the doctest module (if this is absent, it will continue to look > in the doc string). I like to separate the doctest script from the > actual documentation string because, when used for rigorous unit > testing (as in Jim&Tim's talk at PyCON), the doctest script is too > long to double as reasonable documentation. Maybe this should be > imported from the doctest module. Let me try to squash this one before it gains traction from people who don't use doctest anyway . Burying a large triple-quoted string inside a function call inside an indented "list" isn't going to help the readability of anything. Zope use is moving toward putting large doctests into their own .py file, or, more recently, into their own .txt file. An example of the former: http://cvs.zope.org/ZODB/src/ZODB/tests/testmvcc.py The idea that "too long" makes for unreasonable documentation doesn't make sense to me: these long doctests read like tutorials liberally sprinkled with Python examples, and are the best docs some of these subsystems will ever have. Indeed, the Python Tutorial is a prime example of something that "should be" a doctest, and I think that's exceptionally good documentation. Of course this doesn't preclude putting a judiciously few examples in docstrings either (also for doctest chewing), but I'd rather encourage people who want more than that to go all the way, and write a tutorial-style large doctest in its own file. From dave at boost-consulting.com Fri Apr 2 15:48:07 2004 From: dave at boost-consulting.com (David Abrahams) Date: Fri Apr 2 15:50:31 2004 Subject: [Python-Dev] Re: PEP 318: Let's not give in to FUD References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080912282.5481.158.camel@anthem.wooz.org> <2m1xn6gwzp.fsf@starship.python.net> <200404021159.47419.fdrake@acm.org> Message-ID: "Fred L. Drake, Jr." writes: > Barry Warsaw writes: > > At least with decorator-before-colon trying to run the code in older > > interpreters will barf loudly. > > On Friday 02 April 2004 11:34 am, Michael Hudson wrote: > > I think this is a good point that hadn't occurred to me: silent > > failures suck. > > There's also the little matter that using syntax only, without new keywords, > allows us to avoid __future__ imports and all the implementation cruft that > entails. > > Decorator-before-colon wins out big time for me. +1 -- Dave Abrahams Boost Consulting www.boost-consulting.com From rowen at cesmail.net Fri Apr 2 16:01:03 2004 From: rowen at cesmail.net (Russell E. Owen) Date: Fri Apr 2 16:01:11 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> <1080833042.22892.122.camel@localhost.localdomain> <20040402105535.0526c27d.casey@zope.com> Message-ID: In article <20040402105535.0526c27d.casey@zope.com>, Casey Duncan wrote: > Or how about as? > > as classmethod: > > def jinkies(cls): > pass > > Which seems nice in the face of other declarations: > > as protected(some_permission): > > def zoinks(self, scooby, snack): > pass > > def jinkies(self): > pass > > or > > [protected_by(some_permission)]: > > def dangerous(self, risk): > pass I like this a lot. The effort to avoid new keywords, though laudable in many ways, comes at the cost of increased use of punctuation. I generally find punctuation harder to read than keywords. Also, it is usually incomprehensible to someone unfamiliar with the language. The relative lack of punctuation in python as compared, say, to C is one if its great strengths, to my mind. It is one reason Python reads so much like pseudocode (or did before some of the recent changes such as list comprehensions -- very useful but a classic example of the problem of overloading punctuation). I'd be very sad to see [] used for yet some other purpose. -- Russell From pedronis at bluewin.ch Fri Apr 2 16:16:05 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Fri Apr 2 16:11:09 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <20040402105535.0526c27d.casey@zope.com> References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> <1080833042.22892.122.camel@localhost.localdomain> Message-ID: <5.2.1.1.0.20040402231022.030c0058@pop.bluewin.ch> At 10:55 02.04.2004 -0500, Casey Duncan wrote: > [protected_by(some_permission)]: > > def dangerous(self, risk): > pass I think the syntax (of which the proposed one is a subcase) expr: suite is better reserved for even more mischievous endeavors, even if just fantasized. From bob at redivi.com Fri Apr 2 16:22:13 2004 From: bob at redivi.com (Bob Ippolito) Date: Fri Apr 2 16:18:10 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <5.2.1.1.0.20040402231022.030c0058@pop.bluewin.ch> References: <1080759573.2043.49.camel@heinlein> <97B52378-834E-11D8-8201-000A95686CD8@redivi.com> <20040331234312.GA14993@mems-exchange.org> <2mekr7hp57.fsf@starship.python.net> <1080833042.22892.122.camel@localhost.localdomain> <5.2.1.1.0.20040402231022.030c0058@pop.bluewin.ch> Message-ID: On Apr 2, 2004, at 4:16 PM, Samuele Pedroni wrote: > At 10:55 02.04.2004 -0500, Casey Duncan wrote: > >> [protected_by(some_permission)]: >> >> def dangerous(self, risk): >> pass > > I think the syntax (of which the proposed one is a subcase) > > expr: > suite > > is better reserved for even more mischievous endeavors, even if just > fantasized. +1 :) From aahz at pythoncraft.com Fri Apr 2 20:16:47 2004 From: aahz at pythoncraft.com (Aahz) Date: Fri Apr 2 20:17:00 2004 Subject: [Python-Dev] PEP 328 -- relative and multi-line import In-Reply-To: <200403302259.i2UMxLi10184@guido.python.org> References: <200403302218.i2UMIEV10058@guido.python.org> <20040330225324.GA1947@panix.com> <200403302259.i2UMxLi10184@guido.python.org> Message-ID: <20040403011647.GA8816@panix.com> On Tue, Mar 30, 2004, Guido van Rossum wrote: > > Consider it Pronounced. Okay, I've filed the updated PEP. Who wants to implement it? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From dave at boost-consulting.com Fri Apr 2 13:16:46 2004 From: dave at boost-consulting.com (David Abrahams) Date: Fri Apr 2 20:38:55 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon -- another new syntax idea In-Reply-To: <007501c418c0$655eb6f0$6402a8c0@arkdesktop> (Andrew Koenig's message of "Fri, 2 Apr 2004 09:40:08 -0500") References: <007501c418c0$655eb6f0$6402a8c0@arkdesktop> Message-ID: "Andrew Koenig" writes: >> Has something along these lines been discussed? >> >> with [staticmethod, classmethod]: >> >> def foo(x): >> pass >> >> def bar(x): >> pass >> >> IIUC, the PyObjC application needs whole swathes of functions with >> the same decoration, but this syntax isn't much worse for one >> function than for many. > > If I understand it correctly, the motivation for this idea is to try to > avoid cluttering the definitions with large numbers of decorators. Actually no. It was that other syntaxes such as as: [whatever] def foo(): pass seemed to lack the scoping that needed to make the "as" part apply to the definition. -- Dave Abrahams Boost Consulting www.boost-consulting.com From pythondev at bitfurnace.com Fri Apr 2 15:22:02 2004 From: pythondev at bitfurnace.com (pythondev@bitfurnace.com) Date: Fri Apr 2 20:38:58 2004 Subject: [Python-Dev] PEP 318: Is decorators the right term for these things? Message-ID: <26182.216.254.103.139.1080937322.squirrel@my.modwest.com> I know this discussion has focussed on the term "decorator", but I think the term is a misnomer. These things arent just passive, static decorations, but are rather more dynamic active things; possibly "transformations" or "transforms". From bob at redivi.com Fri Apr 2 20:51:35 2004 From: bob at redivi.com (Bob Ippolito) Date: Fri Apr 2 20:47:30 2004 Subject: [Python-Dev] PEP 318: Is decorators the right term for these things? In-Reply-To: <26182.216.254.103.139.1080937322.squirrel@my.modwest.com> References: <26182.216.254.103.139.1080937322.squirrel@my.modwest.com> Message-ID: <706B841B-8511-11D8-AE8C-000A95686CD8@redivi.com> On Apr 2, 2004, at 3:22 PM, wrote: > I know this discussion has focussed on the term "decorator", but I > think > the term is a misnomer. These things arent just passive, static > decorations, but are rather more dynamic active things; possibly > "transformations" or "transforms". > > Sometimes they are, sometimes they aren't. Many of the useful cases actually are transforms (classmethod, staticmethod, objc.selector), not merely decoration (as an interface declaration may be, it could return the original function unchanged, it really depends on the adaptation framework). I do pretty much agree with you though. -bob From edloper at gradient.cis.upenn.edu Fri Apr 2 21:19:17 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Fri Apr 2 21:17:41 2004 Subject: [Python-Dev] Re: PEP 318: Let's propose some useful built-in decorators In-Reply-To: References: Message-ID: <406E1F25.9090702@gradient.cis.upenn.edu> > deprecated(comment_string) -- mostly for documentation, but would set > the deprecated attribute. The comment_string argument is a string > explaining why it is deprecated. Maybe this should also have > arguments specifying the first Python release in which it was > considered deprecated and the last Python release (if known) where it > will be available. Should this be a python-specific decorator? Or should it also be used to mark deprecated objects in external libraries? (The answer will affect the arguments specifing when it became depreciated and when it will go away.) If we add this, we might want to also define a standard way of marking classes and modules as depreciated. (Perhaps by setting the __deprecated__ variable/property in all 3 cases.) > overrides -- indicates that this overrides a base class method. Maybe > the default metaclass could check that if this is used there actually > is a corresponding base class method, and we might have a "strict" > metaclass that checks this is set for all overriding methods. Alternatively, it could be checked by an external inspection tool like pychecker. (Since metaclasses are so difficult to mix, it's probably better not to use them when we don't need to.) -Edward From python at rcn.com Sat Apr 3 04:27:52 2004 From: python at rcn.com (Raymond Hettinger) Date: Sat Apr 3 04:28:31 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: <200404021950.i32JoZN21741@guido.python.org> Message-ID: <004601c4195d$f03255a0$9433c797@oemcomputer> > While I ponder the decorator syntax, let's propose some built-in > decorators. Okay, I'll play (despite my thumbs down on the pep and especially the syntax using a list prefixing the function). Several people have emphasized that the timing of decorator application is only an implementation detail. So, that leaves open the possibility of using function decorators as compilation pragmas. One possible pragma tells the compiler that the function may assume that specified global variables are constant and already known. This allows the globals to be loaded in the constant table and accessed with LOAD_GLOBAL instead of LOAD_CONSTANT. Another pragma tells the compiler to assume that specified attribute lookup targets won't change outside the function. This allows the lookups to be precomputed as soon as the root is available or whenever it changes. Example ------- X = 20 [earlybind(int, X), cacheattr("mylist.append", "result.append", "random.random", "self.score")] def meth(self, data) result = [] for sub in data: mylist = [] for elem in sub: if int(elem) == X: mylist.append(elem) else: elem += random.random() self.score(elem) result.append(mylist) return result Transformation -------------- def meth(self, data, int=int, X=20) _rr = random.random _ss = self.score result = [] _ra = result.append for sub in data: mylist = [] _ma = mylist.append for elem in sub: if int(elem) == X: _ma(elem) else: elem += _rr() _ss(elem) _ra(mylist) return result Adding the decorator line is sufficient to localize everything in the function, eliminate all lookups from the inner loop, and still keep the code clean and readable. with-this-much-weirdness-who-needs-to-go-pysco-ly yours, Raymond Hettinger From mwh at python.net Sat Apr 3 05:58:57 2004 From: mwh at python.net (Michael Hudson) Date: Sat Apr 3 05:59:02 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: <200404021950.i32JoZN21741@guido.python.org> (Guido van Rossum's message of "Fri, 02 Apr 2004 11:50:35 -0800") References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> Message-ID: <2mwu4xfhu6.fsf@starship.python.net> Guido van Rossum writes: > While I ponder the decorator syntax, let's propose some built-in > decorators. [...] > We could also add a standard implementation of synchronized. Or > perhaps that should be imported from threading. (But is that really > a good thing to copy from Java?) I don't want to sound FUDdy, but I was under the impression that people didn't think this is something we actually want... > Other possibilities (all these are pretty much thinking aloud and very > much up for votes; and not original, I've seen these proposed > before!): [...] > overrides -- indicates that this overrides a base class method. Maybe > the default metaclass could check that if this is used there actually > is a corresponding base class method, and we might have a "strict" > metaclass that checks this is set for all overriding methods. I don't get the point of this. > I'm still torn whether to promote defining properties this way: > > [propget] > def x(self): > "Doc string for x" > return self.__x > > [propset] > def x(self, newx): > self.__x = newx > > [propdel] > def x(self): > del self.__x > > but if people like this (whatever the decorator syntax :) we might as > well make this the recommended way to define properties. - "a bit" I'm not sure I like the idea of promoting something that uses sys._getframe. > Should there be a a separate module from which all those decorators > are imported, or should we make them built-ins, following the trend > set by classmethod etc.? I'm not sure any of the above are important enough to be builtins. funcattrs, maybe, but there's sod all reason to implement that in C, and getting Python implementations into the builtins seems likely to be pain. Could do it in site.py, I guess. Cheers, mwh -- Also, remember to put the galaxy back when you've finished, or an angry mob of astronomers will come round and kneecap you with a small telescope for littering. -- Simon Tatham, ucam.chat, from Owen Dunn's review of the year From goodger at python.org Sat Apr 3 07:52:03 2004 From: goodger at python.org (David Goodger) Date: Sat Apr 3 07:52:15 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200403302218.i2UMIEV10058@guido.python.org> References: <200403302218.i2UMIEV10058@guido.python.org> Message-ID: <406EB373.2030401@python.org> Guido van Rossum wrote: > At least "from . import X" and "from .. import X" are completely > clear and more levels up are not likely to occur in practice... I have Aahz's revision ready to check in, but I'm unclear on one point. Is the accepted/pronounced relative import syntax dots only, or can there be a branch too? In other words, is the "Y" here acceptable (where Y is a sibling to the current module's package)? from ..Y import X I think the PEP needs a clear statement (and examples) of which syntax **and semantics** have been chosen. -- David Goodger From srichter at cosmos.phy.tufts.edu Sat Apr 3 08:35:20 2004 From: srichter at cosmos.phy.tufts.edu (Stephan Richter) Date: Sat Apr 3 08:35:42 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <406EB373.2030401@python.org> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> Message-ID: <200404030835.20989.srichter@cosmos.phy.tufts.edu> On Saturday 03 April 2004 07:52, David Goodger wrote: > Guido van Rossum wrote: > ?> At least "from . import X" and "from .. import X" are completely > ?> clear and more levels up are not likely to occur in practice... > > I have Aahz's revision ready to check in, but I'm unclear on one > point. ?Is the accepted/pronounced relative import syntax dots only, > or can there be a branch too? ?In other words, is the "Y" here > acceptable (where Y is a sibling to the current module's package)? > > ? ? ?from ..Y import X > > I think the PEP needs a clear statement (and examples) of which syntax > **and semantics** have been chosen. If Y would not be allowed, the new syntax would be cumbersome and only half as productive. In Zope 3 we often have the following case: packageX ? utilities.py (contains a bunch of helper functions) ? browser (package containing browser presentation code) ? ? ?someview.py (a specific browser view) We often want to reach utilities from someview. So the ideal import for me would be: from .utilities import somefunc Also, as mentioned by Jim before, we have used this type of syntax for specifying Python objects in ZCML for a while, and it works well for us. Regards, Stephan -- Stephan Richter CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student) Web2k - Web Software Design, Development and Training From aahz at pythoncraft.com Sat Apr 3 09:50:51 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 3 09:51:01 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <406EB373.2030401@python.org> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> Message-ID: <20040403145050.GA8626@panix.com> On Sat, Apr 03, 2004, David Goodger wrote: > > I have Aahz's revision ready to check in, but I'm unclear on one > point. Is the accepted/pronounced relative import syntax dots only, > or can there be a branch too? In other words, is the "Y" here > acceptable (where Y is a sibling to the current module's package)? > > from ..Y import X > > I think the PEP needs a clear statement (and examples) of which syntax > **and semantics** have been chosen. Good point. Here's what I *think* the semantics are; as soon as I get agreement, I'll update the PEP. Given a package layout:: package subpackage1 moduleX moduleY subpackage2 moduleZ moduleA Each leading "." refers to one level of parent. Assuming that the current file is ``moduleX.py``, following are correct usages of the new syntax:: from .moduleY import spam from ..subpackage1 import moduleY from ..subpackage2.moduleZ import eggs from ..moduleA import foo from ...package import bar Actually, I believe that this is the only possible useful semantic; I would welcome clear reasoning for another semantic. Here's a question. Should the following be legal? :: from ...sys import path If not, do we test for it? What kind of error do we raise? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From dave at boost-consulting.com Sat Apr 3 09:57:50 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sat Apr 3 09:58:37 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> Message-ID: David Goodger writes: > Guido van Rossum wrote: > > At least "from . import X" and "from .. import X" are completely > > clear and more levels up are not likely to occur in practice... > > I have Aahz's revision ready to check in, but I'm unclear on one > point. Is the accepted/pronounced relative import syntax dots only, > or can there be a branch too? In other words, is the "Y" here > acceptable (where Y is a sibling to the current module's package)? > > from ..Y import X Wouldn't that have to be from ...Y import X ^-------------separator, as in A.B ?? Or was that what you intended, David? I'm thinking some other syntax is needed to represent upward travel in the hierarchy, e.g. from ^.Y import X from -.Y import X -- Dave Abrahams Boost Consulting www.boost-consulting.com From goodger at python.org Sat Apr 3 10:44:13 2004 From: goodger at python.org (David Goodger) Date: Sat Apr 3 10:44:18 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> Message-ID: <406EDBCD.30603@python.org> > David Goodger writes: >> In other words, is the "Y" here acceptable >> (where Y is a sibling to the current module's package)? >> >> from ..Y import X David Abrahams wrote: > Wouldn't that have to be > > from ...Y import X > ^-------------separator, as in A.B > ?? Aha! My confusion is confirmed. I hadn't thought of requiring a separator "."; that seems like overloading of meaning. I understood the syntax as Aahz describes it, so a single leading "." means "current package" (i.e., sibling of current module), ".." means "parent package" or "up one level", etc. > I'm thinking some other syntax is needed to represent upward travel > in the hierarchy, e.g. > > from ^.Y import X That has a certain elegance to it. So ".Y" would mean import from current package, "^.Y" would mean import from parent package, "^^.Y" from parent's parent package, etc. I don't want to reopen debate needlessly, but the overloading of the meaning of "." *is* a bit confusing here. Whatever makes sense to Guido will eventually make sense to me, so I'm not worried. -- David Goodger From martin at v.loewis.de Sat Apr 3 11:15:40 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Apr 3 11:15:47 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: <200404021950.i32JoZN21741@guido.python.org> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> Message-ID: <406EE32C.9000207@v.loewis.de> Guido van Rossum wrote: > I'm still torn whether to promote defining properties this way: > > [propget] > def x(self): > "Doc string for x" > return self.__x > > [propset] > def x(self, newx): > self.__x = newx > > [propdel] > def x(self): > del self.__x > > but if people like this (whatever the decorator syntax :) we might as > well make this the recommended way to define properties. Does that actually work? I.e. is there an implementation of propget, propset, propdel so that this code introduces a property x? My understanding is that above syntax would be short for> [propget] def x(self): "Doc string for x" return self.__x x = propget(x) def x(self, newx): self.__x = newx x = propset(x) def x(self): del self.__x x = propdel(x) Later assignments to x would override earlier ones, so that only the propdel survives. Regards, Martin From pje at telecommunity.com Sat Apr 3 11:32:45 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Apr 3 11:27:09 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: <406EE32C.9000207@v.loewis.de> References: <200404021950.i32JoZN21741@guido.python.org> <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> Message-ID: <5.1.0.14.0.20040403112915.038069e0@mail.telecommunity.com> At 06:15 PM 4/3/04 +0200, Martin v. L?wis wrote: >Guido van Rossum wrote: >>I'm still torn whether to promote defining properties this way: >> [propget] >> def x(self): >> "Doc string for x" >> return self.__x >> [propset] >> def x(self, newx): >> self.__x = newx >> [propdel] >> def x(self): >> del self.__x >>but if people like this (whatever the decorator syntax :) we might as >>well make this the recommended way to define properties. > >Does that actually work? I.e. is there an implementation of propget, >propset, propdel so that this code introduces a property x? Yes, using sys._getframe().f_locals[function.func_name]. Someone posted a link to an implementation earlier this week. >My understanding is that above syntax would be short for> [propget] > def x(self): > "Doc string for x" > return self.__x > x = propget(x) > > def x(self, newx): > self.__x = newx > x = propset(x) > > def x(self): > del self.__x > x = propdel(x) > >Later assignments to x would override earlier ones, so that only >the propdel survives. Technically, what you show is not the actual expansion of the new syntax. The new syntax applies decorators before binding 'x' to the new function. So, the old value of 'x' is available to a decorator via sys._getframe().f_locals. This technique is also useful for implementing generic functions and/or multimethods, signature-based overloading, etc. From guido at python.org Sat Apr 3 11:56:52 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 11:56:58 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 03 Apr 2004 07:52:03 EST." <406EB373.2030401@python.org> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> Message-ID: <200404031656.i33Guq524029@guido.python.org> > Guido van Rossum wrote: > > At least "from . import X" and "from .. import X" are completely > > clear and more levels up are not likely to occur in practice... > > I have Aahz's revision ready to check in, but I'm unclear on one > point. Is the accepted/pronounced relative import syntax dots only, > or can there be a branch too? In other words, is the "Y" here > acceptable (where Y is a sibling to the current module's package)? > > from ..Y import X Yes. > I think the PEP needs a clear statement (and examples) of which syntax > **and semantics** have been chosen. Ah, I haven't had the time to read the PEP carefully. Apparently it can use a better specification (like so many PEPs these days... :-( ). Don't wait for me; it's clear that someone needs to do some more work on this (if just to spell out what seems obvious). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 3 11:57:59 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 11:58:05 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 03 Apr 2004 09:50:51 EST." <20040403145050.GA8626@panix.com> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <20040403145050.GA8626@panix.com> Message-ID: <200404031657.i33Gvxl24044@guido.python.org> > Good point. Here's what I *think* the semantics are; as soon as I get > agreement, I'll update the PEP. Given a package layout:: > > package > subpackage1 > moduleX > moduleY > subpackage2 > moduleZ > moduleA > > Each leading "." refers to one level of parent. Assuming that the > current file is ``moduleX.py``, following are correct usages of the new > syntax:: > > from .moduleY import spam > from ..subpackage1 import moduleY > from ..subpackage2.moduleZ import eggs > from ..moduleA import foo > from ...package import bar > > Actually, I believe that this is the only possible useful semantic; I > would welcome clear reasoning for another semantic. All correct. > Here's a question. Should the following be legal? :: > > from ...sys import path > > If not, do we test for it? What kind of error do we raise? Legal but insane. Not an error. --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Sat Apr 3 11:58:26 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 3 11:58:31 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> Message-ID: <20040403165826.GA8803@panix.com> On Sat, Apr 03, 2004, David Abrahams wrote: > > I'm thinking some other syntax is needed to represent upward travel > in the hierarchy, e.g. > > from ^.Y import X > from -.Y import X > > That's been suggested, and it'll show up in the revised PEP as soon as it's pushed out to python.org. I'm not Uncle Timmy, but I'll channel Guido: no chance. I don't particularly like dots-only mechanism, but I have to agree that it works well enough for one-to-three levels. More than that is a strong indication of a design that needs reworking. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From guido at python.org Sat Apr 3 11:58:51 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 11:58:57 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 03 Apr 2004 09:57:50 EST." References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> Message-ID: <200404031658.i33Gwps24056@guido.python.org> > > from ..Y import X > > Wouldn't that have to be > > from ...Y import X > ^-------------separator, as in A.B NO, NO, NO! Each dot is one uplevel. Exactly like Mac OS 9 paths do with ':'. (Read macpath.py.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 3 11:59:35 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 11:59:41 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 03 Apr 2004 10:44:13 EST." <406EDBCD.30603@python.org> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <406EDBCD.30603@python.org> Message-ID: <200404031659.i33GxZV24069@guido.python.org> > > I'm thinking some other syntax is needed to represent upward travel > > in the hierarchy, e.g. > > > > from ^.Y import X > > That has a certain elegance to it. Let me nip that one in the bud. NO WAY. --Guido van Rossum (home page: http://www.python.org/~guido/) From Scott.Daniels at Acm.Org Sat Apr 3 11:59:38 2004 From: Scott.Daniels at Acm.Org (Scott David Daniels) Date: Sat Apr 3 11:59:47 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: References: Message-ID: <406EED7A.5090908@Acm.Org> One more (possibly unclear) case to consider: import . as package Is this a way to name your parent package? -- -Scott David Daniels Scott.Daniels@Acm.Org From guido at python.org Sat Apr 3 12:01:27 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 12:01:33 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: Your message of "Sat, 03 Apr 2004 18:15:40 +0200." <406EE32C.9000207@v.loewis.de> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> <406EE32C.9000207@v.loewis.de> Message-ID: <200404031701.i33H1Ra24093@guido.python.org> > > I'm still torn whether to promote defining properties this way: > > > > [propget] > > def x(self): > > "Doc string for x" > > return self.__x > > > > [propset] > > def x(self, newx): > > self.__x = newx > > > > [propdel] > > def x(self): > > del self.__x > > > > but if people like this (whatever the decorator syntax :) we might as > > well make this the recommended way to define properties. > > Does that actually work? I.e. is there an implementation of propget, > propset, propdel so that this code introduces a property x? Yes, but it involves sys._getframe(). > My understanding is that above syntax would be short for> [propget] > def x(self): > "Doc string for x" > return self.__x > x = propget(x) > > def x(self, newx): > self.__x = newx > x = propset(x) > > def x(self): > del self.__x > x = propdel(x) > > Later assignments to x would override earlier ones, so that only > the propdel survives. No, the semantics of decorators are that the function name isn't actually assigned to for the first time until all decorators have been called. So the previous binding of that name is available to devious decorators like these. (This wasn't my idea -- someone else proposed it here first. :) --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Sat Apr 3 12:02:27 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 3 12:02:31 2004 Subject: [Python-Dev] python.sf.net Message-ID: <20040403170227.GA9380@panix.com> Could someone with admin privs update python.sf.net to include a link to the Python project on SF? http://www.sf.net/projects/python/ Just seems like a useful convenience. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From dave at boost-consulting.com Sat Apr 3 12:02:05 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sat Apr 3 12:02:51 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200404031658.i33Gwps24056@guido.python.org> (Guido van Rossum's message of "Sat, 03 Apr 2004 08:58:51 -0800") References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <200404031658.i33Gwps24056@guido.python.org> Message-ID: Guido van Rossum writes: >> > from ..Y import X >> >> Wouldn't that have to be >> >> from ...Y import X >> ^-------------separator, as in A.B > > NO, NO, NO! Each dot is one uplevel. Exactly like Mac OS 9 paths do > with ':'. (Read macpath.py.) That's a fine convention; I'm familiar with MacOS paths. The fact that David Goodger and I were both confused might indicate that the proposed syntax might not be as intuitive as you'd like, but I don't think it'll take long to get used to. -- Dave Abrahams Boost Consulting www.boost-consulting.com From guido at python.org Sat Apr 3 12:07:14 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 12:07:31 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: Your message of "Sat, 03 Apr 2004 11:58:57 +0100." <2mwu4xfhu6.fsf@starship.python.net> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> <2mwu4xfhu6.fsf@starship.python.net> Message-ID: <200404031707.i33H7El24139@guido.python.org> > > We could also add a standard implementation of synchronized. Or > > perhaps that should be imported from threading. (But is that really > > a good thing to copy from Java?) > > I don't want to sound FUDdy, but I was under the impression that > people didn't think this is something we actually want... Hence my parenthetical remark. But given that it is quite simple to do using decorators, surely *someone* will implement it. How useful it will be remains to be seen. Maybe someone should take this idea, run with it, and report how it enhanced their project? > > overrides -- indicates that this overrides a base class method. > > Maybe the default metaclass could check that if this is used there > > actually is a corresponding base class method, and we might have a > > "strict" metaclass that checks this is set for all overriding > > methods. > > I don't get the point of this. First, it's sometimes useful to know when subclassing which methods you are overriding and which you are adding -- this may help the reader understanding what's going on. And then of course it would be useful to have automatic verification of this information so that the reader can actually believe it. > - "a bit" > > I'm not sure I like the idea of promoting something that uses > sys._getframe. Me neither. > > Should there be a a separate module from which all those decorators > > are imported, or should we make them built-ins, following the trend > > set by classmethod etc.? > > I'm not sure any of the above are important enough to be builtins. > funcattrs, maybe, but there's sod all reason to implement that in C, > and getting Python implementations into the builtins seems likely to > be pain. Could do it in site.py, I guess. It would be simple enough in C. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 3 12:11:15 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 12:11:24 2004 Subject: [Python-Dev] Re: PEP 318: Let's propose some useful built-in decorators In-Reply-To: Your message of "Fri, 02 Apr 2004 21:19:17 EST." <406E1F25.9090702@gradient.cis.upenn.edu> References: <406E1F25.9090702@gradient.cis.upenn.edu> Message-ID: <200404031711.i33HBGX24158@guido.python.org> > > deprecated(comment_string) -- mostly for documentation, but would set > > the deprecated attribute. The comment_string argument is a string > > explaining why it is deprecated. Maybe this should also have > > arguments specifying the first Python release in which it was > > considered deprecated and the last Python release (if known) where it > > will be available. > > Should this be a python-specific decorator? Or should it also be used > to mark deprecated objects in external libraries? (The answer will > affect the arguments specifing when it became depreciated and when it > will go away.) I guess there could be a C-level equivalent way of getting the same attributes set, so that introspecting tools might be able to understand those, too. The aguments would probably not be machine interpretable, so considerably liberty should be allowed as long as some conventions are established. I'm not sufficiently interested in pursueing this to see it through as a spec though. > If we add this, we might want to also define a standard way of marking > classes and modules as depreciated. (Perhaps by setting the > __deprecated__ variable/property in all 3 cases.) If you or someone else is interested in doing this, please consider writing a separate PEP. > > overrides -- indicates that this overrides a base class method. > > Maybe the default metaclass could check that if this is used there > > actually is a corresponding base class method, and we might have a > > "strict" metaclass that checks this is set for all overriding > > methods. > > Alternatively, it could be checked by an external inspection tool like > pychecker. (Since metaclasses are so difficult to mix, it's probably > better not to use them when we don't need to.) Of course. --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Sat Apr 3 12:12:48 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 3 12:12:54 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <406EED7A.5090908@Acm.Org> References: <406EED7A.5090908@Acm.Org> Message-ID: <20040403171247.GC8803@panix.com> On Sat, Apr 03, 2004, Scott David Daniels wrote: > > One more (possibly unclear) case to consider: > import . as package > Is this a way to name your parent package? Nope. All uses of relative imports require ``from import`` (I think the PEP makes that clear). You could, however, do from ... import package [as foo] if Guido permits it; I want a Pronouncement on that before adding it to the PEP. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From pje at telecommunity.com Sat Apr 3 12:32:37 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Apr 3 12:27:03 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators In-Reply-To: <200404031707.i33H7El24139@guido.python.org> References: <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <200404021950.i32JoZN21741@guido.python.org> <2mwu4xfhu6.fsf@starship.python.net> Message-ID: <5.1.0.14.0.20040403122854.0281cec0@mail.telecommunity.com> At 09:07 AM 4/3/04 -0800, Guido van Rossum wrote: > > > We could also add a standard implementation of synchronized. Or > > > perhaps that should be imported from threading. (But is that really > > > a good thing to copy from Java?) > > > > I don't want to sound FUDdy, but I was under the impression that > > people didn't think this is something we actually want... > >Hence my parenthetical remark. > >But given that it is quite simple to do using decorators, surely >*someone* will implement it. How useful it will be remains to be >seen. Maybe someone should take this idea, run with it, and report >how it enhanced their project? For what it's worth, Jim Fulton's ExtensionClass package has had a "synchronized" metaclass implemented in C since about... 1997? I don't know of anybody actually using it, though. For my few dabblings into objects being shared across threads, though, I've typically needed rather precise control over when things get locked and unlocked and how, so I've never bothered implementing a synchronizing decorator. From bh at intevation.de Sat Apr 3 13:24:24 2004 From: bh at intevation.de (Bernhard Herzog) Date: Sat Apr 3 13:24:31 2004 Subject: [Python-Dev] Some comments on PEP 328 (absolute/relative imports) Message-ID: I have some comments on PEP 328 (absolute/relative imports). A brief summary would be "-1", mostly because it unnecessarily breaks backwards compatibility and will cause extra work just to keep up with Python. Missing Rationale for the Incompatibility ----------------------------------------- First of all, I think the rationale for the absolute imports is missing some important points. >From the pep: In current Python, if you're reading a module located inside a package, it is not clear whether :: import foo refers to a top-level module or to another module inside the package. I don't think this is a good description of the actual problem. The problem is that the modules in the package shadow global ones and there's no easy way to bypass the local package. FWIW, though, I don't recall ever running into a problem with that. To resolve the ambiguity, it is proposed that ``foo`` will always be a module or package reachable from ``sys.path``. What I'm missing here is an explanation *why* this is the proposed solution. From my point of view the problem happens too rarely to justify breaking existing code. It's not even difficult to come up with a backwards compatible solution. In the few cases where a local module shadows a global one, the import could flaged as absolute with the global keyword, for instance (as has been suggested in one of the threads referenced in the pep): import global time from global foo import bar Unlike the proposed solution, this not only retains compatibility, it would also be similar to how conflicts between local and global scopes are handled elsewhere in Python. For instance, local variables hide globals and builtins. If you later need to access a shadowed global you need to e.g. rename your local variables or resort to some other way to get at the global. Why, then should local, i.e. relative, imports not shadow global ones by default unless a globel, i.e. absolute, import is explicitly requested? Python-Version Independent Code ------------------------------- I like relative imports and it's good to see that the pep proposes to keep them and even to extend their capabilities. However, I also like to write code that runs on a range of Python versions, e.g. 2.2 and later. Unfortunately, if, as is hinted at in the pep, the new syntax becomes the default in 2.5, there will no way to write relative imports that will work in both 2.3 and 2.5. So, for me the pep effectively abolishes relative imports completely. It also means that I will have to go through all my code and change relative imports to absolute imports, costing me at least a few hours that I would rather spend enhancing my code instead of just keeping up with Python. Deprecation Warnings -------------------- Another thing missing is a note about DeprecationWarnings. Since the change will break exisiting code, DeprecationWarnings should be given for relative imports at some point before the new semantics become the default. Leading Dots ------------ Leading dots were also recently mentioned to be used for a with-statement. Won't two different uses for leading dots be confusing for newbies? E.g. in with some_object: .a = 1 from .magic import xyzzy the leading dot in the import statement won't have anything to do with the with statement. I guess code like the above would be rare, but some people expressed concern about the use of "as" for decorators for similar reasons. Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://sketch.sourceforge.net/ Thuban http://thuban.intevation.org/ From tim.one at comcast.net Sat Apr 3 13:36:42 2004 From: tim.one at comcast.net (Tim Peters) Date: Sat Apr 3 13:36:45 2004 Subject: [Python-Dev] python.sf.net In-Reply-To: <20040403170227.GA9380@panix.com> Message-ID: [Aahz] > Could someone with admin privs update python.sf.net to include a link > to the Python project on SF? > > http://www.sf.net/projects/python/ > > Just seems like a useful convenience. I'm not exactly sure what you're after here. Is this something not already covered by the "Outside links" section on the left-side navigation bar (which includes to "SF Project Page" (the URL you gave above), and links to the SF bug tracker, SF patch tracker, SF CVS instructions, etc). From aahz at pythoncraft.com Sat Apr 3 13:56:49 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 3 13:56:57 2004 Subject: [Python-Dev] python.sf.net In-Reply-To: References: <20040403170227.GA9380@panix.com> Message-ID: <20040403185649.GA1379@panix.com> On Sat, Apr 03, 2004, Tim Peters wrote: > [Aahz] >> >> Could someone with admin privs update python.sf.net to include a link >> to the Python project on SF? >> >> http://www.sf.net/projects/python/ >> >> Just seems like a useful convenience. > > I'm not exactly sure what you're after here. Is this something not > already covered by the "Outside links" section on the left-side > navigation bar (which includes to "SF Project Page" (the URL you gave > above), and links to the SF bug tracker, SF patch tracker, SF CVS > instructions, etc). Ah, you're looking at the redirect page of http://www.python.org/dev/ Someone who turns redirects off just sees a link to /dev/ I figured that someone hitting python.sf.net would be likely interested specifically in the Python page on SF. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From aahz at pythoncraft.com Sat Apr 3 14:11:06 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 3 14:11:14 2004 Subject: [Python-Dev] Some comments on PEP 328 (absolute/relative imports) In-Reply-To: References: Message-ID: <20040403191106.GA4254@panix.com> On Sat, Apr 03, 2004, Bernhard Herzog wrote: > > What I'm missing here is an explanation *why* this is the proposed > solution. From my point of view the problem happens too rarely to > justify breaking existing code. It's not even difficult to come up with > a backwards compatible solution. Barry, would you please write up something for the PEP here? (You're the person who pushed this the most.) I've already updated the PEP to include clarification of the shadowing issue. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From arigo at tunes.org Sat Apr 3 14:59:57 2004 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 3 15:03:28 2004 Subject: [Python-Dev] Python is faster than C Message-ID: <20040403195957.GA30613@vicky.ecs.soton.ac.uk> Hi! This is a rant against the optimization trend of the Python interpreter. Sorting a list of 100000 integers in random order takes: * 0.75 seconds in Python 2.1 * 0.51 seconds in Python 2.2 * 0.46 seconds in Python 2.3 Tim Peters did a great job optimizing list.sort(). If I try with a simple, non-stable pure Python quicksort implementation, in Python 2.3: * 4.83 seconds * 0.21 seconds with Psyco First step towards world domination of high-level languages :-) The reason that Psyco manages to outperform the C implementation is not that gcc is a bad compiler (it is about 10 times better than Psyco's). The reason is that the C implementation must use a generic '<' operator to compare elements, while the Psyco version quickly figures out that it can expect to find ints in the list; it still has to check this assumption, but this is cheap and then the comparison is done with a single machine instruction. Similarily, here are some results about the heapq module, which is rewritten in C in the CVS tree for Python 2.4: l = [random.random() for x in range(200000)] heapq.heapify(l) This code executes on my laptop in: * 1.96 seconds on Python 2.3 (pure Python) * 0.18 seconds on Python 2.4cvs (rewritten in C) * 0.16 seconds on Python 2.3 with Psyco So this is not so much a plug for Psyco as a rant against the current trend of rewriting standard modules in C. Premature optimization and all that. Worse, and more importantly, the optimization starts to become visible to the programmer. Iterators, for example, are great in limited cases but I consider their introduction a significant complication in the language; before, you could expect that some function from which you would expect a sequence returned a list. Python was all lists and dicts, with dicts used as namespaces here and there. Nowadays you have to be careful. Moreover, it is harder to explain: >>> zip([1,2,3], [4,5,6]) # easy to understand and explain [(1, 4), (2, 5), (3, 6)] >>> enumerate([6,7,8,9]) # uh ? I know you can always do list(_). My point is that this is a user-visible optimization. enumerate() should return a normal list, and it should be someone else's job to ensure that it is correctly optimized away if possible (and I'm not even talking about Psyco, it could be done in the current Python implementation with a reasonable amount of effort). Protesting-ly yours, Armin From guido at python.org Sat Apr 3 16:22:33 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 16:22:40 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 03 Apr 2004 12:02:05 EST." References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <200404031658.i33Gwps24056@guido.python.org> Message-ID: <200404032122.i33LMXg24564@guido.python.org> > > NO, NO, NO! Each dot is one uplevel. Exactly like Mac OS 9 paths do > > with ':'. (Read macpath.py.) > > That's a fine convention; I'm familiar with MacOS paths. > The fact that David Goodger and I were both confused might indicate > that the proposed syntax might not be as intuitive as you'd like, but > I don't think it'll take long to get used to. I think the only problem was that it might not have been explained clearly enough. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Sat Apr 3 16:24:57 2004 From: skip at pobox.com (Skip Montanaro) Date: Sat Apr 3 16:25:04 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: <5.1.0.14.0.20040403112915.038069e0@mail.telecommunity.com> References: <200404021950.i32JoZN21741@guido.python.org> <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <5.1.0.14.0.20040403112915.038069e0@mail.telecommunity.com> Message-ID: <16495.11177.951402.175926@montanaro.dyndns.org> Phillip> Technically, what you show is not the actual expansion of the Phillip> new syntax. The new syntax applies decorators before binding Phillip> 'x' to the new function. So, the old value of 'x' is available Phillip> to a decorator via sys._getframe().f_locals. This technique is Phillip> also useful for implementing generic functions and/or Phillip> multimethods, signature-based overloading, etc. How would this be interpreted? x = 42 def x(self) [propget]: "Doc string for x" return self.__x def x(self, newx) [propset]: self.__x = newx def x(self) [propdel]: del self.__x That is, there is already an (otherwise invalid) 'x' in the calling scope when propget() is called. Do the property doodads just need to be bulletproofed or should an exception be raised? Skip From guido at python.org Sat Apr 3 16:25:30 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 16:25:38 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 03 Apr 2004 12:12:48 EST." <20040403171247.GC8803@panix.com> References: <406EED7A.5090908@Acm.Org> <20040403171247.GC8803@panix.com> Message-ID: <200404032125.i33LPU024587@guido.python.org> > > One more (possibly unclear) case to consider: > > import . as package > > Is this a way to name your parent package? > > Nope. All uses of relative imports require ``from import`` Right. > (I think the PEP makes that clear). You could, however, do > > from ... import package [as foo] > > if Guido permits it; I want a Pronouncement on that before adding it > to the PEP. Yes, even if ... refers to the root package (but not if it tries to reach beyond that). But this doesn't handle his case: what he wanted (if it were allowed) would rename the current package to 'package' no matter what its real name. Not that I care about that use case. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 3 16:50:45 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 16:50:53 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: Your message of "Sat, 03 Apr 2004 15:24:57 CST." <16495.11177.951402.175926@montanaro.dyndns.org> References: <200404021950.i32JoZN21741@guido.python.org> <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <5.1.0.14.0.20040403112915.038069e0@mail.telecommunity.com> <16495.11177.951402.175926@montanaro.dyndns.org> Message-ID: <200404032150.i33Lojb24741@guido.python.org> > How would this be interpreted? > > x = 42 > > def x(self) [propget]: > "Doc string for x" > return self.__x > > def x(self, newx) [propset]: > self.__x = newx > > def x(self) [propdel]: > del self.__x > > That is, there is already an (otherwise invalid) 'x' in the calling scope > when propget() is called. Do the property doodads just need to be > bulletproofed or should an exception be raised? It's broken. I expect this to raise an exception at some point. Beyond that, who cares? --Guido van Rossum (home page: http://www.python.org/~guido/) From dave at boost-consulting.com Sat Apr 3 16:55:05 2004 From: dave at boost-consulting.com (David Abrahams) Date: Sat Apr 3 16:55:50 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200404032122.i33LMXg24564@guido.python.org> (Guido van Rossum's message of "Sat, 03 Apr 2004 13:22:33 -0800") References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <200404031658.i33Gwps24056@guido.python.org> <200404032122.i33LMXg24564@guido.python.org> Message-ID: Guido van Rossum writes: >> > NO, NO, NO! Each dot is one uplevel. Exactly like Mac OS 9 paths do >> > with ':'. (Read macpath.py.) >> >> That's a fine convention; I'm familiar with MacOS paths. >> The fact that David Goodger and I were both confused might indicate >> that the proposed syntax might not be as intuitive as you'd like, but >> I don't think it'll take long to get used to. > > I think the only problem was that it might not have been explained > clearly enough. Sure, that would handle it. Fails to meet a high standard set by other elements of Python that can be grasped immediately, without any explanation, but of course not everything can be that simple. -- Dave Abrahams Boost Consulting www.boost-consulting.com From pje at telecommunity.com Sat Apr 3 17:42:54 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat Apr 3 17:36:49 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: <16495.11177.951402.175926@montanaro.dyndns.org> References: <5.1.0.14.0.20040403112915.038069e0@mail.telecommunity.com> <200404021950.i32JoZN21741@guido.python.org> <200404020334.i323YAGZ024235@cosc353.cosc.canterbury.ac.nz> <1080880130.22892.158.camel@localhost.localdomain> <1080912282.5481.158.camel@anthem.wooz.org> <1080925513.22892.173.camel@localhost.localdomain> <1080932935.5481.282.camel@anthem.wooz.org> <5.1.0.14.0.20040403112915.038069e0@mail.telecommunity.com> Message-ID: <5.1.0.14.0.20040403174138.0209f020@mail.telecommunity.com> At 03:24 PM 4/3/04 -0600, Skip Montanaro wrote: >That is, there is already an (otherwise invalid) 'x' in the calling scope >when propget() is called. Do the property doodads just need to be >bulletproofed or should an exception be raised? If I recall correctly, the implementation someone contributed earlier this week (sorry I don't remember his name) raised an exception if the previous binding of the function name was not an instance of 'property'. From ishnigarrab at earthlink.net Sat Apr 3 18:08:28 2004 From: ishnigarrab at earthlink.net (foo) Date: Sat Apr 3 18:08:32 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? Message-ID: <12499735.1081033709200.JavaMail.root@kermit.psp.pas.earthlink.net> Here are some examples I've come up with... -------------- next part -------------- A non-text attachment was scrubbed... Name: functionals_dec.py Type: text/x-python Size: 4809 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040403/7283ebd1/functionals_dec.py From jcarlson at uci.edu Sat Apr 3 19:37:55 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat Apr 3 19:42:20 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: <12499735.1081033709200.JavaMail.root@kermit.psp.pas.earthlink.net> References: <12499735.1081033709200.JavaMail.root@kermit.psp.pas.earthlink.net> Message-ID: <20040403163615.FA19.JCARLSON@uci.edu> > ########## > # > # Decorations > # > class Memoized(dict): > """ A function wrapper to cache the results of functions that take a long > time to complete, like database querries or intensive mathematical > computations. > > To wipe the cache, just call m.clear() > """ > def __init__(self, func, init={}): > """ Accepts the function to be memoized and an optional initial > dictionary of known results. > """ > dict.__init__(self, init) > self.func = func > > def __call__(self, *args, **kwds): > key = (args, tuple(kwds.items())) # hope everything's hashable... > return ( self.get(key) > or self.setdefault(key, self.func(*args, **kwds)) ) I believe that with standard dictionaries, kwds is not guaraneed to have any particular order. Perhaps sorting kwds.items() makes sense? - Josiah From skip at pobox.com Sat Apr 3 20:01:16 2004 From: skip at pobox.com (Skip Montanaro) Date: Sat Apr 3 20:01:32 2004 Subject: [Python-Dev] Help with PEP 318 Message-ID: <16495.24156.587861.468059@montanaro.dyndns.org> There's been way too much PEP 318 material on the mailing list during the past week for me to absorb (*). I'm tied up with a new job and can't really devote much time to keeping the PEP updated. If someone could step up and summarize some of the new material (or better yet, update the file directly) it would be much appreciated. Thx, Skip (*) Much of the recent discussion with "PEP 318" in the subject has actually been about a syntax for setting function attributes which requires its own PEP. From edloper at gradient.cis.upenn.edu Sat Apr 3 21:17:16 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Sat Apr 3 21:15:52 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: References: Message-ID: <406F702C.8070700@gradient.cis.upenn.edu> > How would this be interpreted? > > x = 42 > > def x(self) [propget]: > "Doc string for x" > return self.__x It should raise an exception here. See the implementation that I posted a few days ago: . The same should apply for the generic decorator, and any other decorator that "combines" the new value with the old. Errors-should-never-pass-silently-yours, Edward From aahz at pythoncraft.com Sat Apr 3 21:18:00 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 3 21:18:14 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200404032125.i33LPU024587@guido.python.org> References: <406EED7A.5090908@Acm.Org> <20040403171247.GC8803@panix.com> <200404032125.i33LPU024587@guido.python.org> Message-ID: <20040404021800.GA25210@panix.com> On Sat, Apr 03, 2004, Guido van Rossum wrote: >Aahz: >> >> (I think the PEP makes that clear). You could, however, do >> >> from ... import package [as foo] >> >> if Guido permits it; I want a Pronouncement on that before adding it >> to the PEP. > > Yes, even if ... refers to the root package (but not if it tries to > reach beyond that). Great! Then PEP 328 is almost set except for Bernhard's wish for more clarification of the rationale for making the default import absolute. I've gone ahead and updated the PEP, and I think we should move forward on implementation even without the clarification, but I've stuck a note in my calendar to bug people regularly until the rationale is finished. I've added the following: Timeline ======== In Python 2.4, you must enable the new absolute import behavior with :: from __future__ import absolute_import You may use relative imports freely. In Python 2.5, any ``import`` statement that results in an intra-package import will generate a ``PendingDeprecation`` warning (this also applies to ``from <> import`` that fails to use the relative import syntax). In Python 2.6, ``import`` will always be an absolute import. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From guido at python.org Sat Apr 3 21:58:35 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 3 21:58:42 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 03 Apr 2004 21:18:00 EST." <20040404021800.GA25210@panix.com> References: <406EED7A.5090908@Acm.Org> <20040403171247.GC8803@panix.com> <200404032125.i33LPU024587@guido.python.org> <20040404021800.GA25210@panix.com> Message-ID: <200404040258.i342wZs25281@guido.python.org> > Great! Then PEP 328 is almost set except for Bernhard's wish for more > clarification of the rationale for making the default import absolute. > I've gone ahead and updated the PEP, and I think we should move forward > on implementation even without the clarification, but I've stuck a note > in my calendar to bug people regularly until the rationale is finished. > > I've added the following: > > Timeline > ======== > > In Python 2.4, you must enable the new absolute import behavior with :: > > from __future__ import absolute_import > > You may use relative imports freely. In Python 2.5, any ``import`` > statement that results in an intra-package import will generate a > ``PendingDeprecation`` warning (this also applies to ``from <> import`` > that fails to use the relative import syntax). In Python 2.6, ``import`` > will always be an absolute import. Perfect! --Guido van Rossum (home page: http://www.python.org/~guido/) From ishnigarrab at earthlink.net Sat Apr 3 19:00:38 2004 From: ishnigarrab at earthlink.net (Isaac Freeman) Date: Sat Apr 3 23:58:55 2004 Subject: [Python-Dev] Re: PEP 318: Let's propose some useful built-in decorators In-Reply-To: References: Message-ID: <406F5026.6080509@earthlink.net> Here are a few ideas (including a meta-decoration! :) Of course, I'm biased to first-before-colon style. ;) --- decorators.py --- import inspect class decorator: """ An obscenely gratuitous example of decorator usage. Meta-decoration? "One mans use is another man's abuse" """ class decoration: """ The object that represents the decoration itself. """ def __init__(decor, func, **state): self.func = func self.state = {'func': func.__name__} def __call__(self, *args, **kwds): """ Called when the function that is decorated by this decoration is called. (whew) """ self.state.update(inspect.getargvalues(sys._getframe(1))[3]) # better way? return self.decor(self.func, self.state, *args, **kwds) def options(self, opts=None, **kwd_opts): self.state.update(opts or kwd_opts) return self def __init__(self, decor): self.decor = decor def __call__(self, func): """ Called when the decorator is called to decorate a desired function. """ init = self.init init.update(kwds) return self.decoration(decor, func) def fromstate(cls, state=None, **kwd_state) [classmethod]: """ Make a decoration with a specified initial state """ return lambda fn: cls(fn, state or kwd_state) # or... # Smaller, but can be confusing. (But I guess the whole # meta-function-generator thing is to begin with!) def decorator2(decor=None, **state): """ Decoration factory. """ def decoration2(func=None, **opts): """ Decorating function wrapper. """ state.update(opts) state['func'] = func.__name__ def callit(*args, **kwds): """ Function replacement. """ state.update(inspect.getargvalues(sys._getframe(1))[3]) # better way? decor(func, state, *args, **kwds) if func: return callit else: return lambda fn: decoration2(fn, **opts) if decor: return decoration else: return lambda dec: decorator2(dec, state) def memoized(func, state, *args, **kwds) [decorator]: """ Function wrapper to cache the results of previously evaluated arguments. """ items = kwds.items items.sort() key = (args, tuple(items)) # hope everything is hashable! return ( state.get(key) or state.setdefault(key, func(*args, **kwds)) ) def memoized2(func, state, *args, **kwds) [decorator2]: key = (args, tuple(kwds.items())) return ( state.get(key) or state.setdefault(key, func(*args, **kwds)) ) import sys, time def logged(func, state, *args, **kwds) [decorator.fromstate( output=sys.stdout.write, prefix='%(func)s: ')]: """ Usage for default formatting and output: def func(*args) [logged]: pass Usage for custom formatting and output: def func(a, b) [logged.options(output=output_func, prefix="%(func)s %(a)s %(b)i")]: pass String replacements include 'func' which is the function's name, as well as all arguments to the function such that "%(a)s" would become the value of the argument named 'a'. """ p = '[%s] %s' % (time.time(), state['prefix'] % state) state['output'](p) def logged2(func, state, *args, **kwds) [decorator2( output=sys.stdout.write, prefix='%(func)s: ')]: """ Usage for default formatting and output: def func(*args) [logged2]: pass Usage for custom formatting and output: def func(a, b) [logged2(output=output_func, prefix="%(func)s %(a)s %(b)i")]: pass String replacements include 'func' which is the function's name, as well as all arguments to the function such that "%(a)s" would become the value of the argument named 'a'. """ p = '[%s] %s' % (time.time(), state['prefix'] % state) state['output'](p) # demo function def factorial(n) [memoized, logged.options(output=sys.stdout.write, prefix='factorial for %(n)s: ')]: """ Computes the factorial of n """ print "fact(%i)" % n if n == 1: return 1 else: return n * fact(n - 1) # make tail_recursive decorator(?) def factorial2(n) [memoized, logged2(output=sys.stdout.write, prefix='factorial for %(s): ')]: """ Computes the factorial of n """ print "fact(%i)" % n if n == 1: return 1 else: return n * fact(n - 1) From marktrussell at btopenworld.com Sun Apr 4 06:18:56 2004 From: marktrussell at btopenworld.com (Mark Russell) Date: Sun Apr 4 06:19:02 2004 Subject: [Python-Dev] Re: PEP 318 bake-off? In-Reply-To: <20040403163615.FA19.JCARLSON@uci.edu> References: <12499735.1081033709200.JavaMail.root@kermit.psp.pas.earthlink.net> <20040403163615.FA19.JCARLSON@uci.edu> Message-ID: <1081073936.2509.6.camel@localhost> On Sun, 2004-04-04 at 01:37, Josiah Carlson wrote: > > def __call__(self, *args, **kwds): > > key = (args, tuple(kwds.items())) # hope everything's hashable... > > return ( self.get(key) > > or self.setdefault(key, self.func(*args, **kwds)) ) > > I believe that with standard dictionaries, kwds is not guaraneed to have > any particular order. Perhaps sorting kwds.items() makes sense? Also the call to self.func() is needlessly repeated if the return value is 0, None, or anything else that is treated as False. I realise that this is off the point slightly, but using "e1 or e2" like this is such an attractive nuisance it warrants being stomped on wherever possible. Mark Russell From aahz at pythoncraft.com Sun Apr 4 12:21:00 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Apr 4 12:21:05 2004 Subject: [Python-Dev] PEP 328: __path__ Message-ID: <20040404162100.GA6059@panix.com> I just realized that the new relative import semantics are likely to interact oddly with uses of ``__path__``. I therefore suggest that the following text be added to PEP 328, although I'm perfectly happy to adopt some other course: Between the addition of ``as`` in Python 2.0 and the new syntax & semantics for ``import``, there is no longer a need for ``__path__``, and it is now officially deprecated. In Python 2.4 and 2.5, changing ``__path__`` raises ``PendingDeprecationWarning``. In Python 2.6, ``__path__`` becomes a read-only attribute. In the presence of ``absolute_import`` or with a relative import, ``__path__`` is ignored. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From pedronis at bluewin.ch Sun Apr 4 12:35:28 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Sun Apr 4 12:30:31 2004 Subject: [Python-Dev] PEP 328: __path__ In-Reply-To: <20040404162100.GA6059@panix.com> Message-ID: <5.2.1.1.0.20040404183116.031c83a8@pop.bluewin.ch> At 12:21 04.04.2004 -0400, Aahz wrote: >I just realized that the new relative import semantics are likely to >interact oddly with uses of ``__path__``. I therefore suggest that the >following text be added to PEP 328, although I'm perfectly happy to >adopt some other course: > >Between the addition of ``as`` in Python 2.0 and the new syntax & >semantics for ``import``, there is no longer a need for ``__path__``, and >it is now officially deprecated. In Python 2.4 and 2.5, changing >``__path__`` raises ``PendingDeprecationWarning``. In Python 2.6, >``__path__`` becomes a read-only attribute. In the presence of >``absolute_import`` or with a relative import, ``__path__`` is ignored. >-- that's an issue because people use __path__ to spread a package over many directories and not just in ther relative Btw I thought (I admit I have not read the PEP) that relative imports are an operation over package names import .c .# in a.b -> a.c not about concrete directories. regards. From pedronis at bluewin.ch Sun Apr 4 12:38:14 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Sun Apr 4 12:33:14 2004 Subject: [Python-Dev] PEP 328: __path__ In-Reply-To: <5.2.1.1.0.20040404183116.031c83a8@pop.bluewin.ch> References: <20040404162100.GA6059@panix.com> Message-ID: <5.2.1.1.0.20040404183738.02854208@pop.bluewin.ch> sorry too fast. At 18:35 04.04.2004 +0200, Samuele Pedroni wrote: that's an issue because people use __path__ to spread a package >over many directories and not just in ther relative ... in the relative case. >Btw I thought (I admit I have not read the PEP) that relative imports are >an operation over package names > >import .c .# in a.b -> a.c > >not about concrete directories. > >regards. From guido at python.org Sun Apr 4 12:43:26 2004 From: guido at python.org (Guido van Rossum) Date: Sun Apr 4 12:43:33 2004 Subject: [Python-Dev] PEP 328: __path__ In-Reply-To: Your message of "Sun, 04 Apr 2004 12:21:00 EDT." <20040404162100.GA6059@panix.com> References: <20040404162100.GA6059@panix.com> Message-ID: <200404041643.i34GhQY02513@guido.python.org> > I just realized that the new relative import semantics are likely to > interact oddly with uses of ``__path__``. I therefore suggest that the > following text be added to PEP 328, although I'm perfectly happy to > adopt some other course: > > Between the addition of ``as`` in Python 2.0 and the new syntax & > semantics for ``import``, there is no longer a need for ``__path__``, and > it is now officially deprecated. In Python 2.4 and 2.5, changing > ``__path__`` raises ``PendingDeprecationWarning``. In Python 2.6, > ``__path__`` becomes a read-only attribute. In the presence of > ``absolute_import`` or with a relative import, ``__path__`` is ignored. Uh? Maybe you should have some coffee and then try again. __path__ has nothing to do with relative import. When I say "import foo.bar" then bar is searched for in foo.__path__. __path__ defines which directories comprise the package contents, nothing more or less. --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Sun Apr 4 15:57:56 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Apr 4 15:58:04 2004 Subject: [Python-Dev] PEP 328: __path__ In-Reply-To: <200404041643.i34GhQY02513@guido.python.org> References: <20040404162100.GA6059@panix.com> <200404041643.i34GhQY02513@guido.python.org> Message-ID: <20040404195756.GA20803@panix.com> On Sun, Apr 04, 2004, Guido van Rossum wrote: >Aahz: >> >> I just realized that the new relative import semantics are likely to >> interact oddly with uses of ``__path__``. I therefore suggest that the >> following text be added to PEP 328, although I'm perfectly happy to >> adopt some other course: >> >> Between the addition of ``as`` in Python 2.0 and the new syntax & >> semantics for ``import``, there is no longer a need for ``__path__``, and >> it is now officially deprecated. In Python 2.4 and 2.5, changing >> ``__path__`` raises ``PendingDeprecationWarning``. In Python 2.6, >> ``__path__`` becomes a read-only attribute. In the presence of >> ``absolute_import`` or with a relative import, ``__path__`` is ignored. > > Uh? Maybe you should have some coffee and then try again. Don't drink coffee, remember? ;-) > __path__ has nothing to do with relative import. When I say > "import foo.bar" then bar is searched for in foo.__path__. __path__ > defines which directories comprise the package contents, nothing more > or less. Yeah, I know; I think that ``as`` combined with relative imports can obviate most uses for ``__path__``, but I'm now dealing with a family emergency, so I'm going to just drop it. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From guido at python.org Sun Apr 4 18:27:19 2004 From: guido at python.org (Guido van Rossum) Date: Sun Apr 4 18:27:29 2004 Subject: [Python-Dev] PEP 328: __path__ In-Reply-To: Your message of "Sun, 04 Apr 2004 15:57:56 EDT." <20040404195756.GA20803@panix.com> References: <20040404162100.GA6059@panix.com> <200404041643.i34GhQY02513@guido.python.org> <20040404195756.GA20803@panix.com> Message-ID: <200404042227.i34MRJG02985@guido.python.org> > > Uh? Maybe you should have some coffee and then try again. > > Don't drink coffee, remember? ;-) Well, whatever you do take to improve your brain function. > > __path__ has nothing to do with relative import. When I say > > "import foo.bar" then bar is searched for in foo.__path__. __path__ > > defines which directories comprise the package contents, nothing more > > or less. > > Yeah, I know; I think that ``as`` combined with relative imports can > obviate most uses for ``__path__``, but I'm now dealing with a family > emergency, so I'm going to just drop it. 'as' has nothing to do with __path__ either. __path__ is for distributing a package over multiple directories, and has nothing to do with how you import it. So, yes, please drop it. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim.jewett at eds.com Sun Apr 4 19:24:23 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Sun Apr 4 19:25:04 2004 Subject: [Python-Dev] FuncAttrs or funcattrs or func_attrs Message-ID: > I propose this one to set function attributes: > class func_attrs(objects): Aren't classes supposed to use UpperCase? If this is a more subtle distinction, it should probably be added to PEP 8. FWIW, I think this should go in a decorate module (particularly since this use really is just decoration) rather than a builtin. -jJ From jim.jewett at eds.com Sun Apr 4 19:30:01 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Sun Apr 4 19:31:05 2004 Subject: [Python-Dev] from ...sys import path Message-ID: What is wrong with from ...sys import path? There are several CVS notes about reordering startup so that users can point to a custom sys or os. ...sys Sure looks like a request for a custom sys. Whether to raise ImportError or use the default sys when the custom version isn't there ... I'm not sure. The PEP should decide though, since this will probably happen with common addons. (Ensure that I have *some* PIL.) -jJ From bob at redivi.com Sun Apr 4 19:42:44 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun Apr 4 19:38:38 2004 Subject: [Python-Dev] FuncAttrs or funcattrs or func_attrs In-Reply-To: References: Message-ID: On Apr 4, 2004, at 7:24 PM, Jewett, Jim J wrote: > >> I propose this one to set function attributes: > >> class func_attrs(objects): > > > Aren't classes supposed to use UpperCase? > If this is a more subtle distinction, it should probably be > added to PEP 8. > > FWIW, I think this should go in a decorate module (particularly > since this use really is just decoration) rather than a builtin. The fact that it's a class is really just an implementation deal. It acts more like a "type", such as int, list, itertools.chain, property, etc. -bob From jim.jewett at eds.com Sun Apr 4 19:45:16 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Sun Apr 4 19:46:03 2004 Subject: [Python-Dev] PEP 318: Properties Message-ID: Skip: >> How would this be interpreted? >> x = 42 >> def x(self) [propget]: >> "Doc string for x" >> return self.__x >> That is, there is already an (otherwise invalid) 'x' in the calling scope >> when propget() is called. Do the property doodads just need to be >> bulletproofed or should an exception be raised? Guido: > It's broken. I expect this to raise an exception at some point. > Beyond that, who cares? If so, then this decorator is a bad idea. I would expect 42 to be the initial value of the property x. I could learn that properties are different, and this won't work, but... this would be a very unexpected result. At the very least, an explicit exception should be raised at definition time. -jJ From aahz at pythoncraft.com Sun Apr 4 19:52:12 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Apr 4 19:52:22 2004 Subject: [Python-Dev] from ...sys import path In-Reply-To: References: Message-ID: <20040404235212.GA29398@panix.com> On Sun, Apr 04, 2004, Jewett, Jim J wrote: > > What is wrong with > > from ...sys import path? Your question is lacking sufficient context to answer, assuming that this refers to PEP 328. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "usenet imitates usenet" --Darkhawk From skip at pobox.com Sun Apr 4 20:13:45 2004 From: skip at pobox.com (Skip Montanaro) Date: Sun Apr 4 20:13:58 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: References: Message-ID: <16496.42169.731352.91397@montanaro.dyndns.org> Skip> x = 42 Skip> def x(self) [propget]: Skip> "Doc string for x" Skip> return self.__x Guido> It's broken. I expect this to raise an exception at some point. Guido> Beyond that, who cares? Jim> If so, then this decorator is a bad idea. Jim> I would expect 42 to be the initial value of the property x. Then it makes sense to put "x = 42" after the three property definitions (before the end of the class definition). Would that work? Skip From guido at python.org Sun Apr 4 20:19:30 2004 From: guido at python.org (Guido van Rossum) Date: Sun Apr 4 20:19:40 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: Your message of "Sun, 04 Apr 2004 19:45:16 EDT." References: Message-ID: <200404050019.i350JU903188@guido.python.org> > >> How would this be interpreted? > > >> x = 42 > > >> def x(self) [propget]: > >> "Doc string for x" > >> return self.__x > > >> That is, there is already an (otherwise invalid) 'x' in the calling scope > >> when propget() is called. Do the property doodads just need to be > >> bulletproofed or should an exception be raised? > > Guido: > > > It's broken. I expect this to raise an exception at some point. > > Beyond that, who cares? > > If so, then this decorator is a bad idea. > > I would expect 42 to be the initial value of the property x. Then I think you have much to learn about properties. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 4 20:22:44 2004 From: guido at python.org (Guido van Rossum) Date: Sun Apr 4 20:22:52 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: Your message of "Sun, 04 Apr 2004 19:13:45 CDT." <16496.42169.731352.91397@montanaro.dyndns.org> References: <16496.42169.731352.91397@montanaro.dyndns.org> Message-ID: <200404050022.i350MiG03211@guido.python.org> > Then it makes sense to put "x = 42" after the three property definitions > (before the end of the class definition). Would that work? No, for deep reasons that have to do with the implementation of descriptors. This has nothing to do with decorators and not a lot with properties -- it's the same as class C: def x(self): return self.__x x = 42 today. --Guido van Rossum (home page: http://www.python.org/~guido/) From edloper at gradient.cis.upenn.edu Sun Apr 4 20:33:57 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Sun Apr 4 20:32:16 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 9, Issue 16 In-Reply-To: References: Message-ID: <4070A975.3020505@gradient.cis.upenn.edu> >>>How would this be interpreted? >>> x = 42 >>> def x(self) [propget]: >>> "Doc string for x" >>> return self.__x > >>>That is, there is already an (otherwise invalid) 'x' in the calling scope >>>when propget() is called. Do the property doodads just need to be >>>bulletproofed or should an exception be raised? [GvR said:] >>It's broken. I expect this to raise an exception at some point. >>Beyond that, who cares? [Jim said:] > If so, then this decorator is a bad idea. > > I would expect 42 to be the initial value of the property x. That doesn't make sense. If you want to set a default value, you'd say: __x = 42 A property's getter/setter/deleter can perform arbitrary actions, and in the general case there's no sensible way to define an "initial value" for a property. E.g., consider: class A: def getX(self): return random.choice(['apple', 'orange', 'grape']) x = property(getX) If this still doesn't make sense, go read the docs for properties: [Skip said:] > Then it makes sense to put "x = 42" after the three property definitions > (before the end of the class definition). Would that work? No, that would just overwrite x with the value 42 (throwing away the property object). -Edward From tim.one at comcast.net Sun Apr 4 20:50:05 2004 From: tim.one at comcast.net (Tim Peters) Date: Sun Apr 4 20:50:11 2004 Subject: [Python-Dev] genexps slow? In-Reply-To: <2my8phi9bu.fsf@starship.python.net> Message-ID: [Tim Peters] >> Resuming a generator function is a lot cheaper than calling a >> function, but there's still a non-trivial amount of code to get in >> and out of eval_frame() each time, which the listcomp version gets >> to skip. [Michael Hudson. > This is something that occurred to me a while ago: how many opcodes > does a typical invocation of eval_frame actually execute? A little > script told me that the median length of functions in Lib/*.py was 38 > instructions (or 52 bytes) IIRC, but obviously a dynamic count is far > more interesting. If the number is fairly small (and honestly, I have > no idea), the set up and tear down code becomes much more significant > than one might think. And whatever it is, I expect it will be significantly smaller for functions synthesized to implement generator expressions. > I didn't think much about the code to get *out* of eval_frame. There's a twist there: the functions synthesized for generator expressions get to take the quicker-than-general "fast_yield" exit path. Hmm! It looks like tests against WHY_YIELD before the fast_yield label can no longer succeed, but are still there to slow down non-generator cases. From jim.jewett at eds.com Sun Apr 4 21:07:32 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Sun Apr 4 21:08:06 2004 Subject: [Python-Dev] PEP 318: Properties Message-ID: Skip> x = 42 Skip> def x(self) [propget]: Skip> "Doc string for x" Skip> return self.__x Guido> It's broken. I expect this to raise an exception at some point. Guido> Beyond that, who cares? Jim> If so, then this decorator is a bad idea. Jim> I would expect 42 to be the initial value of the property x. Skip> Then it makes sense to put "x = 42" after the three property Skip> definitions (before the end of the class definition). Skip> Would that work? Not really. The documentation says "Properties are a neat way to implement attributes whose usage resembles attribute access, but whose implementation uses method calls." This suggests that creating a property creates a manager. In this case, the manager is created for an attribute that already exists. If it uses the previous value as the initial current value, that is what I expect. If it raises an exception ("I can't manage that! Someone beat me to it!"), I'll be annoyed, but live with the limitation. If it silently throws the value away, or raises an exception at run time -- that is bad. Yes, this is a variation on the "Properties do not work for classic classes, but you don't get a clear error when you try this." wart. But the code this decorator replaces:: x = property(getx, setx) does explicitly replace x, so I think *this* confusion may be specific to the decorator. -jJ From bob at redivi.com Sun Apr 4 21:22:09 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun Apr 4 21:18:00 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: References: Message-ID: On Apr 4, 2004, at 9:07 PM, Jewett, Jim J wrote: > Skip> x = 42 > > Skip> def x(self) [propget]: > Skip> "Doc string for x" > Skip> return self.__x > > Guido> It's broken. I expect this to raise an exception at some > point. > Guido> Beyond that, who cares? > > Jim> If so, then this decorator is a bad idea. > > Jim> I would expect 42 to be the initial value of the property x. > > Skip> Then it makes sense to put "x = 42" after the three property > Skip> definitions (before the end of the class definition). > Skip> Would that work? > > Not really. The documentation says > > "Properties are a neat way to implement attributes whose usage > resembles attribute access, but whose implementation uses method > calls." > > This suggests that creating a property creates a manager. In this > case, > the manager is created for an attribute that already exists. If it > uses > the previous value as the initial current value, that is what I expect. > If it raises an exception ("I can't manage that! Someone beat me to > it!"), > I'll be annoyed, but live with the limitation. If it silently throws > the > value away, or raises an exception at run time -- that is bad. > > Yes, this is a variation on the "Properties do not work for classic > classes, but you don't get a clear error when you try this." wart. > But the code this decorator replaces:: > > x = property(getx, setx) > > does explicitly replace x, so I think *this* confusion may be specific > to the decorator. ... have you ever used descriptors before? Property descriptors are only functionally applicable to *instances* of the class where the descriptor lives. FooClass.x = 42 will replace the descriptor. It's possible to make a descriptor do something special when you fetch it from its class, but in this case FooClass doesn't even really exist when you are replacing the descriptor with something else! -bob From greg at cosc.canterbury.ac.nz Sun Apr 4 21:23:53 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Apr 4 21:24:18 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Message-ID: <200404050123.i351Nr9R029066@cosc353.cosc.canterbury.ac.nz> Ka-Ping Yee : > Putting the [decorator] on a separate line before the function > changes the stakes entirely. It sets aside real functional issues > in favour of aesthetics. Not only that, but I think the aesthetics of this version are *worse* than anything that's been considered before. It strikes me as deeply wrong and un-Pythonic -- so much so that I can't understand why Guido is even considering it. Was he abducted by aliens the other night and given a personality transplant or something? It's the only plausible explanation I can think of. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From pje at telecommunity.com Sun Apr 4 23:34:44 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun Apr 4 23:34:49 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: Message-ID: <5.1.0.14.0.20040404233010.03b05740@mail.telecommunity.com> At 09:07 PM 4/4/04 -0400, Jewett, Jim J wrote: >Not really. The documentation says > > "Properties are a neat way to implement attributes whose usage > resembles attribute access, but whose implementation uses method > calls." > >This suggests that creating a property creates a manager. In this case, >the manager is created for an attribute that already exists. If it uses >the previous value as the initial current value, that is what I expect. >If it raises an exception ("I can't manage that! Someone beat me to it!"), >I'll be annoyed, but live with the limitation. If it silently throws the >value away, or raises an exception at run time -- that is bad. > >Yes, this is a variation on the "Properties do not work for classic >classes, but you don't get a clear error when you try this." wart. >But the code this decorator replaces:: > > x = property(getx, setx) > >does explicitly replace x, so I think *this* confusion may be specific >to the decorator. Are you saying this is what you think, or what you think that newbies will think? If this is what you think, you probably need to read the descriptor tutorial (and perhaps PEP 252) again. Descriptors don't work anything like what you're talking about. As for the implementation of propget and propset, if you look at Ed Loper's draft implementation, you'll see that an error is raised at function definition time if there's an existing non-property binding for the named function. From pje at telecommunity.com Sun Apr 4 23:39:49 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun Apr 4 23:40:02 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404050123.i351Nr9R029066@cosc353.cosc.canterbury.ac.nz> References: Message-ID: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> At 01:23 PM 4/5/04 +1200, Greg Ewing wrote: >Ka-Ping Yee : > > > Putting the [decorator] on a separate line before the function > > changes the stakes entirely. It sets aside real functional issues > > in favour of aesthetics. > >Not only that, but I think the aesthetics of this version are *worse* >than anything that's been considered before. It strikes me as deeply >wrong and un-Pythonic -- so much so that I can't understand why Guido >is even considering it. There are different kinds of aesthetics. Guido's proposal has grown on me from a *visual* aesthetics point of view. After I worked with it a little bit, I realized it really is much prettier than decorators-before-colon. However, from more intellectual aesthetics (consistency, predictability, etc.) I still dislike it, and don't really see how to sanely reconcile it with the Python syntax of today. I wish that I did, because it really does *look* better for simple decorators. On the other hand, *no* syntax proposed so far has been really that nice to look at when used for multiple decorators. From guido at python.org Mon Apr 5 00:33:38 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 5 00:33:53 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Sun, 04 Apr 2004 23:39:49 EDT." <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> Message-ID: <200404050433.i354XcF03700@guido.python.org> > There are different kinds of aesthetics. Guido's proposal has grown > on me from a *visual* aesthetics point of view. After I worked with > it a little bit, I realized it really is much prettier than > decorators-before-colon. Right. It really is. > However, from more intellectual aesthetics (consistency, > predictability, etc.) I still dislike it, and don't really see how > to sanely reconcile it with the Python syntax of today. I wish that > I did, because it really does *look* better for simple decorators. I, too, wish that there were a way to make it work with current expectations. The * prefix looks so arbitrary: why not /, why not @, etc... > On the other hand, *no* syntax proposed so far has been really that > nice to look at when used for multiple decorators. Yet, decorator-before-def does better there, too, because it gives you more horizontal space to work with. This is useful for decorators that take argument lists, like the ObjC decorator and I believe some of PEAK's decorators. (I don't want to use funcattrs() as an argument here, because I believe that it is simply an inevitable by-product of introducing any kind of decorator syntax -- while far from perfect, in terms of readability setting function attributes in a decorator is so much better than setting them after the function has been defined, that I believe we have no choice but to provide it. (Similar for synchronized, except I feel less pressure to add it; I'd much rather introduce some kind of general block wrapper feature like we have discussed many times here.)) I also note that accepting decorator-before-colon now would make it harder to come up with a decent syntax for declaring the return type, which I still want to do in some future version of Python with optional (!) static typing. But not impossible -- there's enough punctuation available besides '[' and ':'. I also note that the proposed variants that prefix the 'def clause' with another (optional) indented clause, started e.g. by 'with:' or 'decorate:', look much worse. They violate another strong expectation in Python: that a suite is a sequence of statements. (And it really can't be a sequence of statements -- e.g. assignments just really don't make sense as decorators, nor do many other forms of statement.) One worry about every syntax that supports multiple decorators: as soon as there's a transformation among the decorators, they stop being commutative. And that could cause a load of problems with beginners trying their hand at using complicated decorators (e.g. trying to define a class property by combining classmethod and property). --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Mon Apr 5 01:16:26 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 01:16:53 2004 Subject: [Python-Dev] PEP 328: __path__ In-Reply-To: <20040404162100.GA6059@panix.com> Message-ID: <200404050516.i355GQes002218@cosc353.cosc.canterbury.ac.nz> Aahz : > Between the addition of ``as`` in Python 2.0 and the new syntax & > semantics for ``import``, there is no longer a need for ``__path__``, and > it is now officially deprecated. NO! WAIT! I have a project in which I'm modifying a module's __path__, and it's *not* to work around any deficiencies in the relative import mechanism. Rather it's to make a set of directories behave like a single package for import purposes, with the members of the set determined at run time. Will there be any easy way to do this under the new scheme? If not, can you please try to find some way to make __path__ work sensibly with the new rules. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Mon Apr 5 01:44:34 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 01:44:45 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: <16496.42169.731352.91397@montanaro.dyndns.org> Message-ID: <200404050544.i355iYF6002268@cosc353.cosc.canterbury.ac.nz> Skip Montanaro : > Jim> I would expect 42 to be the initial value of the property x. > > Then it makes sense to put "x = 42" after the three property definitions > (before the end of the class definition). Would that work? No, it would wipe out the property descriptor that you'd just carefully set up. I don't think this makes sense, anyway. Being a property implies that its value is computed somehow, so how to establish an initial value, if that concept even applies, will depend entirely on the details of how it's being computed. Sounds like a job for a piece of code in an __init__ method to me. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Mon Apr 5 02:01:04 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 02:01:15 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> Message-ID: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> "Phillip J. Eby" : > There are different kinds of aesthetics. Guido's proposal has grown on me > from a *visual* aesthetics point of view. After I worked with it a little > bit, I realized it really is much prettier than decorators-before-colon. Your tastes must be different from mine, then, because it doesn't strike me as any prettier visually, either. I don't think I can fully separate these different kinds of aesthetics in my mind, anyway. To me, a piece of syntax isn't just something to look at -- it has a grammar, and it has a meaning, and if the grammar and the meaning and the way it looks on the page don't all agree with each other, it strikes a discordant note. When I read the syntax def foo(args) [classmethod]: ... it says to me "Define a function foo, with these args, and which happens to be a classmethod. Here's the body..." On the other hand, when I see [classmethod] def foo(args): ... the little voice in my head doesn't really say anything coherent at all. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From python-dev at zesty.ca Mon Apr 5 02:14:42 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Mon Apr 5 02:14:21 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> References: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> Message-ID: On Mon, 5 Apr 2004, Greg Ewing wrote: > the little voice in my head doesn't really say anything > coherent at all. Funny. I experience that most days. -- ?!ng From fumanchu at amor.org Mon Apr 5 02:37:39 2004 From: fumanchu at amor.org (Robert Brewer) Date: Mon Apr 5 02:39:46 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon Message-ID: Guido van Rossum wrote: > ...while far from perfect, in > terms of readability setting function attributes in a decorator is so > much better than setting them after the function has been defined, > that I believe we have no choice but to provide it. The more I read statements like this, the more I think of replacing: def foo(arg1, arg2): pass foo = classmethod(foo) foo.attrA = 3 ...with code like: foo = classmethod() foo.attrA = 3 def foo(arg1, arg2): pass ...the meaning of "def" changes from "make a function" to "bind this suite to foo". OBVIOUSLY, this would be a huge change as I've written it. But perhaps, having seen this ideal, a middle ground might become more visible. > I also note that accepting decorator-before-colon now would make it > harder to come up with a decent syntax for declaring the return type, > which I still want to do in some future version of Python with > optional (!) static typing. But not impossible -- there's enough > punctuation available besides '[' and ':'. Why would such additional features not be standard attributes of function objects? foo = function() foo.returns(int) def foo(arg1): return int(arg1) All in all, I'd prefer that function decorators and attributes use simple, known techniques like assignment and callables (as shown above), than become monolithic, crystallized all-in-one statements like: def foo(arg1 as str, arg2 as int) returns str [classmethod] {attrA: 3}: I understand the desire to keep all those bits near each other on the page, but stronger is my desire to make them independent statements. If we could bind a "blank" function object and supply its actual codeblock in a later statement, the rest falls magically into place, IMO. Just some thoughts. Robert Brewer MIS Amor Ministries fumanchu@amor.org From shane.holloway at ieee.org Mon Apr 5 03:07:10 2004 From: shane.holloway at ieee.org (Shane Holloway (IEEE)) Date: Mon Apr 5 03:07:38 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200404031657.i33Gvxl24044@guido.python.org> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <20040403145050.GA8626@panix.com> <200404031657.i33Gvxl24044@guido.python.org> Message-ID: <4071059E.2060807@ieee.org> >>Good point. Here's what I *think* the semantics are; as soon as I get >>agreement, I'll update the PEP. Given a package layout:: >> >> package >> subpackage1 >> moduleX >> moduleY >> subpackage2 >> moduleZ >> moduleA >> >>Each leading "." refers to one level of parent. Assuming that the >>current file is ``moduleX.py``, following are correct usages of the new >>syntax:: >> >> from .moduleY import spam >> from ..subpackage1 import moduleY >> from ..subpackage2.moduleZ import eggs >> from ..moduleA import foo >> from ...package import bar What about names inside package __init__? from .. import bar Is this also valid? From dave at boost-consulting.com Mon Apr 5 03:49:32 2004 From: dave at boost-consulting.com (David Abrahams) Date: Mon Apr 5 03:50:28 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> Message-ID: Greg Ewing writes: >> There are different kinds of aesthetics. Guido's proposal has grown on me >> from a *visual* aesthetics point of view. After I worked with it a little >> bit, I realized it really is much prettier than decorators-before-colon. > > Your tastes must be different from mine, then, because > it doesn't strike me as any prettier visually, either. > > I don't think I can fully separate these different kinds of > aesthetics in my mind, anyway. To me, a piece of syntax > isn't just something to look at -- it has a grammar, and > it has a meaning, and if the grammar and the meaning and > the way it looks on the page don't all agree with each > other, it strikes a discordant note. > > When I read the syntax > > def foo(args) [classmethod]: > ... > > it says to me "Define a function foo, with these args, and > which happens to be a classmethod. Here's the body..." > > On the other hand, when I see > > [classmethod] > def foo(args): > ... > > the little voice in my head doesn't really say anything > coherent at all. Lest we get the idea that there's any absolute measure of aesthetics: +1 restoring-nature's-equilibrium-ly, Dave -- Dave Abrahams Boost Consulting www.boost-consulting.com From paul at prescod.net Mon Apr 5 04:17:17 2004 From: paul at prescod.net (Paul Prescod) Date: Mon Apr 5 04:22:57 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> References: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> Message-ID: <4071160D.5060204@prescod.net> Greg Ewing wrote: > "Phillip J. Eby" : > > >>There are different kinds of aesthetics. Guido's proposal has grown on me >>from a *visual* aesthetics point of view. After I worked with it a little >>bit, I realized it really is much prettier than decorators-before-colon. > > Your tastes must be different from mine, then, because > it doesn't strike me as any prettier visually, either. What people find readable depends mostly on what they are used to. The reason Python is "so readable" on first read is because it rips off so much syntax directly from C, the language that is the ancestor for all of the other languages people use today. C doesn't have function decorators so people don't have expectations for them. But they will when they start coming over from C# and Java 1.5 in two or three years. One virtue of Guido's proposal is that it is basically what C# does. Java uses a pretty different syntax but it is also a prefix syntax. If Python uses a postfix syntax it will probably be alone in making that choice. I'm not saying that Python has to do what the other languages do because they do it, but all else equal, being familiar is better than being idiosyncratic (i.e. different for no good reason). So I would rank "like C# and Java" higher than "fits my personal aesthetics in early 2004" because aesthetics are likely to drift towards C# and Java over time. Paul Prescod From bh at intevation.de Mon Apr 5 05:05:08 2004 From: bh at intevation.de (Bernhard Herzog) Date: Mon Apr 5 05:05:18 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040404021800.GA25210@panix.com> (aahz@pythoncraft.com's message of "Sat, 3 Apr 2004 21:18:00 -0500") References: <406EED7A.5090908@Acm.Org> <20040403171247.GC8803@panix.com> <200404032125.i33LPU024587@guido.python.org> <20040404021800.GA25210@panix.com> Message-ID: Aahz writes: > Timeline > ======== > > In Python 2.4, you must enable the new absolute import behavior with :: > > from __future__ import absolute_import > > You may use relative imports freely. In Python 2.5, any ``import`` > statement that results in an intra-package import will generate a > ``PendingDeprecation`` warning (this also applies to ``from <> import`` > that fails to use the relative import syntax). In Python 2.6, ``import`` > will always be an absolute import. Why PendingDeprecation? If the import semantics change in incompatible ways in 2.6, 2.5 should generate a "normal" DeprecationWarning, I think. Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://sketch.sourceforge.net/ Thuban http://thuban.intevation.org/ From marktrussell at btopenworld.com Mon Apr 5 05:16:51 2004 From: marktrussell at btopenworld.com (Mark Russell) Date: Mon Apr 5 05:17:02 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404050433.i354XcF03700@guido.python.org> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> Message-ID: <1081156611.2377.6.camel@localhost> On Mon, 2004-04-05 at 05:33, Guido van Rossum wrote: > I, too, wish that there were a way to make it work with current > expectations. The * prefix looks so arbitrary: why not /, why not @, > etc... What about: [as classmethod] def foo(bar, baz): pass To me this is more obvious (and less like p*rl) that *[classmethod], and it is also currently a syntax error so won't break existing code. Mark Russell From dave at boost-consulting.com Mon Apr 5 06:38:41 2004 From: dave at boost-consulting.com (David Abrahams) Date: Mon Apr 5 06:39:41 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> Message-ID: Mark Russell writes: > On Mon, 2004-04-05 at 05:33, Guido van Rossum wrote: >> I, too, wish that there were a way to make it work with current >> expectations. The * prefix looks so arbitrary: why not /, why not @, >> etc... > > What about: > > [as classmethod] > def foo(bar, baz): > pass > > To me this is more obvious (and less like p*rl) that *[classmethod], and > it is also currently a syntax error so won't break existing code. Not bad. Also tolerable: as [classmethod] def foo(bar, baz): pass -- Dave Abrahams Boost Consulting www.boost-consulting.com From marktrussell at btopenworld.com Mon Apr 5 06:54:31 2004 From: marktrussell at btopenworld.com (Mark Russell) Date: Mon Apr 5 06:54:37 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> Message-ID: <1081162471.3004.4.camel@localhost> On Mon, 2004-04-05 at 11:38, David Abrahams wrote: > Not bad. Also tolerable: > > as [classmethod] > def foo(bar, baz): > pass The trouble with that is that its less clearly an error at the moment (e.g.: as = [ 'pathological' ] classmethod = 0 as [classmethod] def foo(bar, baz): pass is actually legal right now). The nice thing about "[as xxx]" is that there's no way you can confuse it with a subscript expression. Mark From mwh at python.net Mon Apr 5 06:58:15 2004 From: mwh at python.net (Michael Hudson) Date: Mon Apr 5 06:58:19 2004 Subject: [Python-Dev] genexps slow? In-Reply-To: (Tim Peters's message of "Sun, 4 Apr 2004 20:50:05 -0400") References: Message-ID: <2mptamg08o.fsf@starship.python.net> "Tim Peters" writes: > [Tim Peters] >>> Resuming a generator function is a lot cheaper than calling a >>> function, but there's still a non-trivial amount of code to get in >>> and out of eval_frame() each time, which the listcomp version gets >>> to skip. > > [Michael Hudson. >> This is something that occurred to me a while ago: how many opcodes >> does a typical invocation of eval_frame actually execute? A little >> script told me that the median length of functions in Lib/*.py was 38 >> instructions (or 52 bytes) IIRC, but obviously a dynamic count is far >> more interesting. If the number is fairly small (and honestly, I have >> no idea), the set up and tear down code becomes much more significant >> than one might think. > > And whatever it is, I expect it will be significantly smaller for functions > synthesized to implement generator expressions. I expect invocations of most generators (not necessarily synthetic genexp ones) probably execute *very* few instructions -- ~5, maybe? >> I didn't think much about the code to get *out* of eval_frame. > > There's a twist there: the functions synthesized for generator expressions > get to take the quicker-than-general "fast_yield" exit path. Hmm! It looks > like tests against WHY_YIELD before the fast_yield label can no longer > succeed, but are still there to slow down non-generator cases. Hmm. I *think* I agree with you. I also think the tests against WHY_YIELD in the code for END_FINALLY are unnecessary, as we (syntactically) don't allow yields when there are finally: statements in play. If I'm wrong about that, then you're wrong too :-) It's all a bit subtle, to be sure. Once again, I wonder if there was someway we could have two eval_frames: one with tracing support, one without. Unfortunately, I don't think this can fly (how do you *start* tracing?). Maybe we could have a '-notrace' command line flag? But I suspect that this is a silly over-complication. Cheers, mwh -- "An infinite number of monkeys at an infinite number of keyboards could produce something like Usenet." "They could do a better job of it." -- the corollaries to Gene Spafford's Axiom #2 of Usenet From pyth at devel.trillke.net Mon Apr 5 06:58:09 2004 From: pyth at devel.trillke.net (Holger Krekel) Date: Mon Apr 5 06:59:05 2004 Subject: Multiple Decorators? (was: Re: [Python-Dev] Re: PEP 318: Decorators last before colon) In-Reply-To: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> Message-ID: <20040405105809.GE6361@solar.trillke> Phillip J. Eby wrote: > On the other hand, *no* syntax proposed so far has been really that nice to > look at when used for multiple decorators. I agree completly here. Add to it that understanding the semantics of applying multiple decorators might not always be easy in case of a problem (which is a very common sitatuation for a programmer). Maybe this hints at thinking about syntaxes that don't neccesarily work for multiple decorators but work well for one decorator? After all the main point of introducing descriptor syntax seems to be that they should visually come closer to the 'def' statement line and not that multiple decorators are a common problem. cheers, holger From gmccaughan at synaptics-uk.com Mon Apr 5 07:07:45 2004 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Mon Apr 5 07:07:49 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <20040403195957.GA30613@vicky.ecs.soton.ac.uk> References: <20040403195957.GA30613@vicky.ecs.soton.ac.uk> Message-ID: <200404051207.45195.gmccaughan@synaptics-uk.com> On Saturday 2004-04-03 20:59, Armin Rigo wrote: > Hi! > > This is a rant against the optimization trend of the Python interpreter. > > Sorting a list of 100000 integers in random order takes: > > * 0.75 seconds in Python 2.1 > * 0.51 seconds in Python 2.2 > * 0.46 seconds in Python 2.3 > > Tim Peters did a great job optimizing list.sort(). If I try with a > simple, non-stable pure Python quicksort implementation, in Python 2.3: > > * 4.83 seconds > * 0.21 seconds with Psyco > > First step towards world domination of high-level languages :-) Mhm. And what happens if you want to sort something other than integers? Say, Python objects with their own comparison operators? Or even just (int,int) tuples? -- g From Paul.Moore at atosorigin.com Mon Apr 5 08:54:38 2004 From: Paul.Moore at atosorigin.com (Moore, Paul) Date: Mon Apr 5 08:54:38 2004 Subject: [Python-Dev] Python is faster than C Message-ID: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> From: Gareth McCaughan > Mhm. And what happens if you want to sort something other > than integers? Say, Python objects with their own comparison > operators? Or even just (int,int) tuples? I got the impression that Armin's point was that Psyco can optimise the integer case *if* the code is in Python, but not if it's in C. I suspect that sort isn't a good example here, but heapq may be. By recoding it in C, the chance of getting of type-specific specialised speedups is lost, in favour of a (possibly) smaller speedup across the board. To get a feel for whether the right tradeoff has been made, you'd need to analyze real use cases in some detail (which may not be worth it). [Armin, how does Psyco + a Python sort compare against timsort on more general objects?] Hmm. I'm torn. On the one hand, it's really cool that Python+Psyco can out-perform C (another argument against the knee-jerk "recode bits in C" reaction). But my gut feel is that not enough people use Psyco to make this a good answer yet. On the third hand, if we recode chunks of the stdlib in C, do we kill too many chances for Psyco to work its magic? Time for Psyco to go into the stdlib, maybe? :-) Paul. From scav at blueyonder.co.uk Mon Apr 5 09:21:52 2004 From: scav at blueyonder.co.uk (Peter Harris) Date: Mon Apr 5 09:14:46 2004 Subject: [Python-Dev] Re: PEP309 re-written In-Reply-To: References: Message-ID: <40715D70.1010402@blueyonder.co.uk> David Abrams wrote: >> If anyone can think of any really elegant hacks that are naturally >> expressed by partial function application I'd like to see them > > > >There are millions, but AFAICT they're all more natural with lambda, >so... > > "I agree that lambda is usually good enough, just not > always." > >Examples, please? > > Well, you can use partial to special-case classes as shorthand object factories. So: C = partial(Canvas,my_window,width=100,height=100,bg='white') ...gives you a callable C that is a factory for Tkinter Canvases parented to my_window, 100 pixels on a side, and with a white background. How this differs from lambda is that you can override keyword parameters from the supplied defaults: c1 = C() c2 = C(width=200) c3 = C(bg='red') > "And I want the possibility of useful introspection and > subclassing" > >Can you elaborate on that, please? > > > You could maybe sub-class partial to override __call__ , in case you want to do anything fancy like pre-supplying arguments at arbitrary positions. You can inspect or indeed change C.fn, C.kw or C.args in the example above. I'm not speculating on why you might want to, but there's nothing stopping you. :) Peter Harris From jim.jewett at eds.com Mon Apr 5 09:20:24 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Mon Apr 5 09:21:09 2004 Subject: [Python-Dev] PEP 318: Properties Message-ID: >>Yes, this is a variation on the "Properties do not work for classic >>classes, but you don't get a clear error when you try this." wart. >>But the code this decorator replaces:: >> x = property(getx, setx) >>does explicitly replace x, so I think *this* confusion may be specific >>to the decorator. > Are you saying this is what you think, or what you think that newbies will > think? I'm saying that it reads that way. This does means that newbies will think that, but it also means *I* will think that whenever I haven't *recently* read the docs or been burned. It also means that when I'm debugging, I have to look more places to see why my value disappeared, just in case someone used properties. The reason to use a high-level language is that you can concentrate on your problem instead of worrying about implementation details. Python normally supports this very well. It keeps most things explicit and it doesn't usually make you write much in the way of infrastructure/glue. (Remember the complaints about logging? No one has suggested that it doesn't work well, only that it is too hard to use casually.) What python does under the covers doesn't matter, *unless* the details are so broken that I need to remember them. x=asdlfjk looks like x is now set to something. I expect it it to stay that way until the next x=??? statement. The fact that properties change it in a different way under the covers doesn't matter; what matters is that it changes when (and only when) I expect it to change. By getting rid of the x=??? statement, you force the programmer to keep more information in his own head. > As for the implementation of propget and propset, if you look at Ed Loper's > draft implementation, you'll see that an error is raised at function > definition time if there's an existing non-property binding for the named > function. Which is an acceptable solution. I would prefer that it keep the current value, but I understand his objection that a particular getter might make "current value" meaningless. What is not acceptable is a silent failure, planning to have it raise an exception "sometime". That would move the burden of implementation details back from the code that implements properties onto everyone who uses the object. -jJ From mikepy-0404 at mike.de Mon Apr 5 02:50:55 2004 From: mikepy-0404 at mike.de (Mike Pall) Date: Mon Apr 5 09:25:26 2004 Subject: [Python-Dev] The Need for a Declarative Syntax Element (aka Getting PEP 318 back on track) Message-ID: <20040405065055.GA1485@mike.de> Hi, I wanted to share my thoughts about the issue and wrote the following essay. Please note: all of this is IMHO of course. Nothing is cast in stone. I'm very willing to change it to whatever decisions the discussion leads to. Sorry about the length of this posting. :) Any and all comments are appreciated. Please send your comments to the list only, to avoid duplicate copies to my mailbox. Thank you! Bye, Mike The Need for a Declarative Syntax Element in Python --------------------------------------------------- (aka "Getting PEP 318 back on track") Author: Mike Pall Release: 2004-04-05 1. Introduction 1.1 Clean and Lean 1.2 Sugar is Mean 1.3 Pep up Your Life 2. Getting it Straight 2.1 Terminology (n); cf. lack of ~ 2.2 I Hereby Declare ... 2.3 Definitions 2.4 Now, what? 3. Foreign Territories 3.1 Big Brother in Action: C# Attributes 3.1.1 Syntax 3.1.2 Semantics 3.1.3 Use Cases 3.2 Catch Up, Baby: Java Annotations 3.2.1 Syntax 3.2.2 Semantics 3.2.3 Use Cases With JSR-175 Annotations 3.2.4 Use Cases With javadoc Annotations 4. Pythonic (Ab)use Cases 4.1 Assorted Attributes 4.2 Roaring Registries 4.3 Witty Wrappers 4.4 Proper Properties 4.5 To Sync or Not to Sync 4.6 Lexical Liberty 5. Semantic Wonderland 5.1 What? 5.2 How? 5.2.1 Calling the DO 5.2.2 Using __declare__ 5.2.3 Passing The Context 5.3 When? 5.4 What else? 6. Syntax, Syntax on the Wall 6.1 Wishful Thinking 6.2 Round Up The Candidates 6.3 Narrowing the Candidate Set 6.4 And The Winner is ... 6.5 ASCII Art 6.6 Finally 1. Introduction --------------- 1.1 Clean and Lean Python is an imperative language and has very few declarative elements embedded down in the language specification. And most of them are in the form of explicit statements (like "def", "class" or "global"). Only a few are implicit (that's why some people have mixed feelings about "yield"). There are only two declarative delimiters ("*" and "**" in parameter lists) and there are no non-statement declarative keywords (owing to the fact that Python has pure dynamic typing). This is in fact good. That's why we all love Python so much: the syntax is clean and lean! Just compare what other languages have embedded deep down in their lexical analyzer and their grammar: type definitions in C, the massive lexical and grammatical overloading of "*" and "&" in C++, scope-sensitive keywords like "static", "volatile" or "private" in C, C++ or Java. Some of that is pure necessity in languages with static typing, to save some typing (no pun). But most of it can be described with just one word: yuck! Python has (so far) successfully avoided the syntax inflation. Even classic OO stuff like "classmethod" or "staticmethod" are just builtin types. You have to invoke (instantiate) them yourself in the body of a class definition and modify the binding to the method name by assigning to the class dictionary. Alas that works only *after* the method definition is done, i.e. it has to be after the method body. 1.2 Sugar is Mean Unfortunately simplicity comes at a price: a severe shortage of syntactic sugar. This is not necessarily a bad thing as the instance reference dilemma shows: Some languages use implicit scoping (C++, Java) and provide overrides ("this"). Others use explicit delimiters ("@" in Ruby). Python does neither and leaves the name of the identifier for the instance reference up to the programmer (though "self" is a well established convention). In this case most of us are happy about it, because "Explicit is better than Implicit" holds here (you may object that "@" is explicit enough, but well ... then it's implicit in the method definition). However other issues leave something to be desired. The pressure to get a nicer syntax for "classmethod" and "staticmethod" has been traditionally rather low. Probably because neither is *that* common (though this may depend on your programming style). And this is where PEP 318 comes into play ... 1.3 Pep up Your Life The proposal of PEP 318 has resulted in a flurry of activity at the python-dev mailing list culminating during February and March 2004. Even shouting and mutual accusations have been reported. Most everone would be happy to get back to real work, as the subject has been creating an enormous amount of traffic but little consent. PEP 318 goes a bit further than simply proposing a nicer syntax for "classmethod" and "staticmethod". It proposes a generic 'decorator' syntax that allows you to write your own decorator functions. And you get to see the decorator definition *before* the method body. Most of the discussion has centered around what the most desirable syntax would be. However everyone's definition of 'desirable' differs remarkably. And the lack of convincing use cases hasn't helped keeping the discussion on track either. The early syntax proposals have some similarity with the C# language feature "[Attribute] definition". The feature is called C# attributes but cannot be compared directly to decorators nor Python attributes. That of course sparked some discussion about using decorators for defining attributes, but this is an entirely different matter. A few regularly scheduled syntactic duels later, we are still where we started: a solution in search of its problem. Knowing we want 'something', but not spelling it out. My humble hope is that this document helps to improve the quality of the ongoing discussion. 2. Getting it Straight ---------------------- 2.1 Terminology (n); cf. lack of ~ All this talk about decorators and attributes is really missing the point. Well, the word 'attribute' suffers severe proliferation in computer science. Everyone seems to have a different opinion on its meaning. The Pythonic definition is at best misleading when it comes to our discussion. And a 'decorator' is a software design pattern -- this is not a language issue. But if we want to extend syntax we _have_ to talk about language. Computer language design, that is. So let me rephrase our desire in proper terms: ==> We want to define a new kind of DECLARATIVE SYNTAX ELEMENT for Python. Now that it's out of the hat, you can gain new appreciation at the preceeding discussion. Part of the problem: naming conveys meaning. Not having a proper name for something will lead you astray. 2.2 I Hereby Declare ... I sure don't want to patronize the language cracks out there. You all know very well what declarative vs. imperative means. However giving some examples in context is always useful. So here are a few of the declarative notions we would like to express, phrased in natural language statements: "This is a classmethod." "This function needs to be run at exit." "This is an abstract class." "This is the documentation for a class attribute." "This method has been written by John Doe." "This class implements the Foo interface." "This method implements the Visitor pattern for this class." "This parameter must be an integral type." "This compound block shall by synchronized by a lock." "This statement should be supressed when DEBUG is not defined at runtime." All of these statements _declare_ that some other language element (class, method, function, class attribute ...) has a specific property. Usually they tell you something about the _effect_ of that declaration, but most of this knowledge is implied (you ought to know what happens when you use such a statement). However in general they do NOT tell you _how_ this effect is accomplished (i.e. implemented). Nor should they. Declarative syntax owes much of its power to the effect that most goes on behind the scenes. Someone else has of course written up in glorious detail how that is to happen. But you (the user of such a declaration) just get to say what you _want_. Not more, but no less. BTW: the word 'statement' in this context means 'natural language statement'. In computer languages we have more degrees of freedom, like keywords or delimiters. That's why I called it a declarative syntax _element_. The discussion about syntactic alternatives deserves an entire section of its own (see below). Ok ... back to our high tech adventure. 2.3 Definitions Declarative Syntax Element (DSE) The syntax element includes the Declaration itself and any lexical tokens that are required to make it recognizable as such by the parser. It can be either a grammatical production or a compound lexical token. This may be a minor issue from a visual point of view, but has subtle implications when it comes to applicability of the syntax element. Declaration The content of the DSE. As far as the grammar goes it could be anything from an identifier up to a suite. It could even be in a language of its own (don't do that). However looking ahead to the section on semantics it makes most sense to use an expression. The term 'Declaration' is a bit too generic and would apply to other syntactic elements, too (e.g. a method definition includes a declaration). Thus it is suggested that the more specific term be used everywhere. Declarative Expression (DE) The most likely syntactical choice for a Declaration. Any standard Python expression is possible here. Declarative Object (DO) The result of evaluating a Declarative Expression. Target of the DSE The grammatical element that is the target of the DSE. This could be anything from a method definition down to a statement or even an expression. The exact range of grammatical variety that could or should be permissible is discussed in the sections on syntax and semantics. Processing of the DSE Processing of the DSE involves several steps: parsing the DSE, Compilation of the DE, Binding to the Target, Evaluating the DE and Applying the DO. These steps may or may not occur in that order. Each step may be assigned to a different processing phase. The important decision to make is the exact point in time _when_ each step is done. Binding to the Target The process of binding either the DE or the DO to the Target. Evaluation of the DE The process of evaluating a DE gives us a DO. This however does not (yet) affect the Target. Applying the DO The process of applying the DO to the Target of the DSE. 2.4 Now, what? Ok, now that we have set things straight its time to get our hands dirty. I suggest we first take a peek over the fence, have a closer look at use cases, take a refreshing detour to semantic wonderland and then visit syntax hell again. Gee, like all good things, the fun part is at the end. 3. Foreign Territories ---------------------- 3.1 Big Brother in Action: C# Attributes Ignoring C#, just because it comes from the company who brought you EDLIN and BlueScreens is NOT a wise move. You gotta keep up with the news or you loose track. What the heck, the language spec (ECMA 334 -- freely available for download) goes so far as to explicitly spell out that you should NOT use Hungarian notation for identifiers. Whoa! Maybe they made their mind up?!? And for our little discussion C# has quite something to offer. Not only does it have an extensible declarative element. The same company brings you to new heights of enjoyment with their .NET and Longhorn initiatives. And part of that all-new lifestyle is using C# in every imaginable corner. So we get some desperately needed use cases to fuel our discussion. Even overly contorted ones -- promise! So let's explain the way C# attributes work in our terms. 3.1.1 Syntax An attribute section consists of one or more C# DSEs prepended to the target. A C# DSE is an optional attribute-target plus a colon and a list of attributes enclosed in brackets: DSE ::= "[" [attribute-target ":" ] attribute ("," attribute)* [","] "]" A C# attribute cannot be an arbitrary expression. It looks like a call to a constructor, but without "new": attribute ::= type [ "(" arguments ")" ] However the arguments look like any regular call and allow for arbitrary constant expressions. Both positional and named arguments are allowed. I'll save you the gory details of the syntax. Most of the time the target of the DSE is implicitly derived from context. The attribute-target allows you to specify the target explicitly. The permissible targets are: - An assembly or module. - A type declaration. - A method or operator declaration. - A parameter or return value declaration. - A field declaration. - An accessor declaration. - An event declaration. - An enum declaration. Defining an attribute is simple: write a class that derives from the abstract class System.Attribute or any subclass of it. 3.1.2 Semantics No big surprises so far. The semantics however are a bit tricky: During compilation the argument expressions are evaluated. They must resolve to constant arguments. An instance constructor is derived from the type and the type of the arguments. The type, the constructor and the arguments are bound to the target in the image for the module. During runtime when an attribute is accessed its constructor is called and returns the attribute instance. This is available with the GetCustomAttributes() reflection method. C# has static typing so compilation of one module requires access to all referenced modules. Several 'magic' attributes assigned to members of the referenced modules are evaluated at compile time to achieve interesting effects: - [AttributeUsage] specifies the scope of an attribute definition. I.e. which target(s) it applies to, whether it is inherited and if multiple applications to the same target are allowed. - [Conditional] allows conditional compilation. This allows you to omit the call to conditional methods defined in other modules depending on defines set while compiling the module containing the call. So basically it is sufficient to tag the target once and get the effect everywhere else whenever it is used. Confused? Please read section 24.4.2 of the spec. - [Obsolete] marks some elements as obsolete. When the compiler encounters it on a referenced element it emits a warning. - The standard mentions a [Flags] attribute that applies to enums and redefines it to automatically assign bitfield masks instead of values to the enum members. However I have not found enough information to guess the implementation details. There are a couple of other attributes that directly influence the runtime. E.g. the JIT compiler takes hints from the [StructLayout] or the [MarshalAs] attributes. And automatic permission elevation (including permission checks) is performed when code security attributes are present. C# has another interesting feature: it allows you to omit the 'Attribute' suffix from say [FooAttribute]. So when the compiler encounters [Foo] and finds a type named Foo that is not derived from the Attribute base class it searches again for FooAttribute. This is useful for interfaces where you can get both declarative and imperative behaviour using the same name. Ok, so far so good. But you cannot achieve the same kind of effects with user defined attributes in C# since that would either require extending the compiler or force compile time evaluation of attributes. This is where XC# comes in: this extension to the C# compiler opens up the compilation entity tree and allows you to operate on that. It comes packaged with some basic attributes: declarative assertions (contracts), code coverage analysis, design rule verification, spell checking and (oh well) code obfuscation. You can write your own custom attributes and you can inspect and modify every aspect of the grammar on the fly. 3.1.3 Use Cases The following use cases have been culled from various documents at MSDN and other sources. I've stripped the implementations and just left the bare declarations and definitions to avoid clutter. *** An attribute definition uses attribute declarations, too: [AttributeUsage(AttributeTargets.All)] public class HelpAttribute: Attribute { public HelpAttribute(string url) { this.url = url; } public string Topic = null; private string url; public string Url { get { return url; } } } The attribute we just defined can be used like this: [Help("http://.../Class1.html")] public class Class1 { [Help("http://.../Class1.html", Topic = "Method1")] public void Method1() {} } *** A DEBUG Conditional: [Conditional("DEBUG")] public static void DebugMessage(string msg) { ... } // Somewhere else, even in a different module ... // The *call* to the method is omitted if DEBUG is not defined while // compiling *this* module. DebugMessage("foo") ... *** The enum magic: [Flags] public enum FileModes { UserRead, UserWrite, UserExecute, ... } This one gets you 1, 2, 4, ... instead of 0, 1, 2, ... *** Custom serialization (marshaling, pickling) made easy: [Serializable()] public class PartiallySerializableClass { public string serializeMe; public int serializeMeToo; [NonSerialized()] public string leaveMeAlone; } Not only is this easier to specify than imperative serialization support. It is also less error prone because the serializer and deserializer is autogenerated from the type information. *** An XC# example of a declarative assertion (contract) that dynamically adds assertion checking code at compile time: void WriteHashCode([NotNull] object o) { Console.WriteLine(o.GetHashCode()); } ... generates the following code with an imperative assertion: void WriteHashCode(object o) { Debug.Assert(o != null); Console.WriteLine(o.GetHashCode()); } But the really interesting consequences of a declarative assertion are: - It allows for runtime introspection. - If you include it in the interface declaration, it automatically applies to any implementation. This allows for inheritance of declarative aspects. *** Calls to unmanaged code: class NativeMethod { [DllImport("msvcrt.dll", EntryPoint="_getch")] public static extern int GetCh(); } *** Defining security behaviour: [SecurityPermission(SecurityAction.Deny, Flags = SecurityPermissionFlag.UnmanagedCode)] private static void CallUnmanagedCodeWithoutPermission() { ... } [SuppressUnmanagedCodeSecurityAttribute()] [DllImport("msvcrt.dll")] internal static extern int puts(string str); *** WinFS extensions: [Folder(Uri="#System/Drive")] public class Drive { ... [Key] public string DriveName { [Probe] get { return drive; } set { ; } } ... [Probe(Uri="Name=_")] public static Drive GetLogicalDrive(string name) { ... } ... [Probe(Uri="GetAll", ResultType=typeof(Drive))] public static Drive[] GetLogicalDrives() { ... } ... } *** Indigo (Longhorn remote messaging framework): URL: http://msdn.microsoft.com/longhorn/default.aspx?pull=/library/en-us/dnlong/html/indigoattrprog.asp [System.MessageBus.Services.PortTypeChannelAttribute (UsingName="IndigoServiceClient")] public interface IHelloChannel : System.MessageBus.Services.IDatagramPortTypeChannel { [System.MessageBus.Services.WrappedMessageAttribute (Namespace="http://tempuri.org/")] [System.MessageBus.Services.ServiceMethodAttribute()] [return: System.MessageBus.Services.WrappedMessageAttribute (Namespace="http://tempuri.org/")] string Greeting(string name); } [DatagramPortType (Name = "Hello", Namespace = "http://tempuri.org/")] public class IndigoService { [ServiceMethod] public string Greeting (string name) { ... } } 3.2 Catch Up, Baby: Java Annotations Java has plenty of declarative syntax elements. Some people argue that Java is overly declarative. Some like it. Some don't. But up until recently it had no support for an extensible DSE. Ok, ok ... Java always had javadoc tags embedded in comments: /** * An abstract class with foo-like behaviour. * * @author John Q. Doe * @see Bar */ public abstract class Foo { ... } These are processed by an extra tool (javadoc) to autogenerate the docs from the sources. Calling this a DSE is ok, but it is not part of the language itself and as such has no access to its structure (you cannot pass constant expressions). Some say it was a quick hack initially. But it stuck. And it got abused. A lot. Examples below. It seems now we have a plethora of tools that scan for javadoc tags and then do some pre- or post-processing of your source and class files. But then came JSR-175 (public review ended November 2003) ... 3.2.1 Syntax JSR-175 introduces a new syntax for 'Annotations'. An annotation definition starts with the "@interface" keyword. A severly restricted variant of the interface definition syntax is used. It has implicit and final inheritance from java.lang.annotation.Annotation. Well ... to me it looks just like a glorified struct declaration anyway. An annotation can be applied to any declaration: classes, interfaces, fields, methods, parameters, constructors, enums, local variables and enum constants (implied field declarations). However the same annotation type can be applied at most once to any entity. Application of an annotation uses one of three syntactic variants: "@" type "(" member-name "=" member-val ["," member-name "=" member-val]* ")" "@" type "@" type "(" single-member-value ")" The latter two are just shorthands for variants of the first one with an empty list or for a single member type. This muddies the distinction between types and constructors quite a bit though. The values must be constant expressions of course. 3.2.2 Semantics During compilation the constant expression arguments for annotations are evaluated and combined into a set of initializers for the structure defined by the annotation interface. There are three retention policies for this set of initializers: - Source-only annotations are dropped at compile time. This is supposed to replace/augment javadoc tags in source comments. Local-variable annotations are always source-only. - Class-only annotations are stored in the class file but not loaded into the runtime. - Runtime annotations are stored in the class file and are loaded by the runtime. Reflection support via java.lang.reflect.AnnotatedElement works only for runtime annotations. Reflection is easy to use because only a single annotation for each type may be present for each element. The classes used to represent annotations are created at runtime using dynamic proxies. Owing to static typing the Java compiler has to read referenced class files and understands a few special annotations (excerpt): - @Target specifies the allowable targets for an annotation definition. - @RetentionPolicy() specifies the retention policy for an annotation definition. - @Inherited allows for inheritance of annotations from superclasses (this is NOT inheritance for annotation definitions themselves). To summarize: nothing spectacular and not very dynamic. It shows that the feature has been added to the language as an afterthought. The key improvement over javadoc tags is that external tools do not have to parse Java source files (unless pre-processing is required) and that annotations can be stored in the class file. This however does not obviate the need for some post-processors either. 3.2.3 Use Cases With JSR-175 Annotations Since the spec is pretty young you won't find many use cases right now: *** Giving a new tune to javadoc stuff: public @interface RequestForEnhancement { int id(); String synopsis(); String engineer(); String date(); } @RequestForEnhancement( id = 2868724, synopsis = "Provide time-travel functionality", engineer = "Mr. Peabody", date = "4/1/2004" ) public static void travelThroughTime(Date destination) { ... } *** This is the definition for the @Retention meta-annotation: public enum RetentionPolicy { SOURCE, CLASS, RUNTIME } @Documented @Retention(RUNTIME) @Target(ANNOTATION_TYPE) public @interface Retention { RetentionPolicy value(); } 3.2.4 Use Cases With javadoc Annotations *** Some stuff for BEA WebLogic using Java web services annotations: /** * @jws:location http-url="http://localhost:7001/webapp/Bank.jws" * @jws:protocol http-soap="true" */ public interface BankControl extends ServiceControl { } *** Security annotations and conversational support for web services: /** * @common:security single-principal="false" */ public class PurchaseSupplies implements com.bea.jws.WebService { /** * @common:operation * @jws:conversation phase="start" */ public void requestPurchase() { } /** * @common:operation * @jws:conversation phase="continue" */ public void approvePurchase() { } /** * @common:operation * @jws:conversation phase="finish" */ public void executePurchase() { } } *** Embedded SQL for Java: /** * @jc:sql statement:: * SELECT name * FROM employees * WHERE name LIKE {partialName} * :: */ public String[] partialNameSearch(String partialName); 4. Pythonic (Ab)use Cases ------------------------- Leaving the exact choice for the syntax aside I used my personal favourite everywhere. I suggest you substitute it with your favourite choice just to see the visual effect. I'm sorry, but I gave up trying to identify the original contributor for each of these examples. Let's say it's a true community effort. A big thank you to everyone for your input! 4.1 Assorted Attributes Attribute setters modify the target attributes but not the target itself. *** The first one is generic, the second one could inherit from it: <| funcattrs(foo = 1, bar = "baz") |> def foobar(x): ... <| rstattrs( arguments = (1, 0, 1), options = {'class': directives.class_option}, content = 1) |> def admonition(*args): return make_admonition(nodes.admonition, *args) <| rstattrs(content = 1) |> def attention(*args): return make_admonition(nodes.attention, *args) *** Here is a nice example to show how neatly the vertical bars line up: <| Author("Joe Q. Random") |> <| Copyright("ACME Inc.") |> <| Release("2004-04-01") |> <| Version(0, 0, 1) |> def industrialStrengthMethod(self): raise BlueScreenError(random.Random()) *** Hints for mad metaclasses with magic mangling motives: <|override|> def overrideMe(self, foo): ... *** With clever definition of the DO application process (see semantics) we get docstrings *before* the corresponding definition. I guess nobody will like the idea, but anyway: <|"""This is a new style docstring for a class. Bla bla bla ... Bla bla bla ... Bla bla bla ... """|> class HeavilyDocumented(object): <|"Class attribute docstrings now have the proper position."|> BUFSIZE = 8192 <|"This is a new style docstring for a method."|> def method1(): pass <|"Dynamic %s [%s]." % ("docstring" % time.ctime()) |> def thisReallyWorks(): pass 4.2 Roaring Registries Registry functions add a reference to the target in some other object. Often combined with attribute setters. *** This one shows where it would be convenient to have a declarative type with the same name as an inheritable imperative type: <| WebService("soap://localhost:8888/mywebservice") |> class MyWebService(WebService): <| ServiceMethod(None, str) |> def setName(self, name): self.name = name <| ServiceMethod(str) |> def sayHello(self): return "Hello %s!" % self.name; *** For more complicated interfaces to environments with static typing it would be nice to have declarations that apply to parameters and/or the return value: <| ServiceMethod |> def <|str|> method(self, <|str|> foo, <|int|> bar): .... 4.3 Witty Wrappers Wrappers generally wrap functions or methods with their own functions. The latter is then stored in the current dictionary to replace the original function. Oh BTW: the 'synchronized' wrapper has a section of its own. See below. *** We know these two well enough by now: <|classmethod|> def AClassMethod(cls, foo): pass <|staticmethod|> def AStaticMethod(foo): pass *** Using multiple decorators: <| Decorator1(), Decorator2("Just a random string") |> def someMethod(...): ... # which is equivalent to: <| Decorator1() |> <| Decorator2("Just a random string") |> def someMethod(...): ... *** Generics for Python? Well, if you must: <|generic(int)|> def f(x): print x, 'is an int' <|generic(str)|> def f(x): print x, 'is a string' *** A variation of the theme: runtime type checking: <|CheckSignature(int, int)|> def f(x, y): return x+y+1 *** Interface evolution does not need to be messy: <|Context("version", 1)|> def movePlayerMessage(self, arg1): .... <|Context("version", 2)|> def movePlayerMessage(self, arg1, arg2): .... Yes, the second declaration needs to be able to get at the dict entry for the first one (see semantics). *** Defining a native method interface: # The first one should be applicable to the module. TODO: But how? <|native.Library("gtk-x11-2.0")|> <|native.ClassMap()|> class GtkScrolledWindow(GtkWindow): # Yow, the great renaming ... the GNOME people would love it ... :) <|native.EntryPoint("gtk_scrolled_window_set_policy")|> def setPolicy(self, <|GtkPolicyType|> hScrollbarPolicy, <|GtkPolicyType|> vScrollbarPolicy): pass <|native.EntryPoint("gtk_scrolled_window_get_shadow_type")|> def <|GtkShadowType|> getShadowType(self): pass 4.4 Proper Properties The following has been proposed to make property definition easier: class Foo(object): <|propget|> def x(self): return self.__x <|propset|> def x(self, newx): self.__x = val <|propdel|> def x(self): del self.__x The problem with this is that propxxx needs to go through some hoops to update the previous definition of x. This includes using sys._getframe() and then searching the local and global dicts of the frame. Oh dear! This is a good argument why the DO should get the target object _and_ the dict where the TO is to be stored. A different idea would be to follow the C# precedent which provides an extra nesting level for properties: class Foo { private int x_; public int x { // <-- !! get { return x_; } set { x_ = value; } // value is an implicit identifier } } Foo foo = new Foo() foo.x = 1 int y = foo.x However I don't know how to do this in Python. Declaring a method as a property and using nested functions for the getter/setter does not work because the locals (i.e. the nested functions) are lost: class Foo(object): <|property|> def x(self): # Does not work! def get(): return self._x def set(value): self._x = value Doing the same thing with an inner class might work somehow, but I'm not sure. We could get inheritance for properties, too!? TODO: I haven't explored this any further. More input is welcome. 4.5 To Sync or Not to Sync Java has special declarative syntax for synchronized methods and blocks: public abstract class FancyControlStream extends FancyStream { public synchronized void putMsg(byte[] data) { } public int kickBack() { ... synchronized (obj) { obj += len; obj.notify(); } ... } } This would be a good use case for our DSE, too. But ... being only applicable to a method definition just doesn't cut it: <|Synchronized(lock)|> def put(self, item): ... This is way too tedious for some stuff. So, what about: <|Synchronized()|> class ThreadSafeQueue(Queue): pass # Yes, this is all that's needed. On the other hand you really want to use fine grained locking for more ambitious endeavors. And this is where you need to use locks at the compound statement level (called suites in Python and blocks elsewhere): def stopStream(self): <| Synchronized(self.recv_lock) |> if self.recv_len<=0: if self.recv_thread: self.recv_thread.interrupt() self.recv_thread=None self.send_close=True <| Synchronized(self.send_lock) |> while self.send_len>=0 && self.send_close: self.send_lock.notify() Getting it at the statement level would be nice, too. But you could work around this with 'if 1: SUITE'. How you would _implement_ this is another story. And this is where the bad reputation from Java's 'synchronized' comes from: there have been some initial design flaws. However just because you have one bad precedent does not mean we can't do it better. IMHO declaring some parts of a program to be regions with synchronized access is inherently useful. 4.6 Lexical Liberty To get at the lexical level you may need a different delimiter which is evaluated at compile time. I have opted not to replace it with a different one in the following examples. Please don't flame me. I'm just documenting some strange ideas: class Whatever(object): # Uh, imports and assignments in a declarative compile-time context? <| from declarative.lexical import * |> <| T = Conditional("TRACE") |> # These statements are optimized away at module load time. def run(self): <|T|> self.tracePrint("Startup") ... <|T|> self.tracePrint("Shutdown") # Combining the power of Python with the syntax of SQL. Or vice versa? def query1(self, start, end) return <|SQL|> "SELECT * FROM table WHERE name BETWEEN :start AND :end" def query2(self, start, end) return <|SQLPy|> SELECT * FROM table WHERE name BETWEEN :start AND :end def query3(self, table, a, b) return <|PySQL|> SELECT "*" FROM table WHERE "f1" == a*2 AND "f2" >= a+b # C has inline assembler. Python has inline C. def fast1(self): <|C|> "{ for (int i=0; i<100; i++) somefunc(i) }" def fast2(self): <|C|> { for (int i=0; i<100; i++) somefunc(i) } def fast3(self, a, b): x = self.foo + <|C|> int dummy(int a, int b) { for (int i=0; i<1000; i++) { a=a+3*i+(b&15); b+=a; } return b; } <||> return x <|C|> int fast4(int a, int b) { for (int i=0; i<1000; i++) { a=a+3*i+(b&15); b+=a; } return b; } <||> # Sssshhh! Don't mention the r-word! <|LexicalSubstitute("@", ("self", "."))|> <|Grammar("parameter_list").afterParsing( lambda p: p.insert(0, "self") )|> class Wherever(object): def __init__(): @flags=[] def setFlag(flag): @flags.append(flag) 5. Semantic Wonderland ---------------------- 5.1 What? First we need to specify what kind of grammatical element a DSE may contain. I could go on an reason about all the possibilities. But nobody ever mentioned anything else than an expression. It seems to be the most straightforward approach. This document is already far too long. A DSE contains a DE. So be it. :) Expressions evaluate to a single object. This is the DO in our case. For multiple declarations in a single DSE there are two possibilities: - Allow a plain expression and add the commas to the DSE grammar. - Allow an expression_list and add tuple handling. Multiple DOs from multiple adjacent DSEs result in a DO list (in the given order). The target the DSE applies to is to be specified by the syntax. To be able to bind the DSE to the target it needs to be bundled into an object. We call this the Target Object (TO). What the TO needs to contain depends on the target. The issues with arbitrary grammatical elements are discussed below (see 'What Else?'). So let's restrict the target to functions, methods and classes for now. Each one of them already bundles its contents into an object. But _defining_ one of them involves adding its name to a dictionary, too: - The Target Dictionary (TD) is required by the DO for some use cases. - The name to be defined is accessible from the TO for all three types. But it is either read-only (for functions) and/or modifying it may not get you the desired effect. It is unclear whether we have any use case that requires storing a different name in the TD instead of the one used in the target. Being able to supress the store might be desirable, too. TODO: Ideas? 5.2 How? [My bet is that this section will be the most controversial.] We need to specify how each of the required steps is to be performed. First the straightforward stuff: - Parsing the DSE: since this is a plain Python expression it is to be performed by the compiler. - Compilation of the DE: dito. - Binding to the target: the Pythonic way to do this is to emit code for the module or class initialization code objects. The code gets to mangle the compiled DE, the target, the target name and the TD. [ It is an implementation detail whether ... - a call to each DO is explicitly emitted OR - whether we call a builtin function or emit a new bytecode that gets passed the TO and all DOs. The latter may have some advantages for future extensibility. ] TODO: Code generation needs to be specified in detail. - Evaluating the DE: this is done by the code compiled from the DE. Now the troublesome part: applying the DO. 5.2.1 Calling the DO The original proposal was derived from the way stuff like classmethod works: *** TO is a function: def func(x): ... func=callable(func) *** TO is a method: class Foo(object): def method(args): ... method=callable(method) So we require the DO to be a callable and apply it by calling it, passing the TO. The return value is then used as the new TO and stored under the target name in the TD. I.e.: TO = DO(TO) [ Subtle side effect: The TD may contain a previous element named like the target before the target definition is performed. This is available to the DO because the TO is not stored until all DOs have been applied. This cannot be achieved without some renaming games by doing it the classic way. ] There are three common cases for DEs: *** The DE is a function call that evaluates to an inner function. The DO is the inner function. It may be bound to derivatives of some arguments of the DE. Applying the DO means calling the inner function. def funcattrs(attrs): def inner(func): for name, value in attrs.iteritems(): setattr(func, name, value) return func return inner *** The DE is a class constructor and evaluates to an instance of the class. The DO is an instance. It may hold derivatives of some arguments of the DE. Applying the DO means calling the instance (which only works if you define a __call__ method). class funcattrs(object): def __init__(self, **kwds): self.attrs = kwds def __call__(self, func): for name, value in self.attrs.iteritems(): setattr(func, name, value) return func *** The DE is a class and evaluates to a class. The DO is a class. Applying the DO means to call the constructor of the class. A side effect of this is that the resulting TO is an instance of the class. The primary example for this are classmethod and staticmethod which are builtin types. 5.2.2 Using __declare__ Reusing the callable attribute of objects for declarator application has a few drawbacks: - Any callable may be used as a DO. There is no error checking. Mistaking imperative classes for declarative classes, dropping empty braces, scoping mismatches or just plain carelessness will NOT be caught. This may be very hard to track down since the resulting TO is not checked for validity. Errors may pop up very late or never. Any error message you might get will probably be misleading. - A class cannot have both declarative and imperative behaviour. However as the precedent given by C# shows, this might be quite useful. - Inheritance is a useful concept for declarative behaviour, too. Thus it is most likely that DE is a class constructor. This requires the use of __call__ which is neither self-documenting nor exactly easy to explain to newbies. I propose to use a new kind of slot with the name "__declare__". The DO must define this slot and it must be bound to a callable. Applying the DO means calling the slot and passing the TO. The return value is used as the new TO and stored under the target name in the TD. I.e.: TO = DO.__declare__(TO) This has the following advantages: - Existing objects cannot be accidentially used as declarative expressions. - Error checking is done at module load time and misuse will abort the load with a unique error message exactly pointing out the problem. - Defining a method with the name __declare__ makes your intention crystal clear. Newbies can find out about the meaning easily by searching for __declare__ in the documentation. Compare this with the value the documentation for __call__ would have to a newbie. - Whenever you see a declaration you know exactly where to look in the class definition to find out what it does. - Requiring a __declare__ slot strongly encourages to use classes instead of functions for declarative behaviour. Since classes are inheritable, this will over time improve the quality of the code base. Using functions is still possible but one has to set the __declare__ slot explicitly in the function dictionary. You can even avoid defining an inner function if you don't need bound DE arguments (otherwise a class would be the cleaner solution). - We can easily support the old and the new way to apply classmethod and staticmethod. We also can adapt the new calling sequence to our needs without breaking compatibility with existing uses of classmethod and staticmethod (see the next section). - We may set __declare__ slots on internal types if we want to give them declarative behaviour without necessarily making them callables. E.g. you can have strings behave like docstrings if used declaratively. I know very well that adding a new slot requires changes in several places. But I think the benefit outweighs the required work by far. TODO: Write up what needs to be done to add a new slot. 5.2.3 Passing The Context As you can see from the use cases, some of them need access to the TD. The current workaround involves sys._getframe() and is ugly beyond belief. We can avoid this problem by passing the TO *and* the TD to the DO. I.e.: TO = DO.__declare__(TD, TO) This may be required, too for some future extensions that involve targets other than functions, methods and classes. In general it is not a bad idea to pass the context where the target is defined. Since applying the DOs is usually done at module initialization there is no performance penalty for passing two arguments instead of one. Code generation would be pretty easy, too. Summary: we gain flexibility with minimal cost. TODO: Check the issue mentioned above about modifying the target name before it gets stored in the TD. We could pass a modifiable reference to the name somehow but that would complicate the code generation for this case quite a bit. Ideas? 5.3 When? There are basically two choices for the temporal behaviour of the DSE: - A DSE with access to the lexical level needs compile time evaluated DEs. As indicated in other sections this is not within the current scope of this document. - Any other imaginable DSE is only useful if the DE is evaluated and applied at module initialization time. Any runtime-only behaviour can be implemented on top of it. This implies that all other processing steps need to be done at compilation time (parsing the DSE, compilation of the DE and binding to the target). Phew. That was easy. A subtle issue that has been discussed previously is the order of application of the DEs. Previously you had to write things backwards to get the correct order of application (d first, then e): def f(x): ... f=e(d(f)) Thus the question arised, what the order might be with the new syntax: <|d,e|> def f(x): ... The natural interpretation depends a little bit on the syntax chosen (whether the DSE is before or after the function name). But with my favourite syntax there is no ambiguity: d is applied before e. Also <|d,e|> is equivalent to <|d|> <|e|>. It helps to remember that Pythons existing declarative statements (such as "def") are implemented in an imperative way. And this is clearly top-to-bottom and left-to-right. If the order of application is important, then it is the duty of the user to get the order of the DEs right. It is however deemed unlikely that this is a problem in reality. TODO: I cannot find a way for the DO class author to specify the preferred order of application (other than through documentation). Priorities would not help since they would need to be attached to the DSE, which is pretty useless (remember: the compiler does not see the DO class). Comments? 5.4 What else? The 'synchronized' issue highlights the need for having declarative syntax that is applicable to a wider range of syntactic constructs. But there is some trouble ahead with Python: methods and classes are first class objects. Compound statements are not. Passing a method or a class to a function to modify them is easy. Inserting the returned object (possibly a wrapper) into the module or class dictionary is easy, too. Getting the same level of support for compound statements or even individual statements is way more difficult. You basically have two choices for this: a) Make statements first class objects and somehow optimize this artifact away by using late code combining. b) Allow compile time evaluation of a declarative element giving it access to the lexical level. Ok, so a) is more akin to the way statements inside a class definition incrementally build up the class. You'd have to plan ahead a bit and think how this could be extended to declarations for parameters and other stuff, too. But b) is a really a much more powerful tool. Forget about CPP macros. Forget about SQL preprocessors. You can do all of this yourself now just by writing some Python functions. C has inline assembler, Python could have inline C! Embedding foreign languages, extending the grammer, even redefining your favourite lexical tokens: you get it, if you ask for it. [ Just a thought: you could do b) 'the language way' and open up the compiler or 'the text processing way' and open up the input stream. The latter would be pretty easy to do, I think. But then you may not get to see or modify all the compiler context you need. ] Both issues need A LOT MORE discussion and are certainly NOT something for the 2.4 time frame or for PEP 318. However we should take a mental note about two things: - A new declarative element SHOULD have the potential to be applied to almost any other syntactic element. This must be reflected in the initial syntactic _specification_ but not necessarily in the initial _implementation_. - Having a declarative element evaluated at compile time holds big potential for the future. Although this could emulate any other more traditional kind of declarative element, the framework for this is just not ready today. So in fact I do NOT propose to specify such an element right now. However I propose to either a) allow for a later extension of the declarative element that we are defining now OR b) define a second declarative element later that shares most of the syntax but has different temporal evaluation behaviour. 6. Syntax, Syntax on the Wall ----------------------------- Hey, the first Python beauty contest in history. Quick, place your bets. 6.1 Wishful Thinking The Syntax ... Requirement Requirement Definition: Mnemnonic The Syntax ... ------------------------------------------------------------------------------ |LEXICAL| MUST fit into the existing lexical structure. |PARSER| SHOULD put no extra burden on the existing parser. |GRAMMAR| MUST fit into the existing grammar. |GENERIC| SHOULD allow for the declarative element to be applied to almost any other syntactic element. |EXPR| SHOULD allow for a general expression as the content of the declarative element. |SPLIT| SHOULD allow for lengthy element content that may need to be split across several lines. |SEQ| SHOULD allow for a convenient notation for a sequence of declarative elements. |CONCISE| MUST be concise. |VISUAL| MUST stand out visually. |DISTINCT| SHOULD be immediately recognizable as a distinct language feature. |MIMIC| SHOULD NOT mimic completely unrelated features of other common programming languages. |NEWBIE| SHOULD NOT overly confuse newbies. :) There are a lot more issues, but I think I covered the most important ones from a language perspective. Feedback is appreciated of course. >From a language perspective not much attention has been given to |GENERIC|. This is what worried me most and got me to write this essay. 6.2 Round Up The Candidates The existing lexical and grammatical structure allows ... {STATEMENT} a statement with a (possibly new) keyword: as DECL {COMPOUND} a compound statement with a (possibly new) keyword: as: DECL ... {EXTENSION} to extend an existing grammatical element: def f(x) DECL: {CONTEXT} an existing lexical element in a different context: def [DECL] f(x): {REDEFINE} redefining the meaning of an existing grammatical construct: [DECL] def f(x): {UNARY} a new introducing delimiter: @DECL .DECL {ASYM} a new asymmetric paired delimiter: *[DECL] @[DECL] {SYM} a new symmetric paired delimiter: [|DECL|] <|DECL|> I guess there are some more variations on this theme, but none too relevant. 6.3 Narrowing the Candidate Set Well our BDFL has already spoken out and narrowed the set considerably. But I think it's still worthwile to discuss this in detail: {STATEMENT} has not been received well, because it would almost certainly necessitate a new keyword. And neither has a good one been proposed nor should the introduction of a new keyword be taken lightly. Getting |GENERIC| straight might prove to be difficult, too. And don't forget about |EXPR| and |VISUAL| while you are at it. Even if we manage to find a good name, it's doubtful that everyone seeing such a construct knows that it is evaluated in a different context than almost any other statement (violating |NEWBIE|). {COMPOUND} looks strange to me because it does not fit well into the lexical structure (putting aside the variants that violate |LEXICAL|). Compound blocks usually contain statements and not expressions. And of course it only works if the target of the declarative element is a block (violating |GENERIC|). Prepending a new kind of block to an existing target block introducer is confusing because one may expect another level of nesting here (which would violate |CONCISE|). Adding a new kind of block just _after_ the target block introducer destroys the visual link between the introducer and the body of the block. The compact variant (putting a single declarative element right after the colon) shares the concerns about |NEWBIE| with {STATEMENT}. In short I cannot find a compelling reason to add a new kind of compound statement to the language. And a compound declarative statement is ... well ... awkward (considering the lack of precedent set by other languages). {EXTENSION} has been thrown out early because it violates |CONCISE|, |VISUAL| and |MIMIC|. But I think far more important are the violation of |SEQ| and |EXPR|. {CONTEXT} has been discussed to death but this is just because the initial discussion focused on variants of this syntax. Other than that it has few merits. Since it is an existing lexical construct (list displays) it violates |GENERIC|, |DISTINCT| and |NEWBIE|. Depending on the position it may also violate |PARSER|, |GRAMMAR| and |VISUAL|. But I think the most compelling reason against it, is that it hurts the eyes with |SPLIT|. The existing use cases for C# indicate that this would be pretty common. {REDEFINE} is problematic because it redefines an existing (though useless) grammatical construct and as such violates |GRAMMAR| and |DISTINCT|. The grammatical meaning could be redefined, but it would only work just before a block introducer (violating |GENERIC|). Just because this syntax mimics C# somewhat is not a good enough reason to introduce the same syntax to Python. It cannot mimic it completely anyway due to lexical differences (break-on-white-space vs. break-on-lines-or-sequence-delimiters). It also violates the principle of least surprise: running a program using this construct in an older version of Python won't get you a syntax error. And the desired effect is dropped silently -- which can be a very dangerous thing to do. {UNARY} fails on |VISUAL| for |SPLIT| because the end may be hard to match to the start. It may be hard for |PARSER| in intra-line contexts because of |EXPR|. Restricting the construct to be used only on extra lines would violate |GENERIC| and maybe |CONCISE|, too. I do not think we should waste one of the three remaining unused non-alphanumeric ASCII characters (@, $, ?) for this purpose. It would miserably fail on |MIMIC| anyway (except for the Java precedent). Prepending a dot is not an option because it would not conform to |GENERIC|, |VISUAL|, |DISTINCT|, |MIMIC| and |NEWBIE|. Also GvR indicated that he wanted to reserve this construct for a future "with" statement. {ASYM} doesn't look bad if the construct is used on extra lines. It may be a bit harder to spot in an intra-line context. {SYM} looks good in any context. It would be natural to define it like any other paired sequence delimiter ((), [] and {}). This allows for flexible line breaks and broad grammatical applicability. 6.4 And The Winner is ... Beware: everything I wrote is IMHO of course, but this applies especially to this section! {ASYM} saves a single character to type over {SYM} (which is not much of a gain). OTOH the asymmetry spoils the visual effect quite a bit. So my personal winner is: {SYM} Your mileage may vary though. Even if you disagree with my reasoning, the best the preceeding section buys you is a systematic way to discuss all alternatives now. Go ahead! 6.5 ASCII Art If (BIG IF!) there is a concluding discussion that indeed supports my reasoning about the lexical alternatives we should decide/vote/pronounce on a delimiter pair. Since @, $ and ? usually do not come in pairs, the shortest possible delimiter pair is two characters long each. We should include one of the four visual pairs that ASCII has proudly brought to your home for the past decades ((), [], {} or <>). For best visual effect the pair should be used for the outer characters. The inner characters may be a single non-paired character or another set of paired characters. Of course neither the opening nor the closing delimiter should collide with an existing delimiter. Nor should the inner character be mistaken for the start of an expression or (less important) for the end. That leaves us with: (< DECL >) [< DECL >] {< DECL >} ($ DECL $) (% DECL %) (& DECL &) (/ DECL /) (: DECL :) (= DECL =) (? DECL ?) (@ DECL @) (^ DECL ^) (| DECL |) [$ DECL $] [% DECL %] [& DECL &] [/ DECL /] [: DECL :] [= DECL =] [? DECL ?] [@ DECL @] [^ DECL ^] [| DECL |] {$ DECL $} {% DECL %} {& DECL &} {/ DECL /} {: DECL :} {= DECL =} {? DECL ?} {@ DECL @} {^ DECL ^} {| DECL |} <$ DECL $> <% DECL %> <& DECL &> <: DECL :> <= DECL => <@ DECL @> <^ DECL ^> <| DECL |> Now, before you choose, consider that you can use it without any spaces between the delimiter and the declarative expression. I guess this would be common for e.g. classmethod. That's why my personal favourites (in descending order of preference) are: <|DECL|> <== I like this one most. <:DECL:> (|DECL|) [|DECL|] <%DECL%> The vertical bars line up neatly if you use them on successive lines. And it visually separates the content well, even without spaces. But back to a more scientific analysis: conveying meaning with delimiters is hard. Only precedent may help us here: So is used in XML for processing instructions (PIs). But it's up to the language used for the PI whether the content of the PI has declarative or imperative meaning. We'd fall prey to |MIMIC| I guess. Choosing would not help to clear up matters, either. Most other delimiters have no common precedent (that I know of). But the inner character has. That makes $, @ and = seem awkward choices. Oh and when picking one, we should think about reserving another one for a future compile time evaluated variant, too (IMHO <:DECL:> has some merit here). BTW: choosing smileys as lexical tokens would sure help to entertain slashdot for weeks! :) 6.6 Finally Still with me? Good, because here is the syntax I propose: Lexical definitions: DECL_ELEMENT_START ::= "<|" DECL_ELEMENT_END ::= "|>" The tokens should be defined so they work like the existing paired delimiters (i.e. allowing NEWLINE). Grammar definitions: decl_element ::= "<|" [expression] ("," expression)* [","] "|>" OR decl_element ::= "<|" [expression_list] "|>" The expressions should yield declarative objects of course (see the section on semantics). The DSE should be placed immediately *before* the lexical element to which it applies. Multiple declarative elements may stack up and all apply to the following element. The use cases indicate the following possible applications in descending order of importance: - Method and function definitions ("def"). - Class definitions ("class"). - Related to the module (there is no good place to put it though ...). - Class attributes, i.e. assignments in a class definition. - All compound statements ("if", "while", "for", "try"). - Parameter definitions. - Related to the return value of a method or function (either by applying it to "return" or by putting it between "def" and the function name). - All statements, i.e. assignments, too. The sheer variety indicates that we should not restrict the DSE to be usable only on a line of its own. We might as well make the DSE applicable to *any* grammatical element. >From a language perspective that would certainly be pretty orthogonal. >From an implementation perspective we may disallow it or ignore it for some cases. The exact way to pass the target of the DSE to the declarative object(s) needs to be figured out, though. A more radical thought would be to treat the DSE as a compound lexical token. This could then be used before ANY token. It would work like a C comment block. Yes, this smells a bit like a macro, but it doesn't behave like one (unless we do compile time evaluation). And no, I wouldn't place my bets on this one. TODO: More input is required. Please go ahead. -------- End of Document -------- From arigo at tunes.org Mon Apr 5 09:23:16 2004 From: arigo at tunes.org (Armin Rigo) Date: Mon Apr 5 09:26:54 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> Message-ID: <20040405132316.GA21940@vicky.ecs.soton.ac.uk> Hello Paul, On Mon, Apr 05, 2004 at 01:54:38PM +0100, Moore, Paul wrote: > I got the impression that Armin's point was that Psyco can optimise > the integer case *if* the code is in Python, but not if it's in C. Yes. > [Armin, how does Psyco + a Python sort compare against timsort on > more general objects?] Psyco suffers its current general problem: if you are using an operation that hasn't been explicitely recoded in Psyco (like tuple comparison) then the optimization wanishes. Sorting a list of (integer, complex_object), which is more common in practice, is 1.5x slower with Python+Psyco. I am sure that with 30 minutes of effort in the proper code I can make it faster than Timsort again. But Psyco has the same general problem everywhere. I think that Psyco isn't ready for real uses yet because you just can't predict the resulting performance unless you know its internals. My hope with PyPy is that it will change that, by giving a way to generate all these missing Psyco parts automatically from the PyPy source, but that's still a long way. I won't even mention that Psyco works well only on 386. (It has got an almost-ready platform-independent back-end, but that's slower, so it is only a win in more extreme cases.) > Hmm. I'm torn. On the one hand, it's really cool that Python+Psyco can > out-perform C (another argument against the knee-jerk "recode bits in > C" reaction). But my gut feel is that not enough people use Psyco to > make this a good answer yet. On the third hand, if we recode chunks of > the stdlib in C, do we kill too many chances for Psyco to work its > magic? I think that (re)coding in C is very positive right now. And if we don't actually loose the original Python source (if there is one) then we don't really loose the ability to revert to it if at some point it is consistently better. Essentially, I wanted to rant about the fact that some performance considerations are driving the language's evolution, and this is much harder to revert to if later the performance reasons vanish. A bientot, Armin. From dave at boost-consulting.com Mon Apr 5 09:55:38 2004 From: dave at boost-consulting.com (David Abrahams) Date: Mon Apr 5 09:56:37 2004 Subject: [Python-Dev] Re: PEP309 re-written References: <40715D70.1010402@blueyonder.co.uk> Message-ID: Peter Harris writes: > David Abrams wrote: > >>> If anyone can think of any really elegant hacks that are naturally >>> expressed by partial function application I'd like to see them >> >> >> >>There are millions, but AFAICT they're all more natural with lambda, >>so... >> >> "I agree that lambda is usually good enough, just not >> always." >> >>Examples, please? >> >> > Well, you can use partial to special-case classes as shorthand object > factories. So: > > C = partial(Canvas,my_window,width=100,height=100,bg='white') > > ...gives you a callable C that is a factory for Tkinter Canvases > parented to my_window, > 100 pixels on a side, and with a white background. How this differs > from lambda is that > you can override keyword parameters from the supplied defaults: > > c1 = C() > > c2 = C(width=200) > > c3 = C(bg='red') C = lambda *args, **kw: \ Canvas(my_window, width=100, height=100, bg='white', *args, **kw) OK, partial is a little shorter in that case. I'm not sure that example justifies a PEP, though. def _merge(d1, d2): d1.update(d2) return d1 def partial(f, *args, **kw): return lambda *args2, **kw2: f(args + args2, merge(kw2,kw)) ?? >> "And I want the possibility of useful introspection and >> subclassing" >> >>Can you elaborate on that, please? >> >> >> > You could maybe sub-class partial to override __call__ , in case you > want to do anything fancy like pre-supplying arguments at arbitrary > positions. A *perfect* example of what lambda is already good at. -- Dave Abrahams Boost Consulting www.boost-consulting.com From mcherm at mcherm.com Mon Apr 5 10:03:46 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Apr 5 10:03:43 2004 Subject: [Python-Dev] PEP 318: Let's propose some useful built-in decorators Message-ID: <1081173826.40716742a1265@mcherm.com> Guido writes: > While I ponder the decorator syntax, let's propose some built-in > decorators. [...] > Proposals for other standard decorators are welcome! memoized -- Encloses the function in a wrapper which provides automatic memoization. Provides two different uses. The simple use is just this: def fibonacci(n) [memoized()]: if n == 0: return 0 elif n == 1: return 1 else return fibonacci(n-1) + fibonacci(n-2) ...which automatically memoizes all results. For advanced users who want to limit memory use, there's this: def fibonacci(n) [memoized(cache_size=25)]: [...] ... which will save only the 25 most-recently-used values. Here... I'll try to implement it, since I want to play around with writing a decorator which takes arguments. I'll see if I can do so in a highly readable manner: # WARNING... Untested code follows! def memoized(cache_size=None): if cache_size is None: # No cache size limits, so just store the values in a hash def memoize_decorator(f): memo_cache = {} def wrapper_function(*args, **kwargs): if memo_cache.has_key((args, kwargs)): return memo_cache[(args, kwargs)] else: value = f(*args, **kwargs) memo_cache[(args, kwargs)] = value elif not memo_cache_full: return value return wrapper_function else: # cache_size is given, so we need a LRU cache. # This currently uses a dirt-simple LRU cache... a list # of (key, value) pairs with new items added to the end. # Performance could be improved with a fancier data # structure. def memoize_decorator(f): memo_cache = [] def wrapper_function(*args, **kwargs): for key, value in memo_cache: if key == (args, kwargs): # Found it in the cache! Move to end of list. memo_cache.remove( ((args, kwargs), value) ) memo_cache.append( ((args, kwargs), value) ) # Not found in cache... must calculate value = f(*args, **kwargs) memo_cache.append( ((args, kwargs), value) ) if len(memo_cache) > cache_size: memo_cache.pop(0) return value return wrapper_function return memoize_decorator Hmm.... somewhat awkward, but readable. Nested function definitions and nested scoping rules make this much easier than otherwise. Nevertheless, a class-based approach might have worked better. -- Michael Chermside From tree at basistech.com Mon Apr 5 10:36:26 2004 From: tree at basistech.com (Tom Emerson) Date: Mon Apr 5 10:41:09 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> Message-ID: <16497.28394.944929.536518@magrathea.basistech.com> Moore, Paul writes: > Hmm. I'm torn. On the one hand, it's really cool that Python+Psyco can > out-perform C (another argument against the knee-jerk "recode bits in > C" reaction). But my gut feel is that not enough people use Psyco to > make this a good answer yet. On the third hand, if we recode chunks of > the stdlib in C, do we kill too many chances for Psyco to work its > magic? If I understand correctly, the issue is that when stdlib functions are rewritten in C, they end up being faster but are still slower than Psyco optimized code because they are still polymorphic while the Psyco'ized generates multiple paths. If that is the case then one could certainly hand-hack the C implementations to expand common code paths in an optimal way. -tree -- Tom Emerson Basis Technology Corp. Software Architect http://www.basistech.com "Beware the lollipop of mediocrity: lick it once and you suck forever" From ark-mlist at att.net Mon Apr 5 10:42:40 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Mon Apr 5 10:42:29 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1081162471.3004.4.camel@localhost> Message-ID: <003901c41b1c$3fd9f760$6402a8c0@arkdesktop> [I've mentioned this idea before, but don't recall seeing any responses to it; so I'm mentioning it again in case it got lost the first time] I am concerned about putting decorators before the function definition because of the possibility of such code quietly sneaking through existing compilers: [classmethod] def foo(bar, baz): pass Guido is concerned about putting decorators after the function definition because there may be a lot of them and it may be hard to read: def foo(bar, baz) \ [staticmethod, classmethod, otherattributes(debug=True}) }: pass Here's what I don't understand. I imagine that most of the time, when someone decorates a function with lots of attributes, there is at least the possibility of decorating more than one function with the same set of attributes. In such a case, doesn't it make sense to bind a variable to the attributes and use it instead? Attributes = [staticmethod, classmethod, otherattributes(debug=True)] def foo(bar, baz) Attributes: pass And doesn't this idea answer the objection that too many attributes after the parameter list make the definition hard to read? For that matter, why do we need the brackets if there is only one attribute: def foo(bar, baz) staticmethod: pass I am suggesting that what comes between the ) and the : should be an expression, which must evaluate to either a callable or a sequence of callables. For that matter, why not allow a tuple expression without parentheses: def foo(bar, baz) staticmethod, classmethod: pass Whatever sequence you put there, I think the semantics are clear: Before binding a name to the function, pass it to the callable or in turn to each element of the sequence. From pje at telecommunity.com Mon Apr 5 11:18:34 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Apr 5 11:18:54 2004 Subject: [Python-Dev] PEP 318: Properties In-Reply-To: Message-ID: <5.1.1.6.0.20040405111547.0242bc50@telecommunity.com> At 09:20 AM 4/5/04 -0400, Jewett, Jim J wrote: > x=asdlfjk > >looks like x is now set to something. I expect it it to stay that way until >the next x=??? statement. Or the next 'def x' statement, or 'class x' statement. Assignment is not the only way to bind a name in Python. Even if you know nothing about decorators, it should be apparent that in today's Python, this: x = 42 def x(): pass does *not* leave x equal to 42. From jim.jewett at eds.com Mon Apr 5 11:40:14 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Mon Apr 5 11:41:07 2004 Subject: [Python-Dev] The Need for a Declarative Syntax Element (aka Message-ID: Mike Pall: > It is unclear whether we have any use case that > requires storing a different name ... When adapting another module or library, it is often desirable to expose both the original name (for people familiar with the original) and a cleaned up name (so that your module can be used with internally consistent naming). The current syntax is to bind a second name to the object after it is defined; that second binding could reasonably move into a decorator. Ensuring that it points to a final-enough version of the object is probably the programmer's responsibility. > Using __declare__ [See the original for more detail - basically, decorators must have a __declare__ slot which will be called.] I think this is the first non-arbitrary restriction we have seen on decorators. It does largely solve the old-code- might-run-differently issue, though not the old-interpreters- might-silently-run-code-undecorated issue. Your suggested markers of [| |] or <| |> also solve that, as would a new keyword. A new word has the advantage of stating explicitly what it means, which allows additional extensions (some of which you already suggested). Defining a decorator API also allows more freedom for the decorators; the __declare__ API could accept more than just the target object. For example, it could take keywords giving the dictionary where the object will eventually be defined, or the name that will be used. (These would reduce the need for _getframe.) -jJ From shane at zope.com Mon Apr 5 11:50:57 2004 From: shane at zope.com (Shane Hathaway) Date: Mon Apr 5 11:51:00 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1081156611.2377.6.camel@localhost> References: <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> Message-ID: <40718061.2040607@zope.com> Mark Russell wrote: > What about: > > [as classmethod] > def foo(bar, baz): > pass > > To me this is more obvious (and less like p*rl) that *[classmethod], and > it is also currently a syntax error so won't break existing code. I agree, it's nice. You could even use this syntax at the interactive prompt. The "is" keyword is another choice, too. Shane From barry at python.org Mon Apr 5 12:00:56 2004 From: barry at python.org (Barry Warsaw) Date: Mon Apr 5 12:01:24 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <40718061.2040607@zope.com> References: <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> <40718061.2040607@zope.com> Message-ID: <1081180856.4556.33.camel@anthem.wooz.org> On Mon, 2004-04-05 at 11:50, Shane Hathaway wrote: > Mark Russell wrote: > > What about: > > > > [as classmethod] > > def foo(bar, baz): > > pass > > > > To me this is more obvious (and less like p*rl) that *[classmethod], and > > it is also currently a syntax error so won't break existing code. > > I agree, it's nice. You could even use this syntax at the interactive > prompt. The "is" keyword is another choice, too. Since that is illegal syntax today, it would alleviate my primary concern with decorator-before-def syntax. And the 'as' keyword does help tie it to the following def. -Barry From aahz at pythoncraft.com Mon Apr 5 12:03:09 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 5 12:07:40 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <4071059E.2060807@ieee.org> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <20040403145050.GA8626@panix.com> <200404031657.i33Gvxl24044@guido.python.org> <4071059E.2060807@ieee.org> Message-ID: <20040405160309.GA10904@panix.com> On Mon, Apr 05, 2004, Shane Holloway (IEEE) wrote: >>Aahz: >>> >>>Good point. Here's what I *think* the semantics are; as soon as I get >>>agreement, I'll update the PEP. Given a package layout:: >>> >>> package >>> subpackage1 >>> moduleX >>> moduleY >>> subpackage2 >>> moduleZ >>> moduleA >>> >>>Each leading "." refers to one level of parent. Assuming that the >>>current file is ``moduleX.py``, following are correct usages of the new >>>syntax:: >>> >>> from .moduleY import spam >>> from ..subpackage1 import moduleY >>> from ..subpackage2.moduleZ import eggs >>> from ..moduleA import foo >>> from ...package import bar > > What about names inside package __init__? > > from .. import bar > > Is this also valid? Do you mean subpackage1's __init__.py? If so, yes. For most purposes, a package's __init__.py is treated as if it were a module with the name of the package. Hmmmm... I think I see what you're getting at: how do you access subpackage1 from package's __init__.py using relative imports? Seems to me that you'd have to do :: from .package import subpackage1 Normally, though, subpackages import from their parents, not the other way around. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From hernan at orgmf.com.ar Mon Apr 5 12:09:23 2004 From: hernan at orgmf.com.ar (Hernan Foffani) Date: Mon Apr 5 12:10:27 2004 Subject: [Python-Dev] PEP 318: Decorators last before colon In-Reply-To: Message-ID: > ... The * prefix looks so arbitrary: why not /, why not @, etc... It's curious that '@' was brought. I bet it won't be included in the proposal but it should be mentioned that there's a notorious corner use case using a symetrical syntax with a prefix-postfix pair of characters. With proper semantics and honoring lowercase plus underscore name style, it may look like this: o_o = [wow, it, reminds, me, ol, uncle, Pep] class Giuseppe(object): @[o_o]@ def pepper_peps_up_the_pep_thread(self): pass # heh... where's the "pess" reserved werd? (BIG :-D if it wasn't clear enough) Now seriously... Don't know if it was mentioned before or even if it's relevant but I strongly believe that most .NET programmers will understand its meaning (even no-C# ones) and also java ones will (they might not reckon it though ;-) yet still considering the fact that python's decorators will be more powerful. FWIW: For reasons already given in previous messages I *do* prefer decorator-before-def-plus-star syntax (prefix only ;-) Regards, -Hernan do i send this or not? yes, no, yes, no, yes... From gmccaughan at synaptics-uk.com Mon Apr 5 12:10:42 2004 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Mon Apr 5 12:10:48 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <003901c41b1c$3fd9f760$6402a8c0@arkdesktop> References: <003901c41b1c$3fd9f760$6402a8c0@arkdesktop> Message-ID: <200404051710.42704.gmccaughan@synaptics-uk.com> On Monday 2004-04-05 at 15:42, Andrew Koenig wrote: > Here's what I don't understand. > > I imagine that most of the time, when someone decorates a function with lots > of attributes, there is at least the possibility of decorating more than one > function with the same set of attributes. > > In such a case, doesn't it make sense to bind a variable to the attributes > and use it instead? > > Attributes = [staticmethod, classmethod, > otherattributes(debug=True)] > > def foo(bar, baz) Attributes: > pass > > And doesn't this idea answer the objection that too many attributes after > the parameter list make the definition hard to read? It certainly answers that objection, but it has (so it seems to me) other problems of its own; see below. > For that matter, why do we need the brackets if there is only one attribute: > > def foo(bar, baz) staticmethod: > pass > > I am suggesting that what comes between the ) and the : should be an > expression, which must evaluate to either a callable or a sequence of > callables. For that matter, why not allow a tuple expression without > parentheses: > > def foo(bar, baz) staticmethod, classmethod: > pass > > Whatever sequence you put there, I think the semantics are clear: Before > binding a name to the function, pass it to the callable or in turn to each > element of the sequence. The brackets serve multiple purposes. - They make the decoration *look*, even to novice eyes, like an annotation: "by the way, please make this a static method". - They may possibly help the parser a little. (I'm showing my ignorance here; I haven't really looked at the Python parser at all.) - They give a hint to the user that what goes in between them may be a sequence, not just a single item. - They ensure that all decorated definitions have a somewhat consistent appearance. Omitting them breaks all these things, especially if an arbitrary expression is allowed there. And do we really want to allow def foo(bar, baz) quux(wibble, spong): pass That's not a hideous pathological case; it's what a simple decoration using a parameterized decorator will look like without the brackets. With good decorator names I suppose it becomes somewhat comprehensible: def foo(bar, baz) memoized(25): pass but surely it's still much clearer when written as def foo(bar, baz) [memoized(25)]: pass or even (though I'm a little tempted to agree with whoever it was that was wondering whether Guido has been abducted by aliens) [memoized(25)] def foo(bar, baz): pass We can keep the syntactic distinctiveness while still allowing multiple decorators to be combined, by having a function (a builtin, perhaps, but it's not hard to write) that composes decorators: def compose(*decorators): def _(f): for d in decorators: f = d(f) return f our_attributes = compose(staticmethod, classmethod, otherattributes(debug=True)) def foo(bar, baz) [our_attributes]: pass -- Gareth McCaughan From aahz at pythoncraft.com Mon Apr 5 12:11:09 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 5 12:11:33 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: References: <406EED7A.5090908@Acm.Org> <20040403171247.GC8803@panix.com> <200404032125.i33LPU024587@guido.python.org> <20040404021800.GA25210@panix.com> Message-ID: <20040405161109.GB10904@panix.com> On Mon, Apr 05, 2004, Bernhard Herzog wrote: > Aahz writes: >> >> You may use relative imports freely. In Python 2.5, any ``import`` >> statement that results in an intra-package import will generate a >> ``PendingDeprecation`` warning (this also applies to ``from <> import`` >> that fails to use the relative import syntax). In Python 2.6, ``import`` >> will always be an absolute import. > > Why PendingDeprecation? If the import semantics change in incompatible > ways in 2.6, 2.5 should generate a "normal" DeprecationWarning, I think. Guido can overrule this, but my take is that there will be few enough cases where this is a significant issue. People who want to write cross-version libraries and applications shouldn't have to deal with suppressing the DeprecationWarning. If we were serious about issuing a DeprecationWarning, I think that would need to be in Python 2.6, pushing the default of absolute imports to Python 2.7. That seems too long a timeline to me. Other opinions? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From guido at python.org Mon Apr 5 12:43:30 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 5 12:43:38 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Mon, 05 Apr 2004 01:17:17 PDT." <4071160D.5060204@prescod.net> References: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> <4071160D.5060204@prescod.net> Message-ID: <200404051643.i35GhU005203@guido.python.org> > What people find readable depends mostly on what they are used to. The > reason Python is "so readable" on first read is because it rips off so > much syntax directly from C, the language that is the ancestor for all > of the other languages people use today. Bizarre. I think most of the readability actually comes from places where I *didn't* rip off C, in particular Python's syntax for block structure, lists and dicts, and the lack of anything resembling C's abomination passing for type declarations (an mistake admitted by its author). Just goes to show that taste differs. :) > C doesn't have function decorators so people don't have expectations > for them. But they will when they start coming over from C# and Java > 1.5 in two or three years. One virtue of Guido's proposal is that it > is basically what C# does. Java uses a pretty different syntax but > it is also a prefix syntax. If Python uses a postfix syntax it will > probably be alone in making that choice. That's exactly Jim Hugunin's argument for this syntax. > I'm not saying that Python has to do what the other languages do > because they do it, but all else equal, being familiar is better > than being idiosyncratic (i.e. different for no good reason). So I > would rank "like C# and Java" higher than "fits my personal > aesthetics in early 2004" because aesthetics are likely to drift > towards C# and Java over time. Anyway, personal aesthetics are learned behavior, like it or not. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 5 12:44:20 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 5 12:44:28 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Mon, 05 Apr 2004 10:16:51 BST." <1081156611.2377.6.camel@localhost> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> Message-ID: <200404051644.i35GiKD05216@guido.python.org> > What about: > > [as classmethod] > def foo(bar, baz): > pass > > To me this is more obvious (and less like p*rl) that *[classmethod], and > it is also currently a syntax error so won't break existing code. I appreciate the suggestion, but the current parser can't handle that. --Guido van Rossum (home page: http://www.python.org/~guido/) From ark-mlist at att.net Mon Apr 5 12:49:53 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Mon Apr 5 12:49:41 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1081180856.4556.33.camel@anthem.wooz.org> Message-ID: <000101c41b2e$0519f7d0$6402a8c0@arkdesktop> > > > What about: > > > > > > [as classmethod] > > > def foo(bar, baz): > > > pass > Since that is illegal syntax today, it would alleviate my primary > concern with decorator-before-def syntax. And the 'as' keyword does > help tie it to the following def. Indeed. And we could even extend it to allow a single set of decorators to apply to multiple function definitions, as follows: [as classmethod]: def foo(bar, baz): pass def bar(foo, baz): pass Of course, the colon would be required if and only if an indent follows it. (I still prefer decorators after the parameters, but this version pretty much solves the same problems and answers my non-aesthetic objections) From skip at pobox.com Mon Apr 5 12:58:22 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon Apr 5 12:58:43 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404051643.i35GhU005203@guido.python.org> References: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> <4071160D.5060204@prescod.net> <200404051643.i35GhU005203@guido.python.org> Message-ID: <16497.36910.88010.318537@montanaro.dyndns.org> >> One virtue of Guido's proposal is that it is basically what C# >> does. Java uses a pretty different syntax but it is also a prefix >> syntax. If Python uses a postfix syntax it will probably be alone in >> making that choice. Guido> That's exactly Jim Hugunin's argument for this syntax. Since both C# and Java differ significantly in their typing from Python I suspect what works well for those languages (they already have a lot of declarative "baggage" because of their compile-time type checks, so what's a few more declarations?) may not work as well for Python. Second, Python has a strong tradition of borrowing what"works well" from other languages. I'm skeptical that C# has been around long enough to suggest that its syntax "works well". It's pretty clear that Microsoft is going to ram that down most programmers' throats, so the C# user base is no doubt going to be very large in a year or two. If what you're looking for is to provide a familiar syntactic base for C# refugees, then I suppose that's fine, but from the examples I've seen, C# decorations (annotations? attributes?) can be sprinkled quite liberally through the code (and serve as much to obscure as to highlight what's going on). A C# refugee might be disappointed to see that Python's decorators are limited to class, function and method declarations. With that in mind, it doesn't seem to me that partially mimicing C#'s decorator system is necessarily a good thing. Skip From guido at python.org Mon Apr 5 12:59:13 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 5 12:59:22 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: Your message of "Mon, 05 Apr 2004 14:23:16 BST." <20040405132316.GA21940@vicky.ecs.soton.ac.uk> References: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> <20040405132316.GA21940@vicky.ecs.soton.ac.uk> Message-ID: <200404051659.i35GxDN05308@guido.python.org> > Essentially, I wanted to rant about the fact that some performance > considerations are driving the language's evolution, and this is > much harder to revert to if later the performance reasons vanish. I have to admit that sometimes I too yearn for the days of lists and dicts, when heapq was implemented in Python. But iterators are here to stay, and generator expressions are coming, so we'd better get used to them, and know how to deal with them. And yes, we'll see lots of code using genexps "for performance" while lists would perform just as well given the actual data set sizes. But that's okay, since genexps are a pretty nice way of saying things regardless of performance! And they are rarely if ever *slower* than lists, so we shouldn't fear their performance, but rather use whatever gives the most readable solution. Regarding heapq: once Psyco is part of the standard library, it's easy enough to revive heapq.py and friends. In the mean time, I have no problem with providing practical performance for regular Python users. --Guido van Rossum (home page: http://www.python.org/~guido/) From marktrussell at btopenworld.com Mon Apr 5 13:00:34 2004 From: marktrussell at btopenworld.com (Mark Russell) Date: Mon Apr 5 13:00:41 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404051644.i35GiKD05216@guido.python.org> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> <200404051644.i35GiKD05216@guido.python.org> Message-ID: <1081184434.4045.3.camel@localhost> On Mon, 2004-04-05 at 17:44, Guido van Rossum wrote: > I appreciate the suggestion, but the current parser can't handle that. Would it not be possible to cheat and make the lexer transform "[" WHITESPACE "as" WHITESPACE IDENTIFIER into an LBRACE-AS token (which I assume would make the parser's job simple). Mark From shane.holloway at ieee.org Mon Apr 5 13:19:48 2004 From: shane.holloway at ieee.org (Shane Holloway (IEEE)) Date: Mon Apr 5 13:20:15 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040405160309.GA10904@panix.com> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <20040403145050.GA8626@panix.com> <200404031657.i33Gvxl24044@guido.python.org> <4071059E.2060807@ieee.org> <20040405160309.GA10904@panix.com> Message-ID: <40719534.2040301@ieee.org> Aahz wrote: > On Mon, Apr 05, 2004, Shane Holloway (IEEE) wrote: > >>>Aahz: >>> >>>>Good point. Here's what I *think* the semantics are; as soon as I get >>>>agreement, I'll update the PEP. Given a package layout:: >>>> >>>> package >>>> subpackage1 >>>> moduleX >>>> moduleY >>>> subpackage2 >>>> moduleZ >>>> moduleA >>>> >>>>Each leading "." refers to one level of parent. Assuming that the >>>>current file is ``moduleX.py``, following are correct usages of the new >>>>syntax:: >>>> >>>> from .moduleY import spam >>>> from ..subpackage1 import moduleY >>>> from ..subpackage2.moduleZ import eggs >>>> from ..moduleA import foo >>>> from ...package import bar >> >>What about names inside package __init__? >> >> from .. import bar >> >>Is this also valid? > > > Do you mean subpackage1's __init__.py? If so, yes. For most purposes, > a package's __init__.py is treated as if it were a module with the name > of the package. Actually, no I meant package's __init__.py -- so that:: from .. import bar as barA from ...package import bar as barB assert barA is barB > Hmmmm... I think I see what you're getting at: how do you access > subpackage1 from package's __init__.py using relative imports? Seems to > me that you'd have to do :: > > from .package import subpackage1 > > Normally, though, subpackages import from their parents, not the other > way around. I would intuit that one would import subpackage1 from package as:: from . import subpackage1 but that's not explicitly spelled out. Is this correct? From aahz at pythoncraft.com Mon Apr 5 13:21:03 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 5 13:21:34 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: References: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040405172103.GB11824@panix.com> On Mon, Apr 05, 2004, Ka-Ping Yee wrote: > On Mon, 5 Apr 2004, Greg Ewing wrote: >> >> the little voice in my head doesn't really say anything >> coherent at all. > > Funny. I experience that most days. "I do what the voices in my head tell me to." -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From aahz at pythoncraft.com Mon Apr 5 13:27:19 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 5 13:27:32 2004 Subject: [Python-Dev] PEP 318: Python's readability In-Reply-To: <4071160D.5060204@prescod.net> References: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> <4071160D.5060204@prescod.net> Message-ID: <20040405172719.GC11824@panix.com> On Mon, Apr 05, 2004, Paul Prescod wrote: > > What people find readable depends mostly on what they are used to. The > reason Python is "so readable" on first read is because it rips off so > much syntax directly from C, the language that is the ancestor for all > of the other languages people use today. Speaking as someone whose first languages were BASIC and Pascal, both before C became so hugely popular, my feelings about Python's readability were more along the lines of it being different from C. Yes, I knew enough C to recognize where it was being ripped off -- but that's a far cry from the elegance of the language as a whole. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From aahz at pythoncraft.com Mon Apr 5 13:36:24 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 5 13:36:30 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <40719534.2040301@ieee.org> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <20040403145050.GA8626@panix.com> <200404031657.i33Gvxl24044@guido.python.org> <4071059E.2060807@ieee.org> <20040405160309.GA10904@panix.com> <40719534.2040301@ieee.org> Message-ID: <20040405173624.GD11824@panix.com> On Mon, Apr 05, 2004, Shane Holloway (IEEE) wrote: > Aahz wrote: >>On Mon, Apr 05, 2004, Shane Holloway (IEEE) wrote: >>>>Aahz: >>>> >>>>>Good point. Here's what I *think* the semantics are; as soon as I get >>>>>agreement, I'll update the PEP. Given a package layout:: >>>>> >>>>> package >>>>> subpackage1 >>>>> moduleX >>>>> moduleY >>>>> subpackage2 >>>>> moduleZ >>>>> moduleA >>>>> >>>>>Each leading "." refers to one level of parent. Assuming that the >>>>>current file is ``moduleX.py``, following are correct usages of the new >>>>>syntax:: >>>>> >>>>> from .moduleY import spam >>>>> from ..subpackage1 import moduleY >>>>> from ..subpackage2.moduleZ import eggs >>>>> from ..moduleA import foo >>>>> from ...package import bar >>> >>>What about names inside package __init__? >>> >>> from .. import bar >>> >>>Is this also valid? >> >> >>Do you mean subpackage1's __init__.py? If so, yes. For most purposes, >>a package's __init__.py is treated as if it were a module with the name >>of the package. > > Actually, no I meant package's __init__.py -- so that:: > > from .. import bar as barA > from ...package import bar as barB > > assert barA is barB That makes no sense at all; both of those imports are wrong from package's __init__.py. >>Hmmmm... I think I see what you're getting at: how do you access >>subpackage1 from package's __init__.py using relative imports? Seems to >>me that you'd have to do :: >> >> from .package import subpackage1 >> >>Normally, though, subpackages import from their parents, not the other >>way around. > > I would intuit that one would import subpackage1 from package as:: > > from . import subpackage1 > > but that's not explicitly spelled out. Is this correct? Your intuition is wrong. From package's __init__.py, ``from .`` refers to package's parent; to get back into the package, you need to use ``from .package``. It's precisely the same as the example above with moduleX getting ``bar`` -- but because moduleX is two levels down, it needs two more dots. You're getting confused because you think of __init__.py as being a level below the package that contains it; however, __init__.py *IS* the package level. It's the code *for* the package. If anyone else has been confused by this, please let me know and I'll add clarification to the PEP, but I think these consequences are pretty clear if you understand how packages work. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From nas-python at python.ca Mon Apr 5 13:51:17 2004 From: nas-python at python.ca (Neil Schemenauer) Date: Mon Apr 5 13:51:30 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040405173624.GD11824@panix.com> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <20040403145050.GA8626@panix.com> <200404031657.i33Gvxl24044@guido.python.org> <4071059E.2060807@ieee.org> <20040405160309.GA10904@panix.com> <40719534.2040301@ieee.org> <20040405173624.GD11824@panix.com> Message-ID: <20040405175117.GA21587@mems-exchange.org> On Mon, Apr 05, 2004 at 01:36:24PM -0400, Aahz wrote: > Your intuition is wrong. From package's __init__.py, ``from .`` refers > to package's parent; to get back into the package, you need to use > ``from .package``. That's confusing. I expected '.' to refer to the current package, just as '.' refers to the current directory in most file systems. I think that meaning would be more useful in practice as well. Imagine a package where '.' means the current package: # __init__.py ################################### from .a import A from .b import B ################################################# # a.py ########################################## from .utils import x, y class A: ... ################################################# # b.py ########################################## from .utils import x, z class B: ... ################################################# # utils.py ###################################### def x(): ... def y(): ... def z(): ... ################################################# Notice that the modules that make up the package do not need to know what the package is called. That was one of the major benefits of relative imports that I was looking forward to. Perhaps I misunderstand the PEP. Neil From guido at python.org Mon Apr 5 13:58:21 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 5 13:58:29 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Mon, 05 Apr 2004 12:11:09 EDT." <20040405161109.GB10904@panix.com> References: <406EED7A.5090908@Acm.Org> <20040403171247.GC8803@panix.com> <200404032125.i33LPU024587@guido.python.org> <20040404021800.GA25210@panix.com> <20040405161109.GB10904@panix.com> Message-ID: <200404051758.i35HwLK05504@guido.python.org> > > Why PendingDeprecation? If the import semantics change in incompatible > > ways in 2.6, 2.5 should generate a "normal" DeprecationWarning, I think. > > Guido can overrule this, but my take is that there will be few enough > cases where this is a significant issue. People who want to write > cross-version libraries and applications shouldn't have to deal with > suppressing the DeprecationWarning. If we were serious about issuing a > DeprecationWarning, I think that would need to be in Python 2.6, pushing > the default of absolute imports to Python 2.7. That seems too long a > timeline to me. I think we ought to start it as a normal Deprecation. If there are enough people who get bit by it during alpha testing we can switch it to PendingDeprecation, but IMO PendingDeprecation is not very useful (since it is ignored by default). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 5 14:03:09 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 5 14:03:14 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Mon, 05 Apr 2004 18:00:34 BST." <1081184434.4045.3.camel@localhost> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> <200404051644.i35GiKD05216@guido.python.org> <1081184434.4045.3.camel@localhost> Message-ID: <200404051803.i35I39U05548@guido.python.org> > > I appreciate the suggestion, but the current parser can't handle that. > > Would it not be possible to cheat and make the lexer transform > > "[" WHITESPACE "as" WHITESPACE IDENTIFIER > > into an LBRACE-AS token (which I assume would make the parser's job > simple). I don't think without extreme hackery, but feel free to prove me wrong by producing a patch. (A problem is, there could be newlines and comments inside the WHITESPACE. Backtracking over that would require the lexer to have an arbitrary-length buffer.) --Guido van Rossum (home page: http://www.python.org/~guido/) From pinard at iro.umontreal.ca Mon Apr 5 14:06:41 2004 From: pinard at iro.umontreal.ca (=?iso-8859-1?Q?Fran=E7ois?= Pinard) Date: Mon Apr 5 14:07:51 2004 Subject: [Python-Dev] PEP 328: __path__ In-Reply-To: <200404050516.i355GQes002218@cosc353.cosc.canterbury.ac.nz> References: <20040404162100.GA6059@panix.com> <200404050516.i355GQes002218@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040405180641.GA5876@alcyon.progiciels-bpi.ca> [Greg Ewing] > Aahz : > > Between the addition of ``as`` in Python 2.0 and the new syntax & > > semantics for ``import``, there is no longer a need for ``__path__``, and > > it is now officially deprecated. > NO! WAIT! > I have a project in which I'm modifying a module's __path__, > and it's *not* to work around any deficiencies in the relative > import mechanism. Rather it's to make a set of directories > behave like a single package for import purposes, with the > members of the set determined at run time. Same here. -- Fran?ois Pinard http://www.iro.umontreal.ca/~pinard From tim.one at comcast.net Mon Apr 5 14:45:51 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon Apr 5 14:46:04 2004 Subject: [Python-Dev] genexps slow? In-Reply-To: <2mptamg08o.fsf@starship.python.net> Message-ID: [Tim Peters] >> There's a twist there: the functions synthesized for generator >> expressions get to take the quicker-than-general "fast_yield" exit >> path. Hmm! It looks like tests against WHY_YIELD before the >> fast_yield label can no longer succeed, but are still there to slow >> down non-generator cases. [Michael Hudson] > Hmm. I *think* I agree with you. I also think the tests against > WHY_YIELD in the code for END_FINALLY are unnecessary, I meant that one too: all tests against WHY_YIELD physically preceding the fast_yield label appear unnecessary now. The only obvious assignment of WHY_YIELD to "why" appears in YIELD_VALUE now, and that jumps immediately to fast_yield, from which there's no going backwards. do_raise() can't return WHY_YIELD either. > as we (syntactically) don't allow yields when there are finally: > statements in play. Even if we did, a yield doesn't collapse any pending block structure. > If I'm wrong about that, then you're wrong too :-) It's all > a bit subtle, to be sure. Na . I'll toss in some asserts and check it in if nothing blows up. > Once again, I wonder if there was someway we could have two > eval_frames: one with tracing support, one without. Unfortunately, I > don't think this can fly (how do you *start* tracing?). Maybe we > could have a '-notrace' command line flag? But I suspect that this is > a silly over-complication. Me too. From fdrake at acm.org Mon Apr 5 14:59:09 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Apr 5 14:59:22 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404051803.i35I39U05548@guido.python.org> References: <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> Message-ID: <200404051459.10263.fdrake@acm.org> Mark Russell suggested: > Would it not be possible to cheat and make the lexer transform > > "[" WHITESPACE "as" WHITESPACE IDENTIFIER > > into an LBRACE-AS token (which I assume would make the parser's job > simple). Guido said: > I don't think without extreme hackery, but feel free to prove me wrong > by producing a patch. (A problem is, there could be newlines and > comments inside the WHITESPACE. Backtracking over that would require > the lexer to have an arbitrary-length buffer.) Another possibility might be to require the construct start without the whitespace: "[as" WHITESPACE ... "]" That avoids the buffering issue and matches what most programmer are more likely to write anyway. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From aahz at pythoncraft.com Mon Apr 5 15:01:51 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 5 15:02:03 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040405175117.GA21587@mems-exchange.org> References: <200403302218.i2UMIEV10058@guido.python.org> <406EB373.2030401@python.org> <20040403145050.GA8626@panix.com> <200404031657.i33Gvxl24044@guido.python.org> <4071059E.2060807@ieee.org> <20040405160309.GA10904@panix.com> <40719534.2040301@ieee.org> <20040405173624.GD11824@panix.com> <20040405175117.GA21587@mems-exchange.org> Message-ID: <20040405190151.GA16139@panix.com> On Mon, Apr 05, 2004, Neil Schemenauer wrote: > On Mon, Apr 05, 2004 at 01:36:24PM -0400, Aahz wrote: >> >> Your intuition is wrong. From package's __init__.py, ``from .`` refers >> to package's parent; to get back into the package, you need to use >> ``from .package``. > > That's confusing. I expected '.' to refer to the current package, > just as '.' refers to the current directory in most file systems. I > think that meaning would be more useful in practice as well. > Imagine a package where '.' means the current package: > > # __init__.py ################################### > from .a import A > from .b import B > ################################################# > > # a.py ########################################## > from .utils import x, y > class A: > ... > ################################################# > > # b.py ########################################## > from .utils import x, z > class B: > ... > ################################################# > > # utils.py ###################################### > def x(): > ... > def y(): > ... > def z(): > ... > ################################################# > > Notice that the modules that make up the package do not need to know > what the package is called. That was one of the major benefits of > relative imports that I was looking forward to. > > Perhaps I misunderstand the PEP. Not sure. The only difference between your semantics and mine lies in __init__.py; there -- and only there -- is where the need exists to know the package name. You're making the mistake of thinking that __init__.py exists in the same namespace as a.py, b.py, and utils.py -- it doesn't. Now, we could in theory use your semantics due to "practicality beats purity", but it seems to me that it would lead to more mistakes in understanding how packages work. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From ishnigarrab at earthlink.net Mon Apr 5 15:37:32 2004 From: ishnigarrab at earthlink.net (Isaac) Date: Mon Apr 5 15:38:09 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 9, Issue 17 In-Reply-To: References: Message-ID: <4071B57C.60803@earthlink.net> >From: Guido van Rossum > [...] >I also note that accepting decorator-before-colon now would make it >harder to come up with a decent syntax for declaring the return type, >which I still want to do in some future version of Python with >optional (!) static typing. But not impossible -- there's enough >punctuation available besides '[' and ':'. > [...] >--Guido van Rossum (home page: http://www.python.org/~guido/) > One thing I was thinking about (aloud on #python) was a validator syntax, something resembling: def is_int(i): return isinstance(number, int): def is_hashable(h): try: hash(h) return True except TypeError: return False # or raise exception def func(is_int : number, hashable : key): # code... which translates to: def func(number, key): if isinstance(number, int): raise ValueError, "first argument must be int" try: hash(key) except TypeError: raise ValueError, "second argument must be hashable" # code... I don't know if it would be best to go the boolean or the exception route. Either way, you get the point. This way python retains it's dynamicy but allows for type checking as well as other form of validation (i.e. range checking, etc). Of course, the validator and the colon are optional 'def' NAME '(' [validator ':'] v_name ',' ... ')' ':' suite or some crap like that, I'm not a language lawyer, but you get the idea. And for people concerned with wasting the time with 10 function calls per function call, there could be built-in validators (like is_int, or whatever you want to call it) which of course are written an C, and maybe some shortcutting could be done in the interpreter (or in psyco, maybe) which would allow is_int to be called with out all the pythonic function calling crap (I'm not sure exactly how this all works...). Something like that. Also as an extension of that, there can a def ret_type : func(arg, arg, ...): pass kinda thing... although the colon might confuse things, there idea is there, just needs to be hammered out. Anyways, just some random thoughts. I'd be happy to work with some people to make a PEP out of this, if it gains any traction, just e-mail me. Isaac From marktrussell at btopenworld.com Mon Apr 5 16:08:19 2004 From: marktrussell at btopenworld.com (Mark Russell) Date: Mon Apr 5 16:08:24 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404051459.10263.fdrake@acm.org> References: <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <200404051459.10263.fdrake@acm.org> Message-ID: <1081195698.2377.9.camel@localhost> On Mon, 2004-04-05 at 19:59, Fred L. Drake, Jr. wrote: > Another possibility might be to require the construct start without the > whitespace: > > "[as" WHITESPACE ... "]" > > That avoids the buffering issue and matches what most programmer are more > likely to write anyway. I think that's probably too arbitrary a restriction. Another thought: presumably the parser could handle this syntax if "as" were a keyword. Is that true, and if so, is making "as" a keyword out of the question? The change could presumably be enabled by a "from __future__" directive to avoid breaking existing code. Mark From nhodgson at bigpond.net.au Mon Apr 5 17:31:14 2004 From: nhodgson at bigpond.net.au (Neil Hodgson) Date: Mon Apr 5 17:30:19 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> <4071160D.5060204@prescod.net> <200404051643.i35GhU005203@guido.python.org> <16497.36910.88010.318537@montanaro.dyndns.org> Message-ID: <00fc01c41b55$52ed9ad0$84bb8490@neil> Skip: > Second, Python has a strong tradition of borrowing what"works well" from > other languages. I'm skeptical that C# has been around long enough to > suggest that its syntax "works well". C# inherited this syntax from Microsoft IDL and its predecessor ODL. Unsure of the age of MS IDL but a 1995 edition of "Inside OLE" uses it. I can't recall any problems with the syntax although fewer people would have worked with it when it was only for defining interfaces rather than as part of a general purpose language. Neil From guido at python.org Mon Apr 5 18:09:48 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 5 18:09:57 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: Your message of "Mon, 05 Apr 2004 21:08:19 BST." <1081195698.2377.9.camel@localhost> References: <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <200404051459.10263.fdrake@acm.org> <1081195698.2377.9.camel@localhost> Message-ID: <200404052209.i35M9m606023@guido.python.org> > presumably the parser could handle this syntax if "as" were a keyword. Not really. The parser is really dumb, it han only handle one alternative starting with a particular token at any point. Since '[' can already start an expression and an expression is a valid statement, anything else also starting with '[', even if it isn't a valid expression, cannot be accepted. You may be thinking of Yacc, which has much more liberal rules. --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Mon Apr 5 18:27:40 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Apr 5 18:31:50 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 9, Issue 17 In-Reply-To: <4071B57C.60803@earthlink.net> References: <4071B57C.60803@earthlink.net> Message-ID: <20040405152615.EC80.JCARLSON@uci.edu> > One thing I was thinking about (aloud on #python) was a validator > syntax, something resembling: > > def is_int(i): > return isinstance(number, int): > > def is_hashable(h): > try: > hash(h) > return True > except TypeError: > return False # or raise exception > > def func(is_int : number, hashable : key): > # code... > > which translates to: > > def func(number, key): > if isinstance(number, int): > raise ValueError, "first argument must be int" > > try: > hash(key) > except TypeError: > raise ValueError, "second argument must be hashable" > > # code... > > > I don't know if it would be best to go the boolean or the exception > route. Either way, you get the point. This way python retains it's > dynamicy but allows for type checking as well as other form of > validation (i.e. range checking, etc). Of course, the validator and the > colon are optional > > 'def' NAME '(' [validator ':'] v_name ',' ... ')' ':' suite > > or some crap like that, I'm not a language lawyer, but you get the idea. > And for people concerned with wasting the time with 10 function calls > per function call, there could be built-in validators (like is_int, or > whatever you want to call it) which of course are written an C, and > maybe some shortcutting could be done in the interpreter (or in psyco, > maybe) which would allow is_int to be called with out all the pythonic > function calling crap (I'm not sure exactly how this all works...). > Something like that. > > Also as an extension of that, there can a > > def ret_type : func(arg, arg, ...): pass > > kinda thing... although the colon might confuse things, there idea is > there, just needs to be hammered out. > > Anyways, just some random thoughts. I'd be happy to work with some > people to make a PEP out of this, if it gains any traction, just e-mail me. Isaac, Everything you want with "validators" is going to be available with decorators from PEP 318. - Josiah From djc at object-craft.com.au Mon Apr 5 19:06:16 2004 From: djc at object-craft.com.au (Dave Cole) Date: Mon Apr 5 19:06:23 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <16497.28394.944929.536518@magrathea.basistech.com> References: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> <16497.28394.944929.536518@magrathea.basistech.com> Message-ID: <1081206375.2666.20.camel@echidna.object-craft.com.au> On Tue, 2004-04-06 at 00:36, Tom Emerson wrote: > Moore, Paul writes: > > Hmm. I'm torn. On the one hand, it's really cool that Python+Psyco can > > out-perform C (another argument against the knee-jerk "recode bits in > > C" reaction). But my gut feel is that not enough people use Psyco to > > make this a good answer yet. On the third hand, if we recode chunks of > > the stdlib in C, do we kill too many chances for Psyco to work its > > magic? > > If I understand correctly, the issue is that when stdlib functions are > rewritten in C, they end up being faster but are still slower than > Psyco optimized code because they are still polymorphic while the > Psyco'ized generates multiple paths. > > If that is the case then one could certainly hand-hack the C > implementations to expand common code paths in an optimal way. I have not looked at Psyco so could be operating from a misunderstanding of what it does... Would it make sense to build specialisation into the base Python interpreter so that it builds specialised versions of bytecode for functions. I am assuming that there are only a few types that can be targeted by specialisation (int, float). Bytecodes then could be added to the interpreter which operate on those native machine types rather than the Python objects. This has the advantage of allowing specialisation on all currently supported platforms and does not preclude the further optimisation of translation to machine code. If specialisation was in the base interpreter then I suppose it should possible for the compiler to know when a native machine type can be compiled directly. For example: def foo(): i = 0 while i < 100: bar() i += 1 While noone writes loops like that, it does illustrate the point. It is clear that the local i is always an integer and is not used outside the function. - Dave -- http://www.object-craft.com.au From scav at blueyonder.co.uk Mon Apr 5 19:29:10 2004 From: scav at blueyonder.co.uk (Peter Harris) Date: Mon Apr 5 19:22:02 2004 Subject: [Python-Dev] Re: PEP309 re-written In-Reply-To: References: Message-ID: <4071EBC6.4010103@blueyonder.co.uk> David Abrams wrote: >> >> >C = lambda *args, **kw: \ > Canvas(my_window, width=100, height=100, bg='white', *args, **kw) > > > C(bg='red') Traceback (most recent call last): File "", line 1, in -toplevel- C(bg='red') File "", line 1, in C = lambda *args, **kw: \ TypeError: Canvas constructor got multiple values for keyword argument 'bg' >OK, partial is a little shorter in that case. I'm not sure that >example justifies a PEP, though. > > > PEPs can be for the little things too. > def _merge(d1, d2): > d1.update(d2) > return d1 > > def partial(f, *args, **kw): > return lambda *args2, **kw2: f(args + args2, merge(kw2,kw)) > >?? > > def partial(f, *args, **kw): def _merge(d1, d2): d1.update(d2) return d1 return lambda *args2, **kw2: f(*(args + args2), **_merge(kw2,kw)) A good implementation indeed. I may be inclined to provide both a class called Partial (for people who want to mess with the internals, or for some other reason want a class) and a function called partial. Would that be OK, everyone? >>You could maybe sub-class partial to override __call__ , in case you >>want to do anything fancy like pre-supplying arguments at arbitrary >>positions. >> >> > >A *perfect* example of what lambda is already good at. > > Yup, no argument from me there. I can't foresee what others may one day want to do, however. Peter Harris From eppstein at ics.uci.edu Mon Apr 5 19:24:14 2004 From: eppstein at ics.uci.edu (David Eppstein) Date: Mon Apr 5 19:24:25 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <200404051459.10263.fdrake@acm.org> <1081195698.2377.9.camel@localhost> <200404052209.i35M9m606023@guido.python.org> Message-ID: In article <200404052209.i35M9m606023@guido.python.org>, Guido van Rossum wrote: > > presumably the parser could handle this syntax if "as" were a keyword. > > Not really. The parser is really dumb, it han only handle one > alternative starting with a particular token at any point. Since '[' > can already start an expression and an expression is a valid > statement, anything else also starting with '[', even if it isn't a > valid expression, cannot be accepted. You may be thinking of Yacc, > which has much more liberal rules. > > --Guido van Rossum (home page: http://www.python.org/~guido/) Ok, then how about def ... ? '<' can't start an expression or statement currently, can it? -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From guido at python.org Mon Apr 5 19:30:23 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 5 19:30:30 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 9, Issue 17 In-Reply-To: Your message of "Mon, 05 Apr 2004 15:37:32 EDT." <4071B57C.60803@earthlink.net> References: <4071B57C.60803@earthlink.net> Message-ID: <200404052330.i35NUND06228@guido.python.org> > One thing I was thinking about (aloud on #python) was a validator > syntax, something resembling: > > def is_int(i): > return isinstance(number, int): > > def is_hashable(h): > try: > hash(h) > return True > except TypeError: > return False # or raise exception > > def func(is_int : number, hashable : key): > # code... Please note that you seem to be using the syntax ``type: variable''. We've settled on using ``variable: type'' instead. (How to express types is quite a different story...) --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Mon Apr 5 20:28:38 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 20:28:44 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <4071160D.5060204@prescod.net> Message-ID: <200404060028.i360ScMU003594@cosc353.cosc.canterbury.ac.nz> Paul Prescod : > I'm not saying that Python has to do what the other languages do because > they do it, but all else equal, But all else isn't equal, at least not for me, because I like one better than the other. If I didn't care one way or the other, I might look to another language for a tiebreaker. But I don't need to, because I already know which one I prefer. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Mon Apr 5 20:32:06 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 20:32:17 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1081162471.3004.4.camel@localhost> Message-ID: <200404060032.i360W6ao003613@cosc353.cosc.canterbury.ac.nz> On Mon, 2004-04-05 at 11:38, David Abrahams wrote: > Not bad. Also tolerable: > > as [classmethod] > def foo(bar, baz): > pass I thought we decided long ago that we can't use a keyword because no single keyword could sound right in the context of all possible use cases. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Mon Apr 5 20:38:00 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 20:38:13 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> Message-ID: <200404060038.i360c0Qo003623@cosc353.cosc.canterbury.ac.nz> "Moore, Paul" : > Hmm. I'm torn. On the one hand, it's really cool that Python+Psyco can > out-perform C (another argument against the knee-jerk "recode bits in > C" reaction). But my gut feel is that not enough people use Psyco to > make this a good answer yet. If and when Psyco becomes an integral part of the core Python distribution and works on all supported platforms, then it may make sense to keep more stuff in Python as a matter of course. Until then, keeping pure-Python versions of some modules around as an alternative for Psyco users might be desirable. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From aahz at pythoncraft.com Mon Apr 5 20:42:45 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 5 20:42:52 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <200404060038.i360c0Qo003623@cosc353.cosc.canterbury.ac.nz> References: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> <200404060038.i360c0Qo003623@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040406004244.GA7899@panix.com> On Tue, Apr 06, 2004, Greg Ewing wrote: > > If and when Psyco becomes an integral part of the core Python > distribution and works on all supported platforms, then it may make > sense to keep more stuff in Python as a matter of course. > > Until then, keeping pure-Python versions of some modules around as an > alternative for Psyco users might be desirable. No, that's not the way to look at it, IMO. The point is to keep Python versions around for everyone who isn't a C programmer. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From bob at redivi.com Mon Apr 5 20:50:28 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon Apr 5 20:46:22 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <200404060038.i360c0Qo003623@cosc353.cosc.canterbury.ac.nz> References: <200404060038.i360c0Qo003623@cosc353.cosc.canterbury.ac.nz> Message-ID: <66133DA1-8764-11D8-B3FA-000A95686CD8@redivi.com> On Apr 5, 2004, at 8:38 PM, Greg Ewing wrote: > "Moore, Paul" : > >> Hmm. I'm torn. On the one hand, it's really cool that Python+Psyco can >> out-perform C (another argument against the knee-jerk "recode bits in >> C" reaction). But my gut feel is that not enough people use Psyco to >> make this a good answer yet. > > If and when Psyco becomes an integral part of the core > Python distribution and works on all supported platforms, > then it may make sense to keep more stuff in Python as > a matter of course. > > Until then, keeping pure-Python versions of some modules > around as an alternative for Psyco users might be > desirable. If nothing else pure python versions of modules: (a) serve as great documentation (b) are good for psyco (c) are good for PyPy (d) are good for Pyrex (e) are good for Starkiller (f) are good for Jython (g) are good for IronPython .... I would recommend that all standard library C extensions that can have python implementations should have them somewhere that is easy to find, even if it's in a CVS sandbox somewhere. -bob From greg at cosc.canterbury.ac.nz Mon Apr 5 21:14:27 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 21:14:37 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <003901c41b1c$3fd9f760$6402a8c0@arkdesktop> Message-ID: <200404060114.i361ERiT003676@cosc353.cosc.canterbury.ac.nz> Andrew Koenig : > For that matter, why do we need the brackets if there is only one attribute: > > def foo(bar, baz) staticmethod: > pass Because the decorator doesn't stand out enough that way. It's a matter of opinion, but I think it looks better with the brackets, even if there is only one decorator. The brackets mean "here is something additional", not "here is a list". Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From ishnigarrab at earthlink.net Mon Apr 5 21:30:32 2004 From: ishnigarrab at earthlink.net (Isaac) Date: Mon Apr 5 21:31:08 2004 Subject: [Python-Dev] Validators (was Re: Python-Dev Digest, Vol 9, Issue 17) Message-ID: <40720838.5050905@earthlink.net> Guido: >Please note that you seem to be using the syntax ``type: variable''. >We've settled on using ``variable: type'' instead. (How to express >types is quite a different story...) > I haven't been following the thread on the subject much at all, I've just heard some rumbling about introducing static typing into python, and frankly I'm a bit wary about the whole thing. I was just thinking that in keeping with python's dynamic nature that something more like a generalized way of validating the _nature_ of something as opposed to it's underlying *type* would be more appropriate. It's just that it rings more pythonic to me (not that I have in any way graduated to the rank of Pythonist to make such judgments). As for the order of things, it's really not that important, "name: validator" would be just as useful. The only problem would be that def func(name: int): pass works in terms of type checking, as in def func(name): if not isinstance(name, int): raise TypeError where as translating it as a validator: def func(name): int(name) doesn't make as much sense, int would have to be overloaded or something like is_int would have to be implemented. But the advantages of validators would be numerous, IMO: * custom validators (equivalent to custom 'types' like in_range, or iterable) * still allows for the same functionality as typing systems while retaining dynamic capabilities. * possibly inline conversions (interface mutators, adaptors, I wonder if twisted might like something like this). i.e.: def add(a: int, b: int): return a+b add('45', 5) => 50 if it were established that validators raise exceptions if the input is invalid, and return the value (possibly modified) if valid. Similar to implicit upcasting, i.e.: class MyInt(int): def __init__(self, i): self.i = i def add(self, other): self.i += other.i return self def add(a: MyInt, b: MyInt): return a.add(b) add(4, 4) => MyInt(8) Again, I'm mostly just thinking aloud, hoping to be productive. Isaac From s.percivall at chello.se Mon Apr 5 22:00:50 2004 From: s.percivall at chello.se (Simon Percivall) Date: Mon Apr 5 22:00:55 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: References: <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <200404051459.10263.fdrake@acm.org> <1081195698.2377.9.camel@localhost> <200404052209.i35M9m606023@guido.python.org> Message-ID: <3AA96ED4-876E-11D8-A42F-0003934AD54A@chello.se> On 2004-04-06, at 01.24, David Eppstein wrote: > > Ok, then how about > > def ... > ? > > '<' can't start an expression or statement currently, can it? Yeah. I think that would look better. On the other hand ... On 2004-03-31, at 17.42, Guido van Rossum wrote: > Why does <...> look better than [...]? To me, <...> just reminds me > of XML, which is totally the wrong association. > > There are several parsing problems with <...>: the lexer doesn't see < > and > as matched brackets, so you won't be able to break lines without > using a backslash, and the closing > is ambiguous -- it might be a > comparison operator. From eppstein at ics.uci.edu Mon Apr 5 22:06:58 2004 From: eppstein at ics.uci.edu (David Eppstein) Date: Mon Apr 5 22:07:04 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <200404051459.10263.fdrake@acm.org> <1081195698.2377.9.camel@localhost> <200404052209.i35M9m606023@guido.python.org> <3AA96ED4-876E-11D8-A42F-0003934AD54A@chello.se> Message-ID: In article <3AA96ED4-876E-11D8-A42F-0003934AD54A@chello.se>, Simon Percivall wrote: > On 2004-04-06, at 01.24, David Eppstein wrote: > > > > Ok, then how about > > > > def ... > > ? > > > > '<' can't start an expression or statement currently, can it? > > Yeah. I think that would look better. On the other hand ... > > On 2004-03-31, at 17.42, Guido van Rossum wrote: > > Why does <...> look better than [...]? To me, <...> just reminds me > > of XML, which is totally the wrong association. > > > > There are several parsing problems with <...>: the lexer doesn't see < > > and > as matched brackets, so you won't be able to break lines without > > using a backslash, and the closing > is ambiguous -- it might be a > > comparison operator. Well, Mike Pall's long message "The Need for a Declarative Syntax" convinced me that some kind of bracketed prefix is the best location for these things, but I still don't like [...] because it already has a (useless) meaning, so declarators would pass silently in old Pythons. And the same objection applies to all other existing brackets. So, angle brackets seem like the cleanest remaining choice. Re the lexer, I don't see this as a big problem as long as multiple declarators can be handled by multiple <...> pairs. Re the ambiguity, I think comparisons shouldn't normally appear in declarators, so it's ok if they have to be parenthesized. -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From jcarlson at uci.edu Mon Apr 5 22:06:47 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon Apr 5 22:12:04 2004 Subject: [Python-Dev] Validators (was Re: Python-Dev Digest, Vol 9, Issue 17) In-Reply-To: <40720838.5050905@earthlink.net> References: <40720838.5050905@earthlink.net> Message-ID: <20040405184719.EC83.JCARLSON@uci.edu> Isaac, > >Please note that you seem to be using the syntax ``type: variable''. > >We've settled on using ``variable: type'' instead. (How to express > >types is quite a different story...) > > > > I haven't been following the thread on the subject much at all, I've > just heard some rumbling about introducing static typing into python, Python is not going to have static typing as a part of the base language, so worry not. > and frankly I'm a bit wary about the whole thing. I was just thinking > that in keeping with python's dynamic nature that something more like a > generalized way of validating the _nature_ of something as opposed to > it's underlying *type* would be more appropriate. It's just that it How do you propose to get at the "nature" of what an object is? Currently in Python, we really only have types to determine what an object is. With that, we /can/ do per-call checks on the types of the input, heck, if we're smart, we can even do polymorphism. Such a solution is doable in current Python with various approaches. Making a decorator that would handle type checking and polymorphism has been provided here already, or we can go a class-based route to do the same thing, though not nearly as neat. Still, what do you mean by the "nature" of what something is? - Josiah From greg at cosc.canterbury.ac.nz Mon Apr 5 22:25:18 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 22:25:43 2004 Subject: [Python-Dev] Re: Python-Dev Digest, Vol 9, Issue 17 In-Reply-To: <4071B57C.60803@earthlink.net> Message-ID: <200404060225.i362PIju003785@cosc353.cosc.canterbury.ac.nz> Isaac : > 'def' NAME '(' [validator ':'] v_name ',' ... ')' ':' suite > > def ret_type : func(arg, arg, ...): pass That would be repeating C's mistake of putting the type info in front of the name being declared. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From ishnigarrab at earthlink.net Mon Apr 5 22:29:58 2004 From: ishnigarrab at earthlink.net (Isaac) Date: Mon Apr 5 22:30:37 2004 Subject: [Python-Dev] Validators (was Re: Python-Dev Digest, Vol 9, Issue 17) In-Reply-To: <20040405184719.EC83.JCARLSON@uci.edu> References: <40720838.5050905@earthlink.net> <20040405184719.EC83.JCARLSON@uci.edu> Message-ID: <40721626.8090906@earthlink.net> Josiah Carlson wrote: >Isaac, > > > >>>Please note that you seem to be using the syntax ``type: variable''. >>>We've settled on using ``variable: type'' instead. (How to express >>>types is quite a different story...) >>> >>> >>> >>I haven't been following the thread on the subject much at all, I've >>just heard some rumbling about introducing static typing into python, >> >> > >Python is not going to have static typing as a part of the base language, >so worry not. > > > >>and frankly I'm a bit wary about the whole thing. I was just thinking >>that in keeping with python's dynamic nature that something more like a >>generalized way of validating the _nature_ of something as opposed to >>it's underlying *type* would be more appropriate. It's just that it >> >> > >How do you propose to get at the "nature" of what an object is? >Currently in Python, we really only have types to determine what an >object is. With that, we /can/ do per-call checks on the types of the >input, heck, if we're smart, we can even do polymorphism. Such a >solution is doable in current Python with various approaches. Making a >decorator that would handle type checking and polymorphism has been >provided here already, or we can go a class-based route to do the same >thing, though not nearly as neat. > >Still, what do you mean by the "nature" of what something is? > > - Josiah > > Nature was a horrible word for me to use; what I was trying to convey was that there are so many mays to analyze an object besides just the type (which I assume directly refers to the return value of type(object) as opposed to some loose concept of 'type' as in style) of the object. Like some of the examples I gave earlier, in_range, i.e.: def in_range(*args): r = range(*args) def partial(n): if n in r: return n else: raise TypeError return partial def func(a: in_range(10)): pass which is very different from pure type checking which I assume would be: def func(a: int): pass which simply translates to an 'isinstance(a, int)' call, where 'a' is a variable and 'int' is a type object. From eppstein at ics.uci.edu Mon Apr 5 22:34:38 2004 From: eppstein at ics.uci.edu (David Eppstein) Date: Mon Apr 5 22:34:45 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon References: <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <200404051459.10263.fdrake@acm.org> <1081195698.2377.9.camel@localhost> <200404052209.i35M9m606023@guido.python.org> <3AA96ED4-876E-11D8-A42F-0003934AD54A@chello.se> Message-ID: > > > There are several parsing problems with <...>: the lexer doesn't see < > > > and > as matched brackets, so you won't be able to break lines without > > > using a backslash, and the closing > is ambiguous -- it might be a > > > comparison operator. > > Re the lexer, I don't see this as a big problem as long as multiple > declarators can be handled by multiple <...> pairs. ...besides, how hard can it be to convince the lexer to treat '<' at the start of a non-continuation line differently from other placements of the same char? -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From greg at cosc.canterbury.ac.nz Mon Apr 5 22:49:58 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 22:50:19 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040405175117.GA21587@mems-exchange.org> Message-ID: <200404060249.i362nwXn003846@cosc353.cosc.canterbury.ac.nz> Neil Schemenauer : > On Mon, Apr 05, 2004 at 01:36:24PM -0400, Aahz wrote: > > Your intuition is wrong. From package's __init__.py, ``from .`` refers > > to package's parent; to get back into the package, you need to use > > ``from .package``. > > That's confusing. I expected '.' to refer to the current package, > just as '.' refers to the current directory in most file systems. Hmmm. I think people's intuition is likely to differ here depending on whether the module doing the import is a plain module or a package. In a plain module, the only sensible thing for .foo to mean is a module foo in the same package as the importing module. But in a package, it seems to me the most obvious meaning of .foo is a module foo in the package. I would suggest that this is what it *should* mean, since it's going to be very common for a package to refer to submodules of itself, and requiring the name of the package to be included in all such references would defeat one of the purposes of having relative imports, which is to minimise the amount of renaming needed when moving things around in the module hierarchy. Would it be unreasonable to define "." as meaning the current *package*, rather than the current *module* -- i.e. the package itself if the importer is a package, and the containing package if it's a plain module? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Mon Apr 5 22:55:54 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 22:56:04 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <66133DA1-8764-11D8-B3FA-000A95686CD8@redivi.com> Message-ID: <200404060255.i362tsPG003858@cosc353.cosc.canterbury.ac.nz> > If nothing else pure python versions of modules: > (a) serve as great documentation > (b) are good for psyco > (c) are good for PyPy > (d) are good for Pyrex > (e) are good for Starkiller > (f) are good for Jython > (g) are good for IronPython I've just thought of another one: (h) are useful for platforms where there isn't a compiled C version available yet and it's too much hassle to compile it yourself a case I ran into the other day when I wanted to use the csv module from 2.3 with a 2.2 installation on Windows (don't ask why). I ended up writing my own Python version of a part of it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Mon Apr 5 22:58:05 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 5 22:58:17 2004 Subject: [Python-Dev] Validators (was Re: Python-Dev Digest, Vol 9, Issue 17) In-Reply-To: <40720838.5050905@earthlink.net> Message-ID: <200404060258.i362w5ZE003864@cosc353.cosc.canterbury.ac.nz> While we're on the topic, an idea I had once for type declaration syntax went like def foo(i/int, s/str)/float: ... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From bob at redivi.com Mon Apr 5 23:04:44 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon Apr 5 23:00:39 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <200404060255.i362tsPG003858@cosc353.cosc.canterbury.ac.nz> References: <200404060255.i362tsPG003858@cosc353.cosc.canterbury.ac.nz> Message-ID: <27B0C115-8777-11D8-84F3-000A95686CD8@redivi.com> On Apr 5, 2004, at 10:55 PM, Greg Ewing wrote: >> If nothing else pure python versions of modules: >> (a) serve as great documentation >> (b) are good for psyco >> (c) are good for PyPy >> (d) are good for Pyrex >> (e) are good for Starkiller >> (f) are good for Jython >> (g) are good for IronPython > > I've just thought of another one: > > (h) are useful for platforms where there isn't a compiled > C version available yet and it's too much hassle to > compile it yourself > > a case I ran into the other day when I wanted to use the csv > module from 2.3 with a 2.2 installation on Windows (don't > ask why). I ended up writing my own Python version of a part > of it. Yeah, the backwards portability bit is key, I forgot about that one. I've personally ported (read: copied the documentation into a python file) itertools to pure python so I could use it (and the sets module) from Python 2.2. -bob From DavidA at activestate.com Tue Apr 6 00:22:13 2004 From: DavidA at activestate.com (David Ascher) Date: Tue Apr 6 00:25:29 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404051643.i35GhU005203@guido.python.org> References: <200404050601.i35614f6002297@cosc353.cosc.canterbury.ac.nz> <4071160D.5060204@prescod.net> <200404051643.i35GhU005203@guido.python.org> Message-ID: <40723075.3000808@ActiveState.com> Guido van Rossum wrote: > Anyway, personal aesthetics are learned behavior, like it or not. Maybe yours. Mine are god-given. So there. --david From jcarlson at uci.edu Tue Apr 6 03:04:54 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue Apr 6 03:09:05 2004 Subject: [Python-Dev] Validators (was Re: Python-Dev Digest, Vol 9, Issue 17) In-Reply-To: <40721626.8090906@earthlink.net> References: <20040405184719.EC83.JCARLSON@uci.edu> <40721626.8090906@earthlink.net> Message-ID: <20040405233102.EC87.JCARLSON@uci.edu> > Nature was a horrible word for me to use; what I was trying to convey > was that there are so many mays to analyze an object besides just the > type (which I assume directly refers to the return value of type(object) > as opposed to some loose concept of 'type' as in style) of the object. > Like some of the examples I gave earlier, in_range, i.e.: type(thing) and isinstance(thing, typ) are the only reliable methods we have for determining the "nature" of what an object really is. In terms of general validation of input, that is also solved by the general decorator syntax. You can wrap your function with any arbitrary argument or return validation as you see fit, if I remember correctly, such decorators are even planned to be in the standard library somewhere. > which is very different from pure type checking which I assume would be: > > def func(a: int): pass > > which simply translates to an 'isinstance(a, int)' call, where 'a' is a > variable and 'int' is a type object. Aside from your "validator" syntax, which I hope /never/ makes it into Python, we understand what you are saying, however, as I said before (and as you didn't listen to), such runtime pre and post validation is currently possible in Python. Certainly it isn't part of the /syntax/ of Python, but I don't believe such type checking or input validation should be a part of Python syntax. Considering that what you desire is already possible to do, and will be made easier with decorators, what else do you want? - Josiah From python-dev at zesty.ca Tue Apr 6 08:46:44 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Tue Apr 6 08:46:22 2004 Subject: [Python-Dev] Expert floats In-Reply-To: References: Message-ID: On Tue, 30 Mar 2004, Tim Peters wrote: > I want marshaling of fp numbers to give exact (not approximate) round-trip > equality on a single box, and across all boxes supporting the 754 standard > where C maps "double" to a 754 double. That is a valuable property. I support it and support Python continuing to have that property. I hope it has been made quite clear by now that this property does not constrain how numbers are displayed by the interpreter in human-readable form. The issue of choosing an appropriate string representation of a number is unaffected by the desire for the above property. > [Ping] > > All right. Maybe we can make some progress. > > Probably not -- we have indeed been thru all of this before. I think we *have* made progress. Now we can set aside the red-herring issue of platform-independent serialization and focus on the real issue: human-readable string representation. So let's look at what you said about Python's accessibility: > As I said before (again and again and again ), I'm the one who has > fielded most newbie questions about fp since Python's beginning, and I'm > very happy with the results of changing repr() to produce 17 digits. They > get a little shock at the start now, but potentially save themselves from > catastrophe by being forced to grow some *necessary* caution about fp > results early. Now you are pulling rank. I cannot dispute your longer history and greater experience with Python; it is something i greatly admire and respect. I also don't know your personal experiences teaching Python. But i can tell you my experiences. And i can tell you that i have tried to teach Python to many people, individually and in groups. I taught a class in Python at UC Berkeley last spring to 22 people who had never used Python before. I maintained good communication with the students and their feedback was very positive about the class. How did the class react to floating-point? Seeing behaviour like this: >>> 3.3 3.2999999999999998 >>> confused and frightened them, and continues to confuse and frighten almost everyone i teach. (The rare exceptions are the people who have done lots of computational work before and know how binary floating-point representations work.) Every time this happens, the teaching is derailed and i am forced to go into an explanation of binary floating-point to assuage their fears. Remember, i am trying to teach basic programming skills. How to solve problems; how to break down problems into steps; what's a subroutine; and so on. Aside from this floating-point thing throwing them off, Python is a great first language for new programmers. This is not the time to talk about internal number representation. I am tired of making excuses for Python. I love to tell people about Python and show them what it can do for them. But this floating-point problem is embarrassing. People are confused because no other system they've seen behaves like this. Other languages don't print their numbers like this. Accounting programs and spreadsheets don't print their numbers like this. Matlab and Maple and Mathematica don't print their numbers like this. Only Python insists on being this ugly. And it screws up the most common way that people first get to know Python -- as a handy interactive calculator. And for what? For no gain at all -- because when you limit your focus to the display issue, the only argument you're making is "People should be frightened." That's a pointless reason. Everything in Python -- everything in computers, in fact -- is a *model*. We don't expect the model to be perfectly accurate or to be completely free of limitations. IEEE 754 happens to be the prevalent model for the real number line. We don't print every string with a message after it saying "WARNING: MAXIMUM LENGTH 4294967296", and we shouldn't do the same for floats. "3.2999999999999998" does not give you any more information than "3.3". They both represent exactly the same value. "3.3" is vastly easier to read. The only reason you seem to want to display "3.2999999999999998" is to frighten people. So why not display "3.3 DANGER DANGER!!"? Even that would be much easier to read, but my point is that i hope it exposes the problem with your argument. I'm asking you to be more realistic here. Not everyone runs into floating-point corner cases. In fact, very few people do. I have never encountered such a problem in my entire history of using Python. And if you surveyed the user community, i'm sure you would find that only a small minority cares enough about the 17th decimal place for the discrepancy to be an issue. Now, why should that minority make it everyone else's problem? Is this CP4E or CPFPFWBFP? Computer Programming For Everybody? Or Computer Programming For People Familiar With Binary Floating-Point? You say it's better for people to get "bitten early". What's better: everyone suffering for a problem that will never affect most of them, or just those who care about the issue having to deal with it? Beautiful is better than ugly. Practicality beats purity. Readability counts. -- ?!ng From s.percivall at chello.se Tue Apr 6 09:12:22 2004 From: s.percivall at chello.se (Simon Percivall) Date: Tue Apr 6 09:12:26 2004 Subject: [Python-Dev] Expert floats In-Reply-To: References: Message-ID: <0AAC5E0D-87CC-11D8-A42F-0003934AD54A@chello.se> On 2004-04-06, at 14.46, Ka-Ping Yee wrote: > I think we *have* made progress. Now we can set aside the red-herring > issue of platform-independent serialization and focus on the real > issue: > human-readable string representation. [Snip] > I am tired of making excuses for Python. I love to tell people about > Python and show them what it can do for them. But this floating-point > problem is embarrassing. People are confused because no other system > they've seen behaves like this. Other languages don't print their > numbers like this. Accounting programs and spreadsheets don't print > their numbers like this. Matlab and Maple and Mathematica don't print > their numbers like this. Only Python insists on being this ugly. And > it screws up the most common way that people first get to know Python > -- as a handy interactive calculator. So how should "2.2 - 1.2 - 1" be represented? Matlab (Solaris 9): 2.22044604925031e-16 Octave (MacOS X 10.3): 2.22044604925031e-16 Python 2.3.3 (MacOS X 10.3): 2.2204460492503131e-16 Is this something you accept since Matlab does it? From barry at python.org Tue Apr 6 09:17:11 2004 From: barry at python.org (Barry Warsaw) Date: Tue Apr 6 09:17:22 2004 Subject: [Python-Dev] Some comments on PEP 328 (absolute/relative imports) In-Reply-To: References: Message-ID: <1081257431.30379.22.camel@anthem.wooz.org> On Sat, 2004-04-03 at 13:24, Bernhard Herzog wrote: > I don't think this is a good description of the actual problem. The > problem is that the modules in the package shadow global ones and > there's no easy way to bypass the local package. FWIW, though, I don't > recall ever running into a problem with that. It happens, and it makes for tricky choices. I know to stay away from internal modules like re and email, but as the standard library gets fatter, there's more chance of collision. The only remedy will be to rename an internal module when I find that I suddenly want to use some new standard module. There should be more independence. If all import paths were absolute, then I wouldn't have this problem, and in fact, that's the rule my own code tries to adhere to. See the email package which uses absolute imports internally. I know others have use cases for relative imports, but I've never quite seen the need. Ideally, IMO, all imports would be absolute. If we have to throw a bone to relative import proponents in order to get to absolute imports by default, so be it. -Barry From barry at python.org Tue Apr 6 09:25:16 2004 From: barry at python.org (Barry Warsaw) Date: Tue Apr 6 09:25:23 2004 Subject: [Python-Dev] Some comments on PEP 328 (absolute/relative imports) In-Reply-To: <20040403191106.GA4254@panix.com> References: <20040403191106.GA4254@panix.com> Message-ID: <1081257915.30379.27.camel@anthem.wooz.org> On Sat, 2004-04-03 at 14:11, Aahz wrote: > Barry, would you please write up something for the PEP here? (You're > the person who pushed this the most.) I've already updated the PEP to > include clarification of the shadowing issue. I added a few more sentences. -Barry From python-dev at zesty.ca Tue Apr 6 09:49:05 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Tue Apr 6 09:48:44 2004 Subject: [Python-Dev] Expert floats In-Reply-To: <0AAC5E0D-87CC-11D8-A42F-0003934AD54A@chello.se> References: <0AAC5E0D-87CC-11D8-A42F-0003934AD54A@chello.se> Message-ID: On Tue, 6 Apr 2004, Simon Percivall wrote: > So how should "2.2 - 1.2 - 1" be represented? > > Matlab (Solaris 9): 2.22044604925031e-16 > Octave (MacOS X 10.3): 2.22044604925031e-16 > Python 2.3.3 (MacOS X 10.3): 2.2204460492503131e-16 > > Is this something you accept since Matlab does it? I accept it, but not primarily because Matlab does it. It is fine to show a value different from zero in this case because the result really is different from zero. In this case those digits are necessary to represent the machine number. In other cases (such as 1.1) they are not. -- ?!ng From aahz at pythoncraft.com Tue Apr 6 10:00:53 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Apr 6 10:00:57 2004 Subject: [Python-Dev] Expert floats In-Reply-To: References: <0AAC5E0D-87CC-11D8-A42F-0003934AD54A@chello.se> Message-ID: <20040406140052.GB27004@panix.com> On Tue, Apr 06, 2004, Ka-Ping Yee wrote: > On Tue, 6 Apr 2004, Simon Percivall wrote: >> >> So how should "2.2 - 1.2 - 1" be represented? >> >> Matlab (Solaris 9): 2.22044604925031e-16 >> Octave (MacOS X 10.3): 2.22044604925031e-16 >> Python 2.3.3 (MacOS X 10.3): 2.2204460492503131e-16 >> >> Is this something you accept since Matlab does it? > > I accept it, but not primarily because Matlab does it. > > It is fine to show a value different from zero in this case > because the result really is different from zero. In this case > those digits are necessary to represent the machine number. > In other cases (such as 1.1) they are not. The point is that anyone who relies on floating point will almost certainly end up with the equivalent of Simon's case at some point. The most common *really* ugly case that's hard for new programmers to deal with is: section_size = length / num_sections tmp = 0 while tmp < length: process(section) tmp += section_size and variations thereof. Just because every other programming language hides this essential difficulty is no reason to follow along. I don't have Tim's experience, but until we added that section to the tutorial, I was one of the regular "first responders" on c.l.py who was always dealing with this issue. I stand with Uncle Timmy -- it's been much easier now. If you want to teach Python to complete newcomers, you need to do one of two things: * Avoid floating point * Start by explaining what's going on (use Simon's example above for emphasis) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From FBatista at uniFON.com.ar Tue Apr 6 10:13:43 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 6 10:15:04 2004 Subject: [Python-Dev] PEP 327: Decimal data type Message-ID: [Aahz] #- > where ``value1`` can be int, long, string, tuple or Decimal, #- > ``value1`` can be only float, and ``decimal_digits`` is an optional #- > int. #- #- You mean value2 in the second case, right? Yes, thank you. This and other typos has been fixed by Tim Peters in the process of acceptance of the PEP. So, you can expect a more typo-free PEP in the future, :) . Facundo From aahz at pythoncraft.com Tue Apr 6 10:16:15 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Apr 6 10:16:19 2004 Subject: [Python-Dev] Some comments on PEP 328 (absolute/relative imports) In-Reply-To: <1081257431.30379.22.camel@anthem.wooz.org> References: <1081257431.30379.22.camel@anthem.wooz.org> Message-ID: <20040406141615.GD27004@panix.com> On Tue, Apr 06, 2004, Barry Warsaw wrote: > On Sat, 2004-04-03 at 13:24, Bernhard Herzog wrote: >> >> I don't think this is a good description of the actual problem. The >> problem is that the modules in the package shadow global ones and >> there's no easy way to bypass the local package. FWIW, though, I don't >> recall ever running into a problem with that. > > It happens, and it makes for tricky choices. I know to stay away from > internal modules like re and email, but as the standard library gets > fatter, there's more chance of collision. The only remedy will be to > rename an internal module when I find that I suddenly want to use some > new standard module. There should be more independence. Thanks! Here's my rewrite; comments, anyone? Rationale for Absolute Imports ============================== In Python 2.3 and earlier, if you're reading a module located inside a package, it is not clear whether :: import foo refers to a top-level module or to another module inside the package. As Python's library expands, more and more old modules suddenly shadow builtin modules by accident. It's a particularly difficult problem inside packages because there's no way to specify which module is meant. To resolve the ambiguity, it is proposed that ``foo`` will always be a module or package reachable from ``sys.path``. This is called an absolute import. The python-dev community chose absolute imports as the default because they're the more common use case and because absolute imports can provide all the functionality of relative (intra-package) imports -- albeit at the cost of difficulty when renaming package pieces higher up in the hierarchy or when moving one package inside another. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From jim.jewett at eds.com Tue Apr 6 10:29:10 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Tue Apr 6 10:30:04 2004 Subject: [Python-Dev] Expert floats Message-ID: Ka-Ping Yee: > >>> 3.3 > 3.2999999999999998 > confused and frightened them, and continues to confuse and frighten > almost everyone i teach. It bothered me when I first saw it (not in Python). It was still better than trying to figure out why my code was doing the wrong thing, when my math and logic were both clearly correct. To steal Simon Percivall's example, I assumed 2.20 - 1.20 == 1.00. It didn't even occur to me to check something as obviously true as that -- until I had learned and remembered that floats were not decimals. -jJ From arigo at tunes.org Tue Apr 6 10:33:56 2004 From: arigo at tunes.org (Armin Rigo) Date: Tue Apr 6 10:37:26 2004 Subject: [Python-Dev] Python is faster than C In-Reply-To: <1081206375.2666.20.camel@echidna.object-craft.com.au> References: <16E1010E4581B049ABC51D4975CEDB8803060E52@UKDCX001.uk.int.atosorigin.com> <16497.28394.944929.536518@magrathea.basistech.com> <1081206375.2666.20.camel@echidna.object-craft.com.au> Message-ID: <20040406143356.GA21775@vicky.ecs.soton.ac.uk> Hello Dave, On Tue, Apr 06, 2004 at 09:06:16AM +1000, Dave Cole wrote: > Would it make sense to build specialisation into the base Python > interpreter so that it builds specialised versions of bytecode for > functions. I am assuming that there are only a few types that can be > targeted by specialisation (int, float). Bytecodes then could be added > to the interpreter which operate on those native machine types rather > than the Python objects. This is a kind of project that would give a good speed-up with limited changes in the Python interpreter, though it is still relatively involved. Whether someone shows up to actually implement and maintain it is another matter. Moreover there are a few dark spots that are easily overlooked; for example, any operation between ints could actually return a long. Psyco itself is very different and copes well with this problem (and can do more than only easy-typed locals). A bientot, Armin. From jim.jewett at eds.com Tue Apr 6 10:41:44 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Tue Apr 6 10:42:04 2004 Subject: [Python-Dev] Python is faster than C Message-ID: Dave Cole: > Would it make sense to build specialisation into the base Python > interpreter so that it builds specialised versions of bytecode for > functions. I am assuming that there are only a few types that can be > targeted by specialisation (int, float). Bytecodes then could be added > to the interpreter which operate on those native machine types rather > than the Python objects. If you add too many specializations, you blow the cache. If you add not enough, then you help only certain types of programming. If you're willing to do that, you might consider going even farther, as in http://www.foretec.com/python/workshops/1998-11/proceedings/papers/aycock-21 1/aycock211.html Basically, he creates a large number of pseudo-codes that just mean "do this, then this, then this" without going back to the eval loop in between. Because he knows what is coming next, there are additional chances for optimizations. -jJ From tim.one at comcast.net Tue Apr 6 11:00:27 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue Apr 6 11:00:37 2004 Subject: [Python-Dev] Expert floats In-Reply-To: Message-ID: [Ka-Ping Yee] >> >>> 3.3 >> 3.2999999999999998 > >> confused and frightened them, and continues to confuse and frighten >> almost everyone i teach. [Jim Jewitt] > It bothered me when I first saw it (not in Python). > > It was still better than trying to figure out why my code was doing > the wrong thing, when my math and logic were both clearly correct. > > To steal Simon Percivall's example, I assumed 2.20 - 1.20 == 1.00. > It didn't even occur to me to check something as obviously true as > that -- until I had learned and remembered that floats were not > decimals. I believe Ping would have >>> 2.20 - 1.20 1.0000000000000002 because "1.0000000000000002" is the shortest string that evals to the true machine result. OTOH, he'd have >> 2.20 - 1.10 1.1 The true machine result isn't decimal 1.1, but "1.1" is again the shortest string that evals to the true machine result. Presumably newbies aren't confused or frightened by 1.0000000000000002, because their intuition about shortest-possible reproducing decimal representations of binary floats is finely honed . Most people should use Decimal: >>> print Decimal("2.20") - Decimal("1.20") 1.00 >>> print Decimal("2.20") - Decimal("1.1") 1.10 The output is "what they expect" then, 100% truthful (the internal results are exactly 1. and decimal 1.1, respectively), and even preserves a useful notion of significant trailing zeroes. From ark-mlist at att.net Tue Apr 6 11:09:37 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Tue Apr 6 11:09:22 2004 Subject: [Python-Dev] Expert floats In-Reply-To: Message-ID: <002301c41be9$2dbedeb0$6402a8c0@arkdesktop> > I believe Ping would have > > >>> 2.20 - 1.20 > 1.0000000000000002 As would I. > because "1.0000000000000002" is the shortest string that evals to the true > machine result. OTOH, he'd have > > >> 2.20 - 1.10 > 1.1 Believe it or not, this apparent anomaly doesn't bother me, but >>> 1.1 1.1000000000000001 does. The reason is that the first example does arithmetic, and it is easy to explain that floating-point arithmetic is not completely accurate. From barry at python.org Tue Apr 6 11:38:24 2004 From: barry at python.org (Barry Warsaw) Date: Tue Apr 6 11:38:33 2004 Subject: [Python-Dev] Some comments on PEP 328 (absolute/relative imports) In-Reply-To: <20040406141615.GD27004@panix.com> References: <1081257431.30379.22.camel@anthem.wooz.org> <20040406141615.GD27004@panix.com> Message-ID: <1081265903.30379.92.camel@anthem.wooz.org> On Tue, 2004-04-06 at 10:16, Aahz wrote: > Thanks! Here's my rewrite; comments, anyone? Some minor rephrasing suggestion... > Rationale for Absolute Imports > ============================== > > In Python 2.3 and earlier, if you're reading a module located inside a > package, it is not clear whether :: > > import foo > > refers to a top-level module or to another module inside the package. > As Python's library expands, more and more old modules suddenly shadow > builtin modules by accident. As Python's library expands, more and more existing package internal modules suddenly shadow standard library modules by accident. -Barry From tim.one at comcast.net Tue Apr 6 12:23:09 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue Apr 6 12:23:18 2004 Subject: [Python-Dev] Expert floats In-Reply-To: <002301c41be9$2dbedeb0$6402a8c0@arkdesktop> Message-ID: [Tim] >> I believe Ping would have >> >> >>> 2.20 - 1.20 >> 1.0000000000000002 [Andrew Koenig] > As would I. Only sometimes, right? You earlier gave a list of 4 behaviors you sometimes want, and that particular output isn't suitable for all of them. >> OTOH, he'd have >> >> >>> 2.20 - 1.10 >> 1.1 > Believe it or not, this apparent anomaly doesn't bother me, but > > >>> 1.1 > 1.1000000000000001 > > does. The reason is that the first example does arithmetic, and it > is easy to explain that floating-point arithmetic is not completely > accurate. Wouldn't it bother you to give a wrong explanation? It's simply impossible to explain why 2.2-1.2 is inaccurate without explaining that "2.2" and/or "1.2" are already inaccurate on their own. The *arithmetic* in both examples is exactly correct (if you watched the hardware's inexact flag across the subtractions, you'd see that it doesn't get set; x-y is always exact in 754 binary floating point when x and y are within a factor of 2 of each other). The only cause of "funny results" in these examples is representation error. So trying to "explain" this by telling a newbie that arithmetic is inaccurate may pacify them, but at the cost of increasing their confusion (the explanation they're given then doesn't actually explain anything about what they're seeing, and the true cause has been left untold). OTOH, >>> 2.2 2.2000000000000002 >>> 1.1 1.1000000000000001 >>> points to the true cause, and the Tutorial Appendix on binary fp issues has been successful in bringing newbies to a correct understanding of that. From bv at artima.com Tue Apr 6 16:14:18 2004 From: bv at artima.com (Bill Venners) Date: Tue Apr 6 16:16:25 2004 Subject: [Python-Dev] Python Interest Group Message-ID: Hello All, I am the editor of Artima.com. Guido and others suggested that I send this query to python-dev. I am about to release a new feature on Artima called Interest Groups, and I am looking for a few people who would like to moderate a Python Interest Group. I am wondering if any of you might like to serve as a moderator, or could recommend others from the Python community who might want to act as moderators. Here is the low down on Interest Groups: Artima Interest Groups is a free service that allows communities to keep informed and discuss issues. Each interest group is essentially a news feed that serves a specific community. The news feed is moderated by one or more volunteers from the community. Anyone can submit news items for consideration. The moderators check the accuracy of the submissions, and decide which news items to approve. Approved items appear on web pages at Artima.com, and in two Artima RSS feeds. Each approved item can be discussed by the community in its own Artima Forums topic. Basically, the moderators would get a short email notification when someone submits an item, then they can edit, approve, or reject the item via the web. It is like having a little theserverside.com or slashdot.org for your community. Another way you might think of it is as a web version of python-announce, except that only the topics are moderated. Any discussion that follows each topic is a free-for-all. One benefit of having an Interest Group at Artima is that people outside the Python community proper will also be exposed to the Python community's news. If anyone is interested, or has a recommendation of someone I can approach about moderating a Python Interest Group, please email me. Thanks. Bill -- Bill Venners Artima Software, Inc. http://www.artima.com From lists at hugunin.net Tue Apr 6 18:08:54 2004 From: lists at hugunin.net (Jim Hugunin) Date: Tue Apr 6 18:21:38 2004 Subject: [Python-Dev] PEP 318: More examples of decorator use Message-ID: I'm a little scared of getting involved in the discussion of PEP 318. Nevertheless, I think any language design discussion can benefit from more examples. Here's a list of ways that I would use decorators in my Python code if I had simpler syntax than is available today. *Interoperability with other languages and systems jythonc parses method docstrings looking for a special directive to generate Java-compatible method signatures, i.e. def meth(self, a, b): "@sig public void meth(int a, int b)" pass If Python had structured metadata, this would change to [java_sig("public void meth(int a, int b)")] def meth(self, a, b): pass C# uses this mechanism to provide interoperability with C, COM, and web services. Here's a translation of a web services example from the .NET framework docs. [WebService(Description="Common Server Variables", Namespace="http://www.contoso.com/")] class ServerVariables: [WebMethod(Description="Obtains the Server Computer Name", EnableSession=false)] def GetMachineName(self): return self.Server.MachineName [SoapDocumentMethod(Action="http://www.contoso.com/Sample", RequestNamespace="http://www.contoso.com/Request", RequestElementName="GetUserNameRequest", ResponseNamespace="http://www.contoso.com/Response", ResponseElementName="GetUserNameResponse"), WebMethod(Description="Obtains the User Name")] def GetUserName(self): ... This is the same functionality motivating Bob Ipolito's pleas for this feature to support pyobjc, and it would obviously also help IronPython. *Tracing and other global policies Aspect-oriented programming has had great success capturing global policies for things like tracing in a single module. For a language like Python that already has strong meta-programming support it's possible to program in an AOP style without any language changes (see the tracing implementation in parrotbench/b0.py for an example). However, one lesson learned from actually applying AOP to large systems (http://hugunin.net/papers/pra03-colyer.pdf) was that there was always a need to capture a few local exceptions. Current AOP systems can do this in a clumsy way, but almost everyone agrees that C#/Java attributes are the best way to specify these kinds of exceptions. Here's an example of a tracing policy with a local exception. from tracing import notrace, add_tracing class C: def m1(self): ... def m2(self): ... [notrace] def m0(self): ... if __DEBUG__: add_tracing(globals()) This code will recursively walk all the declarations in a module and wrap all methods and functions with tracing code. Attributes are used here to indicate special handling for specific methods. In this case, m0 should not be traced. *Unit testing The nunit test framework uses attributes instead of naming conventions to label test methods. My favorite part of this is the additional attributes, for example 'expected_exception' to indicate that the correct behavior for a test is to throw a particular exception, i.e. [test, expected_exception(InsufficientFundsError)] def transfer2(self): self.source.transfer(100000000000) vs. def test_transfer2(self): self.assertRaises(InsufficientFundsError, self.source.transfer, 1000000000) This example is extremely compelling for C# or Java code where functions aren't first class and a complicated try/catch block is required instead of assertRaises. It's less compelling than I'd expected for Python; however, I still find the version with attributes a little more clear. *Parsing There are many simple parser implementations for Python that embed grammar rules in doc strings. This would be nicer to have in explicit metadata. Here's an example from Brian Sabbey's dparser for python. [action("exp", "exp '+' exp")] def d_add(t): return t[0] + t[2] [action("exp", '"[0-9]+"')] def d_number(t): return int(t[0]) *Synchronization I think a synchronized decorator would be useful, but more sophisticated locking policies are even more interesting. Here's an example of reader/writer synchronization assuming a metaclass ReadWriteSynchro that cooperates with attributes on specific methods. class DataStore(ReadWriteSynchro): [reader] def read(self, key): .... [writer] def write(self, key, value): ... ... *Other examples that I'd probably use memoized from Michael Chermside to automatically cache the results of idempotent functions accepts and returns type checking wrappers from the PEP until/unless Python gets a better answer for true optional type declarations generic from Edward Loper to support generic functions. I both really like and really hate this decorator. I'd love to have tolerable syntax for generic functions and this gives me something tolerable. On the other hand, I really dislike code that uses sys._getframe() so that might be enough to scare me off of this. *Some examples that I'd like to see in an anti-patterns list onexit from the PEP. I don't think decorators should have side-effects visible outside of the class/function itself. This seems MUCH better accomplished with the explicit call to the registration function. singleton from the PEP. Whenever possible, decorators should return the same kind of thing that they are given. This code seems much less clear than explicitly creating a singleton instance. propget, propset, propdel from Edward Loper via Guido. To me, these violate the principle of modularity. I much prefer the explicit property declaration that puts all of the parts together. class C(object): def getX(self): return self.__x def setX(self, x): self.__x = x x = property(getX, setX) -Jim From lists at hugunin.net Tue Apr 6 18:09:37 2004 From: lists at hugunin.net (Jim Hugunin) Date: Tue Apr 6 18:21:42 2004 Subject: [Python-Dev] PEP 318: How I would implement decorators Message-ID: I've spent a couple days reading the decorators discussion and thinking about the issues. That's not a lot of time, but I still thought I'd share my opinions on implementing this feature. This all is in my (mostly) humble opinion. When I started looking at PEP 318, I thought that the most important issue was deciding on the correct syntax. After a while I became much more worried about semantics. * Semantics, composition and understandable failures One of the nice properties of C# and Java attributes is that they are declarative rather than transformative so there are no conflicts between different attributes placed on the same object. This is probably too strong of a restriction for Python; nevertheless, it's a valuable reference point. Using the definition of returns from the PEP, I get the following behavior if I confuse its order with classmethod, i.e. class C: [classmethod(m), returns(str)] def m(cls): return repr(cls) C.m() --> TypeError: unbound method new_f() must be called with C instance as first argument (got nothing instead) This is a hard message to understand, but at least it does occur at the first instance of calling the method. Even worse is where you forget to create an instance of a class or to call a function like returns, i.e. class C: [returns] def m(self): return "hello" c = C() print c.m() #there's no error for this call --> function new_f at 0x01278BB0> This kind of error scares me a lot. It can be extremely hard to track down because it doesn't occur either at the point of declaration or the point of calling the method, but only at some later stage where the result from the method is used. I'm also concerned that this error is caused by the person using the attribute not the person writing it and that person may be less sophisticated and more easily confused. A less critical, but important issue is the propagation of doc strings and other function attributes, i.e. class C: [returns(str)] def m(self): "a friendly greeting" return "hello" print C.m.__doc__ --> None This will really mess up programs that rely on doc strings or other function attributes. After thinking about these issues, my first inclination was to sign up for Raymond Hettinger's NO on 318 campaign. Unfortunately, as my previous message showed there are a number of ways that I'd really like to use decorators in my Python code. My next thought was to propose that Python's decorators be restricted to be declarative like C# and Java attributes. They could be used to add meta information to functions, but other meta-tools would be needed for any actual transformations. I'm still open to that idea, but I've come to believe that's not a very pythonic approach. My current idea is that we should design this feature and any documentation around it to strongly encourage people to write safe, composable decorators. Specifically, it should be easier to write safe and composable decorators than to write bad ones. After that, we can rely on decorator developers to do the right thing. 1. Use a "decorate" method instead of a call to avoid the worst problems like [returns]. This has the added benefit in my eyes of making it a little harder to write small decorators and thus encouraging people to think a little more before writing these meta-programs. 2. Provide a small number of helper classes that make it easy to write safe and composable decorators. Here are my top two candidates: class attribute: def __init__(self, **kwds): self.attrs = kwds def decorate(self, decl): if not hasattr(decl, __attrs__): decl.__attrs__ = [] decl.__attrs__.append(self) return decl def __repr__(self): ... class func_wrapper: """subclass this and implement a wrap function""" def decorate(self, func): assert iscallable(func) new_func = self.wrap(func) assert iscallable(new_func) self.copy_attributes(func, new_func) return new_func def copy_attributes(self, old_func, new_func): ... func_wrapper would be used for things like synchronized and memoize, and attribute would be used for most of the other examples in my previous message. 3. Provide a clear set of guidelines for good decorators, i.e. a. decorators should not have side effects outside of the class/function b. decorators should return the same kind of thing they are given c. decorators should be commutative whenever possible (requires b) + sys._getframe() There should be a consensus on whether or not decorators like 'generic' are a good or a bad idea. If they're a good idea then the decorate method should get an extra argument exposing the local namespace so that sys._getframe() won't be required. I'd be in favor of adding that argument to decorate. + classmethod and staticmethod These two decorators are a big problem because they don't return the same kind of thing as they are given. This means they always have to be the last decorator applied in any list. That kind of restriction is never a good thing. Personally, I'd like to see these changed from returning a new object to instead setting an attribute on the existing function/method object. Then the method object could use this attribute to implement its __get__ method correctly. This would make these decorators compose very nicely with all of the other decorators; however, it would be a fairly major change to how Python handles static and class methods and I'm doubtful that would fly. * Syntax I've come to see syntax as less crucial than I originally thought. In particular, the examples I've read on this list of using temporary variables showed me that even if the syntax is too confining there are tolerable solutions. Nevertheless, I'm still strongly in favor of putting the decorator list in front of the def/class. Decorators/attributes are an exciting new technique that's gaining increasing use in different programming languages. They're important enough to be one of the key ideas that Java-1.5 is stealing from C# (turn-about is always fair play). The AOP community often has vehement debates not about the value of attributes, but about how they can be most effectively used. Because decorators are still a relatively new idea, it's hard to know how much room they will need. Certainly if we're only interested in classmethod and staticmethod it's not much. However, I think it's important to give decorators as much room as possible to be prepared for the future. I have some sympathy with the arguments that a bare prefix list is already legal Python syntax and is certainly found in existing code. I'm quite sure that using a bare list would break existing Python programs that have dangling lists left over as they evolved. I like Sam's proposal to add a '*' in front of the list. I could also be persuaded to go along with any of the new grouping constructing, like <| |>; however, those all seem like too much of a change for the value. I'm fairly confident that the *[] form would allow for attributes to optionally be put on the same line as the def. This would be LL1, but perhaps the Python parser has other restrictions that I'm unaware of, i.e. *[java_sig("public void meth(int a, int b)"), staticmethod] def meth(self, a, b): pass or for simple cases *[staticmethod] def meth(self, a, b): pass -Jim From pje at telecommunity.com Tue Apr 6 19:32:53 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 6 19:35:04 2004 Subject: [Python-Dev] PEP 318: How I would implement decorators In-Reply-To: Message-ID: <5.1.1.6.0.20040406191201.02228370@telecommunity.com> At 03:09 PM 4/6/04 -0700, Jim Hugunin wrote: >1. Use a "decorate" method instead of a call to avoid the worst problems >like [returns]. Mike Pall actually suggested this recently, although he called the method '__decorate__' and proposed that it use a type slot. > This has the added benefit in my eyes of making it a little >harder to write small decorators and thus encouraging people to think a >little more before writing these meta-programs. Actually, it's pretty trivial to write a 'decorator_factory' decorator that you could then use to make it easy again, i.e.: [decorator_factory] def returns(typ): ... So, the approach won't actually discourage people from writing small decorators. :) >3. Provide a clear set of guidelines for good decorators, i.e. > a. decorators should not have side effects outside of the class/function This is a use case that people have explicitly requested. IIRC, there have been much better examples than 'atexit' that have been posted here. > b. decorators should return the same kind of thing they are given I think it's probably more appropriate to say that decorators should verify that their input is something useful to them, and return a helpful error message if not. >+ sys._getframe() > >There should be a consensus on whether or not decorators like 'generic' are >a good or a bad idea. If they're a good idea then the decorate method >should get an extra argument exposing the local namespace so that >sys._getframe() won't be required. I'd be in favor of adding that argument >to decorate. I'm +1 on adding the argument, if there's a decoration method, be it 'decorate' or '__decorate__'. As long as decorators are plain functions, though, I think the extra argument makes interoperability harder. By the way, it's not clear whether decorators in the general case might not also require the active globals, in addition to the active locals. >+ classmethod and staticmethod > >These two decorators are a big problem because they don't return the same >kind of thing as they are given. This means they always have to be the last >decorator applied in any list. That kind of restriction is never a good >thing. > >Personally, I'd like to see these changed from returning a new object to >instead setting an attribute on the existing function/method object. Then >the method object could use this attribute to implement its __get__ method >correctly. This would make these decorators compose very nicely with all of >the other decorators; however, it would be a fairly major change to how >Python handles static and class methods and I'm doubtful that would fly. This issue will apply to any decorator that creates a descriptor, so kludging functions for this aren't going to help any other kinds of descriptor-making decorators. But, descriptors (other than functions) aren't usually callable, so as long as decorators that expect callables check for callability, I don't think this is too big of a deal. From pje at telecommunity.com Tue Apr 6 19:43:08 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 6 19:43:25 2004 Subject: [Python-Dev] PEP 318: More examples of decorator use In-Reply-To: Message-ID: <5.1.1.6.0.20040406193816.0223b7c0@telecommunity.com> At 03:08 PM 4/6/04 -0700, Jim Hugunin wrote: >*Some examples that I'd like to see in an anti-patterns list > >onexit from the PEP. I don't think decorators should have side-effects >visible outside of the class/function itself. This seems MUCH better >accomplished with the explicit call to the registration function. > >singleton from the PEP. Whenever possible, decorators should return the >same kind of thing that they are given. This code seems much less clear >than explicitly creating a singleton instance. > >propget, propset, propdel from Edward Loper via Guido. To me, these violate >the principle of modularity. I much prefer the explicit property >declaration that puts all of the parts together. > > class C(object): > def getX(self): return self.__x > def setX(self, x): self.__x = x > x = property(getX, setX) Hm. IIRC, Visual Basic defines "property set", "property let", and "property get" methods in a similar fashion to the propget/propset/propdel mechanism. What does C# use? Doesn't it have such methods also? Also, most languages that I've encountered that offer generic functions, signature overloading, or other similar dispatch mechanisms use variations of defining the same function/method over and over again, each variation with the same name, just a different signature, matching pattern, etc. A bigger problem with propget et al is that they don't follow a sensible inheritance mechanism. That is, you can't inherit the get method while overriding the set method. But this is really more a limitation of the way Python descriptors work, than any issue with decorators. From raymond.hettinger at verizon.net Tue Apr 6 21:22:19 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue Apr 6 21:23:25 2004 Subject: [Python-Dev] More imformative iterator representations Message-ID: <005101c41c3e$cdded760$e841fea9@oemcomputer> Armin's note reminded me that the only thing I do not like about using iterators is not being able to see their contents at the interactive prompt. So, I posted an idea about changing the print part of the read-eval-print loop to display more information: >>> enumerate('abcdefgh') There a couple of problems with the approach. One is how to identify an iterator object -- the presence of __iter__() and next() is a good but not perfect indicator. The other issue was trying to leave the iterator undisturbed while fetching the first three entries. The least imperfect solution I found was to replace the iterator with an equivalent object: firstfew=list(islice(it, 3)); it=chain(firstfew, it). Unfortunately, now the "it" variable is a chain object rather than the original iterator and the object id is different. An alternative to the read-eval-print approach is providing a custom __repr__() method for the builtin iterators. The good news is that this approach is very clean. The bad news is that it won't help user defined iterators or builtin iterators like enumerate() that do not know their future until next() is called. So, this approach is only applicable to a few iterators; it is no problem for lists, tuples, dictionaries, sets, sequence iterators, xrange iterator objects, reversed objects, itertools.repeat(), itertools.count(), and collections.deque() iterators. Eventhough this solution cannot be applied uniformly for all iterators, I think is worth considering for places where it does apply: >>> reversed('abcdefgh') reversed(['h', 'g', 'f', ...]) This could make iterators more pleasant to use interactively. What do you guys think? Raymond Hettinger Q&A: --- Q: What if .next() is labor intensive to compute? A: It is only being proposed for objects where the entries are immediately accessible. Q: Are there any other candidate output formats? A: Yes, we could add length information for a more mathematical style output: reversed(['h', 'g', 'f', ... to 8 objects]) Q: What if .next() returns three objects whose representations consume a lot of visual space? A: This is not unique to iterators. The same issue occurs to a larger extent when displaying lists. It is sometimes a PITA but usually not a problem. Q: What prevents enumerate(), imap(), izip(), etc. from implementing the more informative representations? A: They could, but it is messy to go through copy=list(islice(it,3)) and it=chain(copy,it) for every input. Q: What if calling next() initiates some sort of activity that we wished to defer until later? A: This would be a rare situation for interactive work, but if known to be an issue, printing the repr() is easily avoided by making an assignment: _ = launch1000ships(). Q: Do you really want this? A: It is nice to have but not essential. From greg at cosc.canterbury.ac.nz Tue Apr 6 21:33:21 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Apr 6 21:33:41 2004 Subject: [Python-Dev] Expert floats In-Reply-To: Message-ID: <200404070133.i371XL8d005666@cosc353.cosc.canterbury.ac.nz> > How did the class react to floating-point? Seeing behaviour like this: > > >>> 3.3 > 3.2999999999999998 > >>> > > confused and frightened them, and continues to confuse and frighten > almost everyone i teach. ... Every time this happens, the teaching is > derailed and i am forced to go into an explanation of binary > floating-point to assuage their fears. This is an argument for making some form of decimal floating point the default in Python, with binary FP an option for advanced users -- not an argument for trying to hide the fact that binary FP is being used. Because you *can't* hide that fact completely, and as has been pointed out, it *will* rear up and bite these people eventually. It's much better if that happens early, while there is someone on hand who understands the issues and can guide them gently through the shock-and-awe phase. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From aahz at pythoncraft.com Tue Apr 6 21:52:51 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Apr 6 21:52:55 2004 Subject: [Python-Dev] Some comments on PEP 328 (absolute/relative imports) In-Reply-To: <1081265903.30379.92.camel@anthem.wooz.org> References: <1081257431.30379.22.camel@anthem.wooz.org> <20040406141615.GD27004@panix.com> <1081265903.30379.92.camel@anthem.wooz.org> Message-ID: <20040407015251.GA15148@panix.com> On Tue, Apr 06, 2004, Barry Warsaw wrote: > On Tue, 2004-04-06 at 10:16, Aahz wrote: >> >> Rationale for Absolute Imports >> ============================== >> >> In Python 2.3 and earlier, if you're reading a module located inside a >> package, it is not clear whether :: >> >> import foo >> >> refers to a top-level module or to another module inside the package. >> As Python's library expands, more and more old modules suddenly shadow >> builtin modules by accident. > > As Python's library expands, more and more existing package internal > modules suddenly shadow standard library modules by accident. Done. Seems that the final remaining issue is the meaning of "." inside __init__.py -- does it refer to the current package or the parent? Guido, if you don't want to Pronounce, I'll post to c.l.py requesting feedback. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From greg at cosc.canterbury.ac.nz Tue Apr 6 21:58:33 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Apr 6 21:58:59 2004 Subject: [Python-Dev] Expert floats In-Reply-To: <002301c41be9$2dbedeb0$6402a8c0@arkdesktop> Message-ID: <200404070158.i371wXKk005694@cosc353.cosc.canterbury.ac.nz> Andrew Koenig : > The reason is that the first example does arithmetic, and it is easy > to explain that floating-point arithmetic is not completely accurate. Then how would you explain this to them: >>> 1.1 < 1.1000000000000001 False There's no arithmetic being done there. And if you're thinking of something like "floats don't have that many digits of precision", you then need to explain >>> 1.2 < 1.2000000000000001 True Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From fumanchu at amor.org Tue Apr 6 22:00:31 2004 From: fumanchu at amor.org (Robert Brewer) Date: Tue Apr 6 22:02:39 2004 Subject: [Python-Dev] More imformative iterator representations Message-ID: Raymond Hettinger wrote: > Armin's note reminded me that the only thing I do not like about using > iterators is not being able to see their contents at the interactive > prompt. So, I posted an idea about changing the print part of the > read-eval-print loop to display more information: > > >>> enumerate('abcdefgh') > > > There a couple of problems with the approach. One is how to > identify an > iterator object -- the presence of __iter__() and next() is a good but > not perfect indicator. Oh, I don't know. The Library Ref makes it pretty clear: "The iterator objects themselves are required to support the following two methods, which together form the iterator protocol: __iter__() ... next() " If not perfect, it's still a reasonable assumption. Since there's no Iterator superclass from which to inherit, you're going to have to rewrite each repr by hand anyway, and therefore can decide each one independently. Are there reasonable iterator objects which don't follow the "iterator protocol"? > An alternative to the read-eval-print approach is providing a custom > __repr__() method for the builtin iterators. +1. > Q: Are there any other candidate output formats? > A: Yes, we could add length information for a more mathematical style > output: > > reversed(['h', 'g', 'f', ... to 8 objects]) The very presence of __len__ might be a strong argument for providing the new repr, and its absence an indicator to avoid such. Robert Brewer MIS Amor Ministries fumanchu@amor.org From bob at redivi.com Tue Apr 6 22:13:40 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Apr 6 22:09:45 2004 Subject: [Python-Dev] More imformative iterator representations In-Reply-To: References: Message-ID: <2FAA1373-8839-11D8-84F3-000A95686CD8@redivi.com> On Apr 6, 2004, at 10:00 PM, Robert Brewer wrote: > Raymond Hettinger wrote: >> Q: Are there any other candidate output formats? >> A: Yes, we could add length information for a more mathematical style >> output: >> >> reversed(['h', 'g', 'f', ... to 8 objects]) > > The very presence of __len__ might be a strong argument for providing > the new repr, and its absence an indicator to avoid such. This is essentially getting back to what Armin was saying.. if iterators act more like lists, they will be easier to user and will open up some new doors for optimization potential. Adding a way to get the length of some iterators essentially makes them list-like or tuple-like. -bob From aahz at pythoncraft.com Tue Apr 6 22:28:02 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Apr 6 22:28:05 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200404060249.i362nwXn003846@cosc353.cosc.canterbury.ac.nz> References: <20040405175117.GA21587@mems-exchange.org> <200404060249.i362nwXn003846@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040407022802.GA17603@panix.com> On Tue, Apr 06, 2004, Greg Ewing wrote: > > Would it be unreasonable to define "." as meaning the current > *package*, rather than the current *module* -- i.e. the package itself > if the importer is a package, and the containing package if it's a > plain module? Not unreasonable, but I think it complicates the explanation. I'm basically -0 on having "." in __init__.py refer to the current package instead of the parent; we'll see what Guido says. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From tim.one at comcast.net Tue Apr 6 23:10:16 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue Apr 6 23:10:35 2004 Subject: [Python-Dev] Expert floats In-Reply-To: Message-ID: [Tim] >> I want marshaling of fp numbers to give exact (not approximate) >> round-trip equality on a single box, and across all boxes supporting >> the 754 standard where C maps "double" to a 754 double. [Ping] > That is a valuable property. I support it and support Python > continuing to have that property. That's good, since nobody opposes it . > I hope it has been made quite clear by now that this property does not > constrain how numbers are displayed by the interpreter in > human-readable form. The issue of choosing an appropriate string > representation of a number is unaffected by the desire for the above > property. ... > I think we *have* made progress. Now we can set aside the red-herring > issue of platform-independent serialization and focus on the real > issue: human-readable string representation. I don't think that's "the" real issue, but it is one of several. ... >> I'm the one who> has fielded most newbie questions about fp since >> Python's beginning, and I'm very happy with the results of changing >> repr() to produce 17 digits. > Now you are pulling rank. I'm relating my experience, which informs my beliefs about these issues more than any "head argument". > I cannot dispute your longer history and greater experience with Python; > it is something i greatly admire and respect. I also don't know your > personal experiences teaching Python. In person, mostly to hardware geeks and other hardcore software geeks. On mailing lists and newsgroups, to all comers, although I've had decreasing time for that as the years drag on. > But i can tell you my experiences. And i can tell you that i have > tried to teach Python to many people, individually and in groups. I > taught a class in Python at UC Berkeley last spring to 22 people who > had never used Python before. I maintained good communication with > the students and their feedback was very positive about the class. > > How did the class react to floating-point? Seeing behaviour like > this: > > >>> 3.3 > 3.2999999999999998 > >>> > > confused and frightened them, and continues to confuse and frighten > almost everyone i teach. Sorry, but so long as they stick to binary fp, stuff like that can't be avoided, even using "your rule" (other examples of that were posted today, and I won't repeat them again here). I liked Python's *former* rule myself(repr rounds to 12 significant digits), and would like it much better than "shortest possible" (which still shows tons of crap I usually don't care about) most days for my own uses. That's a real problem Python hasn't addressed: its format decisions are often inappropriate and/or undesirable (both can differ by app and by audience and by object type), and there are insufficient hooks for overriding these decisions. sys.displayhook goes a bit in that direction, but not far enough. BTW, if your students *remain* confused & frightened, it could be you're not really trying to explain binary fp reality to them. > (The rare exceptions are the people who have done lots of computational > work before and know how binary floating-point representations work.) > Every time this happens, the teaching is derailed and i am forced to go > into an explanation of binary floating-point to assuage their fears. Then they *don't* remain confused & frightened? Great. Then they've been educated. How long can it take to read the Tutorial Appendix? It's well worth however many years it takes . > Remember, i am trying to teach basic programming skills. How to solve > problems; how to break down problems into steps; what's a subroutine; > and so on. Aside from this floating-point thing throwing them off, > Python is a great first language for new programmers. This is not the > time to talk about internal number representation. Use Decimal instead. That's always been the best idea for newbies (and for most casual users of floating-point, newbie or not). > I am tired of making excuses for Python. I love to tell people about > Python and show them what it can do for them. But this floating-point > problem is embarrassing. People are confused because no other system > they've seen behaves like this. If you're teaching "basic programming skills", what other systems have they seen? Hand calculators for sure -- which is why they should use Decimal instead. Virually nothing about it will surprise them, except the liberating ability to crank up the precision. > Other languages don't print their numbers like this. Accounting > programs and spreadsheets don't print their numbers like this. I don't care -- really. I'm thoroughly in agreement with Kahan on this; see, e.g., section "QPRO 4.0 and QPRO for Windows" in http://www.cs.berkeley.edu/~wkahan/MktgMath.pdf ... the reader can too easily misinterpret a few references to 15 or 16 sig. dec of precision as indications that no more need be said about QPRO's arithmetic. Actually much more needs to be said because some of it is bizarre. Decimal displays of Binary nonintegers cannot always be WYSIWYG. Trying to pretend otherwise afflicts both customers and implementors with bugs that go mostly misdiagnosed, so ?fixing? one bug merely spawns others. ... The correct cure for the @ROUND and @INT (and some other) bugs is not to fudge their argument but to increase from 15 to 17 the maximum number of sig. dec. that users of QPRO may see displayed. But no such cure can be liberated from little annoyances: [snip things that make Ping's skin crawl about Python today] ... For Quattro?s intended market, mostly small businesses with little numerical expertise, a mathematically competent marketing follow- through would have chosen either to educate customers about binary floating-point or, more likely, to adopt decimal floating-point arithmetic even if it runs benchmarks slower. The same cures are appropriate for Python. > Matlab and Maple and Mathematica don't print their numbers like this. Those are designed for experts (although Mathematica pretends not to be). > Only Python insists on being this ugly. And it screws up the most common > way that people first get to know Python -- as a handy interactive > calculator. > > And for what? For no gain at all -- because when you limit your focus > to the display issue, the only argument you're making is "People > should be frightened." That's a pointless reason. Sorry, that's an absurd recharacterization, and I won't bother responding to it. If you really can't see any more to "my side" of the argument than that yet, then repeating it another time isn't going to help. So enough of this. In what time I can make for "stuff like this", I'm going to try to help the Decimal module along instead. Do what you want with interactive display of non-decimal floats, but do try to make it flexible instead of fighting tooth and nail just to replace one often-hated fixed behavior with another to-be-often-hated fixed behavior. ... > Not everyone runs into floating-point corner cases. In fact, very few > people do. Heh. I like to think that part of that has to do with the change to repr()! As I've said many times before, we *used* to get reports of a great variety of relatively *subtle* problems due to binary fp behavior from newbies; we generally get only one now, and the same one every time. They're not stupid, Ping, they just need the bit of education it takes to learn something about that expensive fp hardware they bought. > I have never encountered such a problem in my entire history of using > Python. Pride goeth before the fall ... > And if you surveyed the user community, i'm sure you would find that > only a small minority cares enough about the 17th decimal place for the > discrepancy to be an issue. The result of int() can change by 1 when the last bit changes, and the difference between 2 and 3 can be a disaster -- see Kahan (op. cit.) for a tale of compounded woe following from that one. Aahz's recent example of a loop going around one time more or less "than expected" used to be very common, and is the same thing in a different guise. It's like security that way: nobody gives a shit before they get burned, and then they get livid about it. If a user believes 0.1 is one tenth, they're going to get burned by it. > ... > You say it's better for people to get "bitten early". What's better: > everyone suffering for a problem that will never affect most of them, > or just those who care about the issue having to deal with it? The force of this is lost because you don't have a way to spare users from "unexpected extra digits" either. It comes with the territory! It's inherit in using binary fp in a decimal world. All you're really going on about is showing "funny extra digits" less often -- which will make them all the more mysterious when they show up. I liked the former round-to-12-digits behavior much better on that count. I expect to like Decimal mounds better on all counts except speed. From greg at cosc.canterbury.ac.nz Tue Apr 6 23:19:39 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Apr 6 23:19:55 2004 Subject: [Python-Dev] PEP 318: How I would implement decorators In-Reply-To: Message-ID: <200404070319.i373JdQF005829@cosc353.cosc.canterbury.ac.nz> Jim Hugunin : > I like Sam's proposal to add a '*' in front of the list. > > *[java_sig("public void meth(int a, int b)"), staticmethod] > def meth(self, a, b): > pass Would this have any implications if we were to decide to allow * in unpacking one day, i.e. a, b, *c = some_tuple It might become legal for an expression to start with *, then. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Tue Apr 6 23:52:29 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Apr 6 23:53:10 2004 Subject: [Python-Dev] More imformative iterator representations In-Reply-To: <2FAA1373-8839-11D8-84F3-000A95686CD8@redivi.com> Message-ID: <200404070352.i373qTwh005897@cosc353.cosc.canterbury.ac.nz> Bob Ippolito : > This is essentially getting back to what Armin was saying.. if > iterators act more like lists, they will be easier to user and will > open up some new doors for optimization potential. Adding a way to > get the length of some iterators essentially makes them list-like or > tuple-like. But... if an iterator is sufficiently random-access to be able to report its length and produce selected items without doing anything irreversible, wouldn't it make more sense for it to be a (possibly read-only) sequence rather than an iterator in the first place? In other words, instead of e.g. dict.iteritems() there should be a dict.itemseq() or something that returns a sequence-like view of the dict. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Tue Apr 6 23:58:56 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue Apr 6 23:59:14 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040407022802.GA17603@panix.com> Message-ID: <200404070358.i373wu6R005907@cosc353.cosc.canterbury.ac.nz> Aahz : > > Would it be unreasonable to define "." as meaning the current > > *package*, rather than the current *module* -- > > Not unreasonable, but I think it complicates the explanation. Not if the explanation simply says "'.' refers to the current package", and leaves the reader to deduce what "current package" means when the module is not a package. :-) [1] My intuition is that this will actually cause people less surprises even if they never think about this subtlety before diving in and using the feature. In fact they'll probably be more confused if they *do* think about it, either way... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ Footnotes: [1] A footnote explaining it would be allowed. From guido at python.org Wed Apr 7 00:14:21 2004 From: guido at python.org (Guido van Rossum) Date: Wed Apr 7 00:16:04 2004 Subject: [Python-Dev] Some comments on PEP 328 (absolute/relative imports) In-Reply-To: Your message of "Tue, 06 Apr 2004 21:52:51 EDT." <20040407015251.GA15148@panix.com> References: <1081257431.30379.22.camel@anthem.wooz.org> <20040406141615.GD27004@panix.com> <1081265903.30379.92.camel@anthem.wooz.org> <20040407015251.GA15148@panix.com> Message-ID: <200404070414.i374ELc02448@guido.python.org> > Done. Seems that the final remaining issue is the meaning of "." > inside __init__.py -- does it refer to the current package or the > parent? Guido, if you don't want to Pronounce, I'll post to c.l.py > requesting feedback. It ought to refer to the current package. After all, in __init__.py, relative imports (today) also work the same way as inside other modules of that package. __init__.py is funny that way (funny strange, not funny ha ha :-): its __dict__ namespace is that of the package, i.e. the parent of the package's submodules; but its location in the package tree is equivalent to that of a submodule of the package. That's just how it is, and once you think about it long enough that's the only way it *can* be. --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Wed Apr 7 00:29:47 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed Apr 7 00:29:56 2004 Subject: [Python-Dev] MS open sources WiX Message-ID: <407383BB.6050907@interlink.com.au> Relevant to MvL's new MSI installer work: http://blogs.msdn.com/robmen/archive/2004/04/05/107709.aspx Windows Installer XML (WiX) toolset has released as Open Source on SourceForge.net The Windows Installer Xml (WiX) toolset (pronounced ?wicks toolset?) appears to have finished propagating around the SourceForge.net CVS servers, so I can finally start writing this blog entry. As promised in my blog here, here, here, here, and here the WiX toolset and all of its source code has been released so that you can build Windows Installer databases (MSI and MSM files) the same way most groups inside Microsoft do. However, a funny thing happened on the way to the forum. WiX became the first project from Microsoft to be released under an OSS approved license, namely the Common Public License. Before everyone gets sidetracked by the Open Source implications, let?s talk about exactly what WiX is. WiX is a toolset composed of a compiler, a linker, a lib tool and a decompiler. The compiler, called candle, is used to compile XML source code into object files that contain symbols and references to symbols. The linker, called light, is fed one or more object files and links the references in the object files to the appropriate symbols in other object files. Light is also responsible for collecting all of the binaries, packaging them appropriately, and generating the final MSI or MSM file. The lib tool, called lit, is an optional tool that can be used to combine multiple object files into libraries that can be consumed by light. Finally, the decompiler, called dark, can take existing MSI and MSM files and generate XML source code that represents the package. -- Anthony Baxter It's never too late to have a happy childhood. From aahz at pythoncraft.com Wed Apr 7 00:53:18 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Apr 7 00:53:25 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200404070358.i373wu6R005907@cosc353.cosc.canterbury.ac.nz> References: <20040407022802.GA17603@panix.com> <200404070358.i373wu6R005907@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040407045318.GB173@panix.com> On Wed, Apr 07, 2004, Greg Ewing wrote: > Aahz : >> Greg: >>> >>> Would it be unreasonable to define "." as meaning the current >>> *package*, rather than the current *module* -- >> >> Not unreasonable, but I think it complicates the explanation. > > Not if the explanation simply says "'.' refers to the current > package", and leaves the reader to deduce what "current package" means > when the module is not a package. :-) [1] All right, let's try this: A leading dot indicates a relative import, starting with the current package. Two or more leading dots point to the parent(s) of the current package, one level per dot after the first. Then the examples should explain the meaning well enough. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From fumanchu at amor.org Wed Apr 7 01:27:55 2004 From: fumanchu at amor.org (Robert Brewer) Date: Wed Apr 7 01:30:03 2004 Subject: [Python-Dev] More imformative iterator representations Message-ID: Bob Ippolito wrote: > > Raymond Hettinger wrote: > >> Q: Are there any other candidate output formats? > >> A: Yes, we could add length information for a more > mathematical style > >> output: > >> > >> reversed(['h', 'g', 'f', ... to 8 objects]) > > > > The very presence of __len__ might be a strong argument for > providing > > the new repr, and its absence an indicator to avoid such. > > This is essentially getting back to what Armin was saying.. if > iterators act more like lists, they will be easier to user and will > open up some new doors for optimization potential. Adding a > way to get > the length of some iterators essentially makes them list-like or > tuple-like. Agreed. I don't mind the crux of Armin's idea, I was reacting more to applying such optimizations to enumerate() in particular. Given sufficient clarification like the above, it'd be easier to swallow. Robert Brewer MIS Amor Ministries fumanchu@amor.org From fumanchu at amor.org Wed Apr 7 01:50:00 2004 From: fumanchu at amor.org (Robert Brewer) Date: Wed Apr 7 01:52:07 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon Message-ID: [Guido van Rossum] > > ...while far from perfect, in > > terms of readability setting function attributes in a > decorator is so > > much better than setting them after the function has been defined, > > that I believe we have no choice but to provide it. [Me] > The more I read statements like this, the more I think of replacing: > > def foo(arg1, arg2): > pass > foo = classmethod(foo) > foo.attrA = 3 > > ...with code like: > > foo = classmethod() > foo.attrA = 3 > def foo(arg1, arg2): > pass > > Why would such additional features not be standard attributes of > function objects? >8 > All in all, I'd prefer that function decorators and attributes use > simple, known techniques like assignment and callables (as > shown above), > than become monolithic, crystallized all-in-one statements like: > > def foo(arg1 as str, arg2 as int) returns str [classmethod] > {attrA: 3}: > > I understand the desire to keep all those bits near each other on the > page, but stronger is my desire to make them independent > statements. If > we could bind a "blank" function object and supply its actual > codeblock > in a later statement, the rest falls magically into place, IMO. I sat down and banged out a metaclass solution, just to see if I *could* get the syntax I wanted in the example Shoe class, and to see what might be required behind the scenes. I put Shoe and all the decorator junk in one module to make the syntax look as clean as if Decorable, etc. were builtins. Basically, Decorable (the metaclass) looks at all attributes of the Shoe class; if there's an attribute "x" and an attribute "x_suite", then Shoe.x gets rebound as x(x_suite). The example provides classmethod and arbitrary function attributes. There are ways to clean it up, of course; this was a quick and dirty proof-of-concept. Rather than playing with "_suite", it should probably either reuse the same name "x" or use a token other than def. I also arbitrarily chose to allow a tuple or list for the decorator (but didn't demonstrate it). ----decorate.py---- import new class Decorable(type): def __init__(cls, name, bases, dct): type.__init__(cls, name, bases, dct) for attr in dct: if attr + "_suite" in dct: # Wrap the attr_suite in the attr decorator. inner_func = getattr(cls, attr + "_suite") decorator = getattr(cls, attr) if isinstance(decorator, (tuple, list)): wrapped_func = inner_func for item in decorator: wrapped_func = item.decorate(wrapped_func) else: wrapped_func = decorator.decorate(inner_func) setattr(cls, attr, wrapped_func) # [Optional] Unbind the inner function from the class. delattr(cls, attr + "_suite") class Decorator(object): def __init__(self): self.attributes = {} def __setattr__(self, key, value): if key == 'attributes': self.__dict__['attributes'] = value else: self.__dict__['attributes'][key] = value def decorate(self, function): for key, value in self.attributes.iteritems(): function.__setattr__(key, value) return function class clsmethod(Decorator): def decorate(self, function): newfunc = new.instancemethod(function, function.im_class(), function.im_class()) return super(clsmethod, self).decorate(newfunc) class Shoe(object): __metaclass__ = Decorable sized_shoe = clsmethod() sized_shoe.author = "Robert E Brewer" def sized_shoe_suite(cls, size): """Make a new Shoe with a 'size' attribute.""" newShoe = Shoe() newShoe.size = size return newShoe ----end decorate.py---- >>> import decorate >>> dir(decorate.Shoe) ['__class__', '__delattr__', '__dict__', '__doc__', '__getattribute__', '__hash__', '__init__', '__metaclass__', '__module__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__str__', '__weakref__', 'sized_shoe'] >>> decorate.Shoe.sized_shoe.author 'Robert E Brewer' >>> s = decorate.Shoe.sized_shoe(3) >>> s >>> s.size 3 Robert Brewer MIS Amor Ministries fumanchu@amor.org From python at rcn.com Wed Apr 7 05:16:25 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Apr 7 05:17:20 2004 Subject: [Python-Dev] More imformative iterator representations In-Reply-To: <200404070352.i373qTwh005897@cosc353.cosc.canterbury.ac.nz> Message-ID: <00b001c41c81$00620800$e841fea9@oemcomputer> > But... if an iterator is sufficiently random-access to be able to > report its length and produce selected items without doing anything > irreversible, wouldn't it make more sense for it to be a (possibly > read-only) sequence rather than an iterator in the first place? > > In other words, instead of e.g. dict.iteritems() there should be > a dict.itemseq() or something that returns a sequence-like view > of the dict. Trees, dicts, and sets all know their length and can iterate sequentially without having random access. PyDict_Next() makes it easy to find the first 3 keys, but there is no general purpose way of finding the nth key without looping over them all. Raymond From mcherm at mcherm.com Wed Apr 7 10:51:49 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Wed Apr 7 10:51:39 2004 Subject: [Python-Dev] Expert floats Message-ID: <1081349509.407415852f0c3@mcherm.com> Ka-Ping writes: > How did the class react to floating-point? Seeing behaviour like this: > > >>> 3.3 > 3.2999999999999998 > >>> > > confused and frightened them, and continues to confuse and frighten > almost everyone i teach. As well it should, until they understand it. I'll come back to this point in a moment. > Everything in Python -- everything in computers, in fact -- is a *model*. > We don't expect the model to be perfectly accurate or to be completely > free of limitations. IEEE 754 happens to be the prevalent model for > the real number line. We don't print every string with a message after > it saying "WARNING: MAXIMUM LENGTH 4294967296", and we shouldn't do > the same for floats. Now here is where I disagree with you quite strongly. I am an experienced programmer, and I *DO* expect my computer programs to be "perfectly accurate". If I calculate 217 + 349, I expect to get 566 each and every time. If it sometimes came out to be 567 instead, I would throw a fit! I also expect my program to be free of limitations. For instance, if I calculate 2147483640 + 9, I EXPECT to get 2147483649. In Python, I get 2147483649L instead... a minor wart that I can easily live with. If I am coding in C, I will instead get -2147483646, which is WAY off. I UNDERSTAND why this happens in C, and I am forced to live with it, but I don't LIKE or EXPECT it, and to be honest, trying to work around this limitation gives me headaches some days! For another example, I realize that if I work with a large enough data set it won't fit into my machine's memory. But I would be very disapointed if someone designed a programming language that limited data sizes because of this. When I have a problem, I expect to be able to go out and buy more memory (or upgrade my machine, my OS, or whatever component is the limiting factor) and have things "just work". And it's not just me... computer newbies have an even STRONGER belief that computers are "always accurate" than I do! They don't understand the degree to which human error and bugs can influence results, and most people figure that if the computer calculates some numbers, it's answer is guaranteed to be right. So how can I stand to use floating point numbers? Well, the answer is that I am aware of the limitations of floating point calculations. I would never hesitate to use floating point to plot a graph, and I would never choose to use floating point to perform financial calculations. I'm not sufficiently well-versed in the details of floating point behavior to always know when floating point would be acceptable to use, so I take a conservative approach and use it only when I know that it is safe to use. And I believe that this is exactly what your students ought to do. > Not everyone runs into > floating-point corner cases. In fact, very few people do. I have never > encountered such a problem in my entire history of using Python. And if > you surveyed the user community, i'm sure you would find that only a > small minority cares enough about the 17th decimal place for the > discrepancy to be an issue. That's simply untrue. Intel conducted (unintentionally) an experiment in just this question in 1994: try googling for "pentium bug". The conditions under which that bug affected results were far more obscure than the "corner cases" for floating point. And yet there was a huge hue and cry, and Intel was forced to replace tens of thousands of CPUs. Clearly, people DO care about tiny discrepencies. So, to come back to my point at the beginning, let me tell you what *I* say when teaching Python to beginners and they complain about the odd display of floating point in Python. I tell them that when computers work with decimals, they store things internally in Binary (everyone "knows" that computers work only in 1s and 0s), and this causes tiny rounding errors. I tell them that Python makes a point of SHOWING you those rounding errors, because it doesn't want to lie to you. Then, (depending on my audience) I usually follow up with a quick demo. I launch Microsoft Excel, and in the top cell I enter "-1". Then I enter "=1/9" (don't forget the "=" or Excel will treat it as a date) in each of the next nine cells. In the next cell I type "=sum(A1:A10)" but before pressing enter, I ask the class what the sum is. As soon as they're convinced it will be zero, I display the result. After that demo, people are usually much happier about Python which "looks funny", but "doesn't lie about its own mistakes". I firmly believe that ANYONE (even beginners) who uses floating point needs to understand that its not exact. They don't need to know the details, but they need to grasp the concept that a computer is NOT an Oracle, and this demonstration helps. -- Michael Chermside From kbk at shore.net Wed Apr 7 21:00:13 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Wed Apr 7 21:00:17 2004 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200404080100.i3810C18003532@hydra.localdomain> Patch / Bug Summary ___________________ Patches : 251 open ( +8) / 2364 closed ( +2) / 2615 total (+10) Bugs : 738 open ( +2) / 3972 closed (+12) / 4710 total (+14) RFE : 132 open ( +2) / 124 closed ( +0) / 256 total ( +2) New / Reopened Patches ______________________ Port tests to unittest (Part 2) (2003-05-13) http://python.org/sf/736962 reopened by doerwalter PEP 309 unit tests (2004-04-07) http://python.org/sf/931010 opened by Peter Harris struct.pack() on 64bit architectures (2004-03-30) http://python.org/sf/925932 opened by Aleksander Piotrowski Patch to setup.py to run on x86_64 Linux (2004-03-30) http://python.org/sf/926209 opened by Christopher Petrilli unichr(wide) error on ucs2 build (2004-03-31) CLOSED http://python.org/sf/926375 opened by Hye-Shik Chang Implementation for PEP 318 (Guido's version) (2004-03-31) http://python.org/sf/926860 opened by Guido van Rossum Option to propagate errors from pkgutil (2004-03-31) http://python.org/sf/927232 opened by Jp Calderone Improvements to Bluetooth support (2004-04-04) http://python.org/sf/929192 opened by Paul Clifford Remove ROT_FOUR (2004-04-04) http://python.org/sf/929502 opened by Michael Chermside PEP 309 reference implementation (2004-04-07) http://python.org/sf/931005 opened by Peter Harris PEP 309 LaTeX documentation (2004-04-07) http://python.org/sf/931007 opened by Peter Harris Patches Closed ______________ unichr(wide) error on ucs2 build (2004-03-31) http://python.org/sf/926375 closed by perky Updates to the Misc/RPM spec file. (2004-03-27) http://python.org/sf/924497 closed by loewis bugfix for [Bugs-850964](optparse) (2003-12-01) http://python.org/sf/851902 closed by fdrake New / Reopened Bugs ___________________ re.sub on u'' gives NON-unicode '' (2004-03-30) http://python.org/sf/926075 opened by Alex Martelli pdb sometimes sets breakpoints in the wrong location (2004-03-30) http://python.org/sf/926369 opened by Ilya Sandler socket timeouts + Ctrl-C don't play nice (2004-03-30) http://python.org/sf/926423 opened by David M. Cooke OEM codepage chars in mbcs filenames are misinterpreted (2004-03-30) CLOSED http://python.org/sf/926427 opened by Mike Brown (ref-manual) position docstrings in source not documented (2004-03-31) http://python.org/sf/926501 opened by a.hofkamp pdb sometimes sets breakpoints in the wrong location (2004-03-31) CLOSED http://python.org/sf/926807 opened by Ilya Sandler Overenthusiastic check in Swap? (2004-03-31) http://python.org/sf/926910 opened by benson margulies Python segfaults when freeing "deep" objects (2004-03-31) http://python.org/sf/927248 opened by Jp Calderone platform.libc_ver() fails on Cygwin (2004-04-02) http://python.org/sf/928297 opened by George Yoshida Python interpreter stalled on _PyPclose.WaitForSingleObject (2004-04-02) http://python.org/sf/928332 opened by Popov Typo in 7.2.1: socket.connect_ex (2004-04-02) CLOSED http://python.org/sf/928751 opened by PieOnCar Default encoding harmful (2004-04-03) CLOSED http://python.org/sf/928881 opened by Toni Goeller many os.path functions bahave inconsistently (2004-04-04) http://python.org/sf/929316 opened by Simon Percivall Python 2.3.3: socketmodule doesn't build on UnixWare 7.1.3 (2004-04-05) http://python.org/sf/929689 opened by Larry Rosenman os.path.realpath can't handle symlink loops (2004-04-05) http://python.org/sf/930024 opened by A.M. Kuchling RecenFilesList mysteriously containing str with NULL bytes (2004-04-07) http://python.org/sf/931336 opened by Gustavo Tabares Bugs Closed ___________ sdist ignores scripts argument in setup (2003-08-27) http://python.org/sf/796042 closed by fdrake urllib.urlencode doesn't work for output from cgi.parse_qs (2003-10-30) http://python.org/sf/833405 closed by myers_carpenter Python.h must be included first (2003-11-06) http://python.org/sf/837228 closed by fdrake OEM codepage chars in mbcs filenames can be misinterpreted (2004-03-31) http://python.org/sf/926427 closed by loewis pdb sometimes sets breakpoints in the wrong location (2004-03-31) http://python.org/sf/926807 closed by isandler os.py uses #' - undocumented magic? (2004-03-29) http://python.org/sf/925500 closed by montanaro Wrong reference for specific minidom methods (2003-10-29) http://python.org/sf/832251 closed by fdrake urlopen object's read() doesn't read to EOF (2003-04-21) http://python.org/sf/725265 closed by fdrake Section 13.1 HTMLParser documentation error (2003-08-23) http://python.org/sf/793702 closed by fdrake optparse: OptionParser.__init__'s "prog" argument ignored (2003-11-28) http://python.org/sf/850964 closed by fdrake Typo in 7.2.1: socket.connect_ex (2004-04-03) http://python.org/sf/928751 closed by nnorwitz Default encoding harmful (2004-04-03) http://python.org/sf/928881 closed by lemburg segmentation fault (2004-03-03) http://python.org/sf/909295 closed by tim_one New / Reopened RFE __________________ "with self:" statement (2004-04-01) http://python.org/sf/927543 opened by Thomas Dunne From janssen at parc.com Wed Apr 7 21:33:10 2004 From: janssen at parc.com (Bill Janssen) Date: Wed Apr 7 21:33:40 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Tue, 06 Apr 2004 21:53:18 PDT." <20040407045318.GB173@panix.com> Message-ID: <04Apr7.183319pdt."58612"@synergy1.parc.xerox.com> I really wish I'd been paying more attention to 328, which I've just read. This proposal seems to break lots and lots and lots of existing code. Maybe I'm reading it wrong. I've written many, many packages that use the form import foo where "foo" is a sibling module in the package. Or from foo import bar where, again, foo is a sibling to the current module. Packages that work like this have been submitted to on-line libraries, printed in books, etc., etc., etc. I really hope I'm reading the PEP wrong! Am I correct in thinking that come Python 2.4 or 2.5, these will just break if the package isn't at the top level (reachable from sys.path)? I was thinking (some months ago) that y'all would come up with an absolute form instead, something like from / import os or import /os or import /email.Message and a relative form like from foo import bar to do fuzzy and imperfect matching of the module (what we have now), or from ./foo import bar to do more explicit matching of siblings, or from ../baz.foo import bar or from -/baz.foo import bar to go up a level. And that the fuzzy and imperfect form would be eventually deprecated, but never removed. Bill From b.scharpf at tesionmail.de Wed Apr 7 15:46:14 2004 From: b.scharpf at tesionmail.de (Bertram Scharpf) Date: Wed Apr 7 21:40:17 2004 Subject: [Python-Dev] Python vs. chat (conversation with modem) Message-ID: Hi, in simple serial communication, I found a problem that doesn't arise, when I'm doing exactly the same with the `chat' program. I admit my hardware is a little bit special and may have problems itself. It is an ISDN telephone, Elmeg C200d@ta. With a normal modem, everything works fine. For a speech call, my telephone wants me to send a command 'at&b30' before dialing (otherwise the modem functionality is dialing). When I send the commands with `chat', the device is dialing, but when I send them from a Python script, the communication hangs. My code is (simplified): class Modem: def __init__( self, device = '/dev/ttyS0'): self.modem = open( device, 'w+') self.command( 'atz') def send( self, cmd): print >>self.modem, cmd + '\r' def expect( self, patt): import select ... def command( self, cmd): self.send( cmd) self.expect( 'OK') def dial_orig( self, nr): self.command( 'at&b30') # On this request, the modem won't send an answer. self.command( 'atdt,%s;h0z' % nr) self.command( 'at&b31') def dial_chat( self, nr): "That's working!?" script += [ repr( ''), 'atz', 'OK', r'at\&b30dt,\\T\;h0z', 'OK', r'\\c'] chatcmd = '/usr/sbin/chat -vs -T %s %s <%s >%s' % \ (nr, ' '.join( script), self.modem.name, self.modem.name) import os os.system( chatcmd) dial = dial_chat (The full script is available at `http://www.bertram-scharpf.de/dial.py'.) On the 'atz' an answer of 'OK' is sent back, but not on the 'at&b30' request. `chat' receives an OK. Setting the termcaps exactly to what chat would have chosen didn't help. I have found this behaviour in the versions 2.1, 2.2 and 2.3. $ uname -srmp Linux 2.4.18-bs1 i586 unknown Do I have any chance to find out what happens? Bertram -- Bertram Scharpf Stuttgart, Deutschland/Germany http://www.bertram-scharpf.de From tjreedy at udel.edu Wed Apr 7 22:55:36 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Wed Apr 7 22:55:40 2004 Subject: [Python-Dev] Re: Python vs. chat (conversation with modem) References: Message-ID: "Bertram Scharpf" wrote in message news:slrnc78mk6.46p.b.scharpf@homer.bertram-scharpf... > in simple serial communication, I found a problem that ... > I have found this behaviour in the versions 2.1, 2.2 and 2.3. ... This appears to be a usage question about current and past Python versions rather than anything related to the development of future versions. As such, it would be better posted to comp.lang.python or the corresponding mailing list. tjr From paul at prescod.net Thu Apr 8 01:01:56 2004 From: paul at prescod.net (Paul Prescod) Date: Thu Apr 8 01:07:26 2004 Subject: [Python-Dev] More imformative iterator representations In-Reply-To: <005101c41c3e$cdded760$e841fea9@oemcomputer> References: <005101c41c3e$cdded760$e841fea9@oemcomputer> Message-ID: <4074DCC4.4070106@prescod.net> Raymond Hettinger wrote: >... > An alternative to the read-eval-print approach is providing a custom > __repr__() method for the builtin iterators. The good news is that this > approach is very clean. And worth doing! A few questions: * will it be easy for end-users to write iterators that can repr nicely? Do you anticipate a base class or helper function to help? * might it make sense for enumerate to report something like: ...> If the underlying object's repr is helpful then enumerate at least might not discard the information. Paul Prescod From b.scharpf at tesionmail.de Thu Apr 8 05:04:51 2004 From: b.scharpf at tesionmail.de (Bertram Scharpf) Date: Thu Apr 8 05:05:00 2004 Subject: [Python-Dev] Re: Python vs. chat (conversation with modem) References: Message-ID: Hi Terry, Terry Reedy schrieb: > > "Bertram Scharpf" wrote in message > news:slrnc78mk6.46p.b.scharpf@homer.bertram-scharpf... >> in simple serial communication, I found a problem that ... >> I have found this behaviour in the versions 2.1, 2.2 and 2.3. ... > > This appears to be a usage question ... I thoroughly analysed the problem. I cannot totally exclude it's a usage question, but it made me wonder if it were. Sorry. Bertram -- Bertram Scharpf Stuttgart, Deutschland/Germany http://www.bertram-scharpf.de From martin at v.loewis.de Thu Apr 8 14:47:42 2004 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Apr 8 14:48:07 2004 Subject: [Python-Dev] MS open sources WiX In-Reply-To: <407383BB.6050907@interlink.com.au> References: <407383BB.6050907@interlink.com.au> Message-ID: <40759E4E.6050302@v.loewis.de> Anthony Baxter wrote: > Windows Installer XML (WiX) toolset has released as Open Source on > SourceForge.net Very interesting indeed. The challenge to any MSI generator is the ability to include user interface in a flexible yet easy-to-use manner. I'll take a look; off-hand, my criteria list for packaging tools evaluates like so: - avoid proprietary packagers: being open source, this is free of charge, and we can adjust it if we need to - support Win64: Don't know, probably not (but could be added given that it is open source) - support non-interactive installation, group policy: all done, thanks to MSI Regards, Martin From skip at pobox.com Thu Apr 8 17:29:27 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Apr 8 17:29:43 2004 Subject: [Python-Dev] PEP 318: Singleton decorator In-Reply-To: References: Message-ID: <16501.50231.895012.334330@montanaro.dyndns.org> (Still catching up. I guess no help with the substance of the PEP will be forthcoming, so I'll try to get to things as I can.) ... [snip] ... Ping> But i don't think this is a good way to write a singleton, because Ping> then the user of the class has to get instances by saying Ping> "MyClass" instead of "MyClass()". Better would be: ... [snip] ... Ping> I respectfully suggest that the PEP use the latter implementation Ping> in its singleton example instead. Done. Skip From skip at pobox.com Thu Apr 8 17:31:07 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Apr 8 17:31:22 2004 Subject: [Python-Dev] PEP 318: Singleton decorator In-Reply-To: <5.1.0.14.0.20040330095542.030435e0@mail.telecommunity.com> References: <5.1.0.14.0.20040330095542.030435e0@mail.telecommunity.com> Message-ID: <16501.50331.232161.858966@montanaro.dyndns.org> >> But i don't think this is a good way to write a singleton, because >> then the user of the class has to get instances by saying "MyClass" >> instead of "MyClass()". Phillip> That's a stylistic decision. IMO, it's more pythonic to *not* Phillip> call a constructor. One does not, after all, call modules in Phillip> order to "construct" them, and modules are a prime example of Phillip> singletons in Python. I think Ping's point is that if the user thinks Foo is a class, he's going to want to instantiate it like Foo(). Whether or not you get back a new instance or an existing one is sort of beside the point. It's simply a consistency issue. Skip From skip at pobox.com Thu Apr 8 17:34:55 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Apr 8 17:35:02 2004 Subject: [Python-Dev] PEP 318: Security use case In-Reply-To: References: Message-ID: <16501.50559.174096.437955@montanaro.dyndns.org> Ping> Here is an additional simple use case you could consider. Ping> def private(cls): Ping> def instantiate(*args, **kw): Ping> return cls(*args, **kw) Ping> return instantiate Ping> class DontTouch [private]: Ping> ... Ping> Inner scopes are one of the best places to hide things in Python; Ping> they are very difficult to get at. Are you using "private" to mean "can't be subclassed"? I'm not sure that's the correct term to use here. I guess I'm interpreting "private" in the C++ sense of the word. Skip From marktrussell at btopenworld.com Thu Apr 8 19:51:24 2004 From: marktrussell at btopenworld.com (Mark Russell) Date: Thu Apr 8 19:51:32 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <200404051803.i35I39U05548@guido.python.org> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> <200404051644.i35GiKD05216@guido.python.org> <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> Message-ID: <1081468284.2390.12.camel@localhost> On Mon, 2004-04-05 at 19:03, Guido van Rossum wrote: > > Would it not be possible to cheat and make the lexer transform > > > > "[" WHITESPACE "as" WHITESPACE IDENTIFIER > > > > into an LBRACE-AS token (which I assume would make the parser's job > > simple). > > I don't think without extreme hackery, but feel free to prove me wrong > by producing a patch. (A problem is, there could be newlines and > comments inside the WHITESPACE. Backtracking over that would require > the lexer to have an arbitrary-length buffer.) Unless I'm missing something (very possible) there's no need to backtrack over the whitespace - all you need is a two-entry stack for the "[" and the token that follows. There's no question of INDENT or DEDENT because we're inside a [] pair. I've put a patch at python.org/sf/932100 which does this, plus changes to Grammar/Grammar, compile.c etc - it implements the syntax: [as classmethod] def foo(cls): pass The newline is optional: [as foo, bar, baz] def foo(cls): pass is also legal. The test suite passes as before (there are still two failures but they are unrelated to this patch). The patch also includes your test_decorators.py, modified for the new syntax. Mark From skip at pobox.com Thu Apr 8 20:36:01 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Apr 8 20:36:05 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1081468284.2390.12.camel@localhost> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> <200404051644.i35GiKD05216@guido.python.org> <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <1081468284.2390.12.camel@localhost> Message-ID: <16501.61425.664065.876124@montanaro.dyndns.org> Mark> I've put a patch at python.org/sf/932100 which does this, plus Mark> changes to Grammar/Grammar, compile.c etc - it implements the Mark> syntax: Mark> [as classmethod] Mark> def foo(cls): pass Mark> The newline is optional: Mark> [as foo, bar, baz] def foo(cls): pass Mark> is also legal. Does it treat '[' and 'as' as separate tokens so whitespace can occur between them? Note that the second alternative is likely to break all sorts of auxiliary tools which scan Python source. etags/ctags and python-mode come to mind. Clearly, python-mode is well within our domain, but other syntax coloring tools aren't. Skip From python at rcn.com Thu Apr 8 22:37:18 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Apr 8 22:38:22 2004 Subject: [Python-Dev] Proposed iterator representations Message-ID: <000b01c41ddb$94296280$0507a044@oemcomputer> I checked in two itertool representations that were clear-cut: >>> from itertools import count, repeat >>> count(20) count(20) >>> repeat(None, 12) repeat(None, 12) For xrange iterators, there are a few choices: >>> it = iter(xrange(10, 20, 2)) iter(xrange(10, 20, 2)) or rangeiterator(10, 20, 2) or The first is my favorite. Though odd looking, it can be run through eval() and is completely self-documenting. The second has the virtue of using the actual type name but it cannot be run through eval(). The third is closer to the current format but adds some information. For dictionaries, lists, tuples, and reversed, there are similar choices but the eval() version can only be supported with a full listing of the parent object: >>> iter(dict(a=1, b=2, c=3, d=4, e=5)) iter(dict.fromkeys(['a', 'c', 'b', 'e', 'd'])) or >>> dict(a=1, b=2, c=3, d=4, e=5).iteritems() dict([('a', 1), ('c', 3), ('b', 2), ('e', 5), ('d', 4)]).iteritems() or >>> reversed('abcde') reversed(['a', 'b', 'c', 'd', 'e']) or What do you guys think? Raymond Hettinger From bac at OCF.Berkeley.EDU Fri Apr 9 02:18:28 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Fri Apr 9 02:18:34 2004 Subject: [Python-Dev] Proposed iterator representations In-Reply-To: <000b01c41ddb$94296280$0507a044@oemcomputer> References: <000b01c41ddb$94296280$0507a044@oemcomputer> Message-ID: <40764034.6000306@ocf.berkeley.edu> Raymond Hettinger wrote: > For xrange iterators, there are a few choices: > > >>>>it = iter(xrange(10, 20, 2)) > > iter(xrange(10, 20, 2)) > or > rangeiterator(10, 20, 2) > or > > > The first is my favorite. Though odd looking, it can be run through > eval() and is completely self-documenting. The second has the virtue of > using the actual type name but it cannot be run through eval(). The > third is closer to the current format but adds some information. > Third. If I am at the interpreter I want information presented to me in the best way possible. If I need to type it in again I have readline and a command history for that. =) > > For dictionaries, lists, tuples, and reversed, there are similar choices > but the eval() version can only be supported with a full listing of the > parent object: > > > What do you guys think? > As mentioned above, I want the information presented in the most informative way and not worry about the repr output. -Brett From marktrussell at btopenworld.com Fri Apr 9 05:44:03 2004 From: marktrussell at btopenworld.com (Mark Russell) Date: Fri Apr 9 05:44:39 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <16501.61425.664065.876124@montanaro.dyndns.org> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> <200404051644.i35GiKD05216@guido.python.org> <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <1081468284.2390.12.camel@localhost> <16501.61425.664065.876124@montanaro.dyndns.org> Message-ID: <1081503842.2391.8.camel@localhost> On Fri, 2004-04-09 at 01:36, Skip Montanaro wrote: > Does it treat '[' and 'as' as separate tokens so whitespace can occur > between them? Yes. > Note that the second alternative is likely to break all sorts of auxiliary > tools which scan Python source. etags/ctags and python-mode come to mind. > Clearly, python-mode is well within our domain, but other syntax coloring > tools aren't. It's a trivial change to disallow the second form - I'm pretty much 50/50 on whether it's a good idea to allow it or not. Probably "There's only one way to do it" should apply. (The change is trivial - in Grammar/Grammar, just replace [NEWLINE] with NEWLINE in the decorators entry). Mark From barry at python.org Fri Apr 9 09:10:23 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 9 09:10:32 2004 Subject: [Python-Dev] Re: PEP 318: Decorators last before colon In-Reply-To: <1081503842.2391.8.camel@localhost> References: <5.1.0.14.0.20040404233506.03b08210@mail.telecommunity.com> <200404050433.i354XcF03700@guido.python.org> <1081156611.2377.6.camel@localhost> <200404051644.i35GiKD05216@guido.python.org> <1081184434.4045.3.camel@localhost> <200404051803.i35I39U05548@guido.python.org> <1081468284.2390.12.camel@localhost> <16501.61425.664065.876124@montanaro.dyndns.org> <1081503842.2391.8.camel@localhost> Message-ID: <1081516222.5071.132.camel@anthem.wooz.org> On Fri, 2004-04-09 at 05:44, Mark Russell wrote: > It's a trivial change to disallow the second form - I'm pretty much > 50/50 on whether it's a good idea to allow it or not. Probably "There's > only one way to do it" should apply. (The change is trivial - in > Grammar/Grammar, just replace [NEWLINE] with NEWLINE in the decorators > entry). Personally, I think it's fine to allow them to be on the same line. I wouldn't want to arbitrarily disallow it because we're worried about how the tools will react. If people gravitate toward this syntax because it's more readable, then the tools will eventually adjust. just-because-you-can-doesn't-mean-you-should-ly y'rs, -Barry From aahz at pythoncraft.com Fri Apr 9 11:42:52 2004 From: aahz at pythoncraft.com (Aahz) Date: Fri Apr 9 11:42:58 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <04Apr7.183319pdt."58612"@synergy1.parc.xerox.com> References: <20040407045318.GB173@panix.com> <04Apr7.183319pdt."58612"@synergy1.parc.xerox.com> Message-ID: <20040409154252.GB28808@panix.com> On Wed, Apr 07, 2004, Bill Janssen wrote: > > I really wish I'd been paying more attention to 328, which I've just > read. This proposal seems to break lots and lots and lots of existing > code. Maybe I'm reading it wrong. > > I've written many, many packages that use the form > > import foo > > where "foo" is a sibling module in the package. You won't be able to do that anymore. > Or > > from foo import bar > > where, again, foo is a sibling to the current module. Packages that > work like this have been submitted to on-line libraries, printed in > books, etc., etc., etc. You'll have to change these to from .foo import bar > I really hope I'm reading the PEP wrong! Am I correct in thinking > that come Python 2.4 or 2.5, these will just break if the package > isn't at the top level (reachable from sys.path)? They won't break until Python 2.6. Python 2.5 will issue a DeprecationWarning when it finds relative imports. > I was thinking (some months ago) that y'all would come up with an > absolute form instead, something like > > from / import os Nope. Absolute imports are the most common form, and we do want to encourage absolute imports whenever possible (because of the conflicts with the stdlib). Relative imports should be "marked". -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From tjreedy at udel.edu Fri Apr 9 14:23:15 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Fri Apr 9 14:23:19 2004 Subject: [Python-Dev] Re: Proposed iterator representations References: <000b01c41ddb$94296280$0507a044@oemcomputer> Message-ID: "Raymond Hettinger" wrote in message news:000b01c41ddb$94296280$0507a044@oemcomputer... > I checked in two itertool representations that were clear-cut: ;-) > >>> from itertools import count, repeat > >>> count(20) > count(20) Is the representation (of the dynamic object) static? Does it=count(20); it.next(), it.next(); it still print 'count(20)' as it one was but no longer is? This seems much like lt=iter([1,2,3]); lt.next(); lt.next(); lt printing iter([1,2,3]) instead of iter([3]), or worse, lt=[1,2,3]; lt.pop(); lt.pop(); lt printing [1,2,3] instead of [1]. Terry J. Reedy From python at rcn.com Fri Apr 9 14:49:07 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Apr 9 14:50:19 2004 Subject: [Python-Dev] Re: Proposed iterator representations In-Reply-To: Message-ID: <012001c41e63$5799a860$e435c797@oemcomputer> > Is the representation (of the dynamic object) static? No, it is dynamic. > Does > > it=count(20); it.next(), it.next(); it > > still print 'count(20)' as it one was but no longer is? That prints "count(22)". Raymond "... as a general rule of thumb, the first time a memory location is accessed incurring a cache miss, it will be at least as expensive as a divide operation, and additional accesses to the same location are roughly free." -- The Software Optimization Cookbook, Intel Press From eppstein at ics.uci.edu Fri Apr 9 16:43:29 2004 From: eppstein at ics.uci.edu (David Eppstein) Date: Fri Apr 9 16:43:33 2004 Subject: [Python-Dev] Re: Proposed iterator representations References: <012001c41e63$5799a860$e435c797@oemcomputer> Message-ID: In article <012001c41e63$5799a860$e435c797@oemcomputer>, "Raymond Hettinger" wrote: > > Does > > > > it=count(20); it.next(), it.next(); it > > > > still print 'count(20)' as it one was but no longer is? > > That prints "count(22)". Speaking of itertools.count, where can I find the rationale for the "Does not currently support python long integers." restriction? This bit me the last time I needed count (part of an implementation of the greedy algorithm for Egyptian fractions, so I really needed longs) and I ended up having to roll my own. Not that it was especially difficult to roll my own, but it seems like it would have been harder to put that check in than to just allow longs to work and I don't understand why it was done that way. I tried googling the python-dev archives but no luck. -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From python at rcn.com Fri Apr 9 18:06:38 2004 From: python at rcn.com (Raymond Hettinger) Date: Fri Apr 9 18:07:44 2004 Subject: [Python-Dev] Re: Proposed iterator representations In-Reply-To: Message-ID: <019a01c41e7e$ee96f680$e435c797@oemcomputer> [David Eppstein] > Speaking of itertools.count, where can I find the rationale for the > "Does not currently support python long integers." restriction? The rationale was writing "the simplest thing that could possibly work". At the time, there were no use cases that warranted additional code complexity. I wanted this to be bug free right out of the box. In the docs, I included a cut and pastable pure python version that did support long integers so that no one would be left in the cold. As you said, rolling your own is trivial. Also, the design was also affected by trying to fit itertools.count() into the tiny ecological niche between xrange() and itertools.repeat(). Its competitive advantage was being lightweight, fast, and infinite so you could write things like izip(count(10), data). Raymond "... as a general rule of thumb, the first time a memory location is accessed incurring a cache miss, it will be at least as expensive as a divide operation, and additional accesses to the same location are roughly free." -- The Software Optimization Cookbook, Intel Press From arigo at tunes.org Sat Apr 10 07:22:17 2004 From: arigo at tunes.org (Armin Rigo) Date: Sat Apr 10 07:25:45 2004 Subject: [Python-Dev] Proposed iterator representations In-Reply-To: <000b01c41ddb$94296280$0507a044@oemcomputer> References: <000b01c41ddb$94296280$0507a044@oemcomputer> Message-ID: <20040410112217.GA3001@vicky.ecs.soton.ac.uk> Hello Raymond, On Thu, Apr 08, 2004 at 10:37:18PM -0400, Raymond Hettinger wrote: > I checked in two itertool representations that were clear-cut: > > >>> from itertools import count, repeat > >>> count(20) > count(20) > >>> repeat(None, 12) > repeat(None, 12) Another point of view is that these representation are not very informative if you don't already know count() and repeat(): someone will probably type the above code in order to experiment with them, and the repr doesn't help in this respect. I think I'd prefer a more standard representation for all the iterators for which it is reasonably possible to do so. >>> count(20) >>> repeat(None, 12) >>> iter(xrange(10, 20, 2)) A bit of effort should be made at showing the nicest answer, e.g.: >>> iter(xrange(0)) >>> iter(xrange(1)) >>> iter(xrange(2)) >>> iter(xrange(3)) >>> iter(xrange(4)) >>> iter(xrange(5)) Armin From janssen at parc.com Sat Apr 10 12:38:36 2004 From: janssen at parc.com (Bill Janssen) Date: Sat Apr 10 12:39:01 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Fri, 09 Apr 2004 08:42:52 PDT." <20040409154252.GB28808@panix.com> Message-ID: <04Apr10.093843pdt."58612"@synergy1.parc.xerox.com> > Nope. Absolute imports are the most common form, and we do want to > encourage absolute imports whenever possible (because of the conflicts > with the stdlib). Relative imports should be "marked". Wow. This seems amazingly user-hostile. Users (not Python developers) download some two or three year old code to do something from an online library, like the Vaults of Parnassus, and it just breaks because the meaning of "import" has changed. I hope we aren't developing the kind of attitude towards non-developer users that made lispers call them "lusers". Bill From ark at acm.org Sat Apr 10 12:57:04 2004 From: ark at acm.org (Andrew Koenig) Date: Sat Apr 10 12:57:01 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <04Apr10.093843pdt."58612"@synergy1.parc.xerox.com> Message-ID: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> > Wow. This seems amazingly user-hostile. Users (not Python > developers) download some two or three year old code to do something > from an online library, like the Vaults of Parnassus, and it just > breaks because the meaning of "import" has changed. I hope we aren't > developing the kind of attitude towards non-developer users that made > lispers call them "lusers". This is a tougher issue than it appears, as I have learned from my long experience with C++. The trouble is that if you commit yourself to backward compatibility, you wind up with a growing list of decisions for which the default is wrong but you didn't want to break code by changing it. Each time you make such a decision, you make it more difficult the next time to do anything but go for compatibility again. Ultimately, everyone has to remember to override the defaults in lots of everyday circumstances, a mental burden that hurts everyone from novices to experts. On the other hand, if you don't commit yourself to backward compatibility, you wind up with people downloading old code that breaks because things have changed. The third possibility is to support all alternatives forever, perhaps with a mechanism such as "from __past__ import foo". That's possible in theory, but if changes include changes in semantics, it may be impossible for old and new to coexist. Moreover, even if they can coexist, testing becomes a nightmare. It's a trilemma--a choice between three unpleasant alternatives. I don't know the right answer, but I do wish to caution against assuming the problem is trivial. From aahz at pythoncraft.com Sat Apr 10 14:33:10 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 10 14:33:48 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <04Apr10.093843pdt."58612"@synergy1.parc.xerox.com> References: <20040409154252.GB28808@panix.com> <04Apr10.093843pdt."58612"@synergy1.parc.xerox.com> Message-ID: <20040410183310.GA27946@panix.com> On Sat, Apr 10, 2004, Bill Janssen wrote: >Aahz: >> >> Nope. Absolute imports are the most common form, and we do want to >> encourage absolute imports whenever possible (because of the conflicts >> with the stdlib). Relative imports should be "marked". > > Wow. This seems amazingly user-hostile. Users (not Python > developers) download some two or three year old code to do something > from an online library, like the Vaults of Parnassus, and it just > breaks because the meaning of "import" has changed. I hope we aren't > developing the kind of attitude towards non-developer users that made > lispers call them "lusers". In addition to Andrew's excellent response, I want to point out that this only affects packages; most of the distributed code that I've seen tends to be in single modules. It also only affects packages that rely on relative imports: as Barry points out, there's a significant set of developers that currently uses only absolute imports because of the tricky semantics for relative imports. Also, by the time imports start to fail, what is currently two or three year-old code will be five or six years old. Anyone who is writing code now will be aware of this issue and should be able to future-proof their code if they want. Nobody should expect to download code that's five or six years old and have it just work. Do you still think this is a "user-hostile" attitude? If yes, you'll need to make your case in more detail. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From janssen at parc.com Sat Apr 10 15:26:56 2004 From: janssen at parc.com (Bill Janssen) Date: Sat Apr 10 15:27:16 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 10 Apr 2004 09:57:04 PDT." <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> Message-ID: <04Apr10.122659pdt."58612"@synergy1.parc.xerox.com> > I do wish to caution against assuming the problem is trivial. I've been down this road myself, and I think I understand the problem. However, whenever I've made changes of this break-itude, I've always done it in a major release. Changing things this way in a minor release just seems hostile to me. Remember that one of the major reasons people use Python (instead of, say, Scheme) is the availability of lots of packages written in Python to do various things. Making those packages buggy is a suboptimal way to win friends and influence people. Bill From janssen at parc.com Sat Apr 10 15:44:28 2004 From: janssen at parc.com (Bill Janssen) Date: Sat Apr 10 15:44:52 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 10 Apr 2004 11:33:10 PDT." <20040410183310.GA27946@panix.com> Message-ID: <04Apr10.124434pdt."58612"@synergy1.parc.xerox.com> > In addition to Andrew's excellent response, I want to point out that this > only affects packages; most of the distributed code that I've seen tends > to be in single modules. It also only affects packages that rely on > relative imports: as Barry points out, there's a significant set of > developers that currently uses only absolute imports because of the > tricky semantics for relative imports. It would be interesting to have a survey. You'd want to have usage statistics in there, as well. That is, xx% of packages use relative imports; these packages account for xx% of downloads. > Also, by the time imports start to fail, what is currently two or three > year-old code will be five or six years old. Anyone who is writing code > now will be aware of this issue and should be able to future-proof their > code if they want. Nobody should expect to download code that's five or > six years old and have it just work. I expressed myself poorly; I was thinking of packages written in the future, so they'd still be two or three years old. "Anyone who is writing code now will be aware..." is an unfortunate example of ivory-tower thinking. Many people who are writing code using a wonderful tool like Python have better things to do than keep up with the vagaries of Python releases. They rightly feel that if they learned Python with 2.2, what they know should be pretty stable at least until Python 3.0 comes out. They don't think that in a minor release, Python import semantics will change out from under them. Every few years, when they find some time, they will check out some of the new features, like generators, that the developers find so delightful. But in between those times, they won't be clueful. Most Python developers may be Python users, but most Python users are not Python developers. > Nobody should expect to download code that's five or > six years old and have it just work. One of the attractive features of Python has always been that 5-year-old code *does* just work. That's because the design was largely right from the start, and the designers have been careful to keep things backwards-compatible. > Do you still think this is a "user-hostile" attitude? If yes, you'll > need to make your case in more detail. I think users unfamiliar with the Python development process will perceive Python as being hostile to their concerns if things like this keep happening. I wouldn't attribute any hostility to the developers; it's just a mistake. Bill From fumanchu at amor.org Sat Apr 10 18:14:17 2004 From: fumanchu at amor.org (Robert Brewer) Date: Sat Apr 10 18:16:31 2004 Subject: [Python-Dev] PEP 321 and Gustavo's DateUtil Message-ID: There was some discussion about Gustavo Niemeyer's DateUtil module back in March: http://aspn.activestate.com/ASPN/Mail/Message/python-dev/2022361 ...and an immediate question was whether it fulfilled (or exceeded ;) the requirements of PEP 321: http://www.python.org/peps/pep-0321.html I looked at the dateutil.parser.py module today in some detail; with a few slight modifications, it could very well meet PEP 321. The issues as I see them: 1. I am *not* proposing the whole dateutil module be accepted. Recurrence and relativedelta brought up too many unresolved questions among developers back in March. Therefore, if dateutil.parser were to be considered, it should stand alone, and be named independently, for example, "dateparse" or "dateparser", IMO a sibling of the datetime module, not a child. 2. If it were to stand on its own, it only needs about a dozen lines modified to reduce dependency on: dateutil.relativedelta (1 mention, line 304): ret = ret+relativedelta.relativedelta(weekday=res.weekday) ...which could be replaced inline by simply distilling the relevant chunk of relativedelta.__radd__. The only other dependency is on: dateutil.tz (5 mentions, lines 314-326), for example: ret = ret.replace(tzinfo=tz.tzutc()) ...every use of which simply returns a concrete subclass of datetime.tzinfo. To resolve this, we could: a) include dateutil.tz in the library as its own module, "timezones", or b) include dateutil.tz as "datetime.timezones", or c) drop the 5 necessary tz classes into the new "dateparse" module, or d) strip that particular functionality from dateparse, making it effectively tz-naive. 3. Locale-dependent data is nicely separated into a "parserinfo" class, which might benefit from built-in subclasses for major languages; currently, the default is English. There are minor cleaning issues, but none of them are show-stoppers: 4. dateutil.parser imports os.path and sys but doesn't reference them. 5. _resultbase uses __slots__ for some odd reason. 6. There's a mix of old- and new-style classes. I haven't see any reason in the code not to make them all new-style. 7. Some PEP 8 conventions, like CapWord class names, might be nice. In all, I think we could finish up yet another PEP and get some solid, long-desired functionality out to the Python community with a minimum of effort and even less breakage, since Gustavo's done all the hard work already. ;) Thoughts? Robert Brewer MIS Amor Ministries fumanchu@amor.org From paul at prescod.net Sat Apr 10 18:55:15 2004 From: paul at prescod.net (Paul Prescod) Date: Sat Apr 10 18:57:49 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <04Apr10.124434pdt."58612"@synergy1.parc.xerox.com> References: <04Apr10.124434pdt."58612"@synergy1.parc.xerox.com> Message-ID: <40787B53.9070007@prescod.net> Bill Janssen wrote: >>Also, by the time imports start to fail, what is currently two or three >>year-old code will be five or six years old. Anyone who is writing code >>now will be aware of this issue and should be able to future-proof their >>code if they want. Nobody should expect to download code that's five or >>six years old and have it just work. > > > I expressed myself poorly; I was thinking of packages written in the > future, so they'd still be two or three years old. > > "Anyone who is writing code now will be aware..." is an unfortunate > example of ivory-tower thinking. Many people who are writing code > using a wonderful tool like Python have better things to do than keep > up with the vagaries of Python releases. They rightly feel that if > they learned Python with 2.2, what they know should be pretty stable > at least until Python 3.0 comes out. Most people in the Python community know that Python doesn't really have "minor releases" other than bugfix releases. Just like Linux and Perl, the choice to switch to a big version number is more marketing than anything. (and it says something about our marketing saavy that we've done it exactly once) > ... They don't think that in a minor > release, Python import semantics will change out from under them. > Every few years, when they find some time, they will check out some of > the new features, like generators, that the developers find so > delightful. But in between those times, they won't be clueful. It really just boils down to a question of how much time you give people. First you try to publicize the change via books, tutorials, mailing lists etc. Then you add an annoying warning. Then eventually you break the old code. If you move slow enough then I claim you can make an arbitrarily deep and pervasive change without significant user pain. For example consider the shift over the years from K&R C to ANSI C to C++. It wasn't totally painful but it was never so painful as to make people switch languages on the basis of backwards incompatibility pain. And there were some pretty major changes in there...the deprecation of implicit casting of void * being one that pops to mind. > One of the attractive features of Python has always been that > 5-year-old code *does* just work. That's because the design was > largely right from the start, and the designers have been careful to > keep things backwards-compatible. 5-year-old code OFTEN just works. But not always. Each version of Python has a document that describes incompatibilities. http://www.python.org/1.6/ http://www.python.org/2.2.3/bugs.html e.g. "The special attributes __members__ and __methods__ are no longer supported (for most built-in types). Use the new and improved dir() function instead." Admittedly this relative import change is a big one compared to those from the past. My opinion is that that just means it should be done a very long time after the version with the warning ships. My opinion is that the process is correct and it is just a question of whether the timelines are generous enough or not. Somewhere between "Python 2.6" and "never" there must be a point where people have had enough time to change over their code after getting nagged by the DeprecationWarning. Paul Prescod From bac at OCF.Berkeley.EDU Sat Apr 10 22:27:36 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sat Apr 10 22:27:52 2004 Subject: [Python-Dev] python-dev Summary for 2004-03-16 through 2004-03-31 [rough draft] Message-ID: <4078AD18.5070502@ocf.berkeley.edu> Don't plan on sending this summary out for a while since it is so long (over 8 pages if TextEdit is to be believed) and because of the volume of emails I had to go based on memory on some of the summaries and thus may have missed some of the discussions in the large threads. Anyway, reply with any corrections you have and such. --------------------------------- ===================== Summary Announcements ===================== Still looking for a summer job or internship. Please let me know if have one at your company or happen to know of one somewhere. And yes, this message will go away; either I will actually get a summer job or the summer will come around. =) PyCon happened during the timespan this summary covers. Unlike last year when emails to python-dev actually went down, there was a good amount of traffic at the conference. A good portion of this was due to the sprint on Python's core that took place. A bunch of core developers showed up for the whole weekend and used part of that time to answer emails. Great for python-dev but made my life a little difficult. =) But the sprint overall was a definite gain with various tweaks done, improvements to the profiler, and a bunch of bugs and patches closed (actually had the number of open patches and bugs have a net loss of 7 and 10 from that week, respectively; that has not happened in a while). The release schedule (and thus what is definitely planned and what can go in if it gets finished in time) was set while at PyCon. Read the summary entitled `Python 2.4 release schedule`_ below for details and links. ========= Summaries ========= ---------------------------- "I want to be like Raymond!" ---------------------------- So, you want to speed up Python like Raymond Hettinger has? Well, he posted an email listing various optimizations he has on his list to try out. That specific email can be found at http://mail.python.org/pipermail/python-dev/2004-March/043276.html This also brought back the debate of whether pure Python modules should be recoded in C for performance. See previous threads for a longer discussion on the subject. Contributing threads: - `Joys of Optimization `__ - `todo (was: Joys of Optimization) `__ -------------- Decimal update -------------- `PEP 327`_ was revised to make certain issues more explicit and to discuss possible function naming (summarized below). This lead to a discussion about how to name various things. Although there are historical violations of the coding standard, the names should follow `PEP 7`_ and `PEP 8`_. The testing suite is also finished along with the PEP essentially being finished. .. _PEP 327: http://www.python.org/peps/pep-0327.html Contributing threads: - `Changes to PEP 327: Decimal data type `__ - `PEP 327: Decimal data type `__ -------------------------------------------------------------------------- Follow the coding standard, people -------------------------------------------------------------------------- Discussing naming conventions came up while discussing PEP 327. The basic conclusion is that older modules don't follow the coding standards laid out in `PEP 7`_ and `PEP 8`_, but that does not mean new code should be allowed to break the rules. .. _PEP 7: http://www.python.org/peps/pep-0007.html .. _PEP 8: http://www.python.org/peps/pep-0008.html Contributing threads: - `Changes to PEP 327: Decimal data type `__ - `(class) module names clarification `__ - `(old) module names `__ ---------------------------------------------------- "It depends on what the meaning of the word 'is' is" ---------------------------------------------------- The idea of redefining how 'is' operates when comparing immutable types came up. It was suggested that when two immutable objects of the same type were being compared that it should be an equivalence test instead of an identity test. For instance:: x = 99 y = 99 x is y is true while:: x = 100 y = 100 x is y is false. The point was made, though, that unless you are comparing to None that 'is' should not be used for comparing immutables and instead the equality operator ('==') should be used instead. There is backwards-compatibility issues with changing this. Guido pronounced that 'is' ain't changing. Contributing threads: - `A proposal has surfaced on comp.lang.python to redefine "is" `__ - `redefining is `__ - `(not) redefining is `__ --------------------------------- Getting PyChecker into the stdlib --------------------------------- The idea of adding PyChecker_ to the stdlib came up once again. Support seems to be there but the developers of PyChecker did not speak up. PyChecker 2 is under development and that changes how PyChecker does things significantly (analyzes bytecode directly) which might be why they were mum on the topic. .. _PyChecker: http://pychecker.sf.net/ ------------------------------- Simplifying logging.basicConfig ------------------------------- Simplifying basicConfig in the logging package to making logging to a file simpler was proposed. One person suggested more examples in the documentation ala unittest. Another person disagreed. No change has been made at this point nor any specific statement as to whether any will be. Contributing threads: - `Some changes to logging `__ -------------------------- A new committer amongst us -------------------------- Phillip J. Eby gained CVS commit privileges to help with development of Distutils. But Phillip has been active on python-dev for a while so I am sure he can be put to work on some other things as well. =) Contributing threads: - `new Python CVS committer `__ ---------------- Working on HP-UX ---------------- Cameron Laird is still interested in working on the port of Python on HP-UX. If you are interested in helping out please contact him. Contributing threads: - `Portability: update and oblique apology `__ ------------------------------------------- Binding variables for generator expressions ------------------------------------------- What would you expect in this situation using generator expressions?:: gen_exps = [] for cnt in range(10): gen_exps.append(x**2 for x in range(cnt)) If you thought that gen_exps would contain 10 generators that each return a list longer than the one previous then you like capture binding for generator expressions (capturing the values of all free variables used in the generator expression at definition time). If you though there would be 10 generators that all had the power of 2 from 0 to 9 then you like late binding (using the value of the free variables in the generator expression at execution time of the generator). Guido brought this up in his keynote for PyCon_ and said that he preferred late binding (don't remember if he pronounced; I could lie and say I was under the influence of a hang-over from a drunken stupor from the night before but I actually didn't go drinking once this year at PyCon and so my excuse is I just don't plain remember). The reasoning is that there are no surprises in terms of corner cases and having more obvious values for the free variables. The point about performance of generator expressions was also brought up using the now-standard ``sum(x**2 for range(1000))`` example. Using a list comprehension was actually slightly faster (less than 10%) then the generator expression. This is partially thanks to Raymond's tweaking of list comprehensions by adding the LIST_APPEND opcode. The rest is due to generator expressions creating an actual generator and thus having to create eval frames and playing with them. But it was pointed out that if the number passed into 'range' is raised then generator expressions quickly end up winning because the execution fits in the cache better, especially if xrange is used. Contributing threads: - `An issue recently brought up in patch #872326 `__ - `Possible resolution of generator expression variable... `__ - `genexps slow? `__ -------------------------------------------------- More than you ever wanted to know about decorators -------------------------------------------------- Decorators were the topic for the latter half of March. Initially the discussion focused on where to put the list of decorators (if Guido's patch as found at http://www.python.org/sf/926860 is any indication he has a preference with the decorators coming on the line above the function/method definition) should go. With C# having a similar functionality as what is being proposed roughly following C#'s syntax was considered a good thing, albeit tweaked so that the decorator list goes on its own line (reason being the LL(1) parser is too dumb to handle it coming before the 'def' on the same line and it alleviates the issue of having a long decorator list that would a full line anyway. With the winning syntax, though, there is the issue of preexisting code and the work at interpreter prompt. The former is a problem for code that happens to have a list that is not assigned to anything before a function definition; why anyone would have a list defined that is not assigned is beyond me. As for the latter issue, that is a slight problem. To get around it you can stick the definition in an 'if' statement:: if True: [decorate, this] def marthastewart(): pass Slight pain, but it has been generally accepted that placing decorators on their own line is just so much easier to deal with aesthetically this minor inconvenience has been viewed as acceptable. After PyCon Guido talked to some people who suggested having another way of tacking on metadata separate from using descriptors, partially since C#'s version of decorators is mostly for metadata. But tacking on attributes to a function can be done using the proposed decorators by defining some auxiliary function that takes in named arguments that specify the name of the attribute and the value to set it and returns a callable that will take in a function and assign the attributes. This also brought up the discussion of setting function attributes from within a function since decorators will most likely be used to set metadata. The idea of having attributes set for the function if a name starts with '.' was brought up but Guido said he wanted to save that syntax for within a 'with' block. The other suggestion was to do ``func.attribute`` where "func" is the name of the function is being defined, but Guido gave that a -1000 vote so that ain't about to happen. Contributing threads: - `order of decorator application? `__ - `PEP 318 - posting draft `__ - `PEP 318 `__ - `method decorators (PEP 318) `__ - `PEP 318 and syntax coloring `__ - `decorators (not transformations) `__ - `PEP318 metaclass approach `__ - `method decorators `__ - `Yet Another Decorator Syntax Suggestion (YADSS) `__ - `PEP 318: Singleton decorator `__ - `PEP 318: Preserve function signatures `__ - `PEP 318: Security use case `__ - `PEP 318: Decorators last before colon `__ - `PEP 318: Set attribs with .name = value `__ - `Re: Python-Dev Digest, Vol 8, Issue 91 `__ --------------------------- Python 2.4 release schedule --------------------------- If you read `PEP 320`_ you will notice an alpha is currently planned in July with a possible release in September if the alphas and betas go well. To see what is planned for this release read the PEP. Looks like some really cool new additions to Python (under the hood, stdlib, and to syntax) might make this release. .. _PEP 320: http://python.org/peps/pep-0320.html Contributing threads: - `Timing for Py2.4 `__ ------------------------------------------ Cutting down on string creation at startup ------------------------------------------ Martin v. Loewis discovered that Python in CVS created about 12,000 strings while Python 2.2 created 8,000 strings. Martin subsequently changed 'marshal' so as to share strings that are the same instead of having to constantly create strings, discover they have been interned, and then destroy the second copy. Contributing threads: - `Python startup time: String objects `__ --------------------------------- f_tstate gone the way of the dodo --------------------------------- In case you don't catch the meaning of the title, f_tstate has now been removed. Contributing threads: - `Last chance! `__ --------------------------------------------------------------- Tentative fix for OpenVMS and its dislike of UNIVERSAL_NEWLINES --------------------------------------------------------------- See http://www.python.org/sf/903339 for the proposed patch. Contributing threads: - ` OpenVMS file system and UNIVERSAL_NEWLINES `__ ---------------------------------------------- The hell that is floating point representation ---------------------------------------------- The discussion came up over the output of ``repr(1.1)``, which is '1.1000000000000001' (although, apparently, a few years back this was not the case). The objection was risen that this can surprise people since repr is what 'marshal' uses for writing .pyc files. But the counter-argument was that this forces people to deal with the issues inherent in binary floating point. Plus it makes the value consistent across platforms if you pass around a .pyc file since the value the programmer used is explicitly stated. The status quo is staying put. It was also pointed out that decimal floating point has similar issues as binary floating point, but at least you get to specify the accuracy at the cost of speed. Contributing threads: - `Expert floats `__ - `repr(1.1) `__ ---------------------------------------- Relative imports getting a bunch of dots ---------------------------------------- For `PEP 328`_ (which deals with relative imports of modules), Guido pronounced that he prefers the multiple dots to delineate going up a level in the directory hierarchy for handling relative imports. .. _PEP 328: htttp://www.python.org/peps/pep-0328.html Contributing threads: - `PEP 328 -- relative and multi-line import `__ From aahz at pythoncraft.com Sun Apr 11 02:14:35 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Apr 11 02:14:45 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <04Apr10.122659pdt."58612"@synergy1.parc.xerox.com> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <04Apr10.122659pdt."58612"@synergy1.parc.xerox.com> Message-ID: <20040411061435.GA13165@panix.com> On Sat, Apr 10, 2004, Bill Janssen wrote: >Andrew Koenig: >> >> I do wish to caution against assuming the problem is trivial. > > I've been down this road myself, and I think I understand the > problem. However, whenever I've made changes of this break-itude, > I've always done it in a major release. Changing things this way > in a minor release just seems hostile to me. Remember that one of > the major reasons people use Python (instead of, say, Scheme) is the > availability of lots of packages written in Python to do various > things. Making those packages buggy is a suboptimal way to win > friends and influence people. Python's version changes from 2.0 through 2.3 have all been considered "major" releases, as will the upcoming 2.4 release. Ditto 1.4 to 1.5 (haven't been involved with Python long enough to know about earlier releases). For that matter, 1.5 to 1.5.1 to 1.5.2 contained a fair number of major changes, which led to the creation of PEP 6. The jump from 1.5 to 1.6 to 2.0 was overall driven more by politics than technical issues. Although we talk about plans for Python 3.0, that's very much a pie-in-the-sky release at this point, and that will likely contain about as much breakage as going from FORTRAN II to Fortran 77. I don't see anyone else considering this a major issue in this case as long as we allot a sufficiently long timeline (and if you search the records, I'm generally one of the people more likely to raise objections about causing old code to fail). If you want to pursue this angle, I think you should go to c.l.py to see if anyone agrees with you. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From mwh at python.net Sun Apr 11 06:19:56 2004 From: mwh at python.net (Michael Hudson) Date: Sun Apr 11 06:20:01 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <04Apr7.183319pdt."58612"@synergy1.parc.xerox.com> (Bill Janssen's message of "Wed, 7 Apr 2004 18:33:10 PDT") References: <04Apr7.183319pdt."58612"@synergy1.parc.xerox.com> Message-ID: <2m8yh2byur.fsf@starship.python.net> Bill Janssen writes: > I really wish I'd been paying more attention to 328, which I've just > read. This proposal seems to break lots and lots and lots of existing > code. Lots of *your* code, maybe... I think you might be one of the few people who embraced packages as soon as they were available. > Maybe I'm reading it wrong. > > I've written many, many packages that use the form > > import foo > > where "foo" is a sibling module in the package. Or > > from foo import bar > > where, again, foo is a sibling to the current module. Nevertheless, do you agree or not that it was a mistake that relative and absolute imports are spelt the same? I you do, then surely the longer we leave it, the worse the problem gets... [...] > I was thinking (some months ago) that y'all would come up with an > absolute form instead, something like Then you really haven't been paying attention! Pretty much *everyone* who has contributed to the discussion, and presumably plenty who haven't (like me), has assumed that absolute imports are more common and should the default (certainly, all the imports in my own code are absolute...). Cheers, mwh -- > With Python you can start a thread, but you can't stop it. Sorry. > You'll have to wait until reaches the end of execution. So, just the same as c.l.py, then? -- Cliff Wells & Steve Holden, comp.lang.python From mwh at python.net Sun Apr 11 06:22:21 2004 From: mwh at python.net (Michael Hudson) Date: Sun Apr 11 06:22:25 2004 Subject: [Python-Dev] python-dev Summary for 2004-03-16 through 2004-03-31 [rough draft] In-Reply-To: <4078AD18.5070502@ocf.berkeley.edu> (Brett C.'s message of "Sat, 10 Apr 2004 19:27:36 -0700") References: <4078AD18.5070502@ocf.berkeley.edu> Message-ID: <2m4qrqbyqq.fsf@starship.python.net> "Brett C." writes: > --------------------------------- > f_tstate gone the way of the dodo > --------------------------------- > In case you don't catch the meaning of the title, f_tstate has now > been removed. > > Contributing threads: > - `Last chance! > `__ Only in stackless Python! Cheers, mwh -- /* I'd just like to take this moment to point out that C has all the expressive power of two dixie cups and a string. */ -- Jamie Zawinski from the xkeycaps source From barry at python.org Sun Apr 11 11:22:01 2004 From: barry at python.org (Barry Warsaw) Date: Sun Apr 11 11:22:16 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> Message-ID: <1081696920.17920.76.camel@anthem.wooz.org> On Sat, 2004-04-10 at 12:57, Andrew Koenig wrote: > The trouble is that if you commit yourself to backward compatibility, you > wind up with a growing list of decisions for which the default is wrong but > you didn't want to break code by changing it. Each time you make such a > decision, you make it more difficult the next time to do anything but go for > compatibility again. Ultimately, everyone has to remember to override the > defaults in lots of everyday circumstances, a mental burden that hurts > everyone from novices to experts. > > On the other hand, if you don't commit yourself to backward compatibility, > you wind up with people downloading old code that breaks because things have > changed. > > The third possibility is to support all alternatives forever, perhaps with a > mechanism such as "from __past__ import foo". That's possible in theory, > but if changes include changes in semantics, it may be impossible for old > and new to coexist. Moreover, even if they can coexist, testing becomes a > nightmare. > > It's a trilemma--a choice between three unpleasant alternatives. I don't > know the right answer, but I do wish to caution against assuming the problem > is trivial. This is one of the reasons I suggested early on that there be a way to spell -- at the package level by the package developer -- "I'm using absolute imports here". Without that spelling, the package would still use the old semantics, however broken they are . E.g. if I could put in email/__init__.py something like: from __future__ import i_am_absolutely_resolved_about_my_imports then I'd expect PEP 328 semantics in package email and all subpackages. Older packages wouldn't have this declaration and would operate under the old rules. (I still don't have any need for relative imports. ;) -Barry From aahz at pythoncraft.com Sun Apr 11 11:54:53 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Apr 11 11:54:56 2004 Subject: [Python-Dev] python-dev Summary for 2004-03-16 through 2004-03-31 [rough draft] In-Reply-To: <2m4qrqbyqq.fsf@starship.python.net> References: <4078AD18.5070502@ocf.berkeley.edu> <2m4qrqbyqq.fsf@starship.python.net> Message-ID: <20040411155453.GA18539@panix.com> On Sun, Apr 11, 2004, Michael Hudson wrote: > "Brett C." writes: >> >> --------------------------------- >> f_tstate gone the way of the dodo >> --------------------------------- >> In case you don't catch the meaning of the title, f_tstate has now >> been removed. >> >> Contributing threads: >> - `Last chance! >> `__ > > Only in stackless Python! Sure, but Guido said it should be removed from CPython, too. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From skip at pobox.com Sun Apr 11 12:26:18 2004 From: skip at pobox.com (Skip Montanaro) Date: Sun Apr 11 12:26:26 2004 Subject: [Python-Dev] python-dev Summary for 2004-03-16 through 2004-03-31 [rough draft] In-Reply-To: <20040411155453.GA18539@panix.com> References: <4078AD18.5070502@ocf.berkeley.edu> <2m4qrqbyqq.fsf@starship.python.net> <20040411155453.GA18539@panix.com> Message-ID: <16505.29098.291755.625930@montanaro.dyndns.org> >>> In case you don't catch the meaning of the title, f_tstate has now >>> been removed. >> >> Only in stackless Python! aahz> Sure, but Guido said it should be removed from CPython, too. True, but at this point it still yet to actually be removed. Skip From aahz at pythoncraft.com Sun Apr 11 13:23:51 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Apr 11 13:23:54 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <1081696920.17920.76.camel@anthem.wooz.org> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <1081696920.17920.76.camel@anthem.wooz.org> Message-ID: <20040411172351.GA23114@panix.com> On Sun, Apr 11, 2004, Barry Warsaw wrote: > > This is one of the reasons I suggested early on that there be a way to > spell -- at the package level by the package developer -- "I'm using > absolute imports here". Without that spelling, the package would still > use the old semantics, however broken they are . > > E.g. if I could put in email/__init__.py something like: > > from __future__ import i_am_absolutely_resolved_about_my_imports > > then I'd expect PEP 328 semantics in package email and all subpackages. > Older packages wouldn't have this declaration and would operate under > the old rules. (I still don't have any need for relative imports. ;) Anyone else favor having the ``__future__`` directive apply to the package and all its subpackages? (Currently it follows the standard rules of applying only to each module.) Note that this would make it impossible to do what some people want, where a package becomes another package's subpackage, if the new subpackage relies on relative imports under the old rules. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From bac at OCF.Berkeley.EDU Sun Apr 11 15:55:41 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sun Apr 11 15:55:58 2004 Subject: [Python-Dev] python-dev Summary for 2004-03-16 through 2004-03-31 [rough draft] In-Reply-To: <16505.29098.291755.625930@montanaro.dyndns.org> References: <4078AD18.5070502@ocf.berkeley.edu> <2m4qrqbyqq.fsf@starship.python.net> <20040411155453.GA18539@panix.com> <16505.29098.291755.625930@montanaro.dyndns.org> Message-ID: <4079A2BD.4030104@ocf.berkeley.edu> Skip Montanaro wrote: > >>> In case you don't catch the meaning of the title, f_tstate has now > >>> been removed. > >> > >> Only in stackless Python! > > aahz> Sure, but Guido said it should be removed from CPython, too. > > True, but at this point it still yet to actually be removed. > Sorry, wrong tense to use. It's now future tense. -Brett From ajole-1 at gci.net Sun Apr 11 17:59:14 2004 From: ajole-1 at gci.net (Patrick Stinson) Date: Mon Apr 12 00:09:38 2004 Subject: [Python-Dev] pyc/embedded c++ efficiency wrt binary data Message-ID: <200404111359.14011.ajole-1@gci.net> I'm using pyuic, a tool that comes as part of pyqt, that reads images and creates a python file with the data embedded in it for easy loading. This is based on a similar tool called qembed that embeds image data in C++ header files. Well, I have found that upon loading the py (9.2MB) file, it takes quite a while, but a pyc file is created, and is approx. 1/4 the size. This gives me the indication that the binary encoding is quite good, and importing the pyc file will takes considerably less time. Does anyone have any idea how the binary encoding of pyc files is compared to something like this done in C++? Of course, I'm not asking about the ultimate efficiency including processing the binary data as pixmaps, but just the loading from disk bit. Cheers! -P From aahz at pythoncraft.com Mon Apr 12 00:54:49 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 12 00:54:55 2004 Subject: [Python-Dev] pyc/embedded c++ efficiency wrt binary data In-Reply-To: <200404111359.14011.ajole-1@gci.net> References: <200404111359.14011.ajole-1@gci.net> Message-ID: <20040412045449.GA13027@panix.com> On Sun, Apr 11, 2004, Patrick Stinson wrote: > > Well, I have found that upon loading the py (9.2MB) file, it takes > quite a while, but a pyc file is created, and is approx. 1/4 the > size. This gives me the indication that the binary encoding is > quite good, and importing the pyc file will takes considerably less > time. Does anyone have any idea how the binary encoding of pyc files > is compared to something like this done in C++? Of course, I'm not > asking about the ultimate efficiency including processing the binary > data as pixmaps, but just the loading from disk bit. python-dev is for the work of the Python project, not for questions about using Python. Please post your question to comp.lang.python. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From chrism at plope.com Mon Apr 12 00:18:29 2004 From: chrism at plope.com (Chris McDonough) Date: Mon Apr 12 09:43:49 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040411172351.GA23114@panix.com> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <1081696920.17920.76.camel@anthem.wooz.org> <20040411172351.GA23114@panix.com> Message-ID: <1081743508.2987.118.camel@james> What is the rationale given for not allowing: import .moduleY as a relative import statement meaning "from the package containing the module running the import statement, import moduleY"? The PEP suggests that "relative imports will require listing specific names to import". I'm wondering what about "moduleY" in the above isn't specific? FWIW, I read through much of the meanderings of the old threads but never saw a specific damnation-with-cause of this. - C On Sun, 2004-04-11 at 13:23, Aahz wrote: > On Sun, Apr 11, 2004, Barry Warsaw wrote: > > > > This is one of the reasons I suggested early on that there be a way to > > spell -- at the package level by the package developer -- "I'm using > > absolute imports here". Without that spelling, the package would still > > use the old semantics, however broken they are . > > > > E.g. if I could put in email/__init__.py something like: > > > > from __future__ import i_am_absolutely_resolved_about_my_imports > > > > then I'd expect PEP 328 semantics in package email and all subpackages. > > Older packages wouldn't have this declaration and would operate under > > the old rules. (I still don't have any need for relative imports. ;) > > Anyone else favor having the ``__future__`` directive apply to the > package and all its subpackages? (Currently it follows the standard > rules of applying only to each module.) Note that this would make it > impossible to do what some people want, where a package becomes another > package's subpackage, if the new subpackage relies on relative imports > under the old rules. From barry at python.org Mon Apr 12 10:06:47 2004 From: barry at python.org (Barry Warsaw) Date: Mon Apr 12 10:06:58 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040411172351.GA23114@panix.com> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <1081696920.17920.76.camel@anthem.wooz.org> <20040411172351.GA23114@panix.com> Message-ID: <1081778806.17920.300.camel@anthem.wooz.org> On Sun, 2004-04-11 at 13:23, Aahz wrote: > On Sun, Apr 11, 2004, Barry Warsaw wrote: > > > > This is one of the reasons I suggested early on that there be a way to > > spell -- at the package level by the package developer -- "I'm using > > absolute imports here". Without that spelling, the package would still > > use the old semantics, however broken they are . > > > > E.g. if I could put in email/__init__.py something like: > > > > from __future__ import i_am_absolutely_resolved_about_my_imports > > > > then I'd expect PEP 328 semantics in package email and all subpackages. > > Older packages wouldn't have this declaration and would operate under > > the old rules. (I still don't have any need for relative imports. ;) > > Anyone else favor having the ``__future__`` directive apply to the > package and all its subpackages? (Currently it follows the standard > rules of applying only to each module.) Note that this would make it > impossible to do what some people want, where a package becomes another > package's subpackage, if the new subpackage relies on relative imports > under the old rules. Would that be any different than if modules of the subpackage add the __future__ in them? I'm envisioning semantics such that a __future__ in an __init__.py was the same as if that __future__ was explicitly added to every module (i.e. it's a convenience). -Barry From jeremy at alum.mit.edu Mon Apr 12 09:41:02 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon Apr 12 10:54:34 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <1081778806.17920.300.camel@anthem.wooz.org> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <1081696920.17920.76.camel@anthem.wooz.org> <20040411172351.GA23114@panix.com> <1081778806.17920.300.camel@anthem.wooz.org> Message-ID: <1081777262.2610.2.camel@localhost.localdomain> On Mon, 2004-04-12 at 10:06, Barry Warsaw wrote: > Would that be any different than if modules of the subpackage add the > __future__ in them? I'm envisioning semantics such that a __future__ in > an __init__.py was the same as if that __future__ was explicitly added > to every module (i.e. it's a convenience). I think they belong in every module. A future statement is a compiler gimmick and modules are compiled separately, so each module ought to have its own future statement. If not, the compiler (and human reader) have no idea whether a future statement is in effect. Obviously, the compiler and the reader could learn about some change to import semantics by studying context, but that's extra complexity I'd like to avoid (particular in the case of the compiler). Jeremy From guido at python.org Mon Apr 12 10:55:10 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 12 10:55:25 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Mon, 12 Apr 2004 00:18:29 EDT." <1081743508.2987.118.camel@james> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <1081696920.17920.76.camel@anthem.wooz.org> <20040411172351.GA23114@panix.com> <1081743508.2987.118.camel@james> Message-ID: <200404121455.i3CEtAE19580@guido.python.org> > What is the rationale given for not allowing: > > import .moduleY For all other forms of import, after import XXX.YYY.ZZZ then XXX.YYY.ZZZ is usable in an expression. (Very different from Java!) But .moduleY is not usable in an expression. (This might be worth adding to the PEP.) --Guido van Rossum (home page: http://www.python.org/~guido/) From jim.jewett at eds.com Mon Apr 12 12:11:00 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Mon Apr 12 12:12:05 2004 Subject: [Python-Dev] backwards compatibility (was: PEP 328 -- relative and multi-line import) Message-ID: Bill Janssen: > Wow. This seems amazingly user-hostile. Users (not Python > developers) download some two or three year old code to do something > from an online library, like the Vaults of Parnassus, and it just > breaks because the meaning of "import" has changed. Andrew Koenig: > ... If you commit yourself to backward compatibility, you > wind up with a growing list of decisions for which the > default is wrong ... everyone has to remember to override > the defaults in lots of everyday circumstances, a mental > burden that hurts everyone from novices to experts. > ... If you don't commit yourself to backward compatibility, > you wind up with people downloading old code that breaks > The third possibility is to support all alternatives forever, > perhaps with a mechanism such as "from __past__ import foo". Most changes are for something that was possible but awkward (inconvenient/ugly/inefficient) in the past. Would it be reasonable to add a magic attribute like __target_python__ = 2.3 Then the interpreter (or an externally maintained translator) could translate the code automatically, in at least several cases. Older code wouldn't have this, but that already suggests it is pre-(something), and the FAQ could say to that if downloaded code doesn't work, try the automated translator. In theory, the translator could also work backwards, so that maintainers could start to write True instead of 1. I would be willing to work on such a translator, though I doubt I could get it to full strength on my own. -jJ From aahz at pythoncraft.com Mon Apr 12 12:52:49 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 12 12:53:50 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200404121455.i3CEtAE19580@guido.python.org> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <1081696920.17920.76.camel@anthem.wooz.org> <20040411172351.GA23114@panix.com> <1081743508.2987.118.camel@james> <200404121455.i3CEtAE19580@guido.python.org> Message-ID: <20040412165249.GA4311@panix.com> On Mon, Apr 12, 2004, Guido van Rossum wrote: > > For all other forms of import, after > > import XXX.YYY.ZZZ > > then > > XXX.YYY.ZZZ > > is usable in an expression. (Very different from Java!) But > > .moduleY > > is not usable in an expression. > > (This might be worth adding to the PEP.) Done! -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From aahz at pythoncraft.com Mon Apr 12 12:55:59 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 12 12:56:43 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <1081777262.2610.2.camel@localhost.localdomain> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <1081696920.17920.76.camel@anthem.wooz.org> <20040411172351.GA23114@panix.com> <1081778806.17920.300.camel@anthem.wooz.org> <1081777262.2610.2.camel@localhost.localdomain> Message-ID: <20040412165559.GB4311@panix.com> On Mon, Apr 12, 2004, Jeremy Hylton wrote: > On Mon, 2004-04-12 at 10:06, Barry Warsaw wrote: >> >> Would that be any different than if modules of the subpackage add the >> __future__ in them? I'm envisioning semantics such that a __future__ in >> an __init__.py was the same as if that __future__ was explicitly added >> to every module (i.e. it's a convenience). Right. And my point is that there's at least one corner case where this will have definite drawbacks. > I think they belong in every module. A future statement is a compiler > gimmick and modules are compiled separately, so each module ought to > have its own future statement. If not, the compiler (and human reader) > have no idea whether a future statement is in effect. Obviously, the > compiler and the reader could learn about some change to import > semantics by studying context, but that's extra complexity I'd like to > avoid (particular in the case of the compiler). That's what I expected the reaction to be from others. Barry, if you want your change, lobby Guido. ;-) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From barry at python.org Mon Apr 12 13:35:27 2004 From: barry at python.org (Barry Warsaw) Date: Mon Apr 12 13:35:37 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040412165559.GB4311@panix.com> References: <00db01c41f1c$d9beec10$6402a8c0@arkdesktop> <1081696920.17920.76.camel@anthem.wooz.org> <20040411172351.GA23114@panix.com> <1081778806.17920.300.camel@anthem.wooz.org> <1081777262.2610.2.camel@localhost.localdomain> <20040412165559.GB4311@panix.com> Message-ID: <1081791326.21906.32.camel@anthem.wooz.org> On Mon, 2004-04-12 at 12:55, Aahz wrote: > > I think they belong in every module. A future statement is a compiler > > gimmick and modules are compiled separately, so each module ought to > > have its own future statement. If not, the compiler (and human reader) > > have no idea whether a future statement is in effect. Obviously, the > > compiler and the reader could learn about some change to import > > semantics by studying context, but that's extra complexity I'd like to > > avoid (particular in the case of the compiler). > > That's what I expected the reaction to be from others. Barry, if you > want your change, lobby Guido. ;-) I might be crazy but I'm not insane. :) -Barry From janssen at parc.com Mon Apr 12 18:03:49 2004 From: janssen at parc.com (Bill Janssen) Date: Mon Apr 12 18:04:20 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sat, 10 Apr 2004 15:55:15 PDT." <40787B53.9070007@prescod.net> Message-ID: <04Apr12.150359pdt."58612"@synergy1.parc.xerox.com> > My opinion is that the process is correct and it is just a question of > whether the timelines are generous enough or not. Somewhere between > "Python 2.6" and "never" there must be a point where people have had > enough time to change over their code after getting nagged by the > DeprecationWarning. I couldn't agree with you more, Paul. I think that point is whenever the major version number is bumped to 3. > Most people in the Python community know that Python doesn't really have > "minor releases" other than bugfix releases. This is python-dev thinking again. There are lots of users of Python who don't want to be "in the Python community", and feel that a properly managed programming language is one that doesn't require "membership" to get things right. Bill From janssen at parc.com Mon Apr 12 18:05:58 2004 From: janssen at parc.com (Bill Janssen) Date: Mon Apr 12 18:06:19 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: Your message of "Sun, 11 Apr 2004 08:22:01 PDT." <1081696920.17920.76.camel@anthem.wooz.org> Message-ID: <04Apr12.150604pdt."58612"@synergy1.parc.xerox.com> > E.g. if I could put in email/__init__.py something like: > > from __future__ import i_am_absolutely_resolved_about_my_imports > > then I'd expect PEP 328 semantics in package email and all subpackages. > Older packages wouldn't have this declaration and would operate under > the old rules. Sounds like a good idea to me. Bill From raymond.hettinger at verizon.net Tue Apr 13 07:56:27 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue Apr 13 07:57:24 2004 Subject: [Python-Dev] Candidate Function Decorator Message-ID: <000301c4214e$5a3e1400$e530cb97@oemcomputer> I worked out the details for binding constants and posted a recipe to ASPN: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/277940 Example: [bind_constants] def biased_dotprod(v1, v2): sum = 0 for i in xrange(min(len(v1), len(v2)): sum += v1[i] * (v2[i] - bias) return sum The decorate function binds globals that are known at compile time. So, "xrange", "min", and "len" are added to the constant table, but bias is left as a global lookup (unless it was already defined somewhere before the function definition). The chief advantage of this construct to avoid code like: _len=len, while still getting all of the advantages. The modules in the sre package are prime candidates for taking advantage of constant binding. Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040413/d8a1937f/attachment.html From tws5 at mail.ru Tue Apr 13 10:06:39 2004 From: tws5 at mail.ru (=?koi8-r?Q?=22?=Maxim Krikun=?koi8-r?Q?=22=20?=) Date: Tue Apr 13 10:06:43 2004 Subject: [Python-Dev] pipe syntax for sequential data Message-ID: Helo python-dev. Please take a look at http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/276960 This recipe proposes an alternative syntax for sequential data processing functions, that reflects unix shell pipe syntax. Here is an example: bash_users = ( file('/etc/passwd') | grep('/bin/bash') | aslist ) I don't know if this idea was already explored before; hope this can be useful. For details and sample code please see link above. regards, Maxim From aahz at pythoncraft.com Tue Apr 13 11:41:16 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Apr 13 11:41:28 2004 Subject: [Python-Dev] pipe syntax for sequential data In-Reply-To: References: Message-ID: <20040413154116.GA12847@panix.com> On Tue, Apr 13, 2004, "Maxim Krikun" wrote: > > This recipe proposes an alternative syntax for sequential data > processing functions, that reflects unix shell pipe syntax. Here is an > example: > > bash_users = ( file('/etc/passwd') | grep('/bin/bash') | aslist ) > > I don't know if this idea was already explored before; hope this can > be useful. For details and sample code please see link above. If you're serious about this, you'll need to write up a PEP and push it forward. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From jcarlson at uci.edu Tue Apr 13 12:10:41 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue Apr 13 12:15:03 2004 Subject: [Python-Dev] pipe syntax for sequential data In-Reply-To: <20040413154116.GA12847@panix.com> References: <20040413154116.GA12847@panix.com> Message-ID: <20040413085533.0CF5.JCARLSON@uci.edu> > > This recipe proposes an alternative syntax for sequential data > > processing functions, that reflects unix shell pipe syntax. Here is an > > example: > > > > bash_users = ( file('/etc/passwd') | grep('/bin/bash') | aslist ) > > > > I don't know if this idea was already explored before; hope this can > > be useful. For details and sample code please see link above. > > If you're serious about this, you'll need to write up a PEP and push it > forward. While some would call Python a 'scripting language', I think that using pipes in such a way in Python would be counter to the zen: "There should be one-- and preferably only one --obvious way to do it." We can already use list comprehensions, filter and friends (or ifilter as used by MK), when such functionality is desired. IMO, just because you can make a language behave like a command line, does not necessarily mean that you should. - Josiah From FBatista at uniFON.com.ar Tue Apr 13 12:18:41 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 12:20:47 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: People: Tim Peters reviewed the PEP 327, and asked some questions. Trying to answer those questions, I found that some items needs to be addressed with a comunity consensus. So, the next items are to be included in the PEP when the discussion finishes: Exponent Maximum ---------------- The Decimal number composes of three elements: A sign that can be 0 or 1, a tuple of digits where each can be 0..9, and an exponent. The exponent is an integer, and in the actual implementation exists a maximum value:: DEFAULT_MAX_EXPONENT = 999999999 DEFAULT_MIN_EXPONENT = -999999999 ABSOLUTE_MAX_EXP = 999999999 ABSOLUTE_MIN_EXP = -999999999 The issue is that this limit is artificial: As long it's a long, you should be able to make it as big as your memory let you. In the General Decimal Arithmetic Specification says: In the abstract, there is no upper limit on the absolute value of the exponent. In practice there may be some upper limit, E_limit , on the absolute value of the exponent. So, should we impose an artificial limit to the exponent? This is important, as there're several cases where this maximums are checked and exceptions raised and/or the numbers get changed. New operations -------------- Tim Peters found I missed three operations required by the standard. Those are: a. ``to-scientific-string``: This operation converts a number to a string, using scientific notation if an exponent is needed. The operation is not affected by the context. b. ``to-engineering-string``: This operation converts a number to a string, using engineering notation if an exponent is needed. c. ``to-number``: This operation converts a string to a number, as defined by its abstract representation. First we should agree the names of the methods, I propose: a. to_sci_string b. to_eng_string c. from_string The (a) and (b) methods are different from str, as this method just doesn't adjust the exponent at all. Regarding the method (c), the only difference with creating the decimal with Decimal(string) is that method (c) honors the context (if the literal contains more digits that the current precision the numbers get rounded, and gets rounded according to the round method specified in context, etc). For example, with a precision of 9 and with the name I proposed:: >>> Decimal('112233445566') Decimal( (0, (1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6), 0) ) >>> Decimal.from_string('112233445566') Decimal( (0, (1, 1, 2, 2, 3, 3, 4, 4, 6), 3L) ) Hash behaviour -------------- This item actually wasn't raised by Tim, but I found it when implementing the module. In the PEP I wrote that Decimal must be hashable. But what hash should it give? Should the following be true?:: hash(Decimal(25) == hash(25) hash(Decimal.from_float(25.35) == hash(25.35) hash(Decimal('-33.8')) == hash(-33.8) I don't think so. I think that hash(Decimal(...)) just should return a different value in each case, but no the same value that other data types. Thank you all for your feedback. . Facundo From bob at redivi.com Tue Apr 13 12:39:55 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Apr 13 12:35:51 2004 Subject: [Python-Dev] pipe syntax for sequential data In-Reply-To: <20040413085533.0CF5.JCARLSON@uci.edu> References: <20040413154116.GA12847@panix.com> <20040413085533.0CF5.JCARLSON@uci.edu> Message-ID: <31A9B879-8D69-11D8-B460-000A95686CD8@redivi.com> On Apr 13, 2004, at 12:10 PM, Josiah Carlson wrote: >>> This recipe proposes an alternative syntax for sequential data >>> processing functions, that reflects unix shell pipe syntax. Here is >>> an >>> example: >>> >>> bash_users = ( file('/etc/passwd') | grep('/bin/bash') | aslist ) >>> >>> I don't know if this idea was already explored before; hope this can >>> be useful. For details and sample code please see link above. >> >> If you're serious about this, you'll need to write up a PEP and push >> it >> forward. > > > While some would call Python a 'scripting language', I think that using > pipes in such a way in Python would be counter to the zen: > "There should be one-- and preferably only one --obvious way to do it." But there are others, such as "Although practicality beats purity". This case is already pretty practical using existing syntax though: [line for line in file('/etc/passwd') if '/bin/bash' in line] Though it does pollute the locals.. > We can already use list comprehensions, filter and friends (or ifilter > as used by > MK), when such functionality is desired. IMO, just because you can > make > a language behave like a command line, does not necessarily mean that > you should. Of course you should, but that doesn't mean it should be __builtin__. -bob From skip at pobox.com Tue Apr 13 12:58:27 2004 From: skip at pobox.com (Skip Montanaro) Date: Tue Apr 13 12:59:47 2004 Subject: [Python-Dev] pipe syntax for sequential data In-Reply-To: <20040413154116.GA12847@panix.com> References: <20040413154116.GA12847@panix.com> Message-ID: <16508.7219.539660.287802@montanaro.dyndns.org> >> This recipe proposes an alternative syntax for sequential data >> processing functions, that reflects unix shell pipe syntax. Here is >> an example: >> >> bash_users = ( file('/etc/passwd') | grep('/bin/bash') | aslist ) aahz> If you're serious about this, you'll need to write up a PEP and aahz> push it forward. Also, you should release it as a standalone module (perhaps create a SourceForge project for it) so it can "cook" for awhile and you can get feedback and perhaps some new command objects for your pipe. Skip From tjreedy at udel.edu Tue Apr 13 13:40:27 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Tue Apr 13 13:40:35 2004 Subject: [Python-Dev] Re: Candidate Function Decorator References: <000301c4214e$5a3e1400$e530cb97@oemcomputer> Message-ID: "Raymond Hettinger" [cc'ed]: >I worked out the details for binding constants and posted a recipe http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/277940 I have the impression that it is proposed that the function f will *not* yet be bound to its name (f.func_name) in its environment at the time the decorator is called. If so, f.func_name will not appear in env unless you separately add it. If you do add it, then I would think that recursive functions would be bound to themselves and recursive calls would also go faster, just as with calls to builtins, etc. The only obvious downside to me is that the circular ref would keep function around until gc'ed, but that is why gc was added ;-). Terry J. Reedy From python at rcn.com Tue Apr 13 13:48:34 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Apr 13 13:50:34 2004 Subject: [Python-Dev] Re: Candidate Function Decorator In-Reply-To: Message-ID: <007d01c4217f$8affce40$e530cb97@oemcomputer> > I have the impression that it is proposed that the function f will *not* > yet be bound to its name (f.func_name) in its environment at the time the > decorator is called. Hmm, that would be a change. Currently, staticmethod() and classmethod() take fully formed functions as inputs. So, if your impression is correct, there would be a small semantic difference between the [classmethod] prefix and f=classmethod(f) postfix decoration. Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From john at neggie.net Tue Apr 13 13:52:32 2004 From: john at neggie.net (John Belmonte) Date: Tue Apr 13 13:52:40 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc Message-ID: <407C28E0.7090808@neggie.net> Hello, I'd like to make some observations and proposals about pydoc regarding slots, properties, and descriptors in general. Given this program: class CountingDescriptor(object): def __init__(self, doc=None): self.val = 0 self.__doc__ = doc def __get__(self, obj, objtype): self.val += 1 return self.val class Foo(object): __slots__ = {'bar': 'slot doc test'} counter = CountingDescriptor(doc='descriptor doc test') def _set_counter_multiple(self, val): print 'I hear you want to set me to %s?' % val counter_multiple = property( lambda self: self.counter*2, _set_counter_multiple, doc='property doc test' ) print help(Foo) The output is: Help on class Foo in module __main__: class Foo(__builtin__.object) | Properties defined here: | | counter_multiple | property doc test | | lambdaself | | = _set_counter_multiple(self, val) | | ------------------------------------------------------------ | Data and other attributes defined here: | | __slots__ = {'bar': 'slot doc test'} | | bar = | | counter = 2 Observations (in order from least to most arguable :-): 1) pydoc's property display doesn't handle lambda well 2) pydoc doesn't show the doc string of general descriptors 3) Using dictionary values as a way to set the doc string of slots, as suggested in Guido's "Unifying types and classes", has not been implemented. Currently there is no way to set a doc string on a descriptor generated by __slots__. Even if you could, pydoc would not display it, according to #2. 4) pydoc is displaying implementation details of properties (set and get method names, etc.). Such details should probably not be displayed, similar to underbar-prefixed method and data attributes. Proposals: 1) remove property-specific handling from pydoc 2) extend pydoc with descriptor-specific handling. In a section called "Descriptors defined here", list each attribute name, with associated doc string if it exists. 3) extend the __slots__ handler to set the descriptor's doc string to dictionary values if the slot's value is a dictionary. This is the only Python change being proposed. 4) have pydoc make a special case for the display of __slots__. Alternatives: A) don't display __slots__ if the value is a dictionary. The display of __slots__ is redundant, because all items are displayed according to #2. B) if for some reason distinguishing slots from other descriptors is essential, then always display __slots__, but normalize it to a list of attribute names (show only dictionary keys, etc.) With these changes, the new output of the example would be: Help on class Foo in module __main__: class Foo(__builtin__.object) | Descriptors defined here: | | bar | slot doc test | | counter | descriptor doc test | | counter_multiple | property doc test I'd be willing to work on a PEP. If these issues are covered in or conflict with an existing PEP, please let me know. Thanks, -John -- http:// if ile.org/ From mwh at python.net Tue Apr 13 13:54:44 2004 From: mwh at python.net (Michael Hudson) Date: Tue Apr 13 13:54:47 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: (Facundo Batista's message of "Tue, 13 Apr 2004 13:18:41 -0300") References: Message-ID: <2m65c33grf.fsf@starship.python.net> A comment from the peanut gallery... "Batista, Facundo" writes: > Should the following be true?:: > > hash(Decimal(25) == hash(25) > hash(Decimal.from_float(25.35) == hash(25.35) > hash(Decimal('-33.8')) == hash(-33.8) Well, if Decimal(25) == 25, they'd better hash the same! If not, then what's the point? Cheers, mwh -- I think perhaps we should have electoral collages and construct our representatives entirely of little bits of cloth and papier mache. -- Owen Dunn, ucam.chat, from his review of the year From pje at telecommunity.com Tue Apr 13 13:54:58 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 13 13:55:33 2004 Subject: [Python-Dev] Re: Candidate Function Decorator In-Reply-To: References: <000301c4214e$5a3e1400$e530cb97@oemcomputer> Message-ID: <5.1.1.6.0.20040413134823.025b0870@telecommunity.com> At 01:40 PM 4/13/04 -0400, Terry Reedy wrote: >"Raymond Hettinger" [cc'ed]: > > >I worked out the details for binding constants and posted a recipe > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/277940 > >I have the impression that it is proposed that the function f will *not* >yet be bound to its name (f.func_name) in its environment at the time the >decorator is called. If so, f.func_name will not appear in env unless you >separately add it. If you do add it, then I would think that recursive >functions would be bound to themselves and recursive calls would also go >faster, just as with calls to builtins, etc. > >The only obvious downside to me is that the circular ref would keep >function around until gc'ed, but that is why gc was added ;-). You'd have to follow a more complex process to set that up. First, you'd create a code object with the function-constant filled in as None, then create a function object from it. Then, you create a second code object containing the new function object in its constants. Finally, you assign the new code object to the new function's func_code attribute. From pje at telecommunity.com Tue Apr 13 13:58:08 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 13 13:58:30 2004 Subject: [Python-Dev] Re: Candidate Function Decorator In-Reply-To: <007d01c4217f$8affce40$e530cb97@oemcomputer> References: Message-ID: <5.1.1.6.0.20040413135528.025a9540@telecommunity.com> At 01:48 PM 4/13/04 -0400, Raymond Hettinger wrote: > > I have the impression that it is proposed that the function f will >*not* > > yet be bound to its name (f.func_name) in its environment at the time >the > > decorator is called. > >Hmm, that would be a change. Currently, staticmethod() and >classmethod() take fully formed functions as inputs. So, if your >impression is correct, It is. See PEP 318, under "Current Implementation". Also, several proposed decorators such as propget/propset/propdel and "generic" rely on this distinction for their magic. >there would be a small semantic difference >between the [classmethod] prefix and f=classmethod(f) postfix >decoration. Yes, and that's the desired semantics, as it makes possible many things that are quite awkward with the existing syntax. From FBatista at uniFON.com.ar Tue Apr 13 14:10:33 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 14:12:42 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Michael Hudson] #- > Should the following be true?:: #- > #- > hash(Decimal(25) == hash(25) #- > hash(Decimal.from_float(25.35) == hash(25.35) #- > hash(Decimal('-33.8')) == hash(-33.8) #- #- Well, if Decimal(25) == 25, they'd better hash the same! If #- not, then #- what's the point? Remember that you only can compare Decimal to ints or long, not float nor string. So, This way, hash() must *only* return the same value than the equivalent integer. I'm ok with this. . Facundo From pf_moore at yahoo.co.uk Tue Apr 13 15:20:14 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Apr 13 15:20:23 2004 Subject: [Python-Dev] Re: Decimal data type issues References: Message-ID: "Batista, Facundo" writes: > Exponent Maximum > ---------------- [...] > So, should we impose an artificial limit to the exponent? > > This is important, as there're several cases where this maximums are > checked and exceptions raised and/or the numbers get changed. I don't have a strong opinion either way (as I'll never be likely to hit a limit in practice) but my instinct says that you should not impose arbitrary limits. Are there any benefits to having a limit? You don't mention any in your message... Paul. -- This signature intentionally left blank From john at neggie.net Tue Apr 13 13:10:10 2004 From: john at neggie.net (John Belmonte) Date: Tue Apr 13 15:21:47 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc Message-ID: <407C1EF2.9000301@neggie.net> Hello, I'd like to make some observations and proposals about pydoc regarding slots, properties, and descriptors in general. Given this program: class CountingDescriptor(object): def __init__(self, doc=None): self.val = 0 self.__doc__ = doc def __get__(self, obj, objtype): self.val += 1 return self.val class Foo(object): __slots__ = {'bar': 'slot doc test'} counter = CountingDescriptor(doc='descriptor doc test') def _set_counter_multiple(self, val): print 'I hear you want to set me to %s?' % val counter_multiple = property( lambda self: self.counter*2, _set_counter_multiple, doc='property doc test' ) print help(Foo) The output is: Help on class Foo in module __main__: class Foo(__builtin__.object) | Properties defined here: | | counter_multiple | property doc test | | lambdaself | | = _set_counter_multiple(self, val) | | ------------------------------------------------------------ | Data and other attributes defined here: | | __slots__ = {'bar': 'slot doc test'} | | bar = | | counter = 2 Observations (in order from least to most arguable :-): 1) pydoc's property display doesn't handle lambda well 2) pydoc doesn't show the doc string of general descriptors 3) Using dictionary values as a way to set the doc string of slots, as suggested in Guido's "Unifying types and classes", has not been implemented. Currently there is no way to set a doc string on a descriptor generated by __slots__. Even if you could, pydoc would not display it, according to #2. 4) pydoc is displaying implementation details of properties (set and get method names, etc.). Such details should probably not be displayed, similar to underbar-prefixed method and data attributes. Proposals: 1) remove property-specific handling from pydoc 2) extend pydoc with descriptor-specific handling. In a section called "Descriptors defined here", list each attribute name, with associated doc string if it exists. 3) extend the __slots__ handler to set the descriptor's doc string to dictionary values if the slot's value is a dictionary. This is the only Python change being proposed. 4) have pydoc make a special case for the display of __slots__. Alternatives: A) don't display __slots__ if the value is a dictionary. The display of __slots__ is redundant, because all items are displayed according to #2. B) if for some reason distinguishing slots from other descriptors is essential, then always display __slots__, but normalize it to a list of attribute names (show only dictionary keys, etc.) With these changes, the new output of the example would be: Help on class Foo in module __main__: class Foo(__builtin__.object) | Descriptors defined here: | | bar | slot doc test | | counter | descriptor doc test | | counter_multiple | property doc test I'd be willing to work on a PEP. -John -- http:// if ile.org/ From mcherm at mcherm.com Tue Apr 13 15:31:16 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Tue Apr 13 15:31:20 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: <1081884676.407c4004be701@mcherm.com> Facundo Batista writes: > Exponent Maximum > ---------------- [...] > The exponent is an integer, and in the actual implementation exists a > maximum > value:: > > DEFAULT_MAX_EXPONENT = 999999999 > DEFAULT_MIN_EXPONENT = -999999999 > ABSOLUTE_MAX_EXP = 999999999 > ABSOLUTE_MIN_EXP = -999999999 > > The issue is that this limit is artificial: [...] > So, should we impose an artificial limit to the exponent? > > This is important, as there're several cases where this > maximums are checked > and exceptions raised and/or the numbers get changed. I strongly prefer for my programming language not to impose aribtrary limits, however, I can't see myself using decimal to represent numbers with 1 billion digits. So IMHO, I would avoid the limit if possible, but if imposing the limit permitted substantial code simplification or performance improvements then I wouldn't complain. Much. > New operations > -------------- > > Tim Peters found I missed three operations required by the > standard. Sure. Your proposal sounds good. > Hash behaviour > -------------- [...] > In the PEP I wrote that Decimal must be hashable. But what > hash should it > give? > > Should the following be true?:: > > hash(Decimal(25) == hash(25) > hash(Decimal.from_float(25.35) == hash(25.35) > hash(Decimal('-33.8')) == hash(-33.8) > > I don't think so. I think that hash(Decimal(...)) just should return a > different value in each case, but no the same value that > other data types. I disagree. If x == y, then that should imply that hash(x) == hash(y). So hash(Decimal(25)) == hash(25) had better be true. On the other hand, hash(Decimal('-33')) CANNOT equal hash(Decimal('-33')), since the latter must equal hash(-33). And I'm not particularly concerned about floats... if it's easy to make the hashes equal for those few numbers which can be expressed EXACTLY as both a (binary) float and a (decimal) Decimal, then that's be "kinda nice", but since there are so few values which can be expressed exactly as both a float and a Decimal, I don't see it as a big issue. -- Michael Chermside From FBatista at uniFON.com.ar Tue Apr 13 15:54:49 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 15:56:51 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Paul Moore] #- > So, should we impose an artificial limit to the exponent? #- > #- > This is important, as there're several cases where this #- maximums are #- > checked and exceptions raised and/or the numbers get changed. #- #- I don't have a strong opinion either way (as I'll never be likely to #- hit a limit in practice) but my instinct says that you should not #- impose arbitrary limits. Are there any benefits to having a #- limit? You #- don't mention any in your message... I don't find any benefit. But the implementation I took to finish imposed the limits, and I'm searching community agreement before extract them. . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040413/2fb70c63/attachment.html From bob at redivi.com Tue Apr 13 16:01:22 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Apr 13 15:57:14 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <1081884676.407c4004be701@mcherm.com> References: <1081884676.407c4004be701@mcherm.com> Message-ID: <563EA5F2-8D85-11D8-B460-000A95686CD8@redivi.com> On Apr 13, 2004, at 3:31 PM, Michael Chermside wrote: > Facundo Batista writes: >> Hash behaviour >> -------------- > [...] >> In the PEP I wrote that Decimal must be hashable. But what >> hash should it >> give? >> >> Should the following be true?:: >> >> hash(Decimal(25) == hash(25) >> hash(Decimal.from_float(25.35) == hash(25.35) >> hash(Decimal('-33.8')) == hash(-33.8) >> >> I don't think so. I think that hash(Decimal(...)) just should return a >> different value in each case, but no the same value that >> other data types. > > I disagree. If x == y, then that should imply that hash(x) == hash(y). > So hash(Decimal(25)) == hash(25) had better be true. On the other > hand, hash(Decimal('-33')) CANNOT equal hash(Decimal('-33')), since > the latter must equal hash(-33). And I'm not particularly concerned > about floats... if it's easy to make the hashes equal for those few > numbers which can be expressed EXACTLY as both a (binary) float and > a (decimal) Decimal, then that's be "kinda nice", but since there > are so few values which can be expressed exactly as both a float > and a Decimal, I don't see it as a big issue. >>> hash(float(1.0)) == hash(1) == hash(1L) True Since this is already true, hash(Decimal.from_float(1.0)), hash(Decimal(1)), hash(Decimal(1L)), hash(Decimal('1.0')), hash(Decimal('1')), hash(Decimal(u'1.0')), etc. better be equal as well, whatever that takes. As far as arbitrary floats hashing equally, I don't really care, but the simple ones definitely should. Hopefully at some point, "float constants" (as in coming from source code or user input) in Python will actually be Decimal representations, so you don't lose precision until you ask for it. -bob From fumanchu at amor.org Tue Apr 13 16:05:25 2004 From: fumanchu at amor.org (Robert Brewer) Date: Tue Apr 13 16:07:42 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Paul Moore] #- > So, should we impose an artificial limit to the exponent? #- > #- > This is important, as there're several cases where this #- maximums are #- > checked and exceptions raised and/or the numbers get changed. #- #- I don't have a strong opinion either way (as I'll never be likely to #- hit a limit in practice) but my instinct says that you should not #- impose arbitrary limits. Are there any benefits to having a #- limit? You don't mention any in your message... [Facundo Batista] > I don't find any benefit. But the implementation I took to finish > imposed the limits, and I'm searching community agreement > before extract them. My only "strong opinion" is that, IF there are limits, they should be consumer-modifiable. Given that, I'd still prefer no limits. In plusminusspeak: No limit: +1 Modifiable limits: +0 Concrete limits: -1 Robert Brewer MIS Amor Ministries fumanchu@amor.org From FBatista at uniFON.com.ar Tue Apr 13 16:04:02 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 16:08:19 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Michael Chermside] #- > The issue is that this limit is artificial: #- [...] #- > So, should we impose an artificial limit to the exponent? #- > #- > This is important, as there're several cases where this #- > maximums are checked #- > and exceptions raised and/or the numbers get changed. #- #- I strongly prefer for my programming language not to impose #- aribtrary limits, however, I can't see myself using decimal #- to represent numbers with 1 billion digits. So IMHO, I would #- avoid the limit if possible, but if imposing the limit #- permitted substantial code simplification or performance #- improvements then I wouldn't complain. Much. Actually, imposing the limit means more code and complexity, and I don't find any benefit. But as I answered to Paul, I'm searching for community agreement before changing functionality that I found in the implementation #- I disagree. If x == y, then that should imply that hash(x) #- == hash(y). #- So hash(Decimal(25)) == hash(25) had better be true. On the other #- hand, hash(Decimal('-33')) CANNOT equal hash(Decimal('-33')), since #- the latter must equal hash(-33). And I'm not particularly concerned #- about floats... if it's easy to make the hashes equal for those few #- numbers which can be expressed EXACTLY as both a (binary) float and #- a (decimal) Decimal, then that's be "kinda nice", but since there #- are so few values which can be expressed exactly as both a float #- and a Decimal, I don't see it as a big issue. So, while Decimal(25) == 25 is True, hash(Decimal(25)) should be equal to hash(25). The detail is that you can NOT compare Decimal to floats or strings, so maybe we should not worry about them to give the same hashes. I'm OK to make hash(int or long) == hash(Decimal(int or long)). But only to int or long, not float nor string. Do you agree? . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040413/855b22a6/attachment-0001.html From shane at zope.com Tue Apr 13 16:09:35 2004 From: shane at zope.com (Shane Hathaway) Date: Tue Apr 13 16:09:28 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <563EA5F2-8D85-11D8-B460-000A95686CD8@redivi.com> References: <1081884676.407c4004be701@mcherm.com> <563EA5F2-8D85-11D8-B460-000A95686CD8@redivi.com> Message-ID: <407C48FF.1070000@zope.com> Bob Ippolito wrote: > As far as arbitrary floats hashing equally, I don't really care, but the > simple ones definitely should. Hopefully at some point, "float > constants" (as in coming from source code or user input) in Python will > actually be Decimal representations, so you don't lose precision until > you ask for it. Should we expect this to work? d = {} d[Decimal('1.1')] = 1 print d[float('1.1')] Maybe pathological, maybe not. Shane From FBatista at uniFON.com.ar Tue Apr 13 16:08:01 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 16:10:04 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Robert Brewer] #- My only "strong opinion" is that, IF there are limits, they should be #- consumer-modifiable. Given that, I'd still prefer no limits. In #- plusminusspeak: #- #- No limit: +1 #- Modifiable limits: +0 #- Concrete limits: -1 Nice. Thank you. . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040413/8e36e758/attachment.html From FBatista at uniFON.com.ar Tue Apr 13 16:15:51 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 16:17:54 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Bob Ippolito] #- >>> hash(float(1.0)) == hash(1) == hash(1L) #- True #- #- Since this is already true, hash(Decimal.from_float(1.0)), #- hash(Decimal(1)), hash(Decimal(1L)), hash(Decimal('1.0')), #- hash(Decimal('1')), hash(Decimal(u'1.0')), etc. better be equal as #- well, whatever that takes. #- #- As far as arbitrary floats hashing equally, I don't really care, but #- the simple ones definitely should. Hopefully at some point, "float #- constants" (as in coming from source code or user input) in #- Python will #- actually be Decimal representations, so you don't lose #- precision until #- you ask for it. I agree. The detail is that the "simple ones" means the same Decimal that the ones constructed from ints: >>> Decimal.from_float(1.0) Decimal( (0, (1,), 0L) ) >>> Decimal(1) Decimal( (0, (1,), 0) ) >>> hash(Decimal.Decimal(1)) 1 So, you'll *always* have the same hash there. Regarding user input, treating them as strings, >>> Decimal('1') Decimal( (0, (1,), 0) ) >>> hash('1') 1977051568 There, you won't have the same hash that the string. But I don't think we need this as long as you can not compare Decimal with strings... . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040413/1acbb44d/attachment.html From FBatista at uniFON.com.ar Tue Apr 13 16:20:00 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 16:22:16 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Shane Hathaway] #- Should we expect this to work? #- #- d = {} #- d[Decimal('1.1')] = 1 #- print d[float('1.1')] #- No, because Decimal('1.1') is not the same number than float('1.1')! >>> d = {} >>> d[Decimal('1.1')] = 1 >>> d {Decimal( (0, (1, 1), -1) ): 1} >>> d[float('1.1')] Traceback (most recent call last): File "", line 1, in ? d[float('1.1')] KeyError: 1.1000000000000001 In part, this is why you can not compare Decimal with float, and why you can not create a Decimal directly from float. You can use the alternative method, but you should know what you're doing: >>> Decimal.from_float(1.1) Decimal( (0, (1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 1, 7, 8, 4, 1, 9, 7, 0, 0, 1, 2, 5, 2, 3, 2, 3, 3, 8, 9, 0, 5, 3, 3, 4, 4, 7, 2, 6, 5, 6, 2, 5), -51L) ) >>> Decimal.Decimal.from_float(1.1, 1) Decimal( (0, (1, 1), -1L) ) . Facundo From pf_moore at yahoo.co.uk Tue Apr 13 16:34:51 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Apr 13 16:34:55 2004 Subject: [Python-Dev] Re: Decimal data type issues References: Message-ID: "Batista, Facundo" writes: [...] > Actually, imposing the limit means more code and complexity, and I don't > find any benefit. But as I answered to Paul, I'm searching for community > agreement before changing functionality that I found in the implementation [...] > So, while Decimal(25) == 25 is True, hash(Decimal(25)) should be equal to > hash(25). > > The detail is that you can NOT compare Decimal to floats or strings, so > maybe we should not worry about them to give the same hashes. > > I'm OK to make hash(int or long) == hash(Decimal(int or long)). But only to > int or long, not float nor string. > > Do you agree? So far, I've found your instincts to be pretty good. I'd say 1. Remove the limits. 2. Make hashes the same for int/long, but not str/float. 3. While I'm at it, I'm OK with the names you suggest for the 3 extra functions. Paul. -- This signature intentionally left blank From FBatista at uniFON.com.ar Tue Apr 13 16:48:56 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 16:51:04 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Paul Moore] #- So far, I've found your instincts to be pretty good. I'd say #- #- 1. Remove the limits. #- 2. Make hashes the same for int/long, but not str/float. #- 3. While I'm at it, I'm OK with the names you suggest for the 3 extra #- functions. Thank you very much. . Facundo From bob at redivi.com Tue Apr 13 17:13:22 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Apr 13 17:09:15 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <65040708-8D8F-11D8-B460-000A95686CD8@redivi.com> On Apr 13, 2004, at 4:34 PM, Paul Moore wrote: > "Batista, Facundo" writes: > > [...] > >> Actually, imposing the limit means more code and complexity, and I >> don't >> find any benefit. But as I answered to Paul, I'm searching for >> community >> agreement before changing functionality that I found in the >> implementation > > [...] > >> So, while Decimal(25) == 25 is True, hash(Decimal(25)) should be >> equal to >> hash(25). >> >> The detail is that you can NOT compare Decimal to floats or strings, >> so >> maybe we should not worry about them to give the same hashes. >> >> I'm OK to make hash(int or long) == hash(Decimal(int or long)). But >> only to >> int or long, not float nor string. >> >> Do you agree? > > So far, I've found your instincts to be pretty good. I'd say > > 1. Remove the limits. > 2. Make hashes the same for int/long, but not str/float. Make hash(Decimal('123.1')) == hash(Decimal('123.1')) .. there is no reason not to. What's the point of having a Decimal type if the only way to get an accurate representation of a non-integer is to do some garbage like Decimal(1231)/Decimal(10)? hash(float(1.0)) == hash(Decimal(1)) comes for free, because hash(float(1.0)) == hash(1). For the few other non-integer numbers that can be accurately represented as floating point, it might make sense to keep the same property? Though I suppose that is pretty what-the-heck-does-C-do specific, so I wouldn't blame anyone if this property couldn't be maintained. > 3. While I'm at it, I'm OK with the names you suggest for the 3 extra > functions. -bob From FBatista at uniFON.com.ar Tue Apr 13 17:23:29 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 13 17:25:33 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Bob Ippolito] #- Make hash(Decimal('123.1')) == hash(Decimal('123.1')) .. there is no I'm pretty sure that those two hash will return the same value, as long have the same argument (?). #- hash(float(1.0)) == hash(Decimal(1)) comes for free, because #- hash(float(1.0)) == hash(1). For the few other non-integer numbers #- that can be accurately represented as floating point, it might make #- sense to keep the same property? Though I suppose that is pretty #- what-the-heck-does-C-do specific, so I wouldn't blame anyone if this #- property couldn't be maintained. OK, so I won't promise it. . Facundo From martin at v.loewis.de Tue Apr 13 17:26:40 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Apr 13 17:26:56 2004 Subject: [Python-Dev] pipe syntax for sequential data In-Reply-To: References: Message-ID: <407C5B10.30103@v.loewis.de> Maxim Krikun wrote: > I don't know if this idea was already explored before; hope this can > be useful. For details and sample code please see link above. As you can see from the responses you got so far, readership of python-dev is uncertain what to do with your message, which primarily originates from the fact that you did not indicate what you expect readership to do. If you merely want to educate people to use your recipe, python-dev might be the wrong forum: readers of that list will quickly grasp your interesting approach, but may then chose to ignore it. It may be more educating to people who were unaware of __ror__ before; you would reach a greater audience on news:comp.lang.python. If you also want to release this technique as an actual library, you should find some packaging, and announce it to news:comp.lang.python.announce. If you then also want this library to be included as a part of the standard Python distribution, you should write a PEP. You will probably find that people would complain about the ad-hoc list of filter objects you provide, their ad-hoc naming, and the relative youth of the entire project. I personally find the approach entertaining, but likely will never use it in a real project, because it is too cute for my taste. Regards, Martin From t-meyer at ihug.co.nz Tue Apr 13 18:50:19 2004 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Tue Apr 13 18:50:37 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <1ED4ECF91CDED24C8D012BCF2B034F1305E11770@its-xchg4.massey.ac.nz> Message-ID: <1ED4ECF91CDED24C8D012BCF2B034F13026F2B86@its-xchg4.massey.ac.nz> > c. ``to-number``: This operation converts a string to a > number, as defined by its abstract representation. [...] > c. from_string [...] > Regarding the method (c), the only difference with creating > the decimal with Decimal(string) is that method (c) honors > the context (if the literal contains more digits that the > current precision the numbers get rounded, and gets rounded > according to the round method specified in context, etc). > For example, with a precision of 9 and with the name I proposed:: > > >>> Decimal('112233445566') > Decimal( (0, (1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6), 0) ) > >>> Decimal.from_string('112233445566') > Decimal( (0, (1, 1, 2, 2, 3, 3, 4, 4, 6), 3L) ) As a relative newbie, I think that it would be better if (c) had a name that somehow indicated the difference; however, I can't think of one offhand that does :) It seems to me that it would be easy to assume that the two above cases were the same (and given that they produce the same result in some situations, might be missed by weak testing). Certainly a quick look at documentation would correct that, but if the name could also reflect the difference, that would be better. Even if from_string took a parameter "honour_context", defaulting to True, that would, IMO, be more clear that there's a difference between the two, and what it is. So, (a) +1 (b) +1 (c) +0 =Tony Meyer From raymond.hettinger at verizon.net Tue Apr 13 19:59:53 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Tue Apr 13 20:00:50 2004 Subject: [Python-Dev] String hash function multiplier Message-ID: <001c01c421b3$6acd0460$e841fea9@oemcomputer> Does anyone have any issues with changing the hash multiplier for the string and Unicode hash functions? Instead of 1000003, I would like to use Aho's 65599, a prime near 2**16 that is efficiently expressible as (x << 6) + (x << 16) - x. This replaces a multiply with fast adds and shifts (with the two shifts potentially being executed in parallel). Googling for "hash 65599" shows a long history of widespread use and testing without any problems. Raymond Hettinger From jepler at unpythonic.net Tue Apr 13 21:09:54 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Tue Apr 13 21:10:32 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <001c01c421b3$6acd0460$e841fea9@oemcomputer> References: <001c01c421b3$6acd0460$e841fea9@oemcomputer> Message-ID: <20040414010954.GA8795@unpythonic.net> With -O2 -mcpu=i686 or newer, gcc uses "imul" for both 100003 and 65599, rather than shifts and adds. There may be a few people who care about some other processor, but I wouldn't listen to them. (the only non-x86 CPU I program for on a weekly basis doesn't have hardware multiply, but it's also much too small for Python) The current value goes back a long way: http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Objects/stringobject.c#rev2.31 ... all the way back to when Python did string haching instead of hashing. Other than some abstract beauty to 65599, are there some other practical advantages I'm missing? Jeff From bob at redivi.com Tue Apr 13 21:30:28 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Apr 13 21:26:16 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <20040414010954.GA8795@unpythonic.net> References: <001c01c421b3$6acd0460$e841fea9@oemcomputer> <20040414010954.GA8795@unpythonic.net> Message-ID: <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> On Apr 13, 2004, at 9:09 PM, Jeff Epler wrote: > With -O2 -mcpu=i686 or newer, gcc uses "imul" for both 100003 and > 65599, > rather than shifts and adds. > > There may be a few people who care about some other processor, but I > wouldn't listen to them. (the only non-x86 CPU I program for on a > weekly > basis doesn't have hardware multiply, but it's also much too small for > Python) > > The current value goes back a long way: > http://cvs.sourceforge.net/viewcvs.py/python/python/dist/src/Objects/ > stringobject.c#rev2.31 > ... all the way back to when Python did string haching instead of > hashing. > > Other than some abstract beauty to 65599, are there some other > practical > advantages I'm missing? It's not expected that GCC optimize an integer constant into shifts on its own. Anyways, a practical advantage is that with a sane instruction set, like PPC, it saves you a memory access or some instructions (depending on the compiler I guess). Both 100003 and 65599 are too big to be immediate values in a PPC instruction, but the shift constants are not. I guess the real question for Raymond is, does it really make a measurable difference? And what effect does it have on pickled dicts (or other such hash-using data structures), if any? -bob From jepler at unpythonic.net Tue Apr 13 22:04:10 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Tue Apr 13 22:04:49 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> References: <001c01c421b3$6acd0460$e841fea9@oemcomputer> <20040414010954.GA8795@unpythonic.net> <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> Message-ID: <20040414020410.GC8795@unpythonic.net> On Tue, Apr 13, 2004 at 09:30:28PM -0400, Bob Ippolito wrote: > It's not expected that GCC optimize an integer constant into shifts on > its own. But gcc does! -- with -mcpu=i386, 65599 is optimized into some shifts, and 100003 is optimized into some very painful use of "lea" (x is in edx): lea (%edx,%edx,4),%eax // eax = 5 * edx lea (%eax,%eax,4),%eax // eax = 25 * edx lea (%eax,%eax,4),%eax // eax = 125 * edx lea (%eax,%eax,4),%eax // eax = 625 * edx lea (%eax,%eax,4),%eax // eax = 3125 * edx lea (%eax,%eax,4),%eax // eax = 15625 * edx shl $0x5,%eax // eax = 50000 * edx add %edx,%eax // eax = 50001 * edx lea (%edx,%eax,2),%edx // edx = 100003 * edx On the newer x86 CPUs (starting at i686 / k6) imul is preferred by the optimizer. Here's what 65599 gives, with -mcpu=i386 (x is in edx again): mov %edx,%eax // eax = edx shl $0xa,%eax // eax = edx * 1024 add %edx,%eax // eax = edx * 1025 shl $0x6,%eax // eax = edx * 65600 sub %edx,%eax // eax = edx * 65599 mov %eax,%edx // edx = eax If gcc can emit these tortured instruction sequences, but chooses not to, I have to suspect it knows the properties of the CPU better than me. Jeff From python at rcn.com Tue Apr 13 22:16:16 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue Apr 13 22:17:18 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> Message-ID: <001d01c421c6$77b4df00$e841fea9@oemcomputer> [Jeff Epler] > > With -O2 -mcpu=i686 or newer, gcc uses "imul" for both 100003 and > > 65599, > > rather than shifts and adds. [Bob Ippolito] > It's not expected that GCC optimize an integer constant into shifts on > its own. Right, the actual diff is: *** 1145,1151 **** p = (unsigned char *) a->ob_sval; x = *p << 7; while (--len >= 0) ! x = (1000003*x) ^ *p++; x ^= a->ob_size; if (x == -1) x = -2; --- 1152,1158 ---- p = (unsigned char *) a->ob_sval; x = *p << 7; while (--len >= 0) ! x = (x << 6) + (x << 16) - x + (long)*p++; x ^= a->ob_size; if (x == -1) x = -2; > I guess the real question for Raymond is, does it really make a > measurable difference? Yes, the loop runs 20% faster on my Pentium III. The speedup ought to be much more dramatic on the Pentium IV (where the integer ALU instructions speedup from 1 clock to 0.5 clocks while the latency on integer multiplication slows down from 4 clocks to 14-18 clocks). > And what effect does it have on pickled dicts > (or other such hash-using data structures), if any? The test suite runs fine. Dicts may display in a different order than in previous pythons. That may upset some doctests if the writer didn't take Tim's documented advice about making tests that do not depend on display order. Raymond From jepler at unpythonic.net Tue Apr 13 23:09:38 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Tue Apr 13 23:10:37 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <001d01c421c6$77b4df00$e841fea9@oemcomputer> References: <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> <001d01c421c6$77b4df00$e841fea9@oemcomputer> Message-ID: <20040414030938.GD8795@unpythonic.net> On Tue, Apr 13, 2004 at 10:16:16PM -0400, Raymond Hettinger wrote: > Yes, the loop runs 20% faster on my Pentium III. The speedup ought to > be much more dramatic on the Pentium IV (where the integer ALU > instructions speedup from 1 clock to 0.5 clocks while the latency on > integer multiplication slows down from 4 clocks to 14-18 clocks). I got different results here, also on a Pentium III (but in a laptop) I'm using gcc, I don't know what you're using. I get the best results from -O3 -mcpu=pentium3 for both MUL values and do worse with CLEVER in most cases. In my test, PyStringObject is not quite the same as in Python, so this could explain different results. Use of a different compiler, or gcc with the implied default -mcpu=i386 (?), might explain why you benchmarked the shifts as better, too. (-O{2,3} -mcpu=i386) [* = best alternative for these optimizer flags] -O -mcpu=i386 -DMUL=100003 1.78 -O -mcpu=i386 -DMUL=65599 1.37 * -O -mcpu=i386 -DCLEVER 1.40 -O -mcpu=pentium3 -DMUL=100003 1.27 * -O -mcpu=pentium3 -DMUL=65599 1.27 * -O -mcpu=pentium3 -DCLEVER 1.35 -O2 -mcpu=i386 -DMUL=100003 1.93 -O2 -mcpu=i386 -DMUL=65599 1.54 -O2 -mcpu=i386 -DCLEVER 1.28 * -O2 -mcpu=pentium3 -DMUL=100003 1.11 * -O2 -mcpu=pentium3 -DMUL=65599 1.12 -O2 -mcpu=pentium3 -DCLEVER 1.29 -O3 -mcpu=i386 -DMUL=100003 1.69 -O3 -mcpu=i386 -DMUL=65599 1.28 -O3 -mcpu=i386 -DCLEVER 1.10 * -O3 -mcpu=pentium3 -DMUL=100003 0.90 * -O3 -mcpu=pentium3 -DMUL=65599 0.90 * -O3 -mcpu=pentium3 -DCLEVER 1.05 -Os -mcpu=i386 -DMUL=100003 1.16 * -Os -mcpu=i386 -DMUL=65599 1.16 * -Os -mcpu=i386 -DCLEVER 1.45 -Os -mcpu=pentium3 -DMUL=100003 1.05 * -Os -mcpu=pentium3 -DMUL=65599 1.05 * -Os -mcpu=pentium3 -DCLEVER 1.30 # alternatives.py OPT = [ '-O -mcpu=i386', '-O -mcpu=pentium3', '-O2 -mcpu=i386', '-O2 -mcpu=pentium3', '-O3 -mcpu=i386', '-O3 -mcpu=pentium3', '-Os -mcpu=i386', '-Os -mcpu=pentium3', ] HASH = ['-DMUL=100003', '-DMUL=65599', '-DCLEVER'] import sys, os for o in OPT: for h in HASH: sys.stdout.write("%-20s %-20s " % (o, h)) sys.stdout.flush() os.system("gcc %s %s hashtest.c && TIME=%%U /usr/bin/time ./a.out" % (o, h)) print // hashtest.c #ifndef MUL #define MUL 100003 #endif typedef struct { long ob_shash; unsigned long ob_size; unsigned char ob_sval[16]; } PyStringObject; static long string_hash(PyStringObject *a) { register int len; register unsigned char *p; register long x; if (a->ob_shash != -1) return a->ob_shash; len = a->ob_size; p = (unsigned char *) a->ob_sval; x = *p << 7; while (--len >= 0) #ifdef CLEVER x = (x << 6) + (x << 16) - x + (long)*p++; #else x = (MUL*x) ^ *p++; #endif x ^= a->ob_size; if (x == -1) x = -2; a->ob_shash = x; return x; } int main(void) { PyStringObject s = {-1, 13, "hello raymond"}; int i; for(i=0; i < 1<<23; i++) { string_hash(&s); s.ob_shash = -1; } return 0; } From guido at python.org Tue Apr 13 23:25:03 2004 From: guido at python.org (Guido van Rossum) Date: Tue Apr 13 23:26:33 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: Your message of "Tue, 13 Apr 2004 19:59:53 EDT." <001c01c421b3$6acd0460$e841fea9@oemcomputer> References: <001c01c421b3$6acd0460$e841fea9@oemcomputer> Message-ID: <200404140325.i3E3P4722847@guido.python.org> > Does anyone have any issues with changing the hash multiplier for the > string and Unicode hash functions? > > Instead of 1000003, I would like to use Aho's 65599, a prime near 2**16 > that is efficiently expressible as (x << 6) + (x << 16) - x. This > replaces a multiply with fast adds and shifts (with the two shifts > potentially being executed in parallel). > > Googling for "hash 65599" shows a long history of widespread use and > testing without any problems. It would break the parrot benchmark, which has a re-implementation of the hash in Python and assumes/checks at various points that the two agree. So I'd rather not see this changed before July 30. --Guido van Rossum (home page: http://www.python.org/~guido/) From python-dev at zesty.ca Tue Apr 13 23:29:39 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Tue Apr 13 23:29:11 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <1ED4ECF91CDED24C8D012BCF2B034F13026F2B86@its-xchg4.massey.ac.nz> References: <1ED4ECF91CDED24C8D012BCF2B034F13026F2B86@its-xchg4.massey.ac.nz> Message-ID: Facundo Batista wrote: > c. from_string [...] > Regarding the method (c), the only difference with creating > the decimal with Decimal(string) is that method (c) honors > the context (if the literal contains more digits that the > current precision the numbers get rounded, and gets rounded > according to the round method specified in context, etc). > For example, with a precision of 9 and with the name I proposed:: > > >>> Decimal('112233445566') > Decimal( (0, (1, 1, 2, 2, 3, 3, 4, 4, 5, 5, 6, 6), 0) ) > >>> Decimal.from_string('112233445566') > Decimal( (0, (1, 1, 2, 2, 3, 3, 4, 4, 6), 3L) ) On Wed, 14 Apr 2004, Tony Meyer wrote: > As a relative newbie, I think that it would be better if (c) had a name that > somehow indicated the difference; however, I can't think of one offhand that > does :) It seems to me that it would be easy to assume that the two above > cases were the same (and given that they produce the same result in some > situations, might be missed by weak testing). Certainly a quick look at > documentation would correct that, but if the name could also reflect the > difference, that would be better. > > Even if from_string took a parameter "honour_context", defaulting to True, > that would, IMO, be more clear that there's a difference between the two, > and what it is. I agree with Tony. Having both Decimal() and Decimal.from_string() work in similar but subtly different ways seems likely to be misunderstood. I also think his last suggestion could provide a nice, convenient solution to the conversion-from-float issue, since it raises the same question (a method named "Decimal.from_float" doesn't explain how or why it is different from just constructing with a float). So here's a more detailed suggestion. Suppose s = '12345', f = 12.345, the default context specifies a precision of 9 as given in the PEP, and c is a context object with a precision of 6. Then this: would yield this: Decimal(s) Decimal((0, (1, 2, 3, 4, 5), 0)) Decimal(s, 2) Decimal((0, (1, 2), 4)) Decimal(s, context=c) Decimal((0, (1, 2, 3, 4, 5, 0), -1)) Decimal(s, default_context=1) Decimal((0, (1, 2, 3, 4, 5, 0, 0, 0, 0), -4)) Decimal(f) Decimal((0, (1, 2, 3, 4, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 6, 3, 9, 4, 8, 8, 4, 6, 2, 1, 8, 4, 0, 9, 0, 1, 6, 7, 2, 8, 4, 0, 1, 1, 8, 4, 0, 8, 2, 0, 3, 1, 2, 5), -49)) Decimal(f, 2) Decimal((0, (1, 2), 0)) Decimal(f, context=c) Decimal((0, (1, 2, 3, 4, 5, 0), -4)) Decimal(f, default_context=1) Decimal((0, (1, 2, 3, 4, 5, 0, 0, 0, 0), -6)) How about that? (Do I assume correctly that there will be context objects that wrap up the precision and rounding mode, and there will be some sort of interface for getting and setting the current context?) -- ?!ng From python-dev at zesty.ca Tue Apr 13 23:35:19 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Tue Apr 13 23:34:48 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: References: Message-ID: Sorry if I missed this, but I didn't see a description in the PEP of how Decimal numbers would convert to other types. If d is a Decimal number, I assume int(d) is obvious and float(d) does the best it can with double precision, right? How about str(d)? I would hope that it uses exponential notation only when necessary to preserve the number of significant digits. Could we please have repr(d) use the string in the constructor rather than the tuple form? That would be much easier to read. So, for example: >>> Decimal('12345') Decimal('12345') # instead of Decimal((0, (1, 2, 3, 4, 5, 0)) >>> _ * 10 Decimal('1.2345e1') >>> Decimal('12345', 2) Decimal('1.2e3') >>> Decimal(1.1) Decimal('1.100000000000000088817841970012523233890533447265625') >>> Decimal(1.1, default_context=1) Decimal('1.10000000') -- ?!ng From tim_one at email.msn.com Tue Apr 13 23:40:32 2004 From: tim_one at email.msn.com (Tim Peters) Date: Tue Apr 13 23:56:38 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <001c01c421b3$6acd0460$e841fea9@oemcomputer> Message-ID: [Raymond] > Does anyone have any issues with changing the hash multiplier for the > string and Unicode hash functions? Don't touch it unless you can prove major benefits -- it's a remarkable fact of life that the current multiplier hasn't resulted in any real-life (but non-contrived) pathological cases. > Instead of 1000003, I would like to use Aho's 65599, a prime near 2**16 > that is efficiently expressible as (x << 6) + (x << 16) - x. This > replaces a multiply with fast adds and shifts (with the two shifts > potentially being executed in parallel). It's unclear why that's a good thing. Perhaps you think shifts and adds are faster? I wouldn't -- the imul instruction on modern Pentiums is very fast. It is clear why it may be a bad thing: that it *can* be expressed as just a couple shifts and adds makes it suspect as a good scrambler (read Knuth). > Googling for "hash 65599" shows a long history of widespread use and > testing without any problems. Testing in the context of Python string hashing? I didn't think so . The current multiplier has been studied extensively *in* Python, both via real-life use and via large-scale focused statistical testing (we got some money to do that during BeOpen.com's brief life -- a surprise that came out of that was that CRC made a terrible string hash function as the number of input strings got large). The right thing to compare Python's string hash to is "the standard" Fowler-Noll-Vo string hash, which was developed independently, but which also ended up using a large multiplier. From tim_one at email.msn.com Wed Apr 14 00:53:57 2004 From: tim_one at email.msn.com (Tim Peters) Date: Wed Apr 14 00:54:16 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: Message-ID: [Batista, Facundo] ... > The exponent is an integer, and in the actual implementation exists a > maximum value:: > > DEFAULT_MAX_EXPONENT = 999999999 > DEFAULT_MIN_EXPONENT = -999999999 > ABSOLUTE_MAX_EXP = 999999999 > ABSOLUTE_MIN_EXP = -999999999 > > The issue is that this limit is artificial: As long it's a long, you > should be able to make it as big as your memory let you. > > In the General Decimal Arithmetic Specification says: > > In the abstract, there is no upper limit on the absolute value of the > exponent. In practice there may be some upper limit, E_limit , on the > absolute value of the exponent. > > So, should we impose an artificial limit to the exponent? I expect everyone's initial reaction will be to say "no limit", but think two or three times about that. Eliminating any restriction on exponent magnitude is the same as saying overflow can never happen. But overflow is usually a valuable safety net in real life, telling you that your algorithm has gone insane. OTOH, there's nothing with physical significance that requires an exponent anywhere near as large as 999999999, and REXX has used that exponent limit for years without problems. Everyone should also note that a restriction on exponent size isn't also a restriction on the number of significant digits: if the max exponent is restricted to 999999999, you can still have more than 10**999999999 significant digits (as soon as you buy enough disk space and a CPU with unbounded addresses ). > a. to_sci_string > b. to_eng_string > c. from_string > > The (a) and (b) methods are different from str, as this method just > doesn't adjust the exponent at all. > > Regarding the method (c), the only difference with creating the decimal > with Decimal(string) is that method (c) honors the context (if the > literal contains more digits that the current precision the numbers > get rounded, and gets rounded according to the round method specified > in context, etc). (c) also specifies ways to spell special values, like NaNs and infinities. Does Decimal(string)? I don't believe the PEP covered that. In any case, from_string() should be subsumed by Decimal(string), perhaps with via an optional "use_context" flag argument. It's quite arguable that str() should be exactly the standard's to_sci_string() (and for those who haven't read the spec, do before you complain about that -- no, you don't always get an exponent under to_sci_string()). > Hash behaviour > -------------- Decimals that happen to be exact integers must return the same hash codes as the hash codes of the ints or longs they're equal to. All bets are off wrt hash codes for non-integral Decimals (e.g., hash(Decimal('1.5')) may or may not be the same as hash(1.5)); promising more than that is simply impractical. BTW, hash(Decimal('1.0')) must be the same as hash(Decimal('1.00')) must be the same as hash(Decimal('0.001e3')) despite that they have different internal representations. ... > Should the following be true?:: > > hash(Decimal(25)) == hash(25) Yes. > hash(Decimal.from_float(25.35) == hash(25.35) Not defined. > hash(Decimal('-33.8')) == hash(-33.8) Not defined. From greg at cosc.canterbury.ac.nz Wed Apr 14 02:24:40 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Apr 14 02:24:58 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040409154252.GB28808@panix.com> Message-ID: <200404140624.i3E6OeMg017756@cosc353.cosc.canterbury.ac.nz> Aahz : > Nope. Absolute imports are the most common form I don't agree with that. In every major Python application I've written, the vast majority of imports are from other modules of the application, which are most sensibly referenced relatively. I, too, am quite concerned about the imminent breaking of every existing relative import, apparently because Guido has an ideological objection to relative imports and wants to punish us for having used them so freely over the years. If he goes ahead with this, I suggest we hack his unix system and remove the ability to use relative pathnames. Let's see how he likes using that. :-) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From jepler at unpythonic.net Wed Apr 14 09:08:36 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Apr 14 09:08:41 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <20040414030938.GD8795@unpythonic.net> References: <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> <001d01c421c6$77b4df00$e841fea9@oemcomputer> <20040414030938.GD8795@unpythonic.net> Message-ID: <20040414130834.GS6139@unpythonic.net> I benchmarked this on a Pentium 4 and Raymond's hash does indeed come out ahead, regardless of compiler flags. Pentium IV, 2.4GHz: -O2 -mcpu=i386 -DMUL=100003 1.56 -O2 -mcpu=i386 -DMUL=65599 0.78 -O2 -mcpu=i386 -DCLEVER 0.54 * -O2 -mcpu=pentium3 -DMUL=100003 0.78 -O2 -mcpu=pentium3 -DMUL=65599 0.79 -O2 -mcpu=pentium3 -DCLEVER 0.53 * -O2 -mcpu=pentium4 -DMUL=100003 0.63 -O2 -mcpu=pentium4 -DMUL=65599 0.65 -O2 -mcpu=pentium4 -DCLEVER 0.50 * With AMD CPUs, the current multiplier beats both the new multipler and the version expressed as shifts and adds/subtracts: On a Duron, 1GHz: -O2 -mcpu=i386 -DMUL=100003 2.03 -O2 -mcpu=i386 -DMUL=65599 1.04 -O2 -mcpu=i386 -DCLEVER 0.95 * -O2 -mcpu=athlon -DMUL=100003 0.92 * -O2 -mcpu=athlon -DMUL=65599 1.03 -O2 -mcpu=athlon -DCLEVER 0.94 On an Athlon XP 2600+: -O2 -mcpu=i386 -DMUL=100003 0.95 -O2 -mcpu=i386 -DMUL=65599 0.49 -O2 -mcpu=i386 -DCLEVER 0.44 * -O2 -mcpu=athlon-xp -DMUL=100003 0.43 * -O2 -mcpu=athlon-xp -DMUL=65599 0.48 -O2 -mcpu=athlon-xp -DCLEVER 0.45 If we want a fast hash function, then we should write one that works a "long" at a time. This probably can't be done if hashes must be equal on different machines, but aren't hashes already different on LP64 machines (because they're 64 bits instead of 32)? (I guess the low-order bits would be identical) Long-at-a-time hash, Duron, 1GHz: -O2 -march=athlon-tbird -DMUL=100003 0.35 -O2 -march=athlon-tbird -DMUL=65599 0.41 -O2 -march=athlon-tbird -DCLEVER 0.42 With this code, it would be necessary to allocate strings "rounded up" to 4 bytes, and zero the unused bytes. Or we could do as Tim and Guido suggest: nothing. Jeff /* long-at-a-time hashing */ typedef struct { long ob_shash; unsigned long ob_size; union { unsigned char ob_sval[16]; long ob_svall[4]; }; } PyStringObject; static long string_hash(PyStringObject *a) { register int len; register long *p; register long x; if (a->ob_shash != -1) return a->ob_shash; len = (a->ob_size+3) / 4; p = a->ob_svall; x = *p << 7; while (--len >= 0) #ifdef CLEVER x = (x << 6) + (x << 16) - x + *p++; #else x = (MUL*x) ^ *p++; #endif x ^= a->ob_size; if (x == -1) x = -2; a->ob_shash = x; return x; } From aahz at pythoncraft.com Wed Apr 14 10:39:35 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Apr 14 10:39:43 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: Message-ID: <20040414143935.GB5922@panix.com> On Tue, Apr 13, 2004, Batista, Facundo wrote: > > Exponent Maximum > ---------------- > > The Decimal number composes of three elements: A sign that can be 0 or 1, a > tuple of digits where each can be 0..9, and an exponent. > > The exponent is an integer, and in the actual implementation exists a > maximum > value:: > > DEFAULT_MAX_EXPONENT = 999999999 > DEFAULT_MIN_EXPONENT = -999999999 > ABSOLUTE_MAX_EXP = 999999999 > ABSOLUTE_MIN_EXP = -999999999 > > The issue is that this limit is artificial: As long it's a long, you should > be able to make it as big as your memory let you. As long as it's in Python code, it's easy enough for a whiner to modify it. Leave it as-is. Other than that, I agree with Tim. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From aahz at pythoncraft.com Wed Apr 14 10:42:26 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Apr 14 10:42:32 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: <1ED4ECF91CDED24C8D012BCF2B034F13026F2B86@its-xchg4.massey.ac.nz> Message-ID: <20040414144226.GC5922@panix.com> On Tue, Apr 13, 2004, Ka-Ping Yee wrote: > > So here's a more detailed suggestion. Suppose s = '12345', f = 12.345, > the default context specifies a precision of 9 as given in the PEP, and > c is a context object with a precision of 6. > > Then this: would yield this: > > Decimal(f) Decimal((0, (1, 2, 3, 4, 5, 0, 0, 0, 0, 0, > 0, 0, 0, 0, 0, 0, 0, 6, 3, 9, > 4, 8, 8, 4, 6, 2, 1, 8, 4, 0, > 9, 0, 1, 6, 7, 2, 8, 4, 0, 1, > 1, 8, 4, 0, 8, 2, 0, 3, 1, 2, > 5), -49)) I'll continue to oppose allowing Decimal() to take a float argument. It's just asking for trouble. Re-read the previous thread about repr() of floats if you want to argue with me; I shan't say anything new. ;-) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From aahz at pythoncraft.com Wed Apr 14 10:46:14 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed Apr 14 10:46:21 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: References: Message-ID: <20040414144614.GD5922@panix.com> On Tue, Apr 13, 2004, Ka-Ping Yee wrote: > > Could we please have repr(d) use the string in the constructor > rather than the tuple form? That would be much easier to read. I'm not dead-set on using a tuple form, but I think it's more useful for someone trying to see the internals of a Decimal instance. The string form already contains all the necessary information to reconstruct a Decimal object (unlike floats). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Why is this newsgroup different from all other newsgroups? From jim.jewett at eds.com Wed Apr 14 11:27:12 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Wed Apr 14 11:28:10 2004 Subject: [Python-Dev] PEP 318 - check for consensus Message-ID: The decorator discussion has gotten lively again, and a summary would be a long list of options. I'm hoping that at least some of the decisions are non-controversial. If you have an opinion on the following, please email me directly (to not clutter the list). If several people disagree (or some disagree strongly), I'll consider the issue still undecided. Issue 1: When is the name bound? The current proposal is that the name does not get bound until after the final decorator is applied. This means that decorators do not have access to the function by its name. They can learn the name, perhaps through sys._getframe, but the name will not yet be bound, unless it happens to refer to an unrelated object. (See issue two for a better way to get the name.) Does anyone feel strongly that the name should be bound to the original function before decorators are applied? Does anyone feel strongly that the name should be bound to the result of each intermediate step, if there are multiple decorators? Issue 2: Restrictions on decorators The working assumption had been "any callable, which will be called with a single object as the argument" Issue 2a: name of the entry point Is there a strong feeling (in any direction) about calling the decorator's "decorate" or "__decorate__" attribute, rather than its "__call__" attribute? Issue 2b: context available to decorators Would there be any objection to passing additional context as keywords? decorate(object, name="myfunc") The obvious keywords are the name and the context's globals; decorators should probably accept arbitrary keywords for forward-compatibility. -jJ From python at rcn.com Wed Apr 14 12:05:50 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed Apr 14 12:06:49 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: Message-ID: <001901c4223a$5bab0a40$e841fea9@oemcomputer> > [Raymond] > > Does anyone have any issues with changing the hash multiplier for the > > string and Unicode hash functions? [Tim] > Don't touch it unless you can prove major benefits -- it's a remarkable > fact > of life that the current multiplier hasn't resulted in any real-life (but > non-contrived) pathological cases. Will leave it alone. > Perhaps you think shifts and adds > are > faster? I wouldn't -- the imul instruction on modern Pentiums is very > fast. On the P4, the documented latency went up from 4 cycles to 14 cycles while shifts and adds went down to 0.5 cycles and 1 cycle respectively. Timings confirm the result. It looks like the best bet is to try to speedup the code without changing the multiplier. Intel's software optimization cookbook recommends a partial unrolling and elimination of data dependencies so that a second multiply can start 4 cycles after the previous one started. If practice bears out the theory, the timings could show a three or fourfold speedup without changing the multiplier. > (read Knuth). Of course, I already have :-) > The right thing to compare Python's string hash > to is "the standard" Fowler-Noll-Vo string hash Ditto. Raymond ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From guido at python.org Wed Apr 14 12:30:30 2004 From: guido at python.org (Guido van Rossum) Date: Wed Apr 14 12:30:38 2004 Subject: [Python-Dev] PEP 318 - check for consensus In-Reply-To: Your message of "Wed, 14 Apr 2004 11:27:12 EDT." References: Message-ID: <200404141630.i3EGUUk24252@guido.python.org> > The decorator discussion has gotten lively again, > and a summary would be a long list of options. Really? If feels pretty quiet here on python-dev. > I'm hoping that at least some of the decisions are > non-controversial. If you have an opinion on the > following, please email me directly (to not clutter > the list). If several people disagree (or some > disagree strongly), I'll consider the issue still > undecided. > > Issue 1: When is the name bound? > > The current proposal is that the name does not get > bound until after the final decorator is applied. > This means that decorators do not have access to > the function by its name. > > They can learn the name, perhaps through sys._getframe, > but the name will not yet be bound, unless it happens > to refer to an unrelated object. (See issue two for > a better way to get the name.) A function's name is accessible through its __name__ attribute and also as func_name. I think that's quite enough. ;-) > Does anyone feel strongly that the name should be > bound to the original function before decorators are > applied? > > Does anyone feel strongly that the name should be > bound to the result of each intermediate step, if > there are multiple decorators? No. The function should not be bound until after all decorators have been applied. > Issue 2: Restrictions on decorators > > The working assumption had been "any callable, which > will be called with a single object as the argument" > > Issue 2a: name of the entry point > > Is there a strong feeling (in any direction) about > calling the decorator's "decorate" or "__decorate__" > attribute, rather than its "__call__" attribute? It should be __call__; that's the only backwards compatible way, and makes it easy to create decorators as functions. > Issue 2b: context available to decorators > > Would there be any objection to passing additional > context as keywords? > > decorate(object, name="myfunc") > > The obvious keywords are the name and the context's > globals; decorators should probably accept arbitrary > keywords for forward-compatibility. That's unnecessary; see answer to #1. --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Apr 14 12:46:35 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Apr 14 12:47:06 2004 Subject: [Python-Dev] PEP 318 - check for consensus In-Reply-To: Message-ID: <5.1.1.6.0.20040414123205.021d0380@telecommunity.com> (I'm replying to this on the list because there are a couple of errors in it regarding names and scopes, and you probably don't want the private votes you get to be based on misunderstandings.) At 11:27 AM 4/14/04 -0400, Jewett, Jim J wrote: > Issue 1: When is the name bound? > >The current proposal is that the name does not get >bound until after the final decorator is applied. >This means that decorators do not have access to >the function by its name. > >They can learn the name, perhaps through sys._getframe, Actually, they can learn the name via the 'func_name' or '__name__' attribute of the function. sys._getframe is only needed to access any *previous* binding of the name, or to do other magic in the class or module namespace. >but the name will not yet be bound, unless it happens >to refer to an unrelated object. (See issue two for >a better way to get the name.) > >Does anyone feel strongly that the name should be >bound to the original function before decorators are >applied? > >Does anyone feel strongly that the name should be >bound to the result of each intermediate step, if >there are multiple decorators? A strong -1 on both questions. The name should not be bound until after all decorators have completed. Nobody has presented any use cases for early binding of the name, but several use cases for late binding the name. Also, all actual implementation patches to date implement the late binding. > Issue 2: Restrictions on decorators > >The working assumption had been "any callable, which >will be called with a single object as the argument" Yes. However, Mike Pall's suggestion of an e.g. '__decorate__' member has some very appealing characteristics, with the downside of requiring a more complex implementation. > Issue 2a: name of the entry point > >Is there a strong feeling (in any direction) about >calling the decorator's "decorate" or "__decorate__" >attribute, rather than its "__call__" attribute? +0.5. I like it, but not enough to want to take the time to write a new patch to implement it, at least not this month. :) > Issue 2b: context available to decorators > >Would there be any objection to passing additional >context as keywords? Only that builtin decorators like classmethod and staticmethod would have to change to support it. >decorate(object, name="myfunc") > >The obvious keywords are the name and the context's >globals; Actually, those are not the obvious keywords. The function's name is an attribute of the function, so it's redundant. For that matter, a function's globals are available as 'func_globals'. It's the enclosing *local* scope that the function was defined in that's needed. For a function defined at module level, the globals and locals are the same, but for functions defined in a class body, you need the class locals. So, 'locals' would be a more obvious keyword. > decorators should probably accept arbitrary >keywords for forward-compatibility. I'm not sure this is a good idea. The only additional argument that anyone has proposed that makes sense is the local scope. I can't imagine what kind of change to Python would result in there being anything else that one might want. On the other hand, I *can* imagine accidentally calling a decorator (assuming __call__ is used) and passing it invalid keywords that are then ignored, silencing an otherwise useful error message. From pje at telecommunity.com Wed Apr 14 12:51:31 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Apr 14 12:51:54 2004 Subject: [Python-Dev] PEP 318 - check for consensus In-Reply-To: <200404141630.i3EGUUk24252@guido.python.org> References: Message-ID: <5.1.1.6.0.20040414124720.021d3260@telecommunity.com> At 09:30 AM 4/14/04 -0700, Guido van Rossum wrote: >[much more succinct presentation of the points I just wrote] The time machine strikes again! I'm really going to have to start waiting longer before I post; by the time I finish writing and editing and hit "Send", your reply to the same issue pops up in my mailbox, almost without fail. OTOH, maybe I should just try writing shorter messages. :) (Sadly, writing shorter messages usually takes me longer, because of all the editing-down I have to do...) From theller at python.net Wed Apr 14 13:40:44 2004 From: theller at python.net (Thomas Heller) Date: Wed Apr 14 13:40:50 2004 Subject: [Python-Dev] Inno Setup script for Python debug runtime Message-ID: I've made a short script (74 lines) for Inno Setup, which builds an installer for the Python debug exes, dlls, and pyds. The installer copies these debug images into the correct places when a 'normal' installation of Python is found in the registry. I don't think it would be a good idea to publish this installer on python.org, but it may be convenient for people who build python from source, because they no longer have to copy the files in place manually. Can I check this into the release-23maint branch in the PCBuild directory, or are there any objections? Thomas From fdrake at acm.org Wed Apr 14 14:00:05 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed Apr 14 14:00:26 2004 Subject: [Python-Dev] PEP 318 - check for consensus In-Reply-To: <5.1.1.6.0.20040414123205.021d0380@telecommunity.com> References: <5.1.1.6.0.20040414123205.021d0380@telecommunity.com> Message-ID: <200404141400.05071.fdrake@acm.org> On Wednesday 14 April 2004 12:46 pm, Phillip J. Eby wrote: > Actually, those are not the obvious keywords. The function's name is an > attribute of the function, so it's redundant. For that matter, a > function's globals are available as 'func_globals'. It's the enclosing > *local* scope that the function was defined in that's needed. For a > function defined at module level, the globals and locals are the same, but > for functions defined in a class body, you need the class locals. So, > 'locals' would be a more obvious keyword. Locals would certainly be nice to have. What should be done about the case of: def func(): [decorator] def f(): pass return f I think the decorator would want some way of determining that the locals can't be modified so an appropriate error can be raised, if it's interested in modifying the namespace directly. This could be done by returning a dict proxy that denies writes; we have one of those already. I don't know that there are many examples where we want write access to the locals in practice; the most interesting case for me is the set of property decorators that have been discussed, and read access is sufficient for those. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From andymac at bullseye.apana.org.au Wed Apr 14 09:34:13 2004 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Wed Apr 14 15:23:23 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <20040414130834.GS6139@unpythonic.net> References: <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> <001d01c421c6$77b4df00$e841fea9@oemcomputer> <20040414030938.GD8795@unpythonic.net> <20040414130834.GS6139@unpythonic.net> Message-ID: <20040414232756.Q67555@bullseye.apana.org.au> On Wed, 14 Apr 2004, Jeff Epler wrote: > Pentium IV, 2.4GHz: > -O2 -mcpu=i386 -DMUL=100003 1.56 {...} > -O2 -mcpu=pentium4 -DMUL=100003 0.63 {...} > With AMD CPUs, the current multiplier beats both the new multipler and > the version expressed as shifts and adds/subtracts: {...} > On an Athlon XP 2600+: > -O2 -mcpu=i386 -DMUL=100003 0.95 {...} > -O2 -mcpu=athlon-xp -DMUL=100003 0.43 * {...} > Long-at-a-time hash, Duron, 1GHz: > -O2 -march=athlon-tbird -DMUL=100003 0.35 Ummm... are you showing what you think you're showing here? As I recall, i386 gcc uses -mcpu and -march differently to most other architectures: - -mcpu just sets scheduling parameters, but not instruction set; - -march sets the instruction set. So most of the timings you show are for the i386 instruction set, but with different scheduling. The exception is the long-at-a-time hash. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From martin at v.loewis.de Wed Apr 14 15:32:11 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Apr 14 15:32:24 2004 Subject: [Python-Dev] Inno Setup script for Python debug runtime In-Reply-To: References: Message-ID: <407D91BB.2030603@v.loewis.de> Thomas Heller wrote: > Can I check this into the release-23maint branch in the PCBuild > directory, or are there any objections? What about PCbuild/python.iss? Is it not sufficient? Regards, Martin From jepler at unpythonic.net Wed Apr 14 15:33:59 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Wed Apr 14 15:35:07 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <20040414232756.Q67555@bullseye.apana.org.au> References: <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> <001d01c421c6$77b4df00$e841fea9@oemcomputer> <20040414030938.GD8795@unpythonic.net> <20040414130834.GS6139@unpythonic.net> <20040414232756.Q67555@bullseye.apana.org.au> Message-ID: <20040414193359.GW6139@unpythonic.net> On Wed, Apr 14, 2004 at 11:34:13PM +1000, Andrew MacIntyre wrote: > Ummm... are you showing what you think you're showing here? As I recall, > i386 gcc uses -mcpu and -march differently to most other architectures: > - -mcpu just sets scheduling parameters, but not instruction set; > - -march sets the instruction set. I'm pretty sure I am. (I'm not sure why I used -march on the long-at-a-time version, though) imul is on all x86 architectures, but whether to use it or not depends on the characteristics of the target CPU. With -mcpu=i386, imul is considered quite slow and a shift sequence is (almost?) always preferred when one operand is constant. With -mcpu=i686 and newer, imul seems to be preferred. I did actually inspect the generated assembly with -mcpu=i386 and -mcpu=i686, though I didn't look at it for all the combinations I posted. Jeff From theller at python.net Wed Apr 14 15:37:22 2004 From: theller at python.net (Thomas Heller) Date: Wed Apr 14 15:37:30 2004 Subject: [Python-Dev] Inno Setup script for Python debug runtime In-Reply-To: <407D91BB.2030603@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Wed, 14 Apr 2004 21:32:11 +0200") References: <407D91BB.2030603@v.loewis.de> Message-ID: "Martin v. L?wis" writes: > Thomas Heller wrote: > > Can I check this into the release-23maint branch in the PCBuild >> directory, or are there any objections? > > What about PCbuild/python.iss? Is it not sufficient? PCBuild/python.iss is an (currently unused) alternative to the wise setup script which installs the 'official' Python distribution, while the script I mentioned only installs the debug compiled binaries (dll's, pyd's, exe's, and lib's) into an already existing installation. Thomas From niemeyer at conectiva.com Wed Apr 14 15:51:15 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Wed Apr 14 15:50:58 2004 Subject: [Python-Dev] pipe syntax for sequential data In-Reply-To: References: Message-ID: <20040414195114.GA27608@burma.localdomain> > bash_users = ( file('/etc/passwd') | grep('/bin/bash') | aslist ) I don't think I'll be using this myself, but it's undoubtfully clever! :-) -- Gustavo Niemeyer http://niemeyer.net From martin at v.loewis.de Wed Apr 14 16:10:12 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Apr 14 16:10:25 2004 Subject: [Python-Dev] Inno Setup script for Python debug runtime In-Reply-To: References: <407D91BB.2030603@v.loewis.de> Message-ID: <407D9AA4.10108@v.loewis.de> Thomas Heller wrote: > PCBuild/python.iss is an (currently unused) alternative to the wise > setup script which installs the 'official' Python distribution, while > the script I mentioned only installs the debug compiled binaries (dll's, > pyd's, exe's, and lib's) into an already existing installation. It's not really unused: people do submit patches for it from time to time. Also, it does instally pyd files. So I wonder whether you could use it as-is. OTOH, I don't mind having an additional file in the next 2.3 release. Regards, Martin From FBatista at uniFON.com.ar Wed Apr 14 17:29:53 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 14 17:32:02 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Tony Meyer] #- As a relative newbie, I think that it would be better if (c) #- had a name that #- somehow indicated the difference; however, I can't think of #- one offhand that #- does :) It seems to me that it would be easy to assume that The name cames from the specification. A better name it's obviously more useful. If you think one, propose it to the list and if get accepted, I'll change the code. It better be soon, because when the interface get fixed, I don't think we all would want to change it. #- Even if from_string took a parameter "honour_context", #- defaulting to True, #- that would, IMO, be more clear that there's a difference #- between the two, #- and what it is. Don't like it, because the doc specify to honor the context. I don't want the method to comply with the specification regarding the value of an argument. . Facundo From FBatista at uniFON.com.ar Wed Apr 14 17:38:26 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 14 17:40:32 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Ka-Ping Yee] #- I agree with Tony. Having both Decimal() and #- Decimal.from_string() work #- in similar but subtly different ways seems likely to be #- misunderstood. Decimal(string) works the way we want/need/desire. Decimal.from_string(string) works the way the documentation specify. I think we can only change the way the former works or the name of the latter. #- I also think his last suggestion could provide a nice, #- convenient solution #- to the conversion-from-float issue, since it raises the same #- question (a #- method named "Decimal.from_float" doesn't explain how or why #- it is different #- from just constructing with a float). You can NOT construct directly from a float. That's the difference. #- So here's a more detailed suggestion. Suppose s = '12345', #- f = 12.345, #- the default context specifies a precision of 9 as given in #- the PEP, and #- c is a context object with a precision of 6. #- #- Then this: would yield this: #- #- Decimal(s) Decimal((0, (1, 2, 3, 4, 5), 0)) #- ... #- Decimal(s, context=c) Decimal((0, (1, 2, 3, 4, 5, 0), -1)) #- ... #- Decimal(f) Decimal((0, (1, 2, 3, 4, 5, 0, 0, 0, 0, 0,... #- ... The community agreed to not create directly from float. You have the from_float() method instead. Also, remember that the context doesn't affect the creation when you create directly from string (only if you use from_string). Your example means to change the behaviour of the creation when creating directly from string? And when create directly from int or long? #- (Do I assume correctly that there will be context objects #- that wrap up #- the precision and rounding mode, and there will be some sort #- of interface #- for getting and setting the current context?) Yes. You can get the context with getcontext() and then change it attributes. . Facundo From mikepy-0404 at mike.de Wed Apr 14 17:49:45 2004 From: mikepy-0404 at mike.de (Mike Pall) Date: Wed Apr 14 17:50:24 2004 Subject: [Python-Dev] Optimization targets (was: String hash function multiplier) In-Reply-To: <001901c4223a$5bab0a40$e841fea9@oemcomputer> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> Message-ID: <20040414214945.GA3443@mike.de> Hi, Raymond wrote: > It looks like the best bet is to try to speedup the code without > changing the multiplier. Indeed. And while you are at it, there are other optimizations, that seem more promising: I compiled a recent CVS Python with profiling and here is a list of the top CPU hogs (on a Pentium III, your mileage may vary): pystone: CPU% Function Name ---------------------------- 55.44 eval_frame 7.30 lookdict_string 4.34 PyFrame_New 3.73 frame_dealloc 1.73 vgetargs1 1.65 PyDict_SetItem 1.42 string_richcompare 1.15 PyObject_GC_UnTrack 1.11 PyObject_RichCompare 1.08 PyInt_FromLong 1.08 tupledealloc 1.04 insertdict parrotbench: CPU% Function Name ---------------------------- 23.65 eval_frame 8.68 l_divmod 4.43 lookdict_string 2.95 k_mul 2.27 PyType_IsSubtype 2.23 PyObject_Malloc 2.09 x_add 2.05 PyObject_Free 2.05 tupledealloc Arguably parrotbench is a bit unrepresentative here. And beware: due to automatic inlining of static functions the real offender may be hidden (x_divmod is the hog, not l_divmod). Anyway, this just confirms that the most important optimization targets are: 1. eval_frame 2. string keyed dictionaries 3. frame handling I think 3. needs an algorithmic approach (there was some discussion about it a few weeks ago), while 1. and 2. look like a code generation issue. So I took a look at the machine code that GCC generates on x86 with -O3 for these: uh oh, not a pretty sight. The main problems are bad branch predictions and lack of inlining. About branch predictions: The worst offender is the main code path for eval_frame: it gets split across half a dozen segments. The code path for the non-argument bytecodes is in fact the slowest. Similarly the code for lookdict_string branches around like crazy. The most likely code path is not the fastest path. One solution is to use likely()/unlikely() macros (using __builtin_expect). This is a good solution for small, isolated and performance critical code. It does not work well for eval_frame, though (I tried). But GCC has more to offer: read the man page entries for -fprofile-arcs and -fbranch-probabilities. Here is a short recipe: Go to your Python source directory and do this: $ mkdir build_profile $ cd build_profile $ ../configure # add any options you may need $ make $ mv python python_orig Edit the Makefile and add -fprofile-arcs to OPT. $ rm Python/ceval.o $ make $ mv python python_profile Run your favourite benchmark(s), but only invoke python *once*: $ ./python_profile -c 'import test.pystone; test.pystone.main(loops=100000)' Forget about the performance numbers the benchmark reports. Never use an executable compiled with profiling for comparison. But ... you should now have a new (binary) file called Python/ceval.da that contains the profiled branch probabilities. Edit the Makefile and replace -fprofile-arcs with -fbranch-probabilities $ rm Python/ceval.o $ make $ mv python python_opt Then compare the benchmarks: $ ./python_orig -c 'import test.pystone; test.pystone.main(loops=100000)' $ ./python_opt -c 'import test.pystone; test.pystone.main(loops=100000)' On my machine I get a 10-15% speedup. But we only optimized ceval.c ... So repeat the above steps, but now delete Python/ceval.o and Objects/*.o each time. Now I get about 20-25% speedup for pystone and 10% speedup for parrotbench! Not bad, huh? Now the bad news: I don't know how to integrate this into the regular build process. So this is not an option for everyone, but ambitious packagers might want to take the trouble to do this by hand. Oh and of course neither pystone nor parrotbench are representative of the branch probabilities of any particular application. But for predicting eval_frame and lookdict_string, they are probably good enough. On a related note: GCC uses random branch probabilities with -O, when no probability information is present (no __builtin_expect or *.da files). Use -fno-guess-branch-probability if you want predictable timings on recompiles. About inlining: - The opcode decoding with NEXTOP/HAS_ARG/NEXTARG does not compile well. GCC has trouble inferring the lifetimes for opcode and oparg, too. Ideas: - Use a special inline assembly variant for x86. OR - Move NEXTARG to each opcode that needs it and make oparg a local block variable. - Use NEXTOP directly for the switch and refetch opcode into a local block variable where necessary. - The Py_Ticker check and the tracing support should be moved out of the core loop. The Py_Ticker check may be put into a subroutine and called from selected opcodes only (haven't checked if that works). I have no (good) idea about how to get rid of the tracing support, though. - _PyString_Eq() is a good candidate to inline for lookdict_string(). I.e. a new macro in stringobject.h. But wait ... - GCC generates pretty slow code for _PyString_Eq() (at least on x86). This is mainly due to a bad branch prediction (we should use likely()) but also due to the penalty for 8-bit partial register load/compare generated by the ob_sval dereferences. The latter is completely useless because memcmp() is inlined (with repz cmpsb) and duplicates that comparison. Omitting the *ob_sval comparison is faster. - Create a macro that inlines the first part of PyObject_Hash(). Useful for PyDict_SetItem() and others. I bet there are more, but I'm running out of time right now -- sorry. Bye, Mike From FBatista at uniFON.com.ar Wed Apr 14 17:51:08 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 14 17:53:18 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Tim Peters] #- I expect everyone's initial reaction will be to say "no #- limit", but think #- two or three times about that. Eliminating any restriction #- on exponent #- magnitude is the same as saying overflow can never happen. #- But overflow is #- usually a valuable safety net in real life, telling you that #- your algorithm #- has gone insane. OTOH, there's nothing with physical #- significance that #- requires an exponent anywhere near as large as 999999999, #- and REXX has used #- that exponent limit for years without problems. But it's still an artificial restriction. We have no-limit longs, should have limited Decimal? And also implies more code and complexity in the module. But this is a side effect and not the primary concern in the decision. #- (c) also specifies ways to spell special values, like NaNs #- and infinities. #- Does Decimal(string)? I don't believe the PEP covered that. Yes, it does. I missed mention it and an example in the PEP. Already fixed. #- In any case, from_string() should be subsumed by #- Decimal(string), perhaps #- with via an optional "use_context" flag argument. You mean something like Decimal(string, use_context=True)? And when creating from long, for example? Remember that the spec says that the context affect the operations, not creations (with the exception of from_string). #- It's quite arguable that str() should be exactly the standard's #- to_sci_string() (and for those who haven't read the spec, do #- before you #- complain about that -- no, you don't always get an exponent under #- to_sci_string()). I'm lost here. Are you saying that str() should have the same behaviour that to_sci_string()? #- > Hash behaviour #- > -------------- #- #- Decimals that happen to be exact integers must return the #- same hash codes as #- the hash codes of the ints or longs they're equal to. All #- bets are off wrt #- hash codes for non-integral Decimals (e.g., #- hash(Decimal('1.5')) may or may #- not be the same as hash(1.5)); promising more than that is simply #- impractical. #- #- BTW, hash(Decimal('1.0')) must be the same as #- hash(Decimal('1.00')) must be #- the same as hash(Decimal('0.001e3')) despite that they have different #- internal representations. I agree completely with this behaviour. . Facundo From FBatista at uniFON.com.ar Wed Apr 14 17:53:19 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 14 17:55:32 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Aahz] #- > Exponent Maximum #- > ---------------- #- > #- > The Decimal number composes of three elements: A sign that #- can be 0 or 1, a #- > tuple of digits where each can be 0..9, and an exponent. #- > #- > The exponent is an integer, and in the actual #- implementation exists a #- > maximum #- > value:: #- > #- > DEFAULT_MAX_EXPONENT = 999999999 #- > DEFAULT_MIN_EXPONENT = -999999999 #- > ABSOLUTE_MAX_EXP = 999999999 #- > ABSOLUTE_MIN_EXP = -999999999 #- > #- > The issue is that this limit is artificial: As long it's a #- long, you should #- > be able to make it as big as your memory let you. #- #- As long as it's in Python code, it's easy enough for a #- whiner to modify #- it. Leave it as-is. Other than that, I agree with Tim. That means that I should let the limit be, because can be easily modified? If I don't put the limit, nobody will need to modify it, :) . Facundo From FBatista at uniFON.com.ar Wed Apr 14 17:58:11 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 14 18:00:21 2004 Subject: [Python-Dev] Decimal conversion to string Message-ID: [Ka-Ping Yee] #- Could we please have repr(d) use the string in the constructor #- rather than the tuple form? That would be much easier to read. #- #- So, for example: #- #- >>> Decimal('12345') #- Decimal('12345') # instead of Decimal((0, (1, 2, 3, 4, 5, 0)) #- >>> _ * 10 #- Decimal('1.2345e1') #- >>> Decimal('12345', 2) #- Decimal('1.2e3') #- >>> Decimal(1.1) #- Decimal('1.100000000000000088817841970012523233890533447265625') #- >>> Decimal(1.1, default_context=1) #- Decimal('1.10000000') To *me* is more natural the tuples. But that's only because that's what the internal objects are. An intermediate solution can be something like: >>> Decimal('12345') Decimal(0, 12345, 0) a tuple of three values: sign, coefficient, and exponent. The three of them as integers.} Anyway, if you want to change the repr() behaviour, you should push it hard to the list. I'm -0 on it. . Facundo From greg at cosc.canterbury.ac.nz Wed Apr 14 19:51:08 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Apr 14 19:51:19 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <20040411172351.GA23114@panix.com> Message-ID: <200404142351.i3ENp81h019114@cosc353.cosc.canterbury.ac.nz> Aahz : > Anyone else favor having the ``__future__`` directive apply to the > package and all its subpackages? -1. Future statements should only apply to the module that contains them. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Wed Apr 14 20:14:21 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Apr 14 20:14:29 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <1081743508.2987.118.camel@james> Message-ID: <200404150014.i3F0ELhf019167@cosc353.cosc.canterbury.ac.nz> > What is the rationale given for not allowing: > > import .moduleY > > as a relative import statement meaning "from the package containing the > module running the import statement, import moduleY"? Probably because it's not clear what name to bind the imported module to, and in what namespace. With the current import syntax, you refer to the imported module using exactly the same name (or dotted sequence of names) that you used when importing, but that's not possible in this case, because ".moduleY" is not a valid Python expression. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Wed Apr 14 21:45:09 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed Apr 14 21:45:58 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: <20040414144614.GD5922@panix.com> Message-ID: <200404150145.i3F1j9tl019325@cosc353.cosc.canterbury.ac.nz> Aahz : > I'm not dead-set on using a tuple form, but I think it's more useful for > someone trying to see the internals of a Decimal instance. But who cares about the gory details of the internals? The most important thing about a repr() is to unambiguously indicate the object's type; the next is to show something about its value, as far as reasonably possible. What's actually inside the object doesn't matter much. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From edloper at gradient.cis.upenn.edu Wed Apr 14 22:24:13 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Wed Apr 14 22:22:37 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: References: Message-ID: <407DF24D.1000607@gradient.cis.upenn.edu> [Facundo wrote:] > Anyway, if you want to change the repr() behaviour, you should push it hard > to the list. I'm -0 on it. +1 on changing the repr behavior to show: Decimal('12345') Instead of: Decimal(0, (1, 2, 3, 4, 5, 0)) Using the latter representation puts up an unnecessary barrier to using decimal, that I'm guessing many people (esp newbies) won't want to put up with. In particular, it makes decimal much less friendly for interactive use. -Edward From tim_one at email.msn.com Wed Apr 14 23:13:45 2004 From: tim_one at email.msn.com (Tim Peters) Date: Wed Apr 14 23:14:05 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <001901c4223a$5bab0a40$e841fea9@oemcomputer> Message-ID: [Raymond] > On the P4, the documented latency went up from 4 cycles to 14 cycles > while shifts and adds went down to 0.5 cycles and 1 cycle respectively. > Timings confirm the result. What makes the current stepping of some flavor of P4 overwhelmingly important <0.3 wink>? Tradeoffs at this level change with every chip generation, and within a generation vary across vendors. In particular, they can make int multiply as fast as they want just by throwing more highly regular silicon at it (which is why its relative performance varies so much across chip generations, and is usually worst in the earliest iterations of a new generation). My overwhelming concern isn't micro-efficiency of the generated code for the string-hash loop, it's that the resulting hash have high quality. I don't know that the "little prime" is vulnerable to bad cases in non-contrived real life, but we have years of evidence suggesting that the "big prime" isn't. If we can develop evidence that the little prime doesn't either, great, then I lose all objections to using it. > It looks like the best bet is to try to speedup the code without > changing the multiplier. Intel's software optimization cookbook > recommends a partial unrolling and elimination of data dependencies so > that a second multiply can start 4 cycles after the previous one > started. If practice bears out the theory, the timings could show a > three or fourfold speedup without changing the multiplier. Or leave it alone. Hand unrolling a loop all but guarantees that the next generation of compiler optimizer won't be able to reverse-engineer the original "natural" loop structure, and so won't be able to do the code transformation that works best for the next generation of hardware tricks. IOW, once you start down the path of hand-optimizing for a specific compiler + HW combo, there are two outcomes: (1) someone makes this their job forever after; or, (2) the performance of the code actually gets worse over time (compared to what it would have been if the original code had been left dirt simple). I happen to have an app (spambayes) that makes very heavy use of string-keyed dicts, and where interning is impractical (so the string hashes can't be avoided). Still, speed up string hashing by a factor of 10, and it won't make enough difference in that app's throughput to mention. I also have a hyper-threaded box today, and the imul stalls in this P4 give great opportunities for other processes to get work done <0.9 wink>. From tim_one at email.msn.com Wed Apr 14 23:13:45 2004 From: tim_one at email.msn.com (Tim Peters) Date: Wed Apr 14 23:14:13 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: Message-ID: [Facundo Batista, on max exponent] > But it's still an artificial restriction. We have no-limit longs, > should have limited Decimal? Nobody has a use for unbounded exponents, so it's a silly thing to support if there are costs. There are costs, including: + As mentioned last time, eliminating bounds on exponents effectively means overflow (and underflow) can never happen. But overflow *is* a valuable safety net in real life fp use, like a canary in a coal mine, giving danger signs early when a program goes insane. This is of no real interest to you (your real goal is a Money type), it's of interest to people who will use Decimal *as* a form of floating-point arithmetic (your use cases are more fixed-point). + Virtually all implementations of 854 use (and as IBM's standard even suggests) "forbidden" exponent values to encode non-finite numbers (infinities and NaNs). A bounded exponent can do this at virtually no extra storage cost. If the exponent is unbounded, then additional bits have to be used instead. This cost remains hidden until more time- and space- efficient implementations are attempted. + Big as it is, the IBM standard is a tiny start at supplying a complete numeric facility. Having no bound on exponent size will enormously complicate the implementations of, e.g., decimal sin() and cos() (there's then no a priori limit on how many digits of pi effectively need to be known in order to perform argument reduction). > And also implies more code and complexity in the module. But this > is a side effect and not the primary concern in the decision. I don't know how it's implemented now. "The natural way" to check exponent bounds is in a common routine called at the end of every operation; in that case, the "more code and complexity" doesn't amount to anything worth mentioning. ... > #- In any case, from_string() should be subsumed by > #- Decimal(string), perhaps with via an optional "use_context" flag > #- argument. > You mean something like Decimal(string, use_context=True)? Yes. > And when creating from long, for example? Sure. > Remember that the spec says that the context affect > the operations, Yes. > not creations (with the exception of from_string). I don't understand. The only place the spec uses the word "creation" is in regard to the creation of context objects. What do you mean by "creations"? Context does affect every way of creating a decimal number in the spec; the only ways to create numbers covered by the spec are via operations (and from_string isn't a special case here, from_string is just one of the required operations in the spec, and all operations covered by the spec respect context). > #- It's quite arguable that str() should be exactly the standard's > #- to_sci_string() (and for those who haven't read the spec, do > #- before you complain about that -- no, you don't always get an > #- exponent under to_sci_string()). > I'm lost here. Are you saying that str() should have the same behaviour > that to_sci_string()? Yes, that str(Decimal) should produce exactly the string the spec specifies as being the result of its abstract to-sci-string operation. There's no requirement to have a method *named* "to_sci_string", the only requirement is that *some* way to spell to-sci-string's functionality be supplied. The meaning of to-sci-string is precisely specified by the standard, and is a good choice for both str(Decimal) and repr(Decimal). (It's both human-readable and exact.) From tim_one at email.msn.com Wed Apr 14 23:13:45 2004 From: tim_one at email.msn.com (Tim Peters) Date: Wed Apr 14 23:14:17 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: Message-ID: [Ping] >> Could we please have repr(d) use the string in the constructor >> rather than the tuple form? That would be much easier to read. [Batista, Facundo] > To *me* is more natural the tuples. But that's only because that's what > the internal objects are. The standard's to-scientific-string operation produces a human-readable string with 1-to-1 correspondence between strings and internal representations. So it's suitable for both human use and as repr() output; it's also suitable for str() output. Tuples are (IMO) *only* interesting to implementers. That's OK -- give 'em an as_tuple() method instead. The repr() and str() forms really should be readable by end users. > An intermediate solution can be something like: > > >>> Decimal('12345') > Decimal(0, 12345, 0) > > a tuple of three values: sign, coefficient, and exponent. The three of > them as integers.} Then we've got quadratic-time requirements for eval(repr(Decimal)), because the coeffecient takes time quadratic in the number of decimal digits to convert to a Python integer, and then there's *another* quadratic-time operation needed to break the Python integer back into its digits. The string and tuple forms are linear-time, so don't suffer this ugliness. The to-scientific-string form is also readable by end-users with no education on the topic. > Anyway, if you want to change the repr() behaviour, you should push it > hard to the list. I'm -0 on it. +1 here, for the reasons above. Also +0 on adding an .as_tuple() method. I can't be stronger than +0 on that, because tuples of single decimal digits reveal *too* much about the current implementation. For example, a speedier implementation could use base 100 or base 10000 (or ...) internally, and then its idea of a natural tuple representation would be correspondingly different. The standard already goes to great pains to define a canonical and readable string representation, so let's use it. From bsder at allcaps.org Thu Apr 15 02:05:27 2004 From: bsder at allcaps.org (Andrew P. Lentvorski, Jr.) Date: Thu Apr 15 02:01:32 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: Message-ID: <20040414225101.A4530@mail.allcaps.org> On Tue, 13 Apr 2004, Batista, Facundo wrote: > The issue is that this limit is artificial: As long it's a long, you should > be able to make it as big as your memory let you. I agree with Tim that underflow/overflow are useful flags of something going wrong. Very few people would connect Decimal overflow with "Gee, why is Python allocating all of my system's memory?" I would like to see a soft default limit that can be modified. As for the value of the default limits, I would actually choose both a default precision and exponent limit which would be within the range represented by a double precision floating point number. This helps in two cases: 1) People will try to interconvert FP and Decimal at various points. Having one with a significantly different range will certainly give some surprises. (example: Decimal would get converted to double precision FP before being used in OpenGL or Direct3D to draw a pie chart, graph, etc.) 2) Having a default range somewhere inside double precision means that some mythical "efficient" implementation could actually use double precision FP operations to get close and then use a cleanup step to check the final digit. (Very useful for exponentials) -a From python-dev at zesty.ca Thu Apr 15 05:49:17 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Thu Apr 15 05:48:49 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: Message-ID: On Wed, 14 Apr 2004, Batista, Facundo wrote: > Decimal(string) works the way we want/need/desire. > Decimal.from_string(string) works the way the documentation specify. > > I think we can only change the way the former works or the name of the > latter. Or you can add an optional argument, which is what I suggested. Aren't you going to need an optional argument anyway to let the user specify a context? > > I also think his last suggestion could provide a nice, convenient > > solution to the conversion-from-float issue, since it raises the > > same question (a method named "Decimal.from_float" doesn't explain > > how or why it is different from just constructing with a float). > > You can NOT construct directly from a float. That's the difference. I understand that you want to make it hard for people to construct a Decimal from a float. But what you just stated does not explain anything. I assume the purpose of from_float is to prevent people from being confused about whether Decimal(1.1) produces exactly 1.1 or the machine value of the double-precision rounded float closest to 1.1. Is that correct? If so, then the real exceptional case is *exact* conversion, not float conversion. Decimal(1.1, precision=4) is unambiguous. Therefore: Decimal(value, default_context=True) Decimal(value, context=c) Decimal(value, precision=p) should all work for integer, float, string, or Decimal values. To protect against the ambiguity of Decimal(1.1), Decimal(value) should work for integer, string, or Decimal values, and raise an exception for a float, such as "precision must be specified for conversion from float to Decimal". The extra method should be specifically for the exceptional case, and named appropriately. Decimal.exact_float(value) It would serve just one purpose: allow exact conversion from floats when it's really desired. This would both (a) separate the functionality between Decimal() and Decimal.exact_float() in a more logical way; and (b) give the extra method a more meaningful, useful name -- a name that really explains why the extra method exists. > Also, remember that the context doesn't affect the creation when you create > directly from string (only if you use from_string). Your example means to > change the behaviour of the creation when creating directly from string? And > when create directly from int or long? That's what I'm suggesting: that you can supply a context object, or ask for the default context, when constructing any Decimal value. -- ?!ng From aahz at pythoncraft.com Thu Apr 15 06:06:36 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Apr 15 06:06:42 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <20040414225101.A4530@mail.allcaps.org> References: <20040414225101.A4530@mail.allcaps.org> Message-ID: <20040415100636.GA16358@panix.com> On Wed, Apr 14, 2004, Andrew P. Lentvorski, Jr. wrote: > > As for the value of the default limits, I would actually choose both a > default precision and exponent limit which would be within the range > represented by a double precision floating point number. -1 (slow connection, no time to argue.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From mwh at python.net Thu Apr 15 07:05:37 2004 From: mwh at python.net (Michael Hudson) Date: Thu Apr 15 07:05:42 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: <20040414214945.GA3443@mike.de> (Mike Pall's message of "Wed, 14 Apr 2004 23:49:45 +0200") References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> Message-ID: <2mekqp1oxq.fsf@starship.python.net> Mike Pall writes: > Hi, > > Raymond wrote: >> It looks like the best bet is to try to speedup the code without >> changing the multiplier. > > Indeed. And while you are at it, there are other optimizations, that > seem more promising: > > I compiled a recent CVS Python with profiling and here is a list of the > top CPU hogs (on a Pentium III, your mileage may vary): I played this game recently, but on a G3 ibook, which is probably a much more boring processor from a scheduling point of view. > pystone: > > CPU% Function Name > ---------------------------- > 55.44 eval_frame > 7.30 lookdict_string > 4.34 PyFrame_New > 3.73 frame_dealloc > 1.73 vgetargs1 > 1.65 PyDict_SetItem > 1.42 string_richcompare > 1.15 PyObject_GC_UnTrack > 1.11 PyObject_RichCompare > 1.08 PyInt_FromLong > 1.08 tupledealloc > 1.04 insertdict I saw similar results to this, tho' I don't remember lookdict_string being so high on the list. > parrotbench: > > CPU% Function Name > ---------------------------- > 23.65 eval_frame > 8.68 l_divmod > 4.43 lookdict_string > 2.95 k_mul > 2.27 PyType_IsSubtype > 2.23 PyObject_Malloc > 2.09 x_add > 2.05 PyObject_Free > 2.05 tupledealloc > > Arguably parrotbench is a bit unrepresentative here. And beware: due to > automatic inlining of static functions the real offender may be hidden > (x_divmod is the hog, not l_divmod). Probably a fine candidate function for rewriting in assembly too... > Anyway, this just confirms that the most important optimization targets are: > 1. eval_frame > 2. string keyed dictionaries > 3. frame handling [...] > But GCC has more to offer: read the man page entries for -fprofile-arcs > and -fbranch-probabilities. Here is a short recipe: I tried this on the ibook and I found that it made a small difference *on the program you ran to generate the profile data* (e.g. pystone), but made naff all difference for something else. I can well believe that it makes more difference on a P4 or G5. [snippety] > I bet there are more, but I'm running out of time right now -- > sorry. I certainly don't want to discourage people from optimizing Python's current implementation, but... Some months ago (just after I played with -fprofile-arcs, most likely) I wrote a rant about improving Python's performance, which I've finally got around to uploading: http://starship.python.net/crew/mwh/hacks/speeding-python.html Tell me what you think! Cheers, mwh -- Hmmm... its Sunday afternoon: I could do my work, or I could do a Fourier analysis of my computer's fan noise. -- Amit Muthu, ucam.chat (from Owen Dunn's summary of the year) From FBatista at uniFON.com.ar Thu Apr 15 09:27:17 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 15 09:29:34 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Tim Peters] #- [Facundo Batista, on max exponent] #- > But it's still an artificial restriction. We have no-limit longs, #- > should have limited Decimal? #- #- Nobody has a use for unbounded exponents, so it's a silly #- thing to support #- if there are costs. There are costs, including: #- #- + As mentioned last time, eliminating bounds on exponents effectively #- ... #- + Virtually all implementations of 854 use (and as IBM's #- ... #- + Big as it is, the IBM standard is a tiny start at supplying a #- ... The feeling previous to this mail was that an artificial limit was silly if you don't gain nothing. Here are three valuable reasons for keep the limits. So, if nobody opposes strongly, I'll put these reasons in the PEP and keep the limit. #- > not creations (with the exception of from_string). #- #- I don't understand. The only place the spec uses the word #- "creation" is in #- regard to the creation of context objects. What do you mean #- by "creations"? #- Context does affect every way of creating a decimal number #- in the spec; the #- only ways to create numbers covered by the spec are via #- operations (and #- from_string isn't a special case here, from_string is just one of the #- required operations in the spec, and all operations covered #- by the spec #- respect context). Until now, Decimal does not uses context in creation time (with creation I mean when you create an object instantiating the class). This allows to not lose any information at creation time, and only use the context in operations. Do you want to change this behaviour? #- > I'm lost here. Are you saying that str() should have the #- same behaviour #- > that to_sci_string()? #- #- Yes, that str(Decimal) should produce exactly the string the Ok. #- supplied. The #- meaning of to-sci-string is precisely specified by the #- standard, and is a #- good choice for both str(Decimal) and repr(Decimal). (It's both #- human-readable and exact.) The only issue *I* see here, is that you don't have a clear representation of the internals of the object, but yes, you have an exact representation of that number. Let me show an example: >>> d = Decimal.Decimal((1, (1,2,3,4,0), 1)) >>> d Decimal( (1, (1, 2, 3, 4, 0), 1L) ) >>> d.to_sci_string() '-1.2340E+5' There you have to count the decimals and use the exponent in the string to know what the real exponent is (but I don't know how much important this is). . Facundo From mcherm at mcherm.com Thu Apr 15 09:32:23 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Thu Apr 15 09:32:29 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: <1082035943.407e8ee7975ef@mcherm.com> Andrew P. Lentvorski writes: > As for the value of the default limits, I would actually choose both a > default precision and exponent limit which would be within the range > represented by a double precision floating point number. I strongly object. I have one particular use case for decimal, and I consider it to be a very important one.... to store and manipulate monetary amounts. Putting a limit as small as that used for floats would significantly harm my use case. On the other hand, your argument seems to be that it will make it easy to convert between floats and Decimals. I disagree... if there were simple easy ways to convert between floats and Decimals (like a Decimal constructor that took a float) then people who didn't know better would use them and then start running into all sorts of errors due to floating point imprecision -- the very thing that Decimal is supposed to save us from. That's why we EXPLICITLY chose to make it possible but somewhat awkward to convert to/from floats. You can't hide the difficulties inherent in (binary) floating point by just sweeping them under the rug. And Decimal can't make it easier and better (at some cost of performance and additional complexity) if you attempt to tie it to float. As for the limits themselves, I remain -0 on them. Doesn't pose a problem for me but why would you bother? Of course, Tim and others have posted answers to this question, mostly just so people will get overflow errors instead of out-of-memory errors (and possibly some vague future advantage to implementers of a "complete numerical package" including things like sin() and cos()). I-never-make-errors-anyway-lly yours, -- Michael Chermside From mikepy-0404 at mike.de Thu Apr 15 09:36:13 2004 From: mikepy-0404 at mike.de (Mike Pall) Date: Thu Apr 15 09:36:32 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: <2mekqp1oxq.fsf@starship.python.net> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> Message-ID: <20040415133613.GA4735@mike.de> Hi, mwh wrote: > > (x_divmod is the hog, not l_divmod). > > Probably a fine candidate function for rewriting in assembly too... As a data point: I once had the doubtful pleasure to write a long-integer library for cryptography. Hand-crafted x86 assembler outperforms plain (but carefully optimized) C code by a factor of 2 to 3. But Python's long-int code is a lot slower than e.g. gmp (factor 15-25 for mul/div, factor 100 for modular exponentiation). I assume the difference between C and assembler is less pronounced with other processors. The register pressure issue may soon be a moot point with x86-64, though. It has been shown that 64 bit pointers slow things down a bit, but compilers just love the extra registers (R8-R15). > > But GCC has more to offer: read the man page entries for -fprofile-arcs > > and -fbranch-probabilities. Here is a short recipe: > > I tried this on the ibook and I found that it made a small difference > *on the program you ran to generate the profile data* (e.g. pystone), > but made naff all difference for something else. I can well believe > that it makes more difference on a P4 or G5. For x86 even profiling python -c 'pass' makes a major difference. And the speed-ups are applicable to almost any program, since the branch predictions for eval_frame and lookdict_string affect all Python programs. I'm currently engaged in a private e-mail conversation with Raymond on how to convince GCC to generate good code on x86 without the help of profiling. > I wrote a rant about improving Python's performance, which I've > finally got around to uploading: > > http://starship.python.net/crew/mwh/hacks/speeding-python.html > > Tell me what you think! About GC: yes, refcounting is the silent killer. But there's a lot to optimize even without discarding refcounting. E.g. the code generated for Py_DECREF is awful (spread across >3500 locations) and PyObject_GC_UnTrack needs some work, too. About psyco: I think it's wonderful. But, you are right, nobody is using it. Why? Simple: It's not 'on' by default. About type inference: maybe the right way to go with Python is lazy and pure runtime type inference? Close to what psyco does. About type declarations: this is too contrary to the Pythonic way of thinking. And before we start to implement this, we should make sure that it's a lot faster than a pure dynamic type inferencing approach. About PyPy: very interesting, will take a closer look. But there's still a long road ahead ... Bye, Mike From FBatista at uniFON.com.ar Thu Apr 15 09:45:58 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 15 09:48:10 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Ka-Ping Yee] #- Aren't you going to need an optional argument anyway to let the #- user specify a context? What for? #- I assume the purpose of from_float is to prevent people from being #- confused about whether Decimal(1.1) produces exactly 1.1 or #- the machine #- value of the double-precision rounded float closest to 1.1. Is that #- correct? #- #- If so, then the real exceptional case is *exact* conversion, #- not float #- conversion. Decimal(1.1, precision=4) is unambiguous. Therefore: Here I think you're confused. *Precision* is a parameter of the context, and determines in how many digits you will get the coefficient after an operation. The issue when constructing from float, is how many *positions* after the decimal point will be relevant when creating from a float. This list agreed to have a special method to creating from float, to avoid confusing users about the real behaviour there. And in that method only you permit to specify the positions. When constructing from other data types, positions does not have any meaning as those other data types are always exact. #- The extra method should be specifically for the exceptional case, #- and named appropriately. #- #- Decimal.exact_float(value) #- #- It would serve just one purpose: allow exact conversion from floats #- when it's really desired. #- #- This would both (a) separate the functionality between Decimal() #- and Decimal.exact_float() in a more logical way; and (b) give #- the extra method a more meaningful, useful name -- a name that #- really explains why the extra method exists. I think that *maybe* a better name is needed, but just that. #- > Also, remember that the context doesn't affect the #- creation when you create #- > directly from string (only if you use from_string). Your #- example means to #- > change the behaviour of the creation when creating #- directly from string? And #- > when create directly from int or long? #- #- That's what I'm suggesting: that you can supply a context object, #- or ask for the default context, when constructing any Decimal value. Do you want to supply a context that *will affect* the creation process, or just supply one that the Decimal will keep to use in other operations? Remember that you have one context per thread. All the decimal instances in a thread share the same context. This was specified by Aahz, the list agreed, and get into the PEP. Do you want to change it? . Facundo From guido at python.org Thu Apr 15 09:55:56 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 15 09:56:02 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: Your message of "Thu, 15 Apr 2004 13:45:09 +1200." <200404150145.i3F1j9tl019325@cosc353.cosc.canterbury.ac.nz> References: <200404150145.i3F1j9tl019325@cosc353.cosc.canterbury.ac.nz> Message-ID: <200404151355.i3FDtuT26138@guido.python.org> > But who cares about the gory details of the internals? > The most important thing about a repr() is to unambiguously > indicate the object's type; the next is to show something > about its value, as far as reasonably possible. What's > actually inside the object doesn't matter much. I didn't see the start of this, but the guidelines for repr() are and have always been this: if at all possible it should be an expression that, when fed to eval(), (assuming the right imports are in scope), returns an object with the same type and value. In most cases, this means it should resemble a call to the constructor with suitable arguments. If that isn't possible, a form surrounded with <...> should be used, and it should at least show the object's type and some stuff that can be used to distinguish different instances (either the address or some representative property of the instance). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 15 10:27:17 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 15 10:27:22 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: Your message of "Thu, 15 Apr 2004 15:36:13 +0200." <20040415133613.GA4735@mike.de> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> Message-ID: <200404151427.i3FERHo26387@guido.python.org> > > http://starship.python.net/crew/mwh/hacks/speeding-python.html I recently met Mario Wolczko who worked on the Self language, and asked him if he had one sentence of advice for Python. His response was: Just-in-time compilation. > About psyco: I think it's wonderful. But, you are right, nobody is > using it. Why? Simple: It's not 'on' by default. I don't think that's the real reason. Last week Jimmy Retzlaff gave a talk at the Bay Piggies meeting on optimization, using a large commercial application he's written as an example. He mentioned that he had tried to use Psyco, and that it had been turned on in some distributions of the code, but that it was currently turned off again. I don't recall the reason he stated for that, but I suspect that it was mostly Psyco's memory bloat, plus the fact that his code was "fast enough" and he was able to code things in C if needed. (He also showed a nice anecdote about Psyco being faster than his hand-coded C in one case, but that didn't last long when he realized the C code was using doubles and the Python code was using ints. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Thu Apr 15 10:29:41 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Apr 15 10:36:58 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: <200404151355.i3FDtuT26138@guido.python.org> References: <200404150145.i3F1j9tl019325@cosc353.cosc.canterbury.ac.nz> <200404151355.i3FDtuT26138@guido.python.org> Message-ID: <20040415142941.GA22303@panix.com> On Thu, Apr 15, 2004, Guido van Rossum wrote: >Greg Ewing: >> >> But who cares about the gory details of the internals? The most >> important thing about a repr() is to unambiguously indicate the >> object's type; the next is to show something about its value, as far >> as reasonably possible. What's actually inside the object doesn't >> matter much. > > I didn't see the start of this, but the guidelines for repr() are and > have always been this: if at all possible it should be an expression > that, when fed to eval(), (assuming the right imports are in scope), > returns an object with the same type and value. In most cases, this > means it should resemble a call to the constructor with suitable > arguments. If that isn't possible, a form surrounded with <...> > should be used, and it should at least show the object's type and some > stuff that can be used to distinguish different instances (either the > address or some representative property of the instance). Right. Question is, to what extent does "human readable" factor in after these criteria? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From jeremy at alum.mit.edu Thu Apr 15 10:38:26 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu Apr 15 10:38:46 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: <200404151427.i3FERHo26387@guido.python.org> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> <200404151427.i3FERHo26387@guido.python.org> Message-ID: <1082039905.2610.128.camel@localhost.localdomain> On Thu, 2004-04-15 at 10:27, Guido van Rossum wrote: > > > http://starship.python.net/crew/mwh/hacks/speeding-python.html > > I recently met Mario Wolczko who worked on the Self language, and > asked him if he had one sentence of advice for Python. His response > was: Just-in-time compilation. It seems like an obvious, but labor intensive solution to me. I didn't get a chance to pursue discussion much with Jim Hugunin or Mike Salib, but it would be interesting to know why they focus on static analysis; Self started out with static analysis but got better performance with runtime type feedback (http://www.cs.ucsb.edu/labs/oocsb/papers/toplas96.shtml). Another question that I'd love to hear the answer to: What's the difference between Pysco and something like this Self implementation or the HotSpot Java implementation? I've never read a clear explanation of what Pysco does or how it compares to other systems that do dynamic compilation. Jeremy From guido at python.org Thu Apr 15 10:41:37 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 15 10:41:44 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: Your message of "Thu, 15 Apr 2004 10:29:41 EDT." <20040415142941.GA22303@panix.com> References: <200404150145.i3F1j9tl019325@cosc353.cosc.canterbury.ac.nz> <200404151355.i3FDtuT26138@guido.python.org> <20040415142941.GA22303@panix.com> Message-ID: <200404151441.i3FEfb226502@guido.python.org> > Right. Question is, to what extent does "human readable" factor in > after these criteria? If the choice is between a string and a tuple, I'd vote for the string because it is (a) shorter and (b) more readable. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 15 10:42:31 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 15 10:42:42 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: Your message of "Thu, 15 Apr 2004 10:38:26 EDT." <1082039905.2610.128.camel@localhost.localdomain> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> <200404151427.i3FERHo26387@guido.python.org> <1082039905.2610.128.camel@localhost.localdomain> Message-ID: <200404151442.i3FEgVR26515@guido.python.org> > It seems like an obvious, but labor intensive solution to me. Well, all *other* proposals seem just as labor intensive... --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Thu Apr 15 10:34:27 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Apr 15 10:43:33 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: Message-ID: <20040415143427.GB22303@panix.com> On Thu, Apr 15, 2004, Batista, Facundo wrote: > > Remember that you have one context per thread. All the decimal > instances in a thread share the same context. This was specified by > Aahz, the list agreed, and get into the PEP. Do you want to change it? Correction for precision: all Decimal operations in a thread use the same context until the context is changed. Threads inherit context from their starting thread (I think). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From FBatista at uniFON.com.ar Thu Apr 15 10:53:15 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 15 10:55:30 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Aahz] #- Correction for precision: all Decimal operations in a thread use the #- same context until the context is changed. Threads inherit #- context from #- their starting thread (I think). I'm lost. Help me with an example (that later I'll shamelessly stole to use in a test case): d1 = Decimal('123') d2 = Decimal('456') So, long, they have the same context, right? getcontext().prec = 46 The precision should change for both d1 and d2, right? d3 = Decimal('789') d3 also has the precision in 46, right? Thank you! . Facundo From edloper at gradient.cis.upenn.edu Thu Apr 15 11:17:56 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Thu Apr 15 11:16:17 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: Message-ID: <407EA7A4.6070709@gradient.cis.upenn.edu> > #- > DEFAULT_MAX_EXPONENT = 999999999 > #- > DEFAULT_MIN_EXPONENT = -999999999 > #- > ABSOLUTE_MAX_EXP = 999999999 > #- > ABSOLUTE_MIN_EXP = -999999999 I can think of at least one real use for exponents outside this range: probabilities. E.g., if I'm using probabilistic models to estimate P(author|text), I end up multiplying together a large number of very low probabilities, and the total probability could easily get this small. Of course, people usually use tricks to avoid underflow: taking logs, and if necessary performing periodic re-normalization (which means that you don't have probabilities anymore, just some sort of likelihood measures). But it might be nice to not have to do those tricks. I'm +1 for following the lead set by Long, and not adding unnecessary limits. -Edward From FBatista at uniFON.com.ar Thu Apr 15 11:18:58 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 15 11:21:15 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Edward Loper] #- > #- > DEFAULT_MAX_EXPONENT = 999999999 #- > #- > DEFAULT_MIN_EXPONENT = -999999999 #- > #- > ABSOLUTE_MAX_EXP = 999999999 #- > #- > ABSOLUTE_MIN_EXP = -999999999 #- #- I can think of at least one real use for exponents outside #- this range: #- probabilities. E.g., if I'm using probabilistic models to estimate So you need a number smaller than 10**(-999999999)? . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040415/c50b02f5/attachment.html From tim.one at comcast.net Thu Apr 15 11:32:15 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu Apr 15 11:32:25 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: Message-ID: [Batista, Facundo] > I'm lost. Help me with an example (that later I'll shamelessly stole to > use in a test case): > > d1 = Decimal('123') > d2 = Decimal('456') > > So, long, they have the same context, right? Context is associated with operations, not with instances, so d1 and d2 don't "have" any context. It's unclear to me what use (if any) the Decimal() constructor makes of context right now. To the extent that the Decimal() constructor makes of context, to exactly that same extent the Decimal constructor is "an operation", so while context may affect Decimal(), I don't know exactly how it does now. In any case, context has no effect on the instances d1 and d2 after they're constructed (although context both affects and is affected by any operations d1 and d2 may later take part in). > getcontext().prec = 46 > > The precision should change for both d1 and d2, right? Sorry, it simply doesn't make sense to talk about context wrt instances, only wrt operations. If, e.g., you do an operation now, like d4 = d1 + d2 then the result of the operation "d1+d2" has precision 46. > d3 = Decimal('789') > > d3 also has the precision in 46, right? That depends on how Decimal() is implemented, and I don't know the answer to that. From FBatista at uniFON.com.ar Thu Apr 15 11:38:54 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 15 11:41:04 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Tim Peters] #- Context is associated with operations, not with instances, #- so d1 and d2 #- don't "have" any context. It's unclear to me what use (if any) the You're right. My confusion. Sorry. . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040415/57d38a90/attachment.html From mikepy-0404 at mike.de Thu Apr 15 11:52:30 2004 From: mikepy-0404 at mike.de (Mike Pall) Date: Thu Apr 15 11:56:29 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: <1082039905.2610.128.camel@localhost.localdomain> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> <200404151427.i3FERHo26387@guido.python.org> <1082039905.2610.128.camel@localhost.localdomain> Message-ID: <20040415155230.GA692@mike.de> Hi, Jeremy Hylton wrote: > Another question that I'd love to hear the answer to: What's the > difference between Pysco and something like this Self implementation or > the HotSpot Java implementation? I've never read a clear explanation of > what Pysco does or how it compares to other systems that do dynamic > compilation. Here is a link for Java HotSpot: http://java.sun.com/products/hotspot/docs/whitepaper/Java_Hotspot_v1.4.1/Java_HSpot_WP_v1.4.1_1002_4.html It seems that many advantages gained by static typing are lost since the Java codebase encourages strong polymorphism (in contrast to C++). What can we learn from this for Python: static typing is probably not all that helpful for a JIT compiler (in contrast to popular belief). About the CLR (.NET and C# target) JIT: http://www.artima.com/intv/choices.html What I found most interesting: their bytecode is polymorphic because they never targetted an interpreter and the JIT must do type inference anyway. Well, Python bytecode doesn't look that bad now ... I would be very interested to hear what Armin thinks about static vs. dynamic type inferencing and about semi-static (CLR) vs. hotspot (Java) compilation. Bye, Mike From mikepy-0404 at mike.de Thu Apr 15 11:56:20 2004 From: mikepy-0404 at mike.de (Mike Pall) Date: Thu Apr 15 11:56:35 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> Message-ID: <20040415155620.GB692@mike.de> Hi, Christopher Petrilli wrote: > Please do make sure that "better code on x86" doesn't equally worse > code on other platforms. It may be the "majority" platform, but it is > not the only platform. We are aware of that. In fact I gave up on trying to trick GCC into producing good code by rearranging the C source. A single architecture dependent #ifdef plus some inline assembly seems much cleaner. And it can be removed quickly if it ever turns out to be a bad idea for newer x86 processors. Bye, Mike From skip at pobox.com Thu Apr 15 11:59:25 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu Apr 15 11:59:47 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> Message-ID: <16510.45405.16024.437304@montanaro.dyndns.org> Tim> What makes the current stepping of some flavor of P4 overwhelmingly Tim> important <0.3 wink>? Unrelated to this thread, what does "stepping" mean? A quick Google search for "Pentium stepping glossary" yielded this page http://processorfinder.intel.com/scripts/help3.asp but doesn't explain why Intel uses "stepping" instead of "version" or "revision". Skip From edloper at gradient.cis.upenn.edu Thu Apr 15 12:49:51 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Thu Apr 15 12:48:31 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: Message-ID: <407EBD2F.1080402@gradient.cis.upenn.edu> > #- > #- > DEFAULT_MAX_EXPONENT = 999999999 > #- > #- > DEFAULT_MIN_EXPONENT = -999999999 > #- > #- > ABSOLUTE_MAX_EXP = 999999999 > #- > #- > ABSOLUTE_MIN_EXP = -999999999 > #- > #- I can think of at least one real use for exponents outside > #- this range: > #- probabilities. E.g., if I'm using probabilistic models to estimate > > So you need a number smaller than 10**(-999999999)? Not often. And of course, "need" might be a bit strong, since I can use other tricks (logs & renormalization) to avoid underflow. But yes, I can imagine cases where I would get underflow using naive algorithms. E.g., if I'm looking at probabilities for 1 billion words of text (computational linguists are starting to use corpora of this size), and for some reason I want to do language modelling on it, I could end up with numbers on the order of (1e-20)**(1,000,000,000). I.e., 10**(-20,000,000,000). On the other hand, I can't think of a use, off hand, of calculating the probability of a billion word string of text. But I just wanted to point out that there may be cases where extremely large/small numbers can be useful; and as computers get more powerful and space gets cheaper, the numbers people use will tend to grow in size. I understand the argument that this *may* help catch errors, but I just don't think it *will* catch many errors. How many people would have errors that would have been caught if long had an absolute min/max? -Edward From theller at python.net Thu Apr 15 13:11:02 2004 From: theller at python.net (Thomas Heller) Date: Thu Apr 15 13:11:46 2004 Subject: [Python-Dev] Inno Setup script for Python debug runtime In-Reply-To: <407D9AA4.10108@v.loewis.de> ( =?iso-8859-1?q?Martin_v._L=F6wis's_message_of?= "Wed, 14 Apr 2004 22:10:12 +0200") References: <407D91BB.2030603@v.loewis.de> <407D9AA4.10108@v.loewis.de> Message-ID: Most probably I was too short in describing what I want, and what this script is and does, I also got a personal email asking for more, so I'll try again: Inno Setup is a free tool of superb quality to create executable windows installers. See http://www.jrsoftware.org/isinfo.php The script I mention allows to create a windows installer containing *only* the debugging binaries and libraries which are created when you compile Python from the sources in debug mode. When run, it copies these files into an existing, standard Python installation, the location is read from the registry. The standard files are not overwritten, since the debug binaries on windows have an additional _d appended to the filename (python_d.exe instead of python.exe, for example). This allows easy building of debug versions of your own extensions for debugging. The existing python.iss inno setup script is not used in the official Python build, although without doubt it is useful to other people. I want to run the official Python binary distribution (I must, because I build them ;-), and also want to have the debug binaries in the same location. This debug build is also installed by my coworkers, who don't want to build debug Python themselves (at least when the standard extension modules requiring third party code, like zlib, have to be included). That is the reason that I don't want to include this into the existing python.iss script. Thomas From mwh at python.net Thu Apr 15 13:26:44 2004 From: mwh at python.net (Michael Hudson) Date: Thu Apr 15 13:27:25 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: <1082039905.2610.128.camel@localhost.localdomain> (Jeremy Hylton's message of "Thu, 15 Apr 2004 10:38:26 -0400") References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> <200404151427.i3FERHo26387@guido.python.org> <1082039905.2610.128.camel@localhost.localdomain> Message-ID: <2m65c117aj.fsf@starship.python.net> Jeremy Hylton writes: > Another question that I'd love to hear the answer to: What's the > difference between Pysco and something like this Self implementation > or the HotSpot Java implementation? I've never read a clear > explanation of what Pysco does or how it compares to other systems > that do dynamic compilation. I don't know the answer to this... I suspect part of it might have to do with 'abstract interpretation'. I'll see Armin tomorrow (at PythonUK) and beat him up until he answers this question himself :-) Cheers, mwh -- please realize that the Common Lisp community is more than 40 years old. collectively, the community has already been where every clueless newbie will be going for the next three years. so relax, please. -- Erik Naggum, comp.lang.lisp From bsder at allcaps.org Thu Apr 15 14:08:36 2004 From: bsder at allcaps.org (Andrew P. Lentvorski, Jr.) Date: Thu Apr 15 14:04:40 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <1082035943.407e8ee7975ef@mcherm.com> References: <1082035943.407e8ee7975ef@mcherm.com> Message-ID: <20040415093930.F5694@mail.allcaps.org> On Thu, 15 Apr 2004, Michael Chermside wrote: > I strongly object. I have one particular use case for decimal, and I > consider it to be a very important one.... to store and manipulate > monetary amounts. Putting a limit as small as that used for floats > would significantly harm my use case. I presume that you are objecting to only having a default limit of 15 digits of precision? I can certainly understand your position. However, why does having a soft default limit set at this level cause you an issue? If you need more, you can move it up. You have to override the default precision (currently 8) anyway. If you want a default limit much higher than this, then there needs to be a way for people to determine when they leave the realm of "fast, but narrower range" and enter the realm of "slow, but much greater range". What I am trying to establish is a "sensible default" limit which allows most people to use fast operations, does not limit those people who want greater range (at the expense of slower operation), does not impose undue constraints on an optimized version of the module (either software or later hardware), and provides useful feedback for those who suddenly find themselves moving from one group to another (either intentionally or unintentionally). Part of creating Decimal is to provide a useful module for currency and other applications which require accurate Decimal arithmetic. But part of creating Decimal is also to be newbie friendly. Letting people know when they pass outside of "sensible" regimes is friendly. > get overflow errors instead of out-of-memory errors (and possibly > some vague future advantage to implementers of a "complete > numerical package" including things like sin() and cos()). And here I was thinking that someone manipulating financial amounts might actually need speed on a silly operation known as "division". My mistake. -a From jim.jewett at eds.com Thu Apr 15 14:08:34 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Thu Apr 15 14:09:13 2004 Subject: [Python-Dev] Optimization targets - refcount Message-ID: Mike Pall: > About GC: yes, refcounting is the silent killer. >... Py_DECREF is awful ... > 3500 locations Is it always needed? What if a few common (constant, singleton) objects (such as None, -1, 0, 1) were declared immortal at compile-time? They would be created at initial load in a special untracked pool, and their tp_dealloc would do nothing. The slot for tracking references would still be there, but could be ignored -- even if it went negative. Since the reference count no longer has to be correct (for these objects), the reference counting macros could optimize to nothing when they know at compile time that they'll have one of these constant objects. -jJ From python at rcn.com Thu Apr 15 14:20:34 2004 From: python at rcn.com (Raymond Hettinger) Date: Thu Apr 15 14:22:09 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <16510.45405.16024.437304@montanaro.dyndns.org> Message-ID: <009001c42316$6a74aae0$e841fea9@oemcomputer> > Tim> What makes the current stepping of some flavor of P4 > overwhelmingly > Tim> important <0.3 wink>? > > Unrelated to this thread, what does "stepping" mean? A quick Google > search > for "Pentium stepping glossary" yielded this page > > http://processorfinder.intel.com/scripts/help3.asp > > but doesn't explain why Intel uses "stepping" instead of "version" or > "revision". [Channeling Tim] In this context, "stepping" means all of the idiosyncratic implementation details of processor workflow. The word "stepping" is natural after staring at Gantt charts showing timings, data dependencies, speculative execution, and parallel operations. Tim is advising me to against focusing on one processor at one point in history because the "stepping" is guaranteed to change over time. Raymond Hettinger From theller at python.net Thu Apr 15 14:31:16 2004 From: theller at python.net (Thomas Heller) Date: Thu Apr 15 14:31:22 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <16510.45405.16024.437304@montanaro.dyndns.org> (Skip Montanaro's message of "Thu, 15 Apr 2004 10:59:25 -0500") References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <16510.45405.16024.437304@montanaro.dyndns.org> Message-ID: Skip Montanaro writes: > Tim> What makes the current stepping of some flavor of P4 overwhelmingly > Tim> important <0.3 wink>? > > Unrelated to this thread, what does "stepping" mean? A quick Google search > for "Pentium stepping glossary" yielded this page > > http://processorfinder.intel.com/scripts/help3.asp > > but doesn't explain why Intel uses "stepping" instead of "version" or > "revision". Here is a possible answer: A stepping is the processor hardware equivalent of a new version. In software, we refer to minor version changes as 1.0, 1.1, 1.2, and so on, while in processors we call these minor revisions steppings. Found in Thomas From martin at v.loewis.de Thu Apr 15 15:07:12 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu Apr 15 15:07:30 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: <1082039905.2610.128.camel@localhost.localdomain> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> <200404151427.i3FERHo26387@guido.python.org> <1082039905.2610.128.camel@localhost.localdomain> Message-ID: <407EDD60.8000101@v.loewis.de> Jeremy Hylton wrote: > Another question that I'd love to hear the answer to: What's the > difference between Pysco and something like this Self implementation or > the HotSpot Java implementation? Psyco is specializing, and that is the main difference compared to Java Hotspot. If you have a program def f(a, b): if a>b: return a else: return a-b f(1,2) f(1.0, 2.0) f([1,2],[3,4]) then psyco generates machine code for *three* functions int f_int(int a, int b){ // uses process int arithmethic throughout if(a>b)return a; else return a-b; } double f_double(double a, double b){ // uses FPU ops throughout if(a>b)return a; else return a-b; } list f_list(list a, list b){ // might invoke C functions if(list_gt(a, b))return a; else raise TypeError("unexpected operands"); } (it actually generates different specializations) In Hotspot, the type of f would already be defined in the Java source code, and Hotspot generates native machine instructions for it. The changes over standard JIT appear to be inlining; it also appears to do inlining of virtual functions, combined with a type check to detect cases where the a different functions should have been called compared to the last time the virtual call was made. Regards, Martin From op73418 at mail.telepac.pt Thu Apr 15 15:22:00 2004 From: op73418 at mail.telepac.pt (Goncalo Rodrigues) Date: Thu Apr 15 15:31:14 2004 Subject: [Python-Dev] Use case for class decorators Message-ID: Here is a use case for class decorators. I find myself writing a lot of proxies and adapters. Usually, they are just a subclass of something as simple as class Proxy(object): """The Proxy base class.""" def __init__(self, obj): """The initializer.""" super(Proxy, self).__init__(obj) #Private attributes. self.__obj = obj #Read-only properties. __obj__ = property(lambda self: self.__obj, doc = "The underlying object of the Proxy.") def __repr__(self): return "%s(%r)" % (self.__class__.__name__, self.__obj__) def __getattr__(self, attrib): return getattr(self.__obj__, attrib) This is all fine and dandy, except that no magic methods are trapped. Usually, I just inject them directly after the class statement via: #Auxiliary getter function. def getter(attrib): return lambda self, *args, **kwargs: getattr(self.__obj__, attrib)(*args, **kwargs) def inject(cls, magicnames): """Inject magic methods directly in a class. Warning(s): - Only magic names are injected. - Does not overwrite already present magic names. """ for magicname in magicnames: if magicname.startswith("__") and magicname.endswith("__"): if not hasattr(cls, magicname): setattr(cls, magicname, getter(magicname)) The function inject is a use case for a class decorator. If it's a good use case that's something I'll leave you to argue at will (). I think it's fairly obvious that using a metaclass to do this simple operation is a clear case of the wrong tool for the job. With my best regards, G. Rodrigues p.s: The syntax for decorators: [dec1, dec2, ...] def foo(arg1, arg2, ...): pass is probably the worst of all syntaxes (with appologies for my bluntness). It just doesn't parse well. What I see in the above proposed syntax is just a list literal and a definition statement. There's no indication that the two are somehow linked (I can't express this any better, English is not my native lingo). The best, IMHO is def foo(arg1, arg2, ...) [dec1, dec2, ...]: pass <\ comment on syntax> From tcdelaney at optusnet.com.au Thu Apr 15 18:07:19 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Thu Apr 15 18:07:29 2004 Subject: [Python-Dev] Inno Setup script for Python debug runtime References: <407D91BB.2030603@v.loewis.de> <407D9AA4.10108@v.loewis.de> Message-ID: <006301c42336$05459970$a100a8c0@ryoko> Thomas Heller wrote: > Most probably I was too short in describing what I want, and what this > script is and does, I also got a personal email asking for more, so > I'll try again: > > Inno Setup is a free tool of superb quality to create executable > windows installers. See http://www.jrsoftware.org/isinfo.php I've found NSIS to be a much better choice for Windows installers. Produces very small installers (adds about 30-40K overhead, can use LZMA), has multiple language support, and are simple to write. http://nsis.sourceforge.net/home/ I keep intending to write an installer for Python in it - maybe next week (I'm on leave at the moment, but I've got the niece (6) and nephew (5) visiting all this week). I've been gradually moving all our installers at work away from WISE and into NSIS. Tim Delaney From barry at barrys-emacs.org Thu Apr 15 19:19:19 2004 From: barry at barrys-emacs.org (Barry Scott) Date: Thu Apr 15 19:31:29 2004 Subject: [Python-Dev] Inno Setup script for Python debug runtime In-Reply-To: <006301c42336$05459970$a100a8c0@ryoko> References: <407D91BB.2030603@v.loewis.de> <407D9AA4.10108@v.loewis.de> <006301c42336$05459970$a100a8c0@ryoko> Message-ID: <6.0.3.0.2.20040416000249.03208e50@torment.chelsea.private> At 15-04-2004 23:07, Tim Delaney wrote: >Thomas Heller wrote: > > > Most probably I was too short in describing what I want, and what this > > script is and does, I also got a personal email asking for more, so > > I'll try again: > > > > Inno Setup is a free tool of superb quality to create executable > > windows installers. See http://www.jrsoftware.org/isinfo.php > >I've found NSIS to be a much better choice for Windows installers. Produces >very small installers (adds about 30-40K overhead, can use LZMA), has >multiple language support, and are simple to write. When I looked a while ago NSIS was overly complex compared to INNO given the same task. I use INNO to install the pysvn extension and it was easy to do. See http://svn.collab.net/repos/pysvn/trunk/pysvn/Extension/Kit/Win32/pysvn.iss Barry From greg at cosc.canterbury.ac.nz Thu Apr 15 20:09:38 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Apr 15 20:09:56 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: Message-ID: <200404160009.i3G09cAX020939@cosc353.cosc.canterbury.ac.nz> Ka-Ping Yee : > The extra method should be specifically for the exceptional case, > and named appropriately. > > Decimal.exact_float(value) That's a confusing name, since it seems to be promising that the conversion will be exact. While this might technically be true for certain values of float, there's no way the implementation can know whether the result is exactly what the user had in mind for the meaning of that float. I think the name from_float is fine, since it says "I know constructing a Decimal from a float isn't necessarily exact, but I want to do it anyway, and I'm prepared to accept the result, whatever it is." Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu Apr 15 21:22:57 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu Apr 15 21:23:09 2004 Subject: [Python-Dev] Decimal conversion to string In-Reply-To: <20040415142941.GA22303@panix.com> Message-ID: <200404160122.i3G1MvjO021056@cosc353.cosc.canterbury.ac.nz> Aahz : > Right. Question is, to what extent does "human readable" factor in > after these criteria? In the case of Decimal, I think it's fairly clear that Decimal("123.45") is the winner overall, since both the type and value are unambiguous and easily readable, and it's eval-able to boot. On the other hand, I see absolutely zero utility in displaying the tuple-form of the internals. >From the point of view of using the interactive interpreter as a desk calculator, it might be slightly more desirable to see Decimal(123.45) but that wouldn't be eval-able. For such use it might be better to have a special calculator-mode interpreter that took all numeric literals as Decimal and displayed them unadorned. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From perky at i18n.org Thu Apr 15 22:56:49 2004 From: perky at i18n.org (Hye-Shik Chang) Date: Thu Apr 15 22:56:54 2004 Subject: [Python-Dev] Inno Setup script for Python debug runtime In-Reply-To: <006301c42336$05459970$a100a8c0@ryoko> References: <407D9AA4.10108@v.loewis.de> <006301c42336$05459970$a100a8c0@ryoko> Message-ID: <20040416025649.GA81432@i18n.org> On Fri, Apr 16, 2004 at 08:07:19AM +1000, Tim Delaney wrote: > Thomas Heller wrote: > > > Most probably I was too short in describing what I want, and what this > > script is and does, I also got a personal email asking for more, so > > I'll try again: > > > > Inno Setup is a free tool of superb quality to create executable > > windows installers. See http://www.jrsoftware.org/isinfo.php > > I've found NSIS to be a much better choice for Windows installers. Produces > very small installers (adds about 30-40K overhead, can use LZMA), has > multiple language support, and are simple to write. > > http://nsis.sourceforge.net/home/ > > I keep intending to write an installer for Python in it - maybe next week > (I'm on leave at the moment, but I've got the niece (6) and nephew (5) > visiting all this week). I've been gradually moving all our installers at > work away from WISE and into NSIS. > CJKPython (http://cjkpython.i18n.org/#CJKPython) is based on NSIS. And .nsi source is available at http://cvs.berlios.de/cgi-bin/viewcvs.cgi/cjkpython/CJKPython/cjkpython.nsi?rev=1.4&content-type=text/vnd.viewcvs-markup As you may find, lack of log-based uninstall feature makes .nsi file too large. Hye-Shik From jeremy at alum.mit.edu Thu Apr 15 23:25:56 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu Apr 15 23:26:19 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: <407EDD60.8000101@v.loewis.de> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> <200404151427.i3FERHo26387@guido.python.org> <1082039905.2610.128.camel@localhost.localdomain> <407EDD60.8000101@v.loewis.de> Message-ID: <1082085955.4953.15.camel@localhost.localdomain> On Thu, 2004-04-15 at 15:07, "Martin v. L?wis" wrote: > Jeremy Hylton wrote: > > Another question that I'd love to hear the answer to: What's the > > difference between Pysco and something like this Self implementation or > > the HotSpot Java implementation? > > Psyco is specializing, and that is the main difference compared > to Java Hotspot. If you have a program That's a helpful clarification, thanks. > In Hotspot, the type of f would already be defined in the Java > source code, and Hotspot generates native machine instructions > for it. > > The changes over standard JIT appear to be inlining; it also > appears to do inlining of virtual functions, combined with > a type check to detect cases where the a different functions > should have been called compared to the last time the virtual > call was made. I'm not sure what you mean by the changes over standard JIT. Do you mean the difference between Hotspot and "standard JIT"? Your descriptions sounds a bit like the Deutsch and Schiffman inline method cache, which doesn't inline the method body but does inline the method lookup. Or does Hotspot actually inline methods? Jeremy From tim_one at email.msn.com Fri Apr 16 00:17:24 2004 From: tim_one at email.msn.com (Tim Peters) Date: Fri Apr 16 00:17:51 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <407EA7A4.6070709@gradient.cis.upenn.edu> Message-ID: >> #- > DEFAULT_MAX_EXPONENT = 999999999 >> #- > DEFAULT_MIN_EXPONENT = -999999999 >> #- > ABSOLUTE_MAX_EXP = 999999999 >> #- > ABSOLUTE_MIN_EXP = -999999999 [Edward Loper] > I can think of at least one real use for exponents outside this range: > probabilities. E.g., if I'm using probabilistic models to estimate > P(author|text), I end up multiplying together a large number of very low > probabilities, and the total probability could easily get this small. No, it couldn't -- and if you worked hard to contrive an example, a probability in the end smaller than the probability that the universe will spontaneously decide to run time backwards for a century is accurately enough represented by 0 (which your example will underflow to). From tim_one at email.msn.com Fri Apr 16 00:17:24 2004 From: tim_one at email.msn.com (Tim Peters) Date: Fri Apr 16 00:17:58 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: Message-ID: [Facundo Batista, on whether to limit exponent magnitude] > The feeling previous to this mail was that an artificial limit was silly > if you don't gain nothing. > > Here are three valuable reasons for keep the limits. > > So, if nobody opposes strongly, I'll put these reasons in the PEP and > keep the limit. I'm only +0 on keeping the limits, btw -- I think they do more good than harm, but it's not going to kill us either way. ... > Until now, Decimal does not uses context in creation time (with creation > I mean when you create an object instantiating the class). > > This allows to not lose any information at creation time, and only use > the context in operations. > > Do you want to change this behaviour? No, I want to spell the spec's mandatory from-string *operation* (which does use and affect context, in the specified ways) via an optional bool argument to the Decimal constructor. This isn't at all useless, although if you're only thinking about a money type it may seem that way. In general number-crunching, literals may be given to high precision, but that precision isn't free and *usually* isn't needed: the operations in the spec don't round back inputs to current context precision before computing, they only round outputs after, and digits feeding into computation are expensive. So, in general, a scientific user will usually *not* want all the digits in pi = Decimal("3.1415926535897932384626433832795", use_context=True) to be used. Instead they'll want to set the working precision once in context, and have their literals automatically get rounded back to that precision. This pattern will be familiar to Fortran77 users, where Fortran REAL ("single precision") literals are usually given to DOUBLE precision, so that changing a single IMPLICIT (static typing) statement can make the same code usable as-is under both REAL and DOUBLE precision. Decimal supports an unbounded number of precisions, and so the need for pleasant precision control is that much more acute (and the spec's from-string operation is one aid toward that end). ... [on what repr() should do] > The only issue *I* see here, is that you don't have a clear > representation of the internals of the object, But nobody cares except implementers. I'm perfectly happy to introduce an .as_tuple() method for them. More generally, Python should (eventually, not necessarily at the start) implement an extension to context, specifying the desired str() and repr() behavior. One size doesn't fit all, and context is the natural place to record individual preferences. Against "one size doesn't fit all", everyone can at least *understand* the to-sci-string format, while only a handful will ever be able to read the tuples with ease. > but yes, you have an exact representation of that number. Let me > show an example: There are more in the spec (for those interested). > >>> d = Decimal.Decimal((1, (1,2,3,4,0), 1)) > >>> d > Decimal( (1, (1, 2, 3, 4, 0), 1L) ) > >>> d.to_sci_string() > '-1.2340E+5' > > There you have to count the decimals and use the exponent in the string > to know what the real exponent is (but I don't know how much important > this is). It's not important to end users; indeed, trying to tell most of them that "the real exponent" *isn't* 5 would be difficult. For most people most of the time, after entering 12345.67, output of Decimal("12345.67") will be much more understandable than output of Decimal( (0, (1, 2, 3, 4, 5, 6, 7), -2) ) From tim_one at email.msn.com Fri Apr 16 00:17:24 2004 From: tim_one at email.msn.com (Tim Peters) Date: Fri Apr 16 00:18:03 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <16510.45405.16024.437304@montanaro.dyndns.org> Message-ID: [Skip] > Unrelated to this thread, what does "stepping" mean? A quick Google > search for "Pentium stepping glossary" yielded this page > > http://processorfinder.intel.com/scripts/help3.asp > > but doesn't explain why Intel uses "stepping" instead of "version" or > "revision". It means "minor revision". Major revisions are usually given names (my current P4 was/is known as "Northwood"). I don't know why minor revisions are called steppings -- I've never been a HW guy, although the term has been in common use among HW guys I've known. From edloper at gradient.cis.upenn.edu Fri Apr 16 00:52:04 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Fri Apr 16 00:51:05 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: Message-ID: <407F6674.4070402@gradient.cis.upenn.edu> Tim Peters wrote: >>I can think of at least one real use for exponents outside this range: >>probabilities. E.g., if I'm using probabilistic models to estimate >>P(author|text), I end up multiplying together a large number of very low >>probabilities, and the total probability could easily get this small. > > No, it couldn't -- and if you worked hard to contrive an example, a > probability in the end smaller than the probability that the universe will > spontaneously decide to run time backwards for a century is accurately > enough represented by 0 (which your example will underflow to). As long as the limits are soft (and can be changed), I won't object too strenuously, because someone who wanted extremely large/small numbers could use them. But... - I do believe that there will be use cases for exponents outside this range. Just because there's no physical quantities that are reasonably measured with these numbers, doesn't mean that there won't be abstract quantities that are reasonably measured with huge numbers. - I don't believe that signaling an error when a number goes outside this range will help catch any errors. What type of error are we expecting to catch here? If this is such a problem, then why doesn't long also have a max/min? Basically, I don't see much loss from adding the constraints, but I also see very little gain. -Edward From martin at v.loewis.de Fri Apr 16 02:00:12 2004 From: martin at v.loewis.de (=?ISO-8859-15?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri Apr 16 02:00:42 2004 Subject: [Python-Dev] Optimization targets In-Reply-To: <1082085955.4953.15.camel@localhost.localdomain> References: <001901c4223a$5bab0a40$e841fea9@oemcomputer> <20040414214945.GA3443@mike.de> <2mekqp1oxq.fsf@starship.python.net> <20040415133613.GA4735@mike.de> <200404151427.i3FERHo26387@guido.python.org> <1082039905.2610.128.camel@localhost.localdomain> <407EDD60.8000101@v.loewis.de> <1082085955.4953.15.camel@localhost.localdomain> Message-ID: <407F766C.3070304@v.loewis.de> Jeremy Hylton wrote: > I'm not sure what you mean by the changes over standard JIT. Do you > mean the difference between Hotspot and "standard JIT"? Your > descriptions sounds a bit like the Deutsch and Schiffman inline method > cache, which doesn't inline the method body but does inline the method > lookup. Or does Hotspot actually inline methods? I lost the link, but on some page describing it, they actually claimed they do inline methods. They went on explaining that you need "deoptimization" in case a different method should be called; I assume they generate a type check just before the inlined method body. I might be that they keep collecting statistics on how often the type guess was wrong to discard the inlining, and come up with a new one. Hotspot operates by only compiling functions selectively, which have been invoked a number of times, and they claim they collect data on where virtual calls go to. So when they eventually do generate machine code, they have the data to do the inlining. A "standard" JIT might chose to compile everything the first time it is invoked, and would then just replace the virtual-call bytecode with the necessary memory fetches and an indirect call. Regards, Martin From paul at prescod.net Fri Apr 16 02:11:00 2004 From: paul at prescod.net (Paul Prescod) Date: Fri Apr 16 02:13:21 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: References: Message-ID: <407F78F4.6000408@prescod.net> Jewett, Jim J wrote: > Mike Pall: > > >>About GC: yes, refcounting is the silent killer. >>... Py_DECREF is awful ... > 3500 locations > > > Is it always needed? > > What if a few common (constant, singleton) objects > (such as None, -1, 0, 1) were declared immortal at > compile-time? They would be created at initial > load in a special untracked pool, and their > tp_dealloc would do nothing. > > The slot for tracking references would still be > there, but could be ignored -- even if it went > negative. > > Since the reference count no longer has to be > correct (for these objects), the reference > counting macros could optimize to nothing when > they know at compile time that they'll have one > of these constant objects. So instead of writing Py_DECREF(foo) would we write code like if(foo!=Py_None&&foo!=Py_Neg1&&foo!=Py_Zero&&foo!=PyOne){ Py_DECREF(foo); } Paul Prescod From tcdelaney at optusnet.com.au Fri Apr 16 04:01:22 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Fri Apr 16 04:01:38 2004 Subject: [Python-Dev] Decimal data type issues References: <407F6674.4070402@gradient.cis.upenn.edu> Message-ID: <001a01c42389$0252a980$a100a8c0@ryoko> Edward Loper wrote: > As long as the limits are soft (and can be changed), I won't object > too > strenuously, because someone who wanted extremely large/small numbers > could use them. But... > > - I do believe that there will be use cases for exponents outside > this range. Just because there's no physical quantities that are > reasonably measured with these numbers, doesn't mean that there > won't be abstract quantities that are reasonably measured with > huge numbers. There's no reason that if these use cases become common enough to care about, that a future version of Python couldn't increase or eliminate the limits. It's a lot easier to remove limits (e.g. the int/long unification) than to add them. I'm +0 on having the limits there initially, even if they complicate the implementation somewhat. I'm highly unlikely to ever encounter them, just as I've never produced a long large enough to cause problems. Tim Delaney From python-dev at zesty.ca Fri Apr 16 04:58:26 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri Apr 16 04:58:43 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <200404160009.i3G09cAX020939@cosc353.cosc.canterbury.ac.nz> References: <200404160009.i3G09cAX020939@cosc353.cosc.canterbury.ac.nz> Message-ID: On Fri, 16 Apr 2004, Greg Ewing wrote: > Ka-Ping Yee : > > The extra method should be specifically for the exceptional case, > > and named appropriately. > > > > Decimal.exact_float(value) > > That's a confusing name, since it seems to be promising that the > conversion will be exact. While this might technically be true for > certain values of float, there's no way the implementation can know > whether the result is exactly what the user had in mind for the > meaning of that float. I think you have misunderstood my suggestion. The name promises that the conversion will be exact precisely because it *is* exact. The exact_float() method would always yield a Decimal number with the exact value of the given floating-point number. My understanding of the reason for disallowing a float argument to the Decimal constructor is that people may be confused by exact conversion: they don't realize that the exact value may be slightly different from the decimal number they entered. My point is that since exact conversion is the confusing case, that's what the special method should be for. I'm arguing very much along the same lines as Tim: let's have an optional argument to the Decimal constructor that specifies the precision of the newly constructed Decimal value. Tim is talking about strings; i am generalizing and saying that, when the precision is specified, all number types should be accepted. -- ?!ng From python-dev at zesty.ca Fri Apr 16 05:14:36 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri Apr 16 05:14:11 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: References: Message-ID: On Thu, 15 Apr 2004, Batista, Facundo wrote: > [Ka-Ping Yee] > > #- Aren't you going to need an optional argument anyway to let the > #- user specify a context? > > What for? So the user can specify how much precision the resulting Decimal value should have. > #- I assume the purpose of from_float is to prevent people from being > #- confused about whether Decimal(1.1) produces exactly 1.1 or the machine > #- value of the double-precision rounded float closest to 1.1. Is that > #- correct? > #- > #- If so, then the real exceptional case is *exact* conversion, not float > #- conversion. Decimal(1.1, precision=4) is unambiguous. Therefore: > > Here I think you're confused. *Precision* is a parameter of the context, and > determines in how many digits you will get the coefficient after an > operation. > > The issue when constructing from float, is how many *positions* after the > decimal point will be relevant when creating from a float. I never mentioned positions, and i don't see any reason why it is necessary to isolate these two ways of specifying precision. Sometimes it will be useful to specify a number of significant digits and other times it will be useful to specify a number of digits after the decimal point. Both kinds of specification are useful as a context for arithmetic operations and useful when constructing from any type including float. So it would be nice if contexts were able to specify precision in either fashion. But that is a separate issue from the one i was trying to raise. To bring up this precision vs. position question is to miss my point. Here is my point, again: Decimal(1.1, significant_digits=4) is unambiguous. Converting from float is not confusing when the precision is specified (either in terms of significant digits or decimal places). Therefore, it is not necessary to make all float conversions inconvenient. It is only necessary to make *exact* float conversions inconvenient. Focusing on the true source of the problem also lets us use a more descriptive name for this special case. > #- That's what I'm suggesting: that you can supply a context object, > #- or ask for the default context, when constructing any Decimal value. > > Do you want to supply a context that *will affect* the creation process, or > just supply one that the Decimal will keep to use in other operations? I want to supply a context for constructing the Decimal value. It would not be persistent. You would use some other method (presumably something you have already conceived and named) for setting the Decimal context for the entire thread. -- ?!ng From Paul.Moore at atosorigin.com Fri Apr 16 05:19:53 2004 From: Paul.Moore at atosorigin.com (Moore, Paul) Date: Fri Apr 16 05:19:55 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: <16E1010E4581B049ABC51D4975CEDB88052CB054@UKDCX001.uk.int.atosorigin.com> From: Ka-Ping Yee > Here is my point, again: Decimal(1.1, significant_digits=4) is unambiguous. > > Converting from float is not confusing when the precision is specified > (either in terms of significant digits or decimal places). Decimal(1.1, significant_digits=34) is unambiguous, but confusing. Paul. __________________________________________________________________________ This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. __________________________________________________________________________ From python-dev at zesty.ca Fri Apr 16 05:34:59 2004 From: python-dev at zesty.ca (Ka-Ping Yee) Date: Fri Apr 16 05:34:39 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <16E1010E4581B049ABC51D4975CEDB88052CB054@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB88052CB054@UKDCX001.uk.int.atosorigin.com> Message-ID: On Fri, 16 Apr 2004, Moore, Paul wrote: > From: Ka-Ping Yee > > Here is my point, again: Decimal(1.1, significant_digits=4) is unambiguous. > > > > Converting from float is not confusing when the precision is specified > > (either in terms of significant digits or decimal places). > > Decimal(1.1, significant_digits=34) is unambiguous, but confusing. It is no more confusing than '%.33f' % 1.1, which gives you exactly the same thing. -- ?!ng From pje at telecommunity.com Fri Apr 16 08:40:11 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Apr 16 08:39:55 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: <407F78F4.6000408@prescod.net> References: Message-ID: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> At 11:11 PM 4/15/04 -0700, Paul Prescod wrote: >Jewett, Jim J wrote: > >>Mike Pall: >> >>>About GC: yes, refcounting is the silent killer. >>>... Py_DECREF is awful ... > 3500 locations >> >>Is it always needed? >>What if a few common (constant, singleton) objects (such as None, -1, 0, >>1) were declared immortal at compile-time? They would be created at >>initial load in a special untracked pool, and their tp_dealloc would do >>nothing. >>The slot for tracking references would still be >>there, but could be ignored -- even if it went >>negative. >>Since the reference count no longer has to be >>correct (for these objects), the reference counting macros could optimize >>to nothing when they know at compile time that they'll have one of these >>constant objects. > >So instead of writing Py_DECREF(foo) would we write code like > >if(foo!=Py_None&&foo!=Py_Neg1&&foo!=Py_Zero&&foo!=PyOne){ > Py_DECREF(foo); >} I think the idea is more that you could skip the 'Py_INCREF(Py_None)', which is a fairly common prelude to 'return Py_None'. You'd set their refcounts to the maximum possible value, and the deallocation function would simply reset the refcount to maximum again. I'm not sure, however, that this would be common enough to be helpful. It seems to me Py_INCREF should effectively translate to only a single machine instruction or two. I mean, it's just incrementing an integer whose address is known at compile time. From FBatista at uniFON.com.ar Fri Apr 16 08:48:34 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Fri Apr 16 08:50:44 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Ka-Ping Yee] #- My understanding of the reason for disallowing a float #- argument to the #- Decimal constructor is that people may be confused by exact #- conversion: #- they don't realize that the exact value may be slightly different #- from the decimal number they entered. #- #- My point is that since exact conversion is the confusing case, that's #- what the special method should be for. I think is deeper than that. For example, *I* don't understand what is "exact conversion" to you: 1.1 -> "1.1" 1.1 -> "1.1000000000000001" 1.1 -> "1.100000000000000088817841970012523233890533447265625" So, what is the expectation that floating point newbies will understand this safely enough as to let them create from: Decimal(float, extra_argument) Decimal.exact_float() To me, is complicate the things to much for the newcomer. I think the following reasoning is simpler: Do you want to create from string? Strings are exact, use Decimal(string) Do you want to create from int/long? Ints and longs are exact, use Decimal(int/long) Do you want to create from float? Oops, floats are NOT exact, so you have to use Decimal.from_float(float), but take note of this, this and this. . Facundo From FBatista at uniFON.com.ar Fri Apr 16 09:07:09 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Fri Apr 16 09:09:28 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Ka-Ping Yee] #- On Thu, 15 Apr 2004, Batista, Facundo wrote: #- >... #- > Here I think you're confused. *Precision* is a parameter #- of the context, and #- > determines in how many digits you will get the coefficient after an #- > operation. #- > #- > The issue when constructing from float, is how many #- *positions* after the #- > decimal point will be relevant when creating from a float. #- #- I never mentioned positions, and i don't see any reason why it is #- necessary to isolate these two ways of specifying precision. Because they are two totally different concepts. And they're not two ways of specifying precision. Precision is the maximum of significant digits the number can have. And positions is the number of digits after the decimal point at which you want to round the number you're creating from and discard the other digits. #- Sometimes #- it will be useful to specify a number of significant digits and other #- times it will be useful to specify a number of digits after #- the decimal #- point. Both kinds of specification are useful as a context #- for arithmetic #- operations and useful when constructing from any type #- including float. #- #- So it would be nice if contexts were able to specify #- precision in either #- fashion. Well. If you want the positions to be included in the context to be used in arithmetic operations, you're out of spec. And the PEP is for implementing the "General Decimal Arithmetic Specification", so the answer to that is NO. #- But that is a separate issue from the one i was trying to raise. To #- bring up this precision vs. position question is to miss my point. #- #- Here is my point, again: Decimal(1.1, significant_digits=4) #- is unambiguous. I don't think so. For example: >>> d = Decimal.Decimal.from_float(25364.1254) >>> str(d) '25364.12540000000080908648669719696044921875' What whould mean here "significant_digits=4"? To have '2536E2'? You use "precision" to achieve this. To have '25364.1254'? You use "position" in from_float method to achieve this. #- Converting from float is not confusing when the precision is #- specified #- (either in terms of significant digits or decimal places). Don't confuse the terminology. Precision means only one thing, and it's specified in Cowlishaw work. You can not change that. Maybe all the discussion doesn't make sense if we talk about different things. #- I want to supply a context for constructing the Decimal value. I'm +1 with this. But the context that you supply affect the PRECISION to be used. It has nothing to do with POSITIONS. . Facundo From casey at zope.com Fri Apr 16 09:36:05 2004 From: casey at zope.com (Casey Duncan) Date: Fri Apr 16 09:34:52 2004 Subject: [Python-Dev] Re: Decimal data type issues References: Message-ID: <20040416093605.78034671.casey@zope.com> On Fri, 16 Apr 2004 00:17:24 -0400 "Tim Peters" wrote: > [Facundo Batista, on whether to limit exponent magnitude] [..] > No, I want to spell the spec's mandatory from-string *operation* > (which does use and affect context, in the specified ways) via an > optional bool argument to the Decimal constructor. This isn't at all > useless, although if you're only thinking about a money type it may > seem that way. In general number-crunching, literals may be given to > high precision, but that precision isn't free and *usually* isn't > needed: the operations in the spec don't round back inputs to current > context precision before computing, they only round outputs after, and > digits feeding into computation are expensive. So, in general, a > scientific user will usually *not* want all the digits in > > pi = Decimal("3.1415926535897932384626433832795", > use_context=True) > > to be used. Instead they'll want to set the working precision once in > context, and have their literals automatically get rounded back to > that precision. [..] I find boolean arguments a general anti-pattern, especially given we have class methods. Why not use an alternate constructor like:: pi = Decimal.rounded_to_context("3.1415926535897932384626433832795") Spelling notwithstanding. -Casey From FBatista at uniFON.com.ar Fri Apr 16 09:53:35 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Fri Apr 16 09:55:51 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Casey Duncan] #- I find boolean arguments a general anti-pattern, especially given we #- have class methods. Why not use an alternate constructor like:: #- #- pi = #- Decimal.rounded_to_context("3.1415926535897932384626433832795") This is a good idea to me. We all must decide which of the following is better: a) Decimal(number) # no context used Decimal(number, context) b) Decimal(number) # no context used Decimal.using_context(number, context) "number" can be string, int, etc., but NOT float. The issue with rounding at a defined position has nothing to do with context. . Facundo From skip at pobox.com Fri Apr 16 10:18:38 2004 From: skip at pobox.com (Skip Montanaro) Date: Fri Apr 16 10:18:51 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? Message-ID: <16511.60222.516830.948141@montanaro.dyndns.org> (I'm not sure where precisely to address this. I'd send it directly to Barry but figure that someone else might also know the answer and have infinitessimally more time to respond.) In my new job I nipped the Python project's syncmail script and added it to our CVSROOT. I then tweaked the loginfo file to run syncmail under the proper conditions. The messages go out just fine, but they come from server@servername instead of from the person who executed the checkin (e.g. skipm@servername). Since that's the only indication in the syncmail message who performed the checkin I'd like to get that corrected. It appears syncmail just sends out the message as the user it's running as. Is this a cvs server setting or some tweak to the SF version of CVS? Locally we're running cvs 1.11.2, which seems to match what SF runs on shell.sf.net. Thanks, Skip From casey at zope.com Fri Apr 16 10:32:14 2004 From: casey at zope.com (Casey Duncan) Date: Fri Apr 16 10:30:47 2004 Subject: [Python-Dev] Re: Decimal data type issues References: Message-ID: <20040416103214.6fdf808f.casey@zope.com> On Fri, 16 Apr 2004 10:53:35 -0300 "Batista, Facundo" wrote: > [Casey Duncan] > > #- I find boolean arguments a general anti-pattern, especially given > #we- have class methods. Why not use an alternate constructor like:: > #- > #- pi = > #- Decimal.rounded_to_context("3.1415926535897932384626433832795") > > This is a good idea to me. > > We all must decide which of the following is better: > > a) Decimal(number) # no context used > Decimal(number, context) > > b) Decimal(number) # no context used > Decimal.using_context(number, context) If the context must be explicitly provided as an argument (rather than just a flag) I think a) is better. It depends also on what other arguments the constructor has, and thus what other contracts it fulfills. Are there other optional arguments? -Casey From jacobs at theopalgroup.com Fri Apr 16 10:32:47 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Fri Apr 16 10:32:52 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <407FEE8F.2090402@theopalgroup.com> Batista, Facundo wrote: >[Casey Duncan] > >#- I find boolean arguments a general anti-pattern, especially given we >#- have class methods. Why not use an alternate constructor like:: >#- >#- pi = >#- Decimal.rounded_to_context("3.1415926535897932384626433832795") > >This is a good idea to me. > >We all must decide which of the following is better: > > a) Decimal(number) # no context used > Decimal(number, context) > > b) Decimal(number) # no context used > Decimal.using_context(number, context) > >"number" can be string, int, etc., but NOT float. The issue with rounding at >a defined position has nothing to do with context. > > Don't forget that many financial applications use fixed scale and precision as their primary mechanism for specifying Decimal types. As such, it would be very nice to have a constructor that took a literal representation as well as scale and precision. While using context is sufficient, it will be unintuitive and may present an undue burden on many users. So-many-people-to-make-happy-ly, -Kevin From guido at python.org Fri Apr 16 10:58:17 2004 From: guido at python.org (Guido van Rossum) Date: Fri Apr 16 10:58:24 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: Your message of "Fri, 16 Apr 2004 08:40:11 EDT." <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> References: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> Message-ID: <200404161458.i3GEwH328527@guido.python.org> > I think the idea is more that you could skip the 'Py_INCREF(Py_None)', > which is a fairly common prelude to 'return Py_None'. You'd set their > refcounts to the maximum possible value, and the deallocation function > would simply reset the refcount to maximum again. > > I'm not sure, however, that this would be common enough to be helpful. It > seems to me Py_INCREF should effectively translate to only a single machine > instruction or two. I mean, it's just incrementing an integer whose > address is known at compile time. I vaguely recall that this was proposed and perhaps even tried before, and not found to make a difference. --Guido van Rossum (home page: http://www.python.org/~guido/) From sjoerd at acm.org Fri Apr 16 11:26:04 2004 From: sjoerd at acm.org (Sjoerd Mullender) Date: Fri Apr 16 11:26:37 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? In-Reply-To: <16511.60222.516830.948141@montanaro.dyndns.org> References: <16511.60222.516830.948141@montanaro.dyndns.org> Message-ID: <407FFB0C.6050200@acm.org> Skip Montanaro wrote: > (I'm not sure where precisely to address this. I'd send it directly to > Barry but figure that someone else might also know the answer and have > infinitessimally more time to respond.) > > In my new job I nipped the Python project's syncmail script and added it to > our CVSROOT. I then tweaked the loginfo file to run syncmail under the > proper conditions. The messages go out just fine, but they come from > server@servername instead of from the person who executed the checkin > (e.g. skipm@servername). Since that's the only indication in the syncmail > message who performed the checkin I'd like to get that corrected. It > appears syncmail just sends out the message as the user it's running as. Is > this a cvs server setting or some tweak to the SF version of CVS? Locally > we're running cvs 1.11.2, which seems to match what SF runs on shell.sf.net. Are you using ssh or pserver to talk to the cvs server? If using ssh, does every user have his/her own account on the server that they use? When using pserver, the cvs server is run under a fixed user id, so that's where the mail will appear to come from. -- Sjoerd Mullender From FBatista at uniFON.com.ar Fri Apr 16 11:32:19 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Fri Apr 16 11:34:29 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Casey Duncan] #- If the context must be explicitly provided as an argument #- (rather than #- just a flag) I think a) is better. It depends also on what other #- arguments the constructor has, and thus what other contracts it #- fulfills. Are there other optional arguments? Until now, it receives just the number. Community seems to agree to be optional to pass a context for construction-time. There isn't other argument by now. . Facundo From FBatista at uniFON.com.ar Fri Apr 16 11:33:38 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Fri Apr 16 11:36:43 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Kevin Jacobs] #- Don't forget that many financial applications use fixed scale and #- precision as #- their primary mechanism for specifying Decimal types. As such, it #- would be very nice to have a constructor that took a literal #- representation #- as well as scale and precision. While using context is What is "fixed scale"? Could you provide examples of the functionality that you're asking for? Thanks. . Facundo From skip at pobox.com Fri Apr 16 11:43:43 2004 From: skip at pobox.com (Skip Montanaro) Date: Fri Apr 16 11:43:57 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? In-Reply-To: <407FFB0C.6050200@acm.org> References: <16511.60222.516830.948141@montanaro.dyndns.org> <407FFB0C.6050200@acm.org> Message-ID: <16511.65327.705618.504369@montanaro.dyndns.org> Sjoerd> Are you using ssh or pserver to talk to the cvs server? pserver Sjoerd> When using pserver, the cvs server is run under a fixed user id, Sjoerd> so that's where the mail will appear to come from. Thanks, that must be it. I'll have to scrounge around for a way to make syncmail figure out the submitter. Skip From fdrake at acm.org Fri Apr 16 12:20:19 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri Apr 16 12:21:14 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? In-Reply-To: <16511.65327.705618.504369@montanaro.dyndns.org> References: <16511.60222.516830.948141@montanaro.dyndns.org> <407FFB0C.6050200@acm.org> <16511.65327.705618.504369@montanaro.dyndns.org> Message-ID: <200404161220.19351.fdrake@acm.org> On Friday 16 April 2004 11:43 am, Skip Montanaro wrote: > Thanks, that must be it. I'll have to scrounge around for a way to make > syncmail figure out the submitter. Hint: the pserver puts the username used to authenticate in the CVS_USER environment variable when running scripts from the admin files (like loginfo). See: http://www.cvshome.org/docs/manual/cvs-1.11.14/cvs_18.html#SEC179 -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jacobs at theopalgroup.com Fri Apr 16 12:56:43 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Fri Apr 16 12:56:47 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <4080104B.6000702@theopalgroup.com> Batista, Facundo wrote: >[Kevin Jacobs] >#- Don't forget that many financial applications use fixed scale and >#- precision as >#- their primary mechanism for specifying Decimal types. As such, it >#- would be very nice to have a constructor that took a literal >#- representation >#- as well as scale and precision. While using context is > >What is "fixed scale"? > >Could you provide examples of the functionality that you're asking for? >Thanks. > > See for example, the SQL Decimal and Numeric data types, which is one of many applications that use scale and precision parameters for decimal representations: [borrowed from the PostgreSQL manual @ http://www.postgresql.org/docs/7.4/interactive/datatype.html#DATATYPE-NUMERIC-DECIMAL] > The /scale/ of a numeric is the count of decimal digits in the > fractional part, to the right > of the decimal point. The /precision/ of a numeric is the total count > of significant digits > in the whole number, that is, the number of digits to both sides of > the decimal point. So > the number 23.5141 has a precision of 6 and a scale of 4. Integers can > be considered to > have a scale of zero. > > Both the precision and the scale of the numeric type can be > configured. To declare a column > of type numeric use the syntax > > NUMERIC(precision, scale) > > The precision must be positive, the scale zero or positive. > Alternatively, > > NUMERIC(precision) > > selects a scale of 0. Specifying > > NUMERIC > > without any precision or scale creates a column in which numeric > values of any precision > and scale can be stored, up to the implementation limit on precision. > A column of this kind > will not coerce input values to any particular scale, whereas numeric > columns with a > declared scale will coerce input values to that scale. (The SQL > standard requires a default > scale of 0, i.e., coercion to integer precision. We find this a bit > useless. If you're concerned > about portability, always specify the precision and scale explicitly.) > > If the precision or scale of a value is greater than the declared > precision or scale of a > column, the system will attempt to round the value. If the value > cannot be rounded > so as to satisfy the declared limits, an error is raised. > Python decimals would do well to support the creation of instances with fixed scale and precision parameters, since this information will be what is provided by databases, other financial and engineering applications, and schema. i.e., these parameters override the natural scale and precision found in the literal values used when constructing decimals. e.g., hypothetically: Decimal('2.4000', precision=2, scale=1) == Decimal('2.4') Decimal('2.4', precision=5, scale=4) == Decimal('2.4000') Remember, these literals are frequently coming from an external source that must be constrained to a given schema. This issue doesn't come up with floating point numbers nearly as often, because those schema use explicitly precision-qualified types (i.e., single, double, quad precision). -Kevin From jim.jewett at eds.com Fri Apr 16 13:05:20 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Fri Apr 16 13:06:06 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: Paul Moore: > Decimal(1.1, significant_digits=34) is unambiguous, > but confusing. What would the preferred solution be? Requiring the user to explicitly type zeros so as to have 34 digits? Implicitly doing a Decimal(str(1.1)) and then adding 32 zeros to the internal representation? (Note: in at least my environment, str(1.1) = '1.1'; it is only repr that adds the extra digits.) Living with the fact that it will have some confusing non-zero digits? Facundo Batista: > ... what is "exact conversion" to you: > 1.1 -> "1.1" > 1.1 -> "1.1000000000000001" > 1.1 -> "1.100000000000000088817841970012523233890533447265625" machine precision of floats, whatever that happens to be. That implementation-dependent part is one reason to use a longer name, like exact_float. But if I say Decimal(1.1, precision=4) Decimal(1.1, positions=3) Those are unambiguous. If the context is 43, then later calculations will effectively fill in the extra digits; I have just said to fill them in with zeros, rather than whatever the float approximations were. Ka-Ping Yee: >> I never mentioned positions, and i don't see any reason >> why it is necessary to isolate these two ways of specifying >> precision. Facundo Batista: > Because they are two totally different concepts. And they're > not two ways of specifying precision. Sure they are; the reason to specify positions is that the underlying data wasn't really floating point -- it was an integer which by convention is written in a larger unit. Example with money: Assume meat is 0.987USD/pound. There are three price digits to multiply by the weight, but the final cost is rounded to an integer number of pennies. 10 pounds cost 9.87, but 1 pound costs 0.99 I don't want to say that my prices have three digits of precision, or I'll keep fractions of a penny. I don't want to say that they have only two, or I'll drop the pennies on expensive items. I want to say that the precision varies depending on the actual price. > What whould mean here "significant_digits=4"? It means precision. In many US classrooms, "precision" applies to the original measurement. Once you start doing calculations, it gets called "significant digits", as a reminder not to let your answer be more precise than your inputs. Ka-Ping Yee: >> Converting from float is not confusing when the >> precision is specified (either in terms of significant >> digits or decimal places). > Don't confuse the terminology. Precision means only one > thing, and it's specified in Cowlishaw work. But for any *specific* value, specifying either the total number of valid digits (signficant figures, precision) or the number of fractional digits (position) is enough to determine both. > "number" can be string, int, etc., but NOT float. > The issue with rounding at a defined position has > nothing to do with context. I assume that a new Decimal would normally be created with as much precision as the context would need for calculations. By passing a context/precision/position, the user is saying "yeah, but this measurement wasn't that precise in the first place. Use zeros for the rest, no matter what this number claims." -jJ From skip at pobox.com Fri Apr 16 14:09:57 2004 From: skip at pobox.com (Skip Montanaro) Date: Fri Apr 16 14:10:07 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? In-Reply-To: References: <16511.65327.705618.504369@montanaro.dyndns.org> Message-ID: <16512.8565.21289.392919@montanaro.dyndns.org> >> Thanks, that must be it. I'll have to scrounge around for a way to >> make syncmail figure out the submitter. Just> If the submitters all have "regular" accounts on the machine Just> running pserver, you can easily have each submitting process run Just> as the proper user, by adding the user to the CVSROOT/passwd file, Just> like so: Just> Just:xxxxxxxx:just Just> this is: Just> :: Hot damn! That might be just the ticket. I'll try it with my own account first. Thanks, Skip From mikepy-0404 at mike.de Fri Apr 16 14:26:05 2004 From: mikepy-0404 at mike.de (Mike Pall) Date: Fri Apr 16 14:26:10 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> References: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> Message-ID: <20040416182605.GA3590@mike.de> Hi, pje wrote: > I think the idea is more that you could skip the 'Py_INCREF(Py_None)', > which is a fairly common prelude to 'return Py_None'. You'd set their > refcounts to the maximum possible value, and the deallocation function > would simply reset the refcount to maximum again. > > I'm not sure, however, that this would be common enough to be helpful. It > seems to me Py_INCREF should effectively translate to only a single machine > instruction or two. I mean, it's just incrementing an integer whose > address is known at compile time. I don't think it would matter much performance-wise. A quick scan does not reveal a single Py_INCREF(Py_None) in any hot code section. But with more than 800 occurences it would reduce the code size a bit (5-10K). So that alone is not a good reason to throw it out. Checking for Py_None in Py_DECREF is counter-productive because that would need another branch. The existing branch is always taken with Py_None since the refcount never goes zero. I.e. this is not an issue. But let's look at the general case. Here's a common fragment of inlined x86 code for Py_DECREF: mov -something(%ebp),%esi ; odep %esi mov (%esi),%eax ; idep %esi, odep %eax dec %eax ; idep %eax, odep %eax test %eax,%eax ; [idep %eax], odep flags mov %eax,(%esi) ; [idep %eax %esi] jne .L1 ; idep flags, predict sub $0xc,%esp mov 0x4(%esi),%eax push %esi call *0x18(%eax) add $0x10,%esp .L1: There are at least 2 to 3 data dependencies that are hard to untangle. Tough luck. The other problem is branch prediction. I've done a little analysis with gcov on ceval.c and found that the situation is not that bad. The Py_DECREF occurences that account for 90% of the time are almost exclusively 0/100 or 100/0 cases. The remaining 10% have a few 30/70 or 50/50 cases. The folded and weighted average is around 4/96. The branch misprediction penalty accounts for less than 0.1% of the runtime. But the deallocation itself is costly. On average the deallocator is called 15% of the time (I checked ceval.c only). This is a lot and accounts for up to 5% of the runtime (this is based on an overall profile). My guess is that allocation is a bit cheaper, but adds another 2-3%. I think tuples are the worst offender (again only a good guess from my profiling experiments). To summarize: I think we can optimize the GC a little more (have not done an in-depth analysis yet), but we really should focus on making Python less malloc-happy. Suggestions welcome! Bye, Mike From jepler at unpythonic.net Fri Apr 16 15:06:22 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Fri Apr 16 15:06:28 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> References: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> Message-ID: <20040416190622.GC3078@unpythonic.net> Maybe something like this: #ifdef __GCC__ #define constant_p(x) __builtin_constant_p(x) #else #define constant_p(x) 0 #endif #define Py_INCREF(x) \ if( ! constant_p(x) || (x != Py_None && x != other immortals)) \ actual Py_INCREF body(x) this should allow the compiler, by constant propagation, to skip actual Py_INCREF body when it sees Py_INCREF(Py_None) and when it's Py_INCREF(nonconstant) to have no test for equality with the immortal values. does this actually work? I didn't check. I think that on some important systems there may be no equivalent to gcc's __builtin_constant_p(), and that Py_None is not a compile-time constant (*cough* DLLs *cough*) Jeff From skip at pobox.com Fri Apr 16 17:30:48 2004 From: skip at pobox.com (Skip Montanaro) Date: Fri Apr 16 17:30:56 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? In-Reply-To: <16512.8565.21289.392919@montanaro.dyndns.org> References: <16511.65327.705618.504369@montanaro.dyndns.org> <16512.8565.21289.392919@montanaro.dyndns.org> Message-ID: <16512.20616.41799.327583@montanaro.dyndns.org> Just> :: Skip> Hot damn! That might be just the ticket. I'll try it with my own Skip> account first. Alas, I was told that wouldn't work in our environment. Permission problems or something. (IOW, more administrative struggle than I was prepared to deal with. ;-) Fred> Hint: the pserver puts the username used to authenticate in the Fred> CVS_USER environment variable when running scripts from the admin Fred> files (like loginfo). This worked perfectly. Thanks for the help... Skip From rowen at cesmail.net Fri Apr 16 19:52:43 2004 From: rowen at cesmail.net (Russell E. Owen) Date: Fri Apr 16 19:52:49 2004 Subject: [Python-Dev] PEP 318, a different approach Message-ID: I had a strange idea for decorator syntax. def.classmethod func(cls,...): Multiple decorators are going to need the usual [] thing (or the current clumsy notation), but the strange thought is allowing def.iter so one could be more explicit about defining iterator functions w/out adding a keyword. Sorry to beat that dead horse. Back to the real notation. -- Russell P.S. one could imagine allowing these, as well,, but I can't imagine users typing them: def.func def.instancemethod From guido at python.org Fri Apr 16 20:31:07 2004 From: guido at python.org (Guido van Rossum) Date: Fri Apr 16 20:31:15 2004 Subject: [Python-Dev] PEP 318, a different approach In-Reply-To: Your message of "Fri, 16 Apr 2004 16:52:43 PDT." References: Message-ID: <200404170031.i3H0V7h29228@guido.python.org> > I had a strange idea for decorator syntax. > > def.classmethod func(cls,...): This would violate the 'greppability' requirement. It's too nice to have the name always be the token immediately following 'def' to allow anything in between the two. Basically, the only two positions that are still in the race are before the def, and between the argument list and the colon. --Guido van Rossum (home page: http://www.python.org/~guido/) From andymac at bullseye.apana.org.au Fri Apr 16 19:36:52 2004 From: andymac at bullseye.apana.org.au (Andrew MacIntyre) Date: Fri Apr 16 21:14:00 2004 Subject: [Python-Dev] String hash function multiplier In-Reply-To: <20040414193359.GW6139@unpythonic.net> References: <4FCB1278-8DB3-11D8-B460-000A95686CD8@redivi.com> <001d01c421c6$77b4df00$e841fea9@oemcomputer> <20040414030938.GD8795@unpythonic.net> <20040414130834.GS6139@unpythonic.net> <20040414232756.Q67555@bullseye.apana.org.au> <20040414193359.GW6139@unpythonic.net> Message-ID: <20040417093341.V89016@bullseye.apana.org.au> On Wed, 14 Apr 2004, Jeff Epler wrote: > imul is on all x86 architectures, but whether to use it or not depends > on the characteristics of the target CPU. With -mcpu=i386, imul is > considered quite slow and a shift sequence is (almost?) always preferred > when one operand is constant. With -mcpu=i686 and newer, imul seems to > be preferred. My misunderstanding. I hadn't really comprehended the full extent of "instruction scheduling" in this context. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au (pref) | Snail: PO Box 370 andymac@pcug.org.au (alt) | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From john at neggie.net Sat Apr 17 01:29:43 2004 From: john at neggie.net (John Belmonte) Date: Sat Apr 17 01:29:53 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <407C28E0.7090808@neggie.net> References: <407C28E0.7090808@neggie.net> Message-ID: <4080C0C7.7070602@neggie.net> I've implemented the proposal, as far as the pydoc modifications. See . > 3) extend the __slots__ handler to set the descriptor's doc string > to dictionary values if the slot's value is a dictionary. I haven't implemented this, because I don't know a thing about Python's C interface. If someone more able would knock off a prototype, I would be grateful. -John -- http:// if ile.org/ From aahz at pythoncraft.com Sat Apr 17 09:58:34 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 17 09:58:40 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: References: Message-ID: <20040417135834.GA8542@panix.com> On Thu, Apr 15, 2004, Jewett, Jim J wrote: > > What if a few common (constant, singleton) objects (such as None, -1, > 0, 1) were declared immortal at compile-time? They would be created > at initial load in a special untracked pool, and their tp_dealloc > would do nothing. > > The slot for tracking references would still be there, but could be > ignored -- even if it went negative. > > Since the reference count no longer has to be correct (for these > objects), the reference counting macros could optimize to nothing when > they know at compile time that they'll have one of these constant > objects. http://mail.python.org/pipermail/python-list/2003-September/thread.html#183710 -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From aahz at pythoncraft.com Sat Apr 17 10:01:13 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 17 10:01:16 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: <20040416182605.GA3590@mike.de> References: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> <20040416182605.GA3590@mike.de> Message-ID: <20040417140113.GB8542@panix.com> On Fri, Apr 16, 2004, Mike Pall wrote: > > To summarize: I think we can optimize the GC a little more (have not done > an in-depth analysis yet), but we really should focus on making Python > less malloc-happy. Suggestions welcome! Could you expand on this? What do you mean by "less malloc-happy"? What you were talking about earlier was ``None``, which doesn't go through malloc hoops. I think you might find it useful to at least read old threads about refcounting vs. GC. (I managed to dig up the old thread about getting rid of refcounting for ``None``; don't have time for more.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From aahz at pythoncraft.com Sat Apr 17 10:03:46 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 17 10:03:49 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <407FEE8F.2090402@theopalgroup.com> References: <407FEE8F.2090402@theopalgroup.com> Message-ID: <20040417140346.GC8542@panix.com> On Fri, Apr 16, 2004, Kevin Jacobs wrote: > > Don't forget that many financial applications use fixed scale and > precision as their primary mechanism for specifying Decimal types. As > such, it would be very nice to have a constructor that took a literal > representation as well as scale and precision. While using context is > sufficient, it will be unintuitive and may present an undue burden on > many users. Could you use an example to explain what you mean? This doesn't make sense to me given that there are already two mechanisms for precisely specifying the input to Decimal(). -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From aahz at pythoncraft.com Sat Apr 17 10:07:13 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 17 10:07:22 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <4080104B.6000702@theopalgroup.com> References: <4080104B.6000702@theopalgroup.com> Message-ID: <20040417140712.GD8542@panix.com> [That'll teach me to respond before reading a thread. ;-) ] On Fri, Apr 16, 2004, Kevin Jacobs wrote: > > Python decimals would do well to support the creation of instances > with fixed scale and precision parameters, since this information will > be what is provided by databases, other financial and engineering > applications, and schema. i.e., these parameters override the natural > scale and precision found in the literal values used when constructing > decimals. e.g., hypothetically: > > Decimal('2.4000', precision=2, scale=1) == Decimal('2.4') > Decimal('2.4', precision=5, scale=4) == Decimal('2.4000') I'd say it makes more sense for someone else to implement these features as helper functions or classes. They're not directly part of the Decimal system. Facundo got roped into this because he wanted a Money class and was persuaded to do the finishing work for getting Decimal into Python as the basis. Seems to me this is in the same category of building some additional capabilities on top of Decimal. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From aahz at pythoncraft.com Sat Apr 17 10:09:51 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 17 10:09:54 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <20040417140951.GE8542@panix.com> On Fri, Apr 16, 2004, Batista, Facundo wrote: > > We all must decide which of the following is better: > > a) Decimal(number) # no context used > Decimal(number, context) > > b) Decimal(number) # no context used > Decimal.using_context(number, context) Based on my reading of Tim's posts, there's also c) Decimal(number, [context]) # context always used, with # optional different from default -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From guido at python.org Sat Apr 17 13:33:04 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 17 13:33:11 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: Your message of "Sat, 17 Apr 2004 01:29:43 EDT." <4080C0C7.7070602@neggie.net> References: <407C28E0.7090808@neggie.net> <4080C0C7.7070602@neggie.net> Message-ID: <200404171733.i3HHX4B30513@guido.python.org> > > 3) extend the __slots__ handler to set the descriptor's doc string > > to dictionary values if the slot's value is a dictionary. > > I haven't implemented this, because I don't know a thing about Python's > C interface. If someone more able would knock off a prototype, I would > be grateful. If I understand this proposal correctly (the dict values would provide the docstrings for the slots) I'd rather not make this default behavior; different metaclasses may want to use the dict value for different purposes, and subsuming it by default for the doc string would make that harder to do. --Guido van Rossum (home page: http://www.python.org/~guido/) From fumanchu at amor.org Sat Apr 17 15:17:36 2004 From: fumanchu at amor.org (Robert Brewer) Date: Sat Apr 17 15:19:58 2004 Subject: [Python-Dev] dict.sortedkeys() Message-ID: I'm willing to pay money for someone to write {}.sortedkeys() and/or sets.Set().sorteditems() (since IANACH (I am not a C hacker)). I'm just *really* tired of writing what should be a one-liner in three lines: From fumanchu at amor.org Sat Apr 17 15:21:18 2004 From: fumanchu at amor.org (Robert Brewer) Date: Sat Apr 17 15:23:39 2004 Subject: [Python-Dev] dict.sortedkeys() Message-ID: I wrote: > I'm willing to pay money for someone to write {}.sortedkeys() and/or > sets.Set().sorteditems() (since IANACH (I am not a C > hacker)). I'm just > *really* tired of writing what should be a one-liner in three lines: ... e.g. dkeys = d.keys() dkeys.sort() for key in dkeys: instead of: for key in d.sortedkeys(): Robert Brewer MIS Amor Ministries fumanchu@amor.org From jepler at unpythonic.net Sat Apr 17 16:08:17 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Sat Apr 17 16:08:31 2004 Subject: [Python-Dev] dict.sortedkeys() In-Reply-To: References: Message-ID: <20040417200817.GB18480@unpythonic.net> Factor sorted into a function that returns a sorted copy: def sorted(seq): seq = list(seq) seq.sort() return seq or one that sorts in place: def sorted(seq): seq.sort() return seq Now you can write your one-liner: for i in sorted({}.keys()): ... Jeff From bob at redivi.com Sat Apr 17 16:23:35 2004 From: bob at redivi.com (Bob Ippolito) Date: Sat Apr 17 16:19:21 2004 Subject: [Python-Dev] dict.sortedkeys() In-Reply-To: References: Message-ID: <1A2EBB04-90AD-11D8-8EF5-000A95686CD8@redivi.com> On Apr 17, 2004, at 3:17 PM, Robert Brewer wrote: > I'm willing to pay money for someone to write {}.sortedkeys() and/or > sets.Set().sorteditems() (since IANACH (I am not a C hacker)). I'm just > *really* tired of writing what should be a one-liner in three lines: I'm pretty sure that in Python 2.4 you will be able to say sorted(seq). -bob From fumanchu at amor.org Sat Apr 17 16:21:26 2004 From: fumanchu at amor.org (Robert Brewer) Date: Sat Apr 17 16:23:48 2004 Subject: [Python-Dev] dict.sortedkeys() Message-ID: [Jeff Epler] > Factor sorted into a function that returns a sorted copy: > def sorted(seq): > seq = list(seq) > seq.sort() > return seq > > or one that sorts in place: > def sorted(seq): > seq.sort() > return seq > > Now you can write your one-liner: > for i in sorted({}.keys()): > ... Yes, of course. But I'm working on a large business app framework. That means either: 1. Writing sorted into every module where it's needed. 2. Putting sorted into a package somewhere, which makes it longer and uglier with absolute referencing. 3. Customizing consumers' Python install. ...none of which are good options. [Bob Ippolito] > I'm pretty sure that in Python 2.4 you will be able to say sorted(seq). Excellent! That'll do nicely. :) -another bob From fdrake at acm.org Sat Apr 17 16:46:26 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sat Apr 17 16:46:36 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? In-Reply-To: <16512.20616.41799.327583@montanaro.dyndns.org> References: <16511.65327.705618.504369@montanaro.dyndns.org> <16512.8565.21289.392919@montanaro.dyndns.org> <16512.20616.41799.327583@montanaro.dyndns.org> Message-ID: <200404171646.26885.fdrake@acm.org> On Friday 16 April 2004 05:30 pm, Skip Montanaro wrote: > Alas, I was told that wouldn't work in our environment. Permission > problems or something. (IOW, more administrative struggle than I was > prepared to deal with. ;-) There's the little matter that CVS users don't necessarily map one-to-one with user accounts on the host with pserver to deal with as well. > Fred> Hint: the pserver puts the username used to authenticate in the > Fred> CVS_USER environment variable when running scripts from the > admin Fred> files (like loginfo). > > This worked perfectly. This is something syncmail should probably support out of the box. If you'd like to submit a patch, we'd love to have it: http://sourceforge.net/projects/cvs-syncmail/ -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one at comcast.net Sat Apr 17 17:37:23 2004 From: tim.one at comcast.net (Tim Peters) Date: Sat Apr 17 17:37:30 2004 Subject: [Python-Dev] No-cost optimizing VC 7.1 Message-ID: Thanks to David Abrahams for passing on this info: a free optimizing cmdline VC 7.1 has been released by MS: http://msdn.microsoft.com/visualc/vctoolkit2003/ The 32MB download appears to contain much of the meat of Visual Studio .NET 2003 Professional, including the compiler, linker, static-link libraries for the C, C++ & STL runtimes, and the .NET Common Language Runtime. From mikepy-0404 at mike.de Sat Apr 17 17:50:00 2004 From: mikepy-0404 at mike.de (Mike Pall) Date: Sat Apr 17 17:50:28 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: <20040417140113.GB8542@panix.com> References: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> <20040416182605.GA3590@mike.de> <20040417140113.GB8542@panix.com> Message-ID: <20040417215000.GA5263@mike.de> Hi, Aahz wrote: > On Fri, Apr 16, 2004, Mike Pall wrote: > > To summarize: I think we can optimize the GC a little more (have not done > > an in-depth analysis yet), but we really should focus on making Python > > less malloc-happy. Suggestions welcome! > > Could you expand on this? What do you mean by "less malloc-happy"? > What you were talking about earlier was ``None``, which doesn't go > through malloc hoops. > > I think you might find it useful to at least read old threads about > refcounting vs. GC. (I managed to dig up the old thread about getting > rid of refcounting for ``None``; don't have time for more.) I'm very well aware of the tradeoffs between refcounting and pure GC schemes. I do *not* suggest that we change CPython's approach. And I do not intend to start another refcount-vs-GC flamewar. You've got me wrong. Also please check the thread -- I never suggested modifying anything about Py_None. I just presented a short quantitative analysis that it's useless to do so (the thread you referenced contains another good reason). What I meant with 'less malloc-happy' is that we should try to reduce the number of allocations/deallocations for internal (!) objects (I'm not talking about objects explicitly managed by the application). Many of them have a very short time-to-live. I guess some of them could be recycled or even optimized away. E.g. global profiling indicates that tuple allocation/deallocation has a noticeable effect on performance. There are probably other objects that have prohibitive setup/teardown cost (but are less pronounced on allocation costs). But since this is only #4 on my list I have not given it a closer look. That's why I said 'suggestions welcome'. Bye, Mike From martin at v.loewis.de Sat Apr 17 18:18:33 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat Apr 17 18:18:45 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: <20040417215000.GA5263@mike.de> References: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> <20040416182605.GA3590@mike.de> <20040417140113.GB8542@panix.com> <20040417215000.GA5263@mike.de> Message-ID: <4081AD39.8090404@v.loewis.de> Mike Pall wrote: > What I meant with 'less malloc-happy' is that we should try to reduce > the number of allocations/deallocations for internal (!) objects (I'm not > talking about objects explicitly managed by the application). Many of them > have a very short time-to-live. I guess some of them could be recycled or > even optimized away. > > E.g. global profiling indicates that tuple allocation/deallocation has a > noticeable effect on performance. There are probably other objects that > have prohibitive setup/teardown cost (but are less pronounced on allocation > costs). The problem is that this has already been done in the past. Anybody contributing to the Python core is certainly aware that avoiding memory allocations, where possible, should be done, and indeed, special cases have been added (e.g. my addition of METH_NONE and METH_O to avoid tuple creation). So unless a specific proposal is made, I'm doubtful that much optimization is possible without breaking core semantic aspects of the language. Regards, Martin From aahz at pythoncraft.com Sat Apr 17 20:53:56 2004 From: aahz at pythoncraft.com (Aahz) Date: Sat Apr 17 20:54:01 2004 Subject: [Python-Dev] Re: PEP 328 -- relative and multi-line import In-Reply-To: <200404140624.i3E6OeMg017756@cosc353.cosc.canterbury.ac.nz> References: <20040409154252.GB28808@panix.com> <200404140624.i3E6OeMg017756@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040418005356.GA27203@panix.com> On Wed, Apr 14, 2004, Greg Ewing wrote: > Aahz : >> >> Nope. Absolute imports are the most common form > > I don't agree with that. In every major Python application I've > written, the vast majority of imports are from other modules of the > application, which are most sensibly referenced relatively. So you're saying that your applications are all put together in a single package? Or that there isn't much in the way of inter-package imports? Could you show us the header of one of your modules as an example? > I, too, am quite concerned about the imminent breaking of every > existing relative import, apparently because Guido has an ideological > objection to relative imports and wants to punish us for having used > them so freely over the years. Actually, it's Barry's hobbyhorse. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From guido at python.org Sat Apr 17 21:04:43 2004 From: guido at python.org (Guido van Rossum) Date: Sat Apr 17 21:05:07 2004 Subject: [Python-Dev] dict.sortedkeys() In-Reply-To: Your message of "Sat, 17 Apr 2004 13:21:26 PDT." References: Message-ID: <200404180104.i3I14ha31011@guido.python.org> > [Bob Ippolito] > > I'm pretty sure that in Python 2.4 you will be able to say > sorted(seq). > > Excellent! That'll do nicely. :) Even better, you'll be able to say sorted(iterable), and a dict is an iterable (yielding its keys), so you can say sorted(dict). $ ./python Python 2.4a0 (#93, Mar 31 2004, 10:08:22) [GCC 2.96 20000731 (Red Hat Linux 7.3 2.96-113)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sorted(sys.modules) ['UserDict', '__builtin__', '__main__', '_codecs', '_sre', 'codecs', 'copy_reg', 'distutils', 'distutils.dep_util', 'distutils.distutils', 'distutils.errors', 'distutils.log', 'distutils.os', 'distutils.re', 'distutils.spawn', 'distutils.string', 'distutils.sys', 'distutils.util', 'encodings', 'encodings.aliases', 'encodings.codecs', 'encodings.exceptions', 'encodings.iso8859_15', 'encodings.types', 'exceptions', 'linecache', 'os', 'os.path', 'posix', 'posixpath', 're', 'readline', 'rlcompleter', 'signal', 'site', 'sre', 'sre_compile', 'sre_constants', 'sre_parse', 'stat', 'string', 'sys', 'types', 'warnings', 'zipimport'] >>> --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Sun Apr 18 05:23:15 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Apr 18 05:23:41 2004 Subject: [Python-Dev] Optimization targets - refcount In-Reply-To: <5.1.1.6.0.20040416083351.02166020@mail.telecommunity.com> Message-ID: <200404180923.i3I9NFiq025149@cosc353.cosc.canterbury.ac.nz> "Phillip J. Eby" : > I'm not sure, however, that this would be common enough to be > helpful. It seems to me Py_INCREF should effectively translate to > only a single machine instruction or two. On a RISC architecture, probably 3 instructions -- load, increment, store. It's the memory accesses that will be the most expensive part of this. But then, if increfing Py_None is really that common, its refcount will likely be in cache. I doubt whether it really is very common, though, compared to increfs of all other objects. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Sun Apr 18 05:40:41 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Apr 18 05:40:49 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: Message-ID: <200404180940.i3I9efOM025212@cosc353.cosc.canterbury.ac.nz> "Batista, Facundo" : > And positions is the number of digits after the decimal point at > which you want to round the number you're creating from and discard > the other digits. Most people call that "decimal places". Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From john at neggie.net Sun Apr 18 08:45:29 2004 From: john at neggie.net (John Belmonte) Date: Sun Apr 18 08:45:52 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <200404171733.i3HHX4B30513@guido.python.org> References: <407C28E0.7090808@neggie.net> <4080C0C7.7070602@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> Message-ID: <40827869.2030300@neggie.net> Guido van Rossum wrote: >>> 3) extend the __slots__ handler to set the descriptor's doc string >>>to dictionary values if the slot's value is a dictionary. >> >>I haven't implemented this, because I don't know a thing about Python's >>C interface. If someone more able would knock off a prototype, I would >>be grateful. > > > If I understand this proposal correctly (the dict values would provide > the docstrings for the slots) I'd rather not make this default > behavior; different metaclasses may want to use the dict value for > different purposes, and subsuming it by default for the doc string > would make that harder to do. Ok. Can anyone suggest an unobtrusive way to set docstrings on slots? Perhaps this (but how could it be implemented?): class Foo(object): __slots__ = docslots( ('slot1': 'description'), ('slot2': """description ...continued """), 'slot3', ) -John -- http:// if ile.org/ From fdrake at acm.org Sun Apr 18 10:13:19 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sun Apr 18 10:13:38 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <40827869.2030300@neggie.net> References: <407C28E0.7090808@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> <40827869.2030300@neggie.net> Message-ID: <200404181013.20045.fdrake@acm.org> On Sunday 18 April 2004 08:45 am, John Belmonte wrote: > Perhaps this (but how could it be implemented?): > > class Foo(object): > __slots__ = docslots( > ('slot1': 'description'), > ('slot2': How about: class Foo(object): docslots(slot1='description', slot2=("Some really long\n" "multi-line description."), ...) The implementation for docslots() would either provide or update __slots__, and defining properities for slots with munged names. This wouldn't actually be difficult to implement, though is would require a sys._getframe() hack. In fact, you could crib from the attached sample. ;-) -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation -------------- next part -------------- A non-text attachment was scrubbed... Name: docslots.py Type: text/x-python Size: 1169 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040418/6aaee156/docslots.py From skip at pobox.com Sun Apr 18 08:53:19 2004 From: skip at pobox.com (Skip Montanaro) Date: Sun Apr 18 11:34:40 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? In-Reply-To: <200404171646.26885.fdrake@acm.org> References: <16511.65327.705618.504369@montanaro.dyndns.org> <16512.8565.21289.392919@montanaro.dyndns.org> <16512.20616.41799.327583@montanaro.dyndns.org> <200404171646.26885.fdrake@acm.org> Message-ID: <16514.31295.926740.594739@montanaro.dyndns.org> Fred> Hint: the pserver puts the username used to authenticate in the Fred> CVS_USER environment variable when running scripts from the >> admin Fred> files (like loginfo). >> >> This worked perfectly. Fred> This is something syncmail should probably support out of the box. Fred> If you'd like to submit a patch, we'd love to have it: Fred> http://sourceforge.net/projects/cvs-syncmail/ Will do when I'm back at work Monday. I added a new command line flag to enable CVS_USER usage. Were you thinking that should be the default? If so, I could delete about 75% of my changes. Skip From john at neggie.net Sun Apr 18 11:35:50 2004 From: john at neggie.net (John Belmonte) Date: Sun Apr 18 11:36:17 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <200404181013.20045.fdrake@acm.org> References: <407C28E0.7090808@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> <40827869.2030300@neggie.net> <200404181013.20045.fdrake@acm.org> Message-ID: <4082A056.8030804@neggie.net> Fred L. Drake, Jr. wrote: > """Sample implementation of the docslots idea.""" > > import sys > > > class slotproperty(object): > def __init__(self, name, docstring): > self.__name__ = name > self.__doc__ = docstring > self._slotname = "_slotproperty_" + name > > def __get__(self, inst, cls=None): > return getattr(inst, self._slotname) > > def __set__(self, inst, value): > setattr(inst, self._slotname, value) > > > def docslots(**kw): > namespace = sys._getframe(1).f_locals > __slots__ = namespace.get("__slots__", ()) > __slots__ = list(__slots__) > for name, docstring in kw.iteritems(): > prop = slotproperty(name, docstring) > if name in __slots__: > __slots__.remove(name) > __slots__.append(prop._slotname) > namespace[name] = prop > namespace["__slots__"] = tuple(__slots__) Thanks, that at least gives me a framework to try out some ideas. I'm concerned about something though. Doesn't this implementation impose an overhead on access of slots with doc strings, or can a C implementation be made just as efficient as normal slots? I'm also wondering about Guido's comment. Even if the __slots__ handler were extended to handle docstrings directly via dict values, wouldn't metaclasses still be free to intercept the dict for other uses? -John -- http:// if ile.org/ From fdrake at acm.org Sun Apr 18 11:55:30 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sun Apr 18 11:55:38 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <4082A056.8030804@neggie.net> References: <407C28E0.7090808@neggie.net> <200404181013.20045.fdrake@acm.org> <4082A056.8030804@neggie.net> Message-ID: <200404181155.30957.fdrake@acm.org> On Sunday 18 April 2004 11:35 am, John Belmonte wrote: > I'm concerned about something though. Doesn't this implementation > impose an overhead on access of slots with doc strings, or can a C > implementation be made just as efficient as normal slots? This implementation does, but a C implementation in the core could avoid that easily enough. It may even be that this could be avoided using a Python implementation by generating an __slots__ that isn't munged so heavily and then playing metaclass tricks to cause the docstrings to be inserted directly on the descriptors generated by the current __slots__ machinery. I wasn't too worried about that for the example code, but you're certainly free to construct something that suits you. I think it would be more interesting to create slot-based properties without the renaming currently needed, but I'm not sure how to do that off-hand. If the slot property type were available for subclassing it would be pretty easy, I'd hope. > I'm also wondering about Guido's comment. Even if the __slots__ handler > were extended to handle docstrings directly via dict values, wouldn't > metaclasses still be free to intercept the dict for other uses? Metaclasses can do what they want to affect the new type. That's what they're for. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From pf_moore at yahoo.co.uk Sun Apr 18 12:13:56 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Sun Apr 18 12:14:31 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 References: Message-ID: "Tim Peters" writes: > Thanks to David Abrahams for passing on this info: a free optimizing > cmdline VC 7.1 has been released by MS: > > http://msdn.microsoft.com/visualc/vctoolkit2003/ > > The 32MB download appears to contain much of the meat of Visual Studio .NET > 2003 Professional, including the compiler, linker, static-link libraries for > the C, C++ & STL runtimes, and the .NET Common Language Runtime. Sadly, a test seems to show that the /MD option links with msvcrt.dll, and not with msvcr71.dll. So this still isn't of immediate use in building Python extensions which will run against a standard Python binary distribution. This would tie in with the information given on the MS website: "C Runtime Library and the C++ Standard Library, including the Standard Template Library. These are the same static-link libraries included with Visual Studio." Note this bit........................^^^^^^^^^^^ I just checked my hard disk (I have MSVC6SP6, a recent platform SDK, .NET framework and SDK, and the VC Toolkit 2003 installed) and I can't find any occurrences of msvcr71.lib, implying that none of these tools offers the possibility of creating code which links against msvcr71.dll... Another oddity - apparently a cast from float to long generates a call to an internal function _ftol2, where it called _ftol in earlier versions. So code that links with /MD fails to link if it contains such a cast (as the compiler generates code to call support functions which are not in the CRT DLL that is being used...) Don't get me wrong - it's great that MS have done this, it's just not the end of the story. Paul. -- This signature intentionally left blank From guido at python.org Sun Apr 18 13:21:29 2004 From: guido at python.org (Guido van Rossum) Date: Sun Apr 18 13:21:36 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: Your message of "Sun, 18 Apr 2004 08:45:29 EDT." <40827869.2030300@neggie.net> References: <407C28E0.7090808@neggie.net> <4080C0C7.7070602@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> <40827869.2030300@neggie.net> Message-ID: <200404181721.i3IHLTp07934@guido.python.org> > Ok. Can anyone suggest an unobtrusive way to set docstrings on slots? > > Perhaps this (but how could it be implemented?): > > class Foo(object): > __slots__ = docslots( > ('slot1': 'description'), > ('slot2': > """description > ...continued > """), > 'slot3', > ) Anything can be done using metaclasses. __slots__ is not special once the class exists -- it is a set of instructions for the default metaclass to create a specific set of descriptors (and associated storage). Another metaclass could use a different convention (although it may have to set __slots__ to let the base metaclass create the associated storage slots). --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Sun Apr 18 20:09:55 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun Apr 18 20:10:44 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: Message-ID: <200404190009.i3J09tfN026216@cosc353.cosc.canterbury.ac.nz> Paul Moore : > Sadly, a test seems to show that the /MD option links with msvcrt.dll, > and not with msvcr71.dll. So this still isn't of immediate use in > building Python extensions which will run against a standard Python > binary distribution. Can I just ask at this point whether it's possible to use MinGW to compile extensions for the Python version under discussion here? If not, I think you're making a big mistake basing the standard distribution on this compiler. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From python-dev-list at cstork.org Sun Apr 18 21:53:48 2004 From: python-dev-list at cstork.org (Christian Stork) Date: Sun Apr 18 21:53:52 2004 Subject: [Python-Dev] iter.index() Message-ID: <20040419015348.GB6807@ics.uci.edu> Hi, I wanted to do something like i = itertools.chain(list1, list2).index(elem) but that gave me ... AttributeError: 'itertools.chain' object has no attribute 'index' If I use the operator module it works just fine. i = operator.indexOf(itertools.chain(list1, list2), elem) Why not add the index method to iterator objects? Of course, internally only next() is used by the default implementation. -- Chris Stork <> Support eff.org! <> http://www.ics.uci.edu/~cstork/ OpenPGP fingerprint: B08B 602C C806 C492 D069 021E 41F3 8C8D 50F9 CA2F From bob at redivi.com Sun Apr 18 22:08:25 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun Apr 18 22:04:12 2004 Subject: [Python-Dev] iter.index() In-Reply-To: <20040419015348.GB6807@ics.uci.edu> References: <20040419015348.GB6807@ics.uci.edu> Message-ID: <70D3F6CF-91A6-11D8-A48C-000A95686CD8@redivi.com> On Apr 18, 2004, at 9:53 PM, Christian Stork wrote: > Hi, > > I wanted to do something like > > i = itertools.chain(list1, list2).index(elem) > > but that gave me > > ... > AttributeError: 'itertools.chain' object has no attribute 'index' > > If I use the operator module it works just fine. > > i = operator.indexOf(itertools.chain(list1, list2), elem) > > Why not add the index method to iterator objects? Of course, > internally > only next() is used by the default implementation. An iterator mutates each time you call its next(). Your call to indexOf does one of three things: exhaust some of your iterator and return a useless integer, exhaust all of your iterator and raise an exception, or never return. If you want an object that acts like a list, you should use a list. -bob From python-dev-list at cstork.org Sun Apr 18 22:21:07 2004 From: python-dev-list at cstork.org (Christian Stork) Date: Sun Apr 18 22:21:12 2004 Subject: [Python-Dev] iter.index() In-Reply-To: <70D3F6CF-91A6-11D8-A48C-000A95686CD8@redivi.com> References: <20040419015348.GB6807@ics.uci.edu> <70D3F6CF-91A6-11D8-A48C-000A95686CD8@redivi.com> Message-ID: <20040419022107.GC6807@ics.uci.edu> On Sun, Apr 18, 2004 at 10:08:25PM -0400, Bob Ippolito wrote: > > On Apr 18, 2004, at 9:53 PM, Christian Stork wrote: > > >Hi, > > > >I wanted to do something like > > > > i = itertools.chain(list1, list2).index(elem) > > > >but that gave me > > > > ... > > AttributeError: 'itertools.chain' object has no attribute 'index' > > > >If I use the operator module it works just fine. > > > > i = operator.indexOf(itertools.chain(list1, list2), elem) > > > >Why not add the index method to iterator objects? Of course, > >internally > >only next() is used by the default implementation. > > An iterator mutates each time you call its next(). Your call to > indexOf does one of three things: exhaust some of your iterator and > return a useless integer, Why useless? It returns the index. Why do I have to build a new concatinated list (or my own special index function) in order to get that information? > exhaust all of your iterator and raise an exception, Yep, just as it would for lists. > or never return. Well, that's the risk we take with iterators, but in my case I'm certain of two sufficient termination criteria: the iterator is finite and I know that elem is in it. > If you want an object that acts like a list, you should use a list. I don't see anything inherently "listy" about index(). It just counts how many elements there are to reach elem. And obviously the functionality is already there in the operator module. I'm just proposing a little convenience. -- Chris Stork <> Support eff.org! <> http://www.ics.uci.edu/~cstork/ OpenPGP fingerprint: B08B 602C C806 C492 D069 021E 41F3 8C8D 50F9 CA2F From john at neggie.net Sun Apr 18 22:29:08 2004 From: john at neggie.net (John Belmonte) Date: Sun Apr 18 22:29:21 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <200404181721.i3IHLTp07934@guido.python.org> References: <407C28E0.7090808@neggie.net> <4080C0C7.7070602@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> <40827869.2030300@neggie.net> <200404181721.i3IHLTp07934@guido.python.org> Message-ID: <40833974.10800@neggie.net> Guido van Rossum wrote: > Anything can be done using metaclasses. __slots__ is not special > once the class exists -- it is a set of instructions for the default > metaclass to create a specific set of descriptors (and associated > storage). Another metaclass could use a different convention > (although it may have to set __slots__ to let the base metaclass > create the associated storage slots). My original proposal was to use __slots__ dict values for docstrings in the default metaclass. You said you'd rather not do that because different metaclasses may want to use the dict value for different purposes. But from what you've explained, metaclasses are free to interpret the value of __slots__ in any way they choose. Metaclasses built on top of the default metaclass could translate their __slots__ value to the __slots__ I proposed. Are optional tuples any better? This wouldn't preclude use of dict values for something else. class Foo(object): __slots__ = [ 'slot1', ('slot2', 'description'), ('slot3', """description ...continued"""), ] -John -- http:// if ile.org/ From python at rcn.com Sun Apr 18 22:37:28 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun Apr 18 22:38:35 2004 Subject: [Python-Dev] iter.index() In-Reply-To: <20040419015348.GB6807@ics.uci.edu> Message-ID: <000b01c425b7$42592d80$e841fea9@oemcomputer> [Christian Stork] > I wanted to do something like > > i = itertools.chain(list1, list2).index(elem) I'm curious about what application needed to do this. > If I use the operator module it works just fine. > > i = operator.indexOf(itertools.chain(list1, list2), elem) Nice solution. It is general purpose, self-documenting, and efficient. Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From pje at telecommunity.com Sun Apr 18 22:44:11 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun Apr 18 22:43:53 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: <200404190009.i3J09tfN026216@cosc353.cosc.canterbury.ac.nz> References: Message-ID: <5.1.1.6.0.20040418224118.0219d470@mail.telecommunity.com> At 12:09 PM 4/19/04 +1200, Greg Ewing wrote: >Paul Moore : > > > Sadly, a test seems to show that the /MD option links with msvcrt.dll, > > and not with msvcr71.dll. So this still isn't of immediate use in > > building Python extensions which will run against a standard Python > > binary distribution. > >Can I just ask at this point whether it's possible to >use MinGW to compile extensions for the Python version >under discussion here? Yes. See: http://mail.python.org/pipermail/python-dev/2004-January/041650.html From bob at redivi.com Sun Apr 18 22:50:13 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun Apr 18 22:45:54 2004 Subject: [Python-Dev] iter.index() In-Reply-To: <20040419022107.GC6807@ics.uci.edu> References: <20040419015348.GB6807@ics.uci.edu> <70D3F6CF-91A6-11D8-A48C-000A95686CD8@redivi.com> <20040419022107.GC6807@ics.uci.edu> Message-ID: <482D62ED-91AC-11D8-A48C-000A95686CD8@redivi.com> On Apr 18, 2004, at 10:21 PM, Christian Stork wrote: > On Sun, Apr 18, 2004 at 10:08:25PM -0400, Bob Ippolito wrote: >> >> On Apr 18, 2004, at 9:53 PM, Christian Stork wrote: >> >>> Hi, >>> >>> I wanted to do something like >>> >>> i = itertools.chain(list1, list2).index(elem) >>> >>> but that gave me >>> >>> ... >>> AttributeError: 'itertools.chain' object has no attribute 'index' >>> >>> If I use the operator module it works just fine. >>> >>> i = operator.indexOf(itertools.chain(list1, list2), elem) >>> >>> Why not add the index method to iterator objects? Of course, >>> internally >>> only next() is used by the default implementation. >> >> An iterator mutates each time you call its next(). Your call to >> indexOf does one of three things: exhaust some of your iterator and >> return a useless integer, > > Why useless? It returns the index. Why do I have to build a new > concatinated list (or my own special index function) in order to get > that information? It returns an index into some sequence that no longer exists. What good is that? >> exhaust all of your iterator and raise an exception, > > Yep, just as it would for lists. The difference is that the iterator's life is more or less over, there's probably nothing useful you can do with it, but a list is not changed by this operation. >> If you want an object that acts like a list, you should use a list. > > I don't see anything inherently "listy" about index(). It just counts > how many elements there are to reach elem. And obviously the > functionality is already there in the operator module. I'm just > proposing a little convenience. I think it's pretty rare that you would want to know this information at the cost of exhausting some/all of your iterator... and if that really is what you want, then you should just use operator.indexOf. There are MANY iterable types, it's not reasonable to change them all. -bob From tim.one at comcast.net Sun Apr 18 22:51:12 2004 From: tim.one at comcast.net (Tim Peters) Date: Sun Apr 18 22:51:21 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: <200404190009.i3J09tfN026216@cosc353.cosc.canterbury.ac.nz> Message-ID: [Greg Ewing] > Can I just ask at this point whether it's possible to > use MinGW to compile extensions for the Python version > under discussion here? Don't know -- someone who uses MinGW would need to answer that. > If not, I think you're making a big mistake basing the > standard distribution on this compiler. Why is that? If it doesn't play nice with the latest MSVC, MinGW doesn't have a future on Windows. The precompiled Python Windows distro has always been built with the flavor of MSVC current at the time. Nothing has changed in that respect, except the specific MSVC flavor in use -- and that MS has made some versions of this flavor available without cost. From python-dev-list at cstork.org Sun Apr 18 23:12:47 2004 From: python-dev-list at cstork.org ('Christian Stork') Date: Sun Apr 18 23:12:52 2004 Subject: [Python-Dev] iter.index() In-Reply-To: <000b01c425b7$42592d80$e841fea9@oemcomputer> References: <20040419015348.GB6807@ics.uci.edu> <000b01c425b7$42592d80$e841fea9@oemcomputer> Message-ID: <20040419031247.GA7895@ics.uci.edu> On Sun, Apr 18, 2004 at 10:37:28PM -0400, Raymond Hettinger wrote: > [Christian Stork] > > I wanted to do something like > > > > i = itertools.chain(list1, list2).index(elem) > > I'm curious about what application needed to do this. It's for a simulation of a peer-to-peer algorithm I'm working on. The lists are lists of peers and could potentially become _very_ large. Therefore concatenation might not be cheap. > > If I use the operator module it works just fine. > > > > i = operator.indexOf(itertools.chain(list1, list2), elem) > > Nice solution. It is general purpose, self-documenting, and efficient. Thanks. :-) -- Chris Stork <> Support eff.org! <> http://www.ics.uci.edu/~cstork/ OpenPGP fingerprint: B08B 602C C806 C492 D069 021E 41F3 8C8D 50F9 CA2F From john at neggie.net Sun Apr 18 23:16:17 2004 From: john at neggie.net (John Belmonte) Date: Sun Apr 18 23:16:47 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <200404181013.20045.fdrake@acm.org> References: <407C28E0.7090808@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> <40827869.2030300@neggie.net> <200404181013.20045.fdrake@acm.org> Message-ID: <40834481.7000002@neggie.net> Fred L. Drake, Jr. wrote: > How about: > > class Foo(object): > docslots(slot1='description', > slot2=("Some really long\n" > "multi-line description."), > ...) My intention was to make something that can completely replace the current __slots__ pattern. Your proposal doesn't support slots that don't have a docstring very naturally (you'd use slot=''). Attached is another strategy that uses optional tuples. Combined with my pydoc patch, class Foo(object): slots( ('slot1', 'description'), ('slot2', """description ...continued"""), 'slot3', ('_slot4', 'hidden'), ) yields this help: class Foo(__builtin__.object) | Data descriptors defined here: | | slot1 | description | slot2 | description | ...continued | slot3 -John -- http:// if ile.org/ -------------- next part -------------- A non-text attachment was scrubbed... Name: slots.py Type: text/x-python Size: 2050 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040418/d4400ae9/slots.py From python-dev-list at cstork.org Sun Apr 18 23:26:32 2004 From: python-dev-list at cstork.org (Christian Stork) Date: Sun Apr 18 23:26:36 2004 Subject: [Python-Dev] iter.index() In-Reply-To: <482D62ED-91AC-11D8-A48C-000A95686CD8@redivi.com> References: <20040419015348.GB6807@ics.uci.edu> <70D3F6CF-91A6-11D8-A48C-000A95686CD8@redivi.com> <20040419022107.GC6807@ics.uci.edu> <482D62ED-91AC-11D8-A48C-000A95686CD8@redivi.com> Message-ID: <20040419032632.GB7895@ics.uci.edu> On Sun, Apr 18, 2004 at 10:50:13PM -0400, Bob Ippolito wrote: ... > >Why useless? It returns the index. Why do I have to build a new > >concatinated list (or my own special index function) in order to get > >that information? > > It returns an index into some sequence that no longer exists. What > good is that? In my special case, it is essentially a ranking. I don't need to access the lists via the index. > >>exhaust all of your iterator and raise an exception, > > > >Yep, just as it would for lists. > > The difference is that the iterator's life is more or less over, > there's probably nothing useful you can do with it, but a list is not > changed by this operation. Note that the mathematical function index: Elems -> Integer has a right to live on its own with the application of list accesses. ;-) > >>If you want an object that acts like a list, you should use a list. > > > >I don't see anything inherently "listy" about index(). It just counts > >how many elements there are to reach elem. And obviously the > >functionality is already there in the operator module. I'm just > >proposing a little convenience. > > I think it's pretty rare that you would want to know this information > at the cost of exhausting some/all of your iterator... and if that > really is what you want, then you should just use operator.indexOf. > There are MANY iterable types, it's not reasonable to change them all. If that is the consensus, so be it. :-) I'm just proposing it as it's in the spirit of itertools. It also eliminates an asymmetry between lists and iterators as demonstrated by operator.index() which is supposed to implement the intrinsic operators. Admittedly, it might not be very common. -- Chris Stork <> Support eff.org! <> http://www.ics.uci.edu/~cstork/ OpenPGP fingerprint: B08B 602C C806 C492 D069 021E 41F3 8C8D 50F9 CA2F From aahz at pythoncraft.com Sun Apr 18 23:52:15 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun Apr 18 23:52:19 2004 Subject: [Python-Dev] iter.index() In-Reply-To: <20040419015348.GB6807@ics.uci.edu> References: <20040419015348.GB6807@ics.uci.edu> Message-ID: <20040419035215.GA7157@panix.com> On Sun, Apr 18, 2004, Christian Stork wrote: > > Why not add the index method to iterator objects? Of course, internally > only next() is used by the default implementation. While not entirely off-topic for python-dev, you'll probably get better discussion by going to c.l.py. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From greg at cosc.canterbury.ac.nz Mon Apr 19 00:19:44 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 19 00:22:31 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: Message-ID: <200404190419.i3J4Ji2H026578@cosc353.cosc.canterbury.ac.nz> > > If not, I think you're making a big mistake basing the > > standard distribution on this compiler. > > Why is that? If it doesn't play nice with the latest MSVC, MinGW doesn't > have a future on Windows. I got scared by an earlier message that said something like "there is currently no free compiler that will generate extension modules for Python 2.4". Fortunately it seems that meant no free *Microsoft* compiler, and doesn't include gcc. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From fdrake at acm.org Mon Apr 19 00:29:57 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Apr 19 00:30:09 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <40834481.7000002@neggie.net> References: <407C28E0.7090808@neggie.net> <200404181013.20045.fdrake@acm.org> <40834481.7000002@neggie.net> Message-ID: <200404190029.57021.fdrake@acm.org> On Sunday 18 April 2004 11:16 pm, John Belmonte wrote: > My intention was to make something that can completely replace the > current __slots__ pattern. Your proposal doesn't support slots that > don't have a docstring very naturally (you'd use slot=''). Please don't misunderstand; I wasn't making a proposal, but was just responding to your question about what can be done. I've no doubt something better can be done, and if anything is done as part of stock Python, it would need to be. That said, docless slots would not be handled any differently with my sample code than they are now; just name them in __slots__. My docslots() function was very careful not to blindly overwrite an existing __slots__. So this: class Sample(object): __slots__ = "slot1", docslots(slot2="description") would work just fine. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From martin at v.loewis.de Mon Apr 19 01:04:21 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Apr 19 01:04:29 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: <200404190419.i3J4Ji2H026578@cosc353.cosc.canterbury.ac.nz> References: <200404190419.i3J4Ji2H026578@cosc353.cosc.canterbury.ac.nz> Message-ID: <40835DD5.5050008@v.loewis.de> Greg Ewing wrote: > I got scared by an earlier message that said something like "there is > currently no free compiler that will generate extension modules for > Python 2.4". > > Fortunately it seems that meant no free *Microsoft* compiler, and > doesn't include gcc. That statement would have been false in any case. It would have talked only about the official binary distribution; if you build Python yourself with a different compiler (say, Borland, or cygwin), you can certainly also build extensions with that compiler. Regards, Martin From bac at OCF.Berkeley.EDU Mon Apr 19 02:56:34 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Apr 19 02:56:42 2004 Subject: [Python-Dev] Possible modules that could use docs Message-ID: <40837822.6060906@ocf.berkeley.edu> Discovering the 'opcode' module thanks to the new PEP 329 caused me to wonder how many modules are undocumented and yet deserve to possibly be (I also was procrastinating from doing schoolwork and felt like doing some light coding =). Below is a list of modules that might deserve some docs. Any with numbers by them mean that a bug report has been filed to have them documented. If any of the modules below should not be documented, please speak up (especially about the ones with SF bug reports since those could then be closed; but they seem like modules that should get some docs). Otherwise I would like to at least mention this list in the next python-dev Summary in hopes someone will want to step forward and document them. I would also like to file a bug report for each one that doesn't have one yet so it at least gets recorded somewhere which modules still need docs beyond just the mailing list archives. DocXMLRPCServer.py ihooks.py imputil.py 515751 modulefinder.py 914375 opcode.py pickletools.py smtpd.py 450803 symtable.py toaiff.py trace.py 779976 737734 clmodule.c puremodule.c symtablemodule.c 896052 timingmodule.c zipimport.c imp (built-in) -Brett From t-meyer at ihug.co.nz Mon Apr 19 03:12:00 2004 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Mon Apr 19 03:12:09 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <1ED4ECF91CDED24C8D012BCF2B034F1305E92F19@its-xchg4.massey.ac.nz> Message-ID: <1ED4ECF91CDED24C8D012BCF2B034F13026F2BAC@its-xchg4.massey.ac.nz> > Otherwise I would like to at least mention this list in the next > python-dev Summary in hopes someone will want to step forward and > document them. +1 to this, although also noting those that have a patch for documentation already would be good (this includes smtpd.py/450803 and zipimport.c/853800), so that people know that they should not bother with those, or modify the existing patch. You also missed one report :) > zipimport.c 853800 =Tony Meyer From Paul.Moore at atosorigin.com Mon Apr 19 04:30:26 2004 From: Paul.Moore at atosorigin.com (Moore, Paul) Date: Mon Apr 19 04:30:38 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 Message-ID: <16E1010E4581B049ABC51D4975CEDB8803060E63@UKDCX001.uk.int.atosorigin.com> From: Tim Peters >[Greg Ewing] >> Can I just ask at this point whether it's possible to >> use MinGW to compile extensions for the Python version >> under discussion here? > > Don't know -- someone who uses MinGW would need to answer that. Yes, it is. However, that's for your own extensions. Grabbing an arbitrary 3rd party extension from the web and trying to compile it yourself isn't always easy. (MSVC-isms in the code, use of 3rd party commercial libraries which don't support mingw, etc) I don't know if anyone has stats on how many of the 3rd party extension authors who currently provide Windows binaries have access to MSVC7, and so can make the switch. It may be "all of them". I don't have MSVC7, although I do have MSVC6 (my company, or at least my group, never upgraded) but I've no idea how unusual my situation is. A heads-up on c.l.p would probably be worth it, but it's also likely to generate a huge amount of FUD, and anti- Microsoft rants. >> If not, I think you're making a big mistake basing the >> standard distribution on this compiler. > > Why is that? If it doesn't play nice with the latest MSVC, MinGW > doesn't have a future on Windows. Mingw is fine in this regard. Paul. __________________________________________________________________________ This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. __________________________________________________________________________ From aleaxit at yahoo.com Mon Apr 19 05:20:25 2004 From: aleaxit at yahoo.com (Alex Martelli) Date: Mon Apr 19 05:21:04 2004 Subject: [Python-Dev] dict.sortedkeys() In-Reply-To: References: Message-ID: <200404191120.25839.aleaxit@yahoo.com> On Saturday 17 April 2004 10:21 pm, Robert Brewer wrote: ... > Yes, of course. But I'm working on a large business app framework. That > means either: > > 1. Writing sorted into every module where it's needed. > 2. Putting sorted into a package somewhere, which makes it longer and > uglier with absolute referencing. > 3. Customizing consumers' Python install. > > ...none of which are good options. Is sorted the ONLY "utility" function your LARGE business app framework needs in addition to what comes with Python 2.3? That would really be utterly astounding to me. In my experience, when coding a large framework I always end up needing somewhere between 0.5 and 2 "utility functions" per tens of thousands of SLOCs (standardized lines of code -- logical lines, ignoring comments, blank lines, docstrings, &c). So, in a moderately large framework of 100,000 lines, I would expect to need somewhere between 5 and 20 utility functions (in addition to Python builtins, of course). I never expect Python's built-ins to swell to accomodate _every_ such minute need, of course. The only truly unacceptable alternative, in my opinion, is to repeat the coding for the utility functions more than once: as long as they're coded _once, and only once_, I'm a pretty happy camper. When I was relatively new as a Pythonista, I used to deal with "the utiliities issue" by injecting names in __builtin__ during the start-up of my framework: import __builtin__ import utils for u in utils.__all__: setattr(__builtin__, u, getattr(utils, u)) or the like. I have since decided that this is not at all a good idea -- as the number of utilities grows, there may be clashes with builtins, names being shadowed, etc, etc -- and the opacity of seeing a barename used and having to guess where it comes from is particularly unacceptable. These days, I far prefer to code, more simply: import __builtin__ # get error msg in case of conflict assert not hasattr(__builtin__, 'ut') import utils as ut __builtin__.ut = ut and access the utilities from everywhere as ut.sorted, ut.scrambled, etc, etc. ONE relatively mysterious barename is OK, it's always 'ut' or 'util' after all, and with nested packages etc it may be worth it to avoid the hassle of importing it repeatedly from different places in the hierarchy. Alternatively, from mywondrousframework_toplevel import utils as ut may be quite acceptable (even though this line _is_ needed in every module of the framework, I don't think it counts as "repetition" -- and it does simplify things if two frameworks are in use at the same time...). > [Bob Ippolito] > > > I'm pretty sure that in Python 2.4 you will be able to say > > sorted(seq). > > Excellent! That'll do nicely. :) Yes, for this one specific need, but surely there will be other tiny utilities you desire to have available. Moreover, I don't think you can expect to release production applications relying on Python 2.4 anytime before this coming fall (optimistically, september -- but I wouldn't base my business on the assumption that this date can't possibly shift to, say, october...) -- and that's only if your customers are uniformly "early adopters" (for many corporations with a large installed base of Python code, it's _always_ a hassle to get them to upgrade their reference Python version, even though backwards compatibility IS targeted and a lot of effort is put into it...). So, I think you should still consider what "utility-functions access strategy" is most appropriate for your needs. The "trickle" of neat utility functions into Python's built-ins, by itself, will not entirely solve the problem for you, IMHO. Alex From raymond.hettinger at verizon.net Mon Apr 19 08:54:41 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Mon Apr 19 08:55:55 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library Message-ID: <00cb01c4260d$80b9b120$e841fea9@oemcomputer> Comments are invited on a new pep: http://www.python.org/peps/pep-0329.html Based on feedback received so far, there were several changes to the original draft: * The module is public * The module name reflects its nature as a bytecode hack * There is a flag to disable binding * Added summaries of how it works, when it is called, and the effect on startup time (near zero). * Included a reference to Jython FAQs supporting the means for automatically doing nothing in environments without bytecodes. From FBatista at uniFON.com.ar Mon Apr 19 09:09:36 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Mon Apr 19 09:11:52 2004 Subject: [Python-Dev] Decimal data type issues Message-ID: [Greg Ewing] #- > And positions is the number of digits after the decimal point at #- > which you want to round the number you're creating from and discard #- > the other digits. #- #- Most people call that "decimal places". Don't know enough english as to know which way most people call something, and I thought that decimal places could get confused with decimal (the module). But, if you all think that "decimal places" is more clear than "positions", I'll fix the PEP. Thanks! From FBatista at uniFON.com.ar Mon Apr 19 09:11:33 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Mon Apr 19 09:13:56 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Kevin Jacobs] #- Python decimals would do well to support the creation of #- instances with #- fixed scale #- and precision parameters, since this information will be what is #- provided by databases, #- other financial and engineering applications, and schema. #- i.e., these #- parameters override #- the natural scale and precision found in the literal values #- used when #- constructing decimals. #- e.g., hypothetically: #- #- Decimal('2.4000', precision=2, scale=1) == Decimal('2.4') #- Decimal('2.4', precision=5, scale=4) == Decimal('2.4000') #- #- Remember, these literals are frequently coming from an #- external source #- that must #- be constrained to a given schema. This issue doesn't come up with #- floating point #- numbers nearly as often, because those schema use explicitly #- precision-qualified #- types (i.e., single, double, quad precision). I like it a lot, but not for Decimal. Money maybe will get something like this. . Facundo From FBatista at uniFON.com.ar Mon Apr 19 09:22:18 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Mon Apr 19 09:24:38 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Aahz] #- On Fri, Apr 16, 2004, Batista, Facundo wrote: #- > #- > We all must decide which of the following is better: #- > #- > a) Decimal(number) # no context used #- > Decimal(number, context) #- > #- > b) Decimal(number) # no context used #- > Decimal.using_context(number, context) #- #- Based on my reading of Tim's posts, there's also #- #- c) Decimal(number, [context]) # context always used, with #- # optional different from default Got it. But I think there're issues: a) You can specifiy the context as an optional argument. Decimal(number, [context]) You can pass context as an argument, and this context will be used at creation time. If you don't pass the context, ?no context is used or is used the "default" one? (with default I mean the context that it's on the thread at that time). With another specific method, that's avoided: b) You have to specify the context in other method. b.1) Decimal(number) This way you don't use the context at creation time. b.2) Decimal.using_context(number, [context]) Always a context is used at creation time. If you specify the context, that's the one used. If you don't, the "default" context is used. I think the (b) syntax is more clear and less error prone. If you all agree, this can be the way to create from Decimal: - Lets you create with no context - Lets you create with default context - Lets you create with an specific context The issue is how to mix this syntax with the from_float one. What do you propose? . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040419/373101ac/attachment.html From fdrake at acm.org Mon Apr 19 09:28:23 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Apr 19 09:28:31 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <40837822.6060906@ocf.berkeley.edu> References: <40837822.6060906@ocf.berkeley.edu> Message-ID: <200404190928.23248.fdrake@acm.org> On Monday 19 April 2004 02:56 am, Brett C. wrote: > pickletools.py SF bug report 873146 > imp (built-in) This one is documented; check the module index. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From pje at telecommunity.com Mon Apr 19 09:30:55 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Apr 19 09:30:45 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <00cb01c4260d$80b9b120$e841fea9@oemcomputer> Message-ID: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> At 08:54 AM 4/19/04 -0400, Raymond Hettinger wrote: >Comments are invited on a new pep: > > http://www.python.org/peps/pep-0329.html I'm not sure I understand the reasoning for implementing this via a bytecode hack, rather than in the compiler or loading mechanisms. I think I'd rather see something like: from __future__ import fast_globals which would mean that globals and builtins could be considered constants unless declared with 'global' at the module level. Then, the compiler could optimize any undeclared builtins, and the 'MAKE_FUNCTION' opcode could bind any constants defined as of the function's declaration. Finally, the module object thus created would ban any __setattr__ on any constant that has been bound into a function. (Since these are the only setattrs that could cause harm.) From FBatista at uniFON.com.ar Mon Apr 19 09:39:20 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Mon Apr 19 09:41:38 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Jewett, Jim J] #- Facundo Batista: #- > ... what is "exact conversion" to you: #- #- > 1.1 -> "1.1" #- > 1.1 -> "1.1000000000000001" #- > 1.1 -> "1.100000000000000088817841970012523233890533447265625" #- #- machine precision of floats, whatever that happens to be. That is not very portable... #- That implementation-dependent part is one reason to use a #- longer name, like exact_float. #- #- But if I say #- #- Decimal(1.1, precision=4) #- Decimal(1.1, positions=3) #- #- Those are unambiguous. If the context is 43, then later #- calculations will effectively fill in the extra digits; #- I have just said to fill them in with zeros, rather than #- whatever the float approximations were. And why you want to fill them with zeros? I think this is a presentation issue and doesn't got nothing to do about how to build the number internals. I want my number to get showed as "1.100", but that is on Money, not on Decimal. #- Sure they are; the reason to specify positions is that the #- underlying data wasn't really floating point -- it was an #- integer which by convention is written in a larger unit. #- Example with money: #- #- Assume meat is 0.987USD/pound. There are three #- price digits to multiply by the weight, but the #- final cost is rounded to an integer number of #- pennies. #- #- 10 pounds cost 9.87, but #- 1 pound costs 0.99 #- #- I don't want to say that my prices have three digits of #- precision, or I'll keep fractions of a penny. I don't #- want to say that they have only two, or I'll drop the #- pennies on expensive items. I want to say that the #- precision varies depending on the actual price. I think that you should have all the math with three decimal places, and then round to two when you have to charge your costumer (how you say "get the money and put it inside the cash register" in english?). But, this is an issue of Money, not Decimal. #- But for any *specific* value, specifying either the total #- number of valid digits (signficant figures, precision) #- or the number of fractional digits (position) is enough #- to determine both. According to your words, sticking with the specification will do your purpose. Why should we add another parameter to the ecuation? Decimal is *now* enough complex. #- > "number" can be string, int, etc., but NOT float. #- > The issue with rounding at a defined position has #- > nothing to do with context. #- #- I assume that a new Decimal would normally be created #- with as much precision as the context would need for #- calculations. By passing a context/precision/position, #- the user is saying "yeah, but this measurement wasn't #- that precise in the first place. Use zeros for the #- rest, no matter what this number claims." I don't get to understand you, and I'm not sure if you have the right concept. Several examples may help: >>> getcontext().prec = 5 >>> Decimal(124) Decimal( (0, (1, 2, 4), 0) ) >>> +Decimal(124) Decimal( (0, (1, 2, 4), 0) ) >>> Decimal('258547.368') Decimal( (0, (2, 5, 8, 5, 4, 7, 3, 6, 8), -3) ) >>> +Decimal('258547.368') Decimal( (0, (2, 5, 8, 5, 5), 1L) ) >>> Decimal.from_float(1.1) Decimal( (0, (1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 1, 7, 8, 4, 1, 9, 7, 0, 0, 1, 2, 5, 2, 3, 2, 3, 3, 8, 9, 0, 5, 3, 3, 4, 4, 7, 2, 6, 5, 6, 2, 5), -51L) ) >>> +Decimal.from_float(1.1) Decimal( (0, (1, 1, 0, 0, 0), -4L) ) . Facundo From pedronis at bluewin.ch Mon Apr 19 09:49:25 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Apr 19 09:44:31 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <00cb01c4260d$80b9b120$e841fea9@oemcomputer> Message-ID: <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> At 08:54 19.04.2004 -0400, Raymond Hettinger wrote: >Comments are invited on a new pep: > > http://www.python.org/peps/pep-0329.html > > > >Based on feedback received so far, there were several changes to the >original draft: > >* The module is public > >* The module name reflects its nature as a bytecode hack > >* There is a flag to disable binding > >* Added summaries of how it works, when it is called, and the effect on >startup time (near zero). > >* Included a reference to Jython FAQs supporting the means for >automatically doing nothing in environments without bytecodes. Raymond, could you please try to tame your speed obsession without devising even more hacks and pissing in other people's pool. Thanks. From gmccaughan at synaptics-uk.com Mon Apr 19 09:56:14 2004 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Mon Apr 19 09:56:31 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> References: <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> Message-ID: <200404191456.14865.gmccaughan@synaptics-uk.com> On Monday 2004-04-19 14:49, Samuele Pedroni wrote: > Raymond, could you please try to tame your speed obsession without > devising even more hacks and pissing in other people's pool. What a gratuitously obnoxious and pointless reply. If you have reason to think Raymond's proposal, if implemented, will cause trouble, then show it. If not, then why shouldn't he work on making Python faster if he wants to? The only drawback I can see to Raymond's proposal is that if you change the values of builtins and expect that to influence the behaviour of the standard library, then that may not work any more. I'd be more inclined to call *that* "pissing in other people's pools" than Raymond's proposal. -- g From aahz at pythoncraft.com Mon Apr 19 10:11:44 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 19 10:11:50 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <20040419141144.GB6494@panix.com> On Mon, Apr 19, 2004, Batista, Facundo wrote: > [Aahz] > > #- Based on my reading of Tim's posts, there's also > #- > #- c) Decimal(number, [context]) # context always used, with > #- # optional different from default > > Got it. But I think there're issues: > > a) You can specifiy the context as an optional argument. > > Decimal(number, [context]) > > You can pass context as an argument, and this context will > be used at creation time. If you don't pass the context, > ?no context is used or is used the "default" one? (with default > I mean the context that it's on the thread at that time). > > With another specific method, that's avoided: > > b) You have to specify the context in other method. > > b.1) Decimal(number) > > This way you don't use the context at creation time. > > b.2) Decimal.using_context(number, [context]) > > Always a context is used at creation time. If you specify > the context, that's the one used. If you don't, the > "default" context is used. I really don't care much about what decision gets made here; I just wanted to make sure all options were clearly specified. > The issue is how to mix this syntax with the from_float one. What do you > propose? This depends on what we choose as the mechanism for Decimal(). If the default is contextless, then from_float() should also be contextless with maximum precision. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From jack at performancedrivers.com Mon Apr 19 10:15:28 2004 From: jack at performancedrivers.com (Jack Diederich) Date: Mon Apr 19 10:15:33 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <00cb01c4260d$80b9b120$e841fea9@oemcomputer> References: <00cb01c4260d$80b9b120$e841fea9@oemcomputer> Message-ID: <20040419141528.GA23507@performancedrivers.com> On Mon, Apr 19, 2004 at 08:54:41AM -0400, Raymond Hettinger wrote: > Comments are invited on a new pep: > > http://www.python.org/peps/pep-0329.html > > Based on feedback received so far, there were several changes to the > original draft: > > * The module is public > > * The module name reflects its nature as a bytecode hack > I dread saying it, but is 'pragma.py' a better name than 'codetweaks.py'? They both rhyme with magic but pragma is more familiar (I think?). -jackdied From aahz at pythoncraft.com Mon Apr 19 10:19:01 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 19 10:19:09 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <200404191456.14865.gmccaughan@synaptics-uk.com> References: <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> <200404191456.14865.gmccaughan@synaptics-uk.com> Message-ID: <20040419141901.GC6494@panix.com> On Mon, Apr 19, 2004, Gareth McCaughan wrote: > On Monday 2004-04-19 14:49, Samuele Pedroni wrote: >> >> Raymond, could you please try to tame your speed obsession without >> devising even more hacks and pissing in other people's pool. Samuele, please remember that this is a "professional" environment and phrase your posts appropriately. > What a gratuitously obnoxious and pointless reply. If you have reason > to think Raymond's proposal, if implemented, will cause trouble, then > show it. If not, then why shouldn't he work on making Python faster if > he wants to? It's gratuitously insulting, but it's certainly not pointless in context: remember that Samuele is responsible for maintaining Jython, and I can certainly understand his bitterness at seeing a proposal that mucks with Python internals like this. I had a similar strong reaction to Raymond's proposal even without Samuele's reasons. Keep in mind that one of the goals of Python development has always been to keep the implementation as simple as possible. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From aahz at pythoncraft.com Mon Apr 19 10:20:44 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 19 10:20:49 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> References: <00cb01c4260d$80b9b120$e841fea9@oemcomputer> <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> Message-ID: <20040419142043.GD6494@panix.com> On Mon, Apr 19, 2004, Phillip J. Eby wrote: > > I'm not sure I understand the reasoning for implementing this via a > bytecode hack, rather than in the compiler or loading mechanisms. I think > I'd rather see something like: > > from __future__ import fast_globals +1 -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From pje at telecommunity.com Mon Apr 19 10:39:06 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Apr 19 10:38:47 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <000201c42615$fb979300$e841fea9@oemcomputer> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040419103223.02a51be0@mail.telecommunity.com> At 09:55 AM 4/19/04 -0400, Raymond Hettinger wrote: > > I'm not sure I understand the reasoning for implementing this via a > > bytecode hack, rather than in the compiler or loading mechanisms. > >The reason is that it is not always possible to know (in general) what >things constants. Take a look at the ASPN recipe example, it ought to >clarify for you how this works. I don't need clarification about the mechanism; I independently implemented *exactly* this hack myself a couple of years ago. (I just didn't propose it become part of standard Python.) See 'bind_func()' in: http://cvs.eby-sarna.com/PEAK/src/peak/util/Code.py?rev=1.1.2.1&content-type=text/vnd.viewcvs-markup > > from __future__ import fast_globals > >The __future__ mechanism is a transitional approach for things that >would break existing code. Right, and the slightly broader proposal I made *would* break existing code, but in a direction that Guido has said before he'd like to move. >Also, __future__ statements go at the beginning of a script and the >proposed bind_all goes at the end. Which is a good argument for why the __future__ statement is a much better idea. :) > > which would mean that globals and builtins could be considered >constants > > unless declared with 'global' at the module level. Then, the compiler > > could optimize any undeclared builtins, and the 'MAKE_FUNCTION' opcode > > could bind any constants defined as of the function's declaration. > > > > Finally, the module object thus created would ban any __setattr__ on >any > > constant that has been bound into a function. (Since these are the >only > > setattrs that could cause harm.) > > >IMO, this is making something hard out of something simple. No, it's trying to advance the *language*, rather than merely its implementation, and thus make it easier for Jython, IronPython, et al to cope. >Try applying the working code in the PEP to a few modules. Given that I've had this hack in my CVS repository for a few years now, you will probably realize that I already have. :) From pedronis at bluewin.ch Mon Apr 19 10:45:37 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon Apr 19 10:40:52 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <200404191456.14865.gmccaughan@synaptics-uk.com> References: <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20040419161441.02d12688@pop.bluewin.ch> At 13:56 19.04.2004 +0000, Gareth McCaughan wrote: >On Monday 2004-04-19 14:49, Samuele Pedroni wrote: > > > Raymond, could you please try to tame your speed obsession without > > devising even more hacks and pissing in other people's pool. > >What a gratuitously obnoxious and pointless reply. If you have >reason to think Raymond's proposal, if implemented, will cause >trouble, then show it. If not, then why shouldn't he work on >making Python faster if he wants to? he's proposing an hack that would speed up CPython and slow down Jython (because it would result in a no-op but OTOH he would like to remove the code when some globals are bound to locals for speed reason. Of course that's not nice too but replacing a hack with another is just sort of a win. Honestly there are probably very convoluted ways to make this, in this form, a non no-op on Jython too, not that I would like to implement them or that Raymond cared). if he has an elegant proposal that works for all of Python that would be a proposal, this seems just a hack. [Not that I particularly like this kind of semi-overt mud slinging] Other people do not completely share Raymond's design taste, so to say, in fact they have politely responded to single proposals, OTOH although it was a bit over-rudely formulated, I think it was the case to bring to the day-light that that is creating some real tension. PS: I'm starting to be really annoyed by the fact that Jython is often cited good PR but a 2nd class citizen for all the rest. I'm aware of the practical circumstances of why that's the case in some occassions, OTOH we are mostly all investing our free time on Python activities, it would be nice if we showed respect beyond just the forms and politeness. I'm back from the ACCU which was fun but tiring and also an occasion of some related heat taking. All I was looking forward is restarting to finish the new-style class work (started in December and January) in Jython,after the stop because of working on the negotations for the PyPy EU proposal together with the others PyPy people, which we hope will conclude positively for the people working on it and looking forward to restart making PyPy a real concrete contribution to Python future. From tim.one at comcast.net Mon Apr 19 10:57:52 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon Apr 19 10:57:58 2004 Subject: [Python-Dev] Decimal data type issues In-Reply-To: <407F6674.4070402@gradient.cis.upenn.edu> Message-ID: [Edward Loper] > ... > - I do believe that there will be use cases for exponents outside > this range. Just because there's no physical quantities that are > reasonably measured with these numbers, doesn't mean that there > won't be abstract quantities that are reasonably measured with > huge numbers. You don't have a real use case, Edward (if you did, you would have given it by now ). YAGNI. Note that GNU GMP's arbitrary-precision float types also have bounds on exponent magnitude (but not on precision) -- this is conventional. > - I don't believe that signaling an error when a number goes > outside this range will help catch any errors. What type of > error are we expecting to catch here? Overflow, and possibly underflow. Typical when a correct iterative algorithm is fed an input outside its radius of convergence, or an incorrect iterative algorithm is fed anything. > If this is such a problem, then why doesn't long also have > a max/min? For the same reason Decimal doesn't have a bound on precision: exact calculations can require any number of digits. Exponents kick in when the calculation becomes approximate (when precision is exceeded). From guido at python.org Mon Apr 19 11:10:59 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 19 11:11:44 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: Your message of "Sun, 18 Apr 2004 22:29:08 EDT." <40833974.10800@neggie.net> References: <407C28E0.7090808@neggie.net> <4080C0C7.7070602@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> <40827869.2030300@neggie.net> <200404181721.i3IHLTp07934@guido.python.org> <40833974.10800@neggie.net> Message-ID: <200404191510.i3JFAxX09635@guido.python.org> > Guido van Rossum wrote: > > Anything can be done using metaclasses. __slots__ is not special > > once the class exists -- it is a set of instructions for the default > > metaclass to create a specific set of descriptors (and associated > > storage). Another metaclass could use a different convention > > (although it may have to set __slots__ to let the base metaclass > > create the associated storage slots). > > My original proposal was to use __slots__ dict values for docstrings in > the default metaclass. You said you'd rather not do that because > different metaclasses may want to use the dict value for different > purposes. But from what you've explained, metaclasses are free to > interpret the value of __slots__ in any way they choose. Metaclasses > built on top of the default metaclass could translate their __slots__ > value to the __slots__ I proposed. Yes, but *if* the default metaclass assumed a dict contained only docstrings, this would be the standard usage, and it would be confusing (and sometimes incompatible) if a custom metaclass had a different interpretation. As long as the default metaclass doesn't have any particular interpretation for the values in the dict, custom metaclasses can do what they like. > Are optional tuples any better? This wouldn't preclude use of dict > values for something else. > > class Foo(object): > __slots__ = [ > 'slot1', > ('slot2', 'description'), > ('slot3', """description > ...continued"""), > ] But that currently doesn't work. Tbe most future-proof solution would be to put some kind of object in the dict values whose attributes can guide various metaclasses. Perhaps: class slotprop(object): def __init__(self, **kwds): self.__dict__.update(kwds) class C(object): __slots__ = {'slot1': slotprop(doc="this is the docstring"), 'slit2': slotprop('doc="another one")} --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 19 11:17:15 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 19 11:17:33 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: Your message of "Sun, 18 Apr 2004 23:56:34 PDT." <40837822.6060906@ocf.berkeley.edu> References: <40837822.6060906@ocf.berkeley.edu> Message-ID: <200404191517.i3JFHFa09689@guido.python.org> > DocXMLRPCServer.py > ihooks.py > imputil.py 515751 > modulefinder.py 914375 > opcode.py > pickletools.py > smtpd.py 450803 > symtable.py > toaiff.py > trace.py 779976 737734 > clmodule.c > puremodule.c > symtablemodule.c 896052 > timingmodule.c > zipimport.c > imp (built-in) I think we're trying to get the timing module to be deprecated. The others indeed ought to be documented IMO. Maybe imp is documented in the docs about import? --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Mon Apr 19 11:23:34 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon Apr 19 11:23:39 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <20040419142043.GD6494@panix.com> References: <00cb01c4260d$80b9b120$e841fea9@oemcomputer> <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> <20040419142043.GD6494@panix.com> Message-ID: <16515.61174.247974.888236@montanaro.dyndns.org> Phillip wrote: > I'm not sure I understand the reasoning for implementing this via a > bytecode hack, rather than in the compiler or loading mechanisms. This sort of stuff is a lot easier to experiment with in Python. If it stabilizes and you figure out a way to decide whether to transform functions automatically then sure, it belongs in the compiler or an optimizer of some sort. Skip From guido at python.org Mon Apr 19 11:25:26 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 19 11:27:27 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: Your message of "Mon, 19 Apr 2004 09:30:55 EDT." <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> Message-ID: <200404191525.i3JFPW009731@guido.python.org> > I'd rather see something like: > > from __future__ import fast_globals > > which would mean that globals and builtins could be considered > constants unless declared with 'global' at the module level. Don't you think that this should be flagged with syntax that doesn't permanently require the use of the word "future"? And I think that reusing the global statement at the global level is hardly the best way to do this. I do think that explicitly flagging "volatile" globals somehow might be the right way to go eventually, but it should only be required for those globals for which the compiler can't tell whether they may be modified (i.e. anything that is assigned to more than once or or from inside a loop or conditional or function is automatically volatile). > Finally, the module object thus created would ban any __setattr__ on > any constant that has been bound into a function. (Since these are > the only setattrs that could cause harm.) Huh? If the object's identity is constant (which is what we're talking about) why should its *contents* be constant? And who says setattr is the only way to modify an object? Or am I misunderstanding what you're trying to say? (I thought that "a constant bound into a function" would be something like a global/builtin name binding.) --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Mon Apr 19 11:36:51 2004 From: barry at python.org (Barry Warsaw) Date: Mon Apr 19 11:37:02 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <200404191517.i3JFHFa09689@guido.python.org> References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> Message-ID: <1082389011.7365.123.camel@anthem.wooz.org> On Mon, 2004-04-19 at 11:17, Guido van Rossum wrote: > > DocXMLRPCServer.py > > ihooks.py > > imputil.py 515751 > > modulefinder.py 914375 > > opcode.py > > pickletools.py > > smtpd.py 450803 > > symtable.py > > toaiff.py > > trace.py 779976 737734 > > clmodule.c > > puremodule.c > > symtablemodule.c 896052 > > timingmodule.c > > zipimport.c > > imp (built-in) > > I think we're trying to get the timing module to be deprecated. The > others indeed ought to be documented IMO. Maybe imp is documented in > the docs about import? puremodule.c could probably be deprecated. Does anybody use Purify anymore -- and if so, have you ever used puremodule? Does it still compile? Are the APIs it uses still up-to-date? I haven't had a license for Purify in years. -Barry From guido at python.org Mon Apr 19 11:36:53 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 19 11:37:07 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: Your message of "Mon, 19 Apr 2004 16:45:37 +0200." <5.2.1.1.0.20040419161441.02d12688@pop.bluewin.ch> References: <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> <5.2.1.1.0.20040419161441.02d12688@pop.bluewin.ch> Message-ID: <200404191536.i3JFasX09776@guido.python.org> [Samuele] > he's proposing an hack that would speed up CPython and slow down Jython > (because it would result in a no-op but OTOH he would like to remove the code > when some globals are bound to locals for speed reason. Of course that's > not nice too > but replacing a hack with another is just sort of a win. Honestly there are > probably > very convoluted ways to make this, in this form, a non no-op on Jython too, > not that I would like to > implement them or that Raymond cared). > > if he has an elegant proposal that works for all of Python that would be a > proposal, this > seems just a hack. > > [Not that I particularly like this kind of semi-overt mud slinging] Other > people do not completely share Raymond's design taste, so to say, in fact > they have politely responded to single proposals, OTOH although it was a > bit over-rudely formulated, I think it was the case to bring to the day-light > that that is creating some real tension. While Samuele's words were inappropriately rude, I'm glad this was brought up (and even to some extent I'm glad about some of the words that Raymond used). Raymond *does* seem to have a "speed obsession", and while I'm sure he has Python's best interests at heart, I'm not sure that accepting everything he proposes will actually be good for Python. I am happy with tweaking of existing internals (like speeding up append) and the addition of new interfaces built on iterators (I don't quite share Armin's views of "iterators considered harmful") but I'm worried that e.g. replacing heapq.py with C code is not useful (raise your hand if you have used heapq.py -- now raise your hand if it was too slow for your purpose -- I expect to see very few hands) and introducing speed-up hacks that will invite many people with less judgement to use them all the time and hence reduce the code quality of Python code samples available to the world with useless encrustations. > PS: I'm starting to be really annoyed by the fact that Jython is > often cited good PR but a 2nd class citizen for all the rest. I'm > aware of the practical circumstances of why that's the case in some > occassions, OTOH we are mostly all investing our free time on Python > activities, it would be nice if we showed respect beyond just the > forms and politeness. I'm back from the ACCU which was fun but > tiring and also an occasion of some related heat taking. All I was > looking forward is restarting to finish the new-style class work > (started in December and January) in Jython,after the stop because > of working on the negotations for the PyPy EU proposal together with > the others PyPy people, which we hope will conclude positively for > the people working on it and looking forward to restart making PyPy > a real concrete contribution to Python future. I hope you don't think *everybody* sees Jython as a 2nd class citizen. I personally see it as very important. Certainly the PSF is considering funding it, whether via the general grants committee (just formed) or via a special grant. --Guido van Rossum (home page: http://www.python.org/~guido/) From neal at metaslash.com Mon Apr 19 11:42:38 2004 From: neal at metaslash.com (Neal Norwitz) Date: Mon Apr 19 11:42:46 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <1082389011.7365.123.camel@anthem.wooz.org> References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> Message-ID: <20040419154238.GC22725@epoch.metaslash.com> On Mon, Apr 19, 2004 at 11:36:51AM -0400, Barry Warsaw wrote: > > puremodule.c could probably be deprecated. Does anybody use Purify > anymore -- and if so, have you ever used puremodule? Does it still > compile? Are the APIs it uses still up-to-date? Although I did use Purify on python years ago, I never use it now. Though, I never used the puremodule. IMO, Purify is not useful compared to valgrind. +1 on deprecating puremodule.c. Neal From nbastin at opnet.com Mon Apr 19 11:47:08 2004 From: nbastin at opnet.com (Nick Bastin) Date: Mon Apr 19 11:47:19 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <20040419154238.GC22725@epoch.metaslash.com> References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> Message-ID: On Apr 19, 2004, at 11:42 AM, Neal Norwitz wrote: > On Mon, Apr 19, 2004 at 11:36:51AM -0400, Barry Warsaw wrote: >> >> puremodule.c could probably be deprecated. Does anybody use Purify >> anymore -- and if so, have you ever used puremodule? Does it still >> compile? Are the APIs it uses still up-to-date? > > Although I did use Purify on python years ago, I never use it now. > Though, I never used the puremodule. IMO, Purify is not useful > compared to valgrind. +1 on deprecating puremodule.c. I think it's useful to point out that Purify runs on a lot more platforms than valgrind does, so we should probably keep puremodule around, and update it if necessary. -- Nick From tcdelaney at optusnet.com.au Mon Apr 19 11:56:41 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Mon Apr 19 11:56:48 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants inthe Standard Library References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> Message-ID: <004301c42626$e8634890$a100a8c0@ryoko> Phillip J. Eby wrote: > At 08:54 AM 4/19/04 -0400, Raymond Hettinger wrote: >> Comments are invited on a new pep: >> >> http://www.python.org/peps/pep-0329.html > > I'm not sure I understand the reasoning for implementing this via a > bytecode hack, rather than in the compiler or loading mechanisms. I > think > I'd rather see something like: > > from __future__ import fast_globals +1. The semantics would have to be that globals *may* be optimised when the above is present (and code should assume that they will be) but there is no guarantee i.e. code should work both in the presence and absence of the above. This allows Jython (for example) to ignore it entirely, partially implement it, or completely implement it as desired. An additional bonus is that this puts it up front, so someone reading the code knows that the author is expecting optimisations to occur - when it's at the end it's much more likely to be missed, and also much less likely to be taken into account when reading the code. > which would mean that globals and builtins could be considered > constants unless declared with 'global' at the module level. This would be a backwards-incompatible change (and hence definitely warranting the __future__) but I presume you mean considered constant within the module that they are defined - hence essentially a non-issue. > Then, the compiler > could optimize any undeclared builtins, and the 'MAKE_FUNCTION' opcode > could bind any constants defined as of the function's declaration. I know it's been suggested before, but I believe the idea of module-level locals - an indexed lookup rather than a dict lookup - might be worthwhile considering as part of this. That would also give improved performance for external modules using module-level names. > Finally, the module object thus created would ban any __setattr__ on > any constant that has been bound into a function. (Since these are the > only setattrs that could cause harm.) It's late (2am), so I'm not entirely sure I understand this - do you mean something like: class X: pass x = X() def func(): print x x.a = 1 would throw an exception? Why would this cause a problem? The above would become almost equivalent to: class X: pass x = X() def func(): global x _x = x print _x x.a = 1 and I don't see a problem with that. Tim Delaney From neal at metaslash.com Mon Apr 19 12:01:49 2004 From: neal at metaslash.com (Neal Norwitz) Date: Mon Apr 19 12:01:56 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> Message-ID: <20040419160149.GD22725@epoch.metaslash.com> On Mon, Apr 19, 2004 at 11:47:08AM -0400, Nick Bastin wrote: > > On Apr 19, 2004, at 11:42 AM, Neal Norwitz wrote: > > >On Mon, Apr 19, 2004 at 11:36:51AM -0400, Barry Warsaw wrote: > >> > >>puremodule.c could probably be deprecated. Does anybody use Purify > >>anymore -- and if so, have you ever used puremodule? Does it still > >>compile? Are the APIs it uses still up-to-date? > > > >Although I did use Purify on python years ago, I never use it now. > >Though, I never used the puremodule. IMO, Purify is not useful > >compared to valgrind. +1 on deprecating puremodule.c. > > I think it's useful to point out that Purify runs on a lot more > platforms than valgrind does, so we should probably keep puremodule > around, and update it if necessary. Valgrind runs on x86 and PPC (still experimental). IIRC, Purify "runs" on Solaris, HPUX, Windows (x86), and SGI? (I tried to verify but the IBM site isn't responding.) IMO that's not a lot. Where Purify "running" is relative to any particular patch you have installed on the system and if you can download a beta. I stopped using Purify because Rational was so awful to deal with. Neal From barry at python.org Mon Apr 19 12:09:46 2004 From: barry at python.org (Barry Warsaw) Date: Mon Apr 19 12:09:55 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <20040419160149.GD22725@epoch.metaslash.com> References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> <20040419160149.GD22725@epoch.metaslash.com> Message-ID: <1082390985.7365.155.camel@anthem.wooz.org> On Mon, 2004-04-19 at 12:01, Neal Norwitz wrote: > On Mon, Apr 19, 2004 at 11:47:08AM -0400, Nick Bastin wrote: > > > > On Apr 19, 2004, at 11:42 AM, Neal Norwitz wrote: > > > > >On Mon, Apr 19, 2004 at 11:36:51AM -0400, Barry Warsaw wrote: > > >> > > >>puremodule.c could probably be deprecated. Does anybody use Purify > > >>anymore -- and if so, have you ever used puremodule? Does it still > > >>compile? Are the APIs it uses still up-to-date? > > > > > >Although I did use Purify on python years ago, I never use it now. > > >Though, I never used the puremodule. IMO, Purify is not useful > > >compared to valgrind. +1 on deprecating puremodule.c. > > > > I think it's useful to point out that Purify runs on a lot more > > platforms than valgrind does, so we should probably keep puremodule > > around, and update it if necessary. > > Valgrind runs on x86 and PPC (still experimental). > IIRC, Purify "runs" on Solaris, HPUX, Windows (x86), and SGI? > (I tried to verify but the IBM site isn't responding.) > IMO that's not a lot. I was able to get to this page: http://www-306.ibm.com/software/awdtools/purifyplus/unix/ and it looks like the suite of tools runs on "Linux" now too. > Where Purify "running" is relative to any particular patch you have > installed on the system and if you can download a beta. I stopped using > Purify because Rational was so awful to deal with. -Barry From pje at telecommunity.com Mon Apr 19 12:21:15 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Apr 19 12:21:32 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <200404191525.i3JFPW009731@guido.python.org> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040419121625.02fd9190@telecommunity.com> At 08:25 AM 4/19/04 -0700, Guido van Rossum wrote: > > I'd rather see something like: > > > > from __future__ import fast_globals > > > > which would mean that globals and builtins could be considered > > constants unless declared with 'global' at the module level. > >Don't you think that this should be flagged with syntax that doesn't >permanently require the use of the word "future"? I thought that the goal was to ultimately make at least builtins non-rebindable by default, without having flagged them as such. >And I think that >reusing the global statement at the global level is hardly the best >way to do this. Okay. >I do think that explicitly flagging "volatile" globals somehow might >be the right way to go eventually, but it should only be required for >those globals for which the compiler can't tell whether they may be >modified (i.e. anything that is assigned to more than once or or from >inside a loop or conditional or function is automatically volatile). The issue I was addressing was whether another module is allowed to rebind them via setattr. > > Finally, the module object thus created would ban any __setattr__ on > > any constant that has been bound into a function. (Since these are > > the only setattrs that could cause harm.) > >Huh? If the object's identity is constant (which is what we're >talking about) why should its *contents* be constant? And who says >setattr is the only way to modify an object? Or am I misunderstanding >what you're trying to say? (I thought that "a constant bound into a >function" would be something like a global/builtin name binding.) Sorry, I was unclear. I meant that the module object could disallow __setattr__(name,value) on *itself* for any name that was considered a "constant", because that name's binding had been copied into a function for use as a constant. I wasn't trying to say that "constants" should be immutable; in fact there are plenty of use cases for mutable (but not rebindable) globals, such as module-level caches and registries. From nbastin at opnet.com Mon Apr 19 12:25:12 2004 From: nbastin at opnet.com (Nick Bastin) Date: Mon Apr 19 12:25:31 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <20040419160149.GD22725@epoch.metaslash.com> References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> <20040419160149.GD22725@epoch.metaslash.com> Message-ID: <21CC4191-921E-11D8-BC61-000393CBDF94@opnet.com> On Apr 19, 2004, at 12:01 PM, Neal Norwitz wrote: > Valgrind runs on x86 and PPC (still experimental). > IIRC, Purify "runs" on Solaris, HPUX, Windows (x86), and SGI? > (I tried to verify but the IBM site isn't responding.) > IMO that's not a lot. That's more than valgrind. Specifically, it includes solaris, which is where we happen to run Purify most often (better results than win32). > Where Purify "running" is relative to any particular patch you have > installed on the system and if you can download a beta. I stopped > using > Purify because Rational was so awful to deal with. While I agree that Rational was terrible to deal with, Purify is the only option available to some users, and it seems like a bad idea to remove something from the distribution while it provides benefit to some people, while not affecting those who don't wish to use it. -- Nick From pje at telecommunity.com Mon Apr 19 12:27:20 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Apr 19 12:27:30 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants inthe Standard Library In-Reply-To: <004301c42626$e8634890$a100a8c0@ryoko> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040419122230.02fdab40@telecommunity.com> At 01:56 AM 4/20/04 +1000, Tim Delaney wrote: >Phillip J. Eby wrote: > > > which would mean that globals and builtins could be considered > > constants unless declared with 'global' at the module level. > >This would be a backwards-incompatible change (and hence definitely >warranting the __future__) but I presume you mean considered constant within >the module that they are defined - hence essentially a non-issue. It's already backwards-incompatible. Currently, you can change the behavior of a module by modifying its attributes. A module that uses this binding approach is immune to changes in its globals. This is a *seriously* backward incompatible change, as it essentially disallows monkeypatching. > > Then, the compiler > > could optimize any undeclared builtins, and the 'MAKE_FUNCTION' opcode > > could bind any constants defined as of the function's declaration. > >I know it's been suggested before, but I believe the idea of module-level >locals - an indexed lookup rather than a dict lookup - might be worthwhile >considering as part of this. That would also give improved performance for >external modules using module-level names. Yes, I gather IronPython uses this technique as one of three alternative approaches to module globals. > > Finally, the module object thus created would ban any __setattr__ on > > any constant that has been bound into a function. (Since these are the > > only setattrs that could cause harm.) > >It's late (2am), so I'm not entirely sure I understand this - do you mean >something like: > > class X: > pass > > x = X() > > def func(): > print x > > x.a = 1 > >would throw an exception? Why would this cause a problem? No. I meant that if your module above were named 'foo', then setting 'foo.x' would be disallowed. From fumanchu at amor.org Mon Apr 19 12:43:07 2004 From: fumanchu at amor.org (Robert Brewer) Date: Mon Apr 19 12:46:49 2004 Subject: [Python-Dev] dict.sortedkeys() Message-ID: Alex Martelli wrote: > On Saturday 17 April 2004 10:21 pm, Robert Brewer wrote: > ... > > Yes, of course. But I'm working on a large business app > framework. That > > means either: > > > > 1. Writing sorted into every module where it's needed. > > 2. Putting sorted into a package somewhere, which makes it > longer and > > uglier with absolute referencing. > > 3. Customizing consumers' Python install. > > > > ...none of which are good options. > > Is sorted the ONLY "utility" function your LARGE business app > framework needs in addition to what comes with Python 2.3? > That would really be utterly astounding to me. That would astound me, too. :) However, I will make the following points (if only for posterity--this is getting off-topic): 1. Despite what I just said, Python 2.3 is astoundingly complete. 2. Although I suck at actual implementation, I can be a pretty good designer. I'd say 95% of the "utility" functions I write end up being specific to one of the (sub)packages in such a framework, so it's logical to shove them in that package. 3. I just checked; I have 4 "utility" modules that the framework uses: one consists of 3 dict subclasses, another has a couple date tools, one provides first-class expressions, and finally, there's a small import helper. Now that I've written that out, I wonder how many other people have similar, generic, yet custom-built code. Which leads me to wonder if a survey on c.l.p. would help hone direction for some standard lib development. > The only truly unacceptable alternative, in my opinion, > is to repeat the coding for the utility functions more > than once: as long as they're coded _once, and only once_, > I'm a pretty happy camper. Yep. There's nothing like forgetting to update both copies to make you wish you had only done it once the first time. :) > These days, I far prefer to code, more simply: > > import __builtin__ > # get error msg in case of conflict > assert not hasattr(__builtin__, 'ut') > import utils as ut > __builtin__.ut = ut > > and access the utilities from everywhere as ut.sorted, > ut.scrambled, etc, etc. > ONE relatively mysterious barename is OK... That runs counter to my sensibilities as a Modern. I have to have clean containers; a module named "utils" starts to cross the line between practicality and purity. I much prefer to group similar code into multiple modules which then get placed in site-packages. Again, if they are app-specific, they go in the __init__.py for the app. The leap from ZERO mysterious barenames to ONE is larger than the leap from one to two, for me at least. ;) > The "trickle" of neat utility functions into Python's > built-ins, by itself, will not entirely solve the > problem for you, IMHO. No; I don't expect it to. I use the alternatives all the time. But there comes a point when the cost of the neat function (implemented in any way other than a builtin) outweighs the cost of the normal "boilerplate" method of doing it. sorted() is just such a case--*extremely* generic and also greatly useful in making code clear and succinct. If you could sort a dictionary's keys in three lines of code by saying: keys sort iterate there wouldn't be a problem, because the cost to the brain and fingers of the programmer is low. But in pre-2.4 one must write: dkeys = mydict.keys() dkeys.sort() for key in dkeys: ...which is extremely costly in terms of usability. Compare to: for key in sorted(mydict): Wow. What a difference. Design of a language is not that different from any other design task: you want to make a better product the easiest choice for the user. Small changes like the above end up returning many times their investment. Robert Brewer MIS Amor Ministries fumanchu@amor.org From barry at python.org Mon Apr 19 13:07:35 2004 From: barry at python.org (Barry Warsaw) Date: Mon Apr 19 13:07:46 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <21CC4191-921E-11D8-BC61-000393CBDF94@opnet.com> References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> <20040419160149.GD22725@epoch.metaslash.com> <21CC4191-921E-11D8-BC61-000393CBDF94@opnet.com> Message-ID: <1082394455.7365.175.camel@anthem.wooz.org> On Mon, 2004-04-19 at 12:25, Nick Bastin wrote: > While I agree that Rational was terrible to deal with, Purify is the > only option available to some users, and it seems like a bad idea to > remove something from the distribution while it provides benefit to > some people, while not affecting those who don't wish to use it. Deprecation isn't the same as removal. What puremodule is really lacking at the moment is a champion; have you actually used it recently? -Barry From nbastin at opnet.com Mon Apr 19 13:41:56 2004 From: nbastin at opnet.com (Nick Bastin) Date: Mon Apr 19 13:42:17 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <1082394455.7365.175.camel@anthem.wooz.org> References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> <20040419160149.GD22725@epoch.metaslash.com> <21CC4191-921E-11D8-BC61-000393CBDF94@opnet.com> <1082394455.7365.175.camel@anthem.wooz.org> Message-ID: On Apr 19, 2004, at 1:07 PM, Barry Warsaw wrote: > On Mon, 2004-04-19 at 12:25, Nick Bastin wrote: > >> While I agree that Rational was terrible to deal with, Purify is the >> only option available to some users, and it seems like a bad idea to >> remove something from the distribution while it provides benefit to >> some people, while not affecting those who don't wish to use it. > > Deprecation isn't the same as removal. What puremodule is really > lacking at the moment is a champion; have you actually used it > recently? Ah, the moment of truth.. :-) Well, uh, no. I didn't know puremodule existed, so I didn't give it a try. It seems like something that would be of use, though, so I'd hate to see it go away. A cursory look at the code indicates that it may need some updating (have to try...the API may not have actually changed significantly). I'm actually more interested in it from the Quantify side of things from the standpoint of being able to provide the user a single look at their application profiling data from both a C and Python perspective. -- Nick From barry at python.org Mon Apr 19 13:52:40 2004 From: barry at python.org (Barry Warsaw) Date: Mon Apr 19 13:52:49 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> <20040419160149.GD22725@epoch.metaslash.com> <21CC4191-921E-11D8-BC61-000393CBDF94@opnet.com> <1082394455.7365.175.camel@anthem.wooz.org> Message-ID: <1082397159.7365.182.camel@anthem.wooz.org> On Mon, 2004-04-19 at 13:41, Nick Bastin wrote: > Ah, the moment of truth.. :-) Well, uh, no. I didn't know puremodule > existed, so I didn't give it a try. It seems like something that would > be of use, though, so I'd hate to see it go away. A cursory look at > the code indicates that it may need some updating (have to try...the > API may not have actually changed significantly). I'm actually more > interested in it from the Quantify side of things from the standpoint > of being able to provide the user a single look at their application > profiling data from both a C and Python perspective. It's been years since I looked at the stuff, but IIRC, the module did work for Quantify too. Well, feel free to update it! Lacking a champion though I still think it should be deprecated (but not removed at least for now). -Barry From nbastin at opnet.com Mon Apr 19 13:56:48 2004 From: nbastin at opnet.com (Nick Bastin) Date: Mon Apr 19 13:57:02 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <1082397159.7365.182.camel@anthem.wooz.org> References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> <20040419160149.GD22725@epoch.metaslash.com> <21CC4191-921E-11D8-BC61-000393CBDF94@opnet.com> <1082394455.7365.175.camel@anthem.wooz.org> <1082397159.7365.182.camel@anthem.wooz.org> Message-ID: On Apr 19, 2004, at 1:52 PM, Barry Warsaw wrote: > On Mon, 2004-04-19 at 13:41, Nick Bastin wrote: > >> Ah, the moment of truth.. :-) Well, uh, no. I didn't know puremodule >> existed, so I didn't give it a try. It seems like something that >> would >> be of use, though, so I'd hate to see it go away. A cursory look at >> the code indicates that it may need some updating (have to try...the >> API may not have actually changed significantly). I'm actually more >> interested in it from the Quantify side of things from the standpoint >> of being able to provide the user a single look at their application >> profiling data from both a C and Python perspective. > > It's been years since I looked at the stuff, but IIRC, the module did > work for Quantify too. Well, feel free to update it! Lacking a > champion though I still think it should be deprecated (but not removed > at least for now). That seems fine. I may pick it up and take a look after I'm done with my current pass of profiler enhancements, but probably not before, so it'll be a little while. It also appears that no one else on python-dev actually uses it... :-) -- Nick From bac at OCF.Berkeley.EDU Mon Apr 19 14:06:04 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Apr 19 14:06:16 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: <200404190928.23248.fdrake@acm.org> References: <40837822.6060906@ocf.berkeley.edu> <200404190928.23248.fdrake@acm.org> Message-ID: <4084150C.3060908@ocf.berkeley.edu> Fred L. Drake, Jr. wrote: > On Monday 19 April 2004 02:56 am, Brett C. wrote: > > pickletools.py > > SF bug report 873146 > > > imp (built-in) > > This one is documented; check the module index. > Oops. My bad. -Brett From bac at OCF.Berkeley.EDU Mon Apr 19 14:24:00 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Apr 19 14:24:13 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <200404191525.i3JFPW009731@guido.python.org> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> <200404191525.i3JFPW009731@guido.python.org> Message-ID: <40841940.4010305@ocf.berkeley.edu> Guido van Rossum wrote: >>I'd rather see something like: >> >> from __future__ import fast_globals >> >>which would mean that globals and builtins could be considered >>constants unless declared with 'global' at the module level. > > > Don't you think that this should be flagged with syntax that doesn't > permanently require the use of the word "future"? And I think that > reusing the global statement at the global level is hardly the best > way to do this. > > I do think that explicitly flagging "volatile" globals somehow might > be the right way to go eventually, but it should only be required for > those globals for which the compiler can't tell whether they may be > modified (i.e. anything that is assigned to more than once or or from > inside a loop or conditional or function is automatically volatile). > Just to make sure I am understanding this, are you suggesting a possible statement like ``volatile len`` to flag that len may be changed? That way all "volatile"-flagged globals and one that are redefined in the module use LOAD_GLOBAL while all other built-ins either get them set to locals or use a LOAD_BUILTIN opcode? Or am I getting the use of volatile reversed (which would make more backwards-compatible)? If we go with the former interpretation I take it we would still need to get that warning Neil worked on a while back for warning people when someone is injecting into a module's global namespace. -Brett From skip at pobox.com Mon Apr 19 12:46:53 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon Apr 19 14:34:43 2004 Subject: [Python-Dev] Possible modules that could use docs In-Reply-To: References: <40837822.6060906@ocf.berkeley.edu> <200404191517.i3JFHFa09689@guido.python.org> <1082389011.7365.123.camel@anthem.wooz.org> <20040419154238.GC22725@epoch.metaslash.com> Message-ID: <16516.637.719565.642835@montanaro.dyndns.org> >>> puremodule.c could probably be deprecated. Does anybody use Purify >>> anymore -- and if so, have you ever used puremodule? Does it still >>> compile? Are the APIs it uses still up-to-date? >> >> Although I did use Purify on python years ago, I never use it now. >> Though, I never used the puremodule. IMO, Purify is not useful >> compared to valgrind. +1 on deprecating puremodule.c. Nick> I think it's useful to point out that Purify runs on a lot more Nick> platforms than valgrind does, so we should probably keep Nick> puremodule around, and update it if necessary. I don't think there's any question that Purify (and perhaps the pure module) are of use to a certain segment of the Python user base. The question is more whether or not it can be properly maintained as part of the distribution. Barry and perhaps Neal are the only two people I remember ever using it, and it appears that neither of them to anymore. Deprecating it doesn't mean it has to go away, just that support has to come from outside the core group of maintainers. Does anyone with checkin privileges: 1. have the necessary Purify license which would allow them to build and exercise the pure module? 2. have the time/desire to support it? If the answer is "no" to either question, I think deprecation is the way to go. Skip From jim.jewett at EDS.COM Mon Apr 19 14:34:10 2004 From: jim.jewett at EDS.COM (Jewett, Jim J) Date: Mon Apr 19 14:35:06 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: Kevin Jacobs: #- Decimal('2.4000', precision=2, scale=1) == Decimal('2.4') #- Decimal('2.4', precision=5, scale=4) == Decimal('2.4000') #- #- Remember, these literals are frequently coming from an #- external source that must be constrained to a given schema. Facundo Batista: > I like it a lot, but not for Decimal. This is another face of "what to do with float" Even if your default context is n digits, there is no reason to assume that all your inputs will be measured that precisely. If someone sends me a list of weights: PartA 1105 kg PartB 3 kg then I don't want to pretend that the 3kg part was weighed any more precisely just because the total is smaller. On the other hand, if the weights are: PartA 3 kg PartB 3.0 kg then I do want to assume that the second weight is more precise. As an example 3000 g 3001 g + 2 g + 1 g ------ ------ 3kg 3002 g #- I assume that a new Decimal would normally be created #- with as much precision as the context would need for #- calculations. By passing a context/precision/position, #- the user is saying "yeah, but this measurement wasn't #- that precise in the first place. Use zeros for the #- rest, no matter what this number claims." I don't get to understand you, and I'm not sure if you have the right concept. Several examples may help: >>> getcontext().prec = 5 >>> Decimal(124) Decimal( (0, (1, 2, 4), 0) ) >>> +Decimal(124) Decimal( (0, (1, 2, 4), 0) ) >>> Decimal('258547.368') Decimal( (0, (2, 5, 8, 5, 4, 7, 3, 6, 8), -3) ) >>> +Decimal('258547.368') Decimal( (0, (2, 5, 8, 5, 5), 1L) ) >>> Decimal.from_float(1.1) Decimal( (0, (1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 8, 8, 8, 1, 7, 8, 4, 1, 9, 7, 0, 0, 1, 2, 5, 2, 3, 2, 3, 3, 8, 9, 0, 5, 3, 3, 4, 4, 7, 2, 6, 5, 6, 2, 5), -51L) ) >>> +Decimal.from_float(1.1) Decimal( (0, (1, 1, 0, 0, 0), -4L) ) . Facundo From guido at python.org Mon Apr 19 14:35:10 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 19 14:35:39 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: Your message of "Mon, 19 Apr 2004 11:24:00 PDT." <40841940.4010305@ocf.berkeley.edu> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> <200404191525.i3JFPW009731@guido.python.org> <40841940.4010305@ocf.berkeley.edu> Message-ID: <200404191835.i3JIZAm10204@guido.python.org> [Phillip Eby] > >>I'd rather see something like: > >> > >> from __future__ import fast_globals > >> > >>which would mean that globals and builtins could be considered > >>constants unless declared with 'global' at the module level. [Guido van Rossum] > > Don't you think that this should be flagged with syntax that doesn't > > permanently require the use of the word "future"? And I think that > > reusing the global statement at the global level is hardly the best > > way to do this. > > > > I do think that explicitly flagging "volatile" globals somehow might > > be the right way to go eventually, but it should only be required for > > those globals for which the compiler can't tell whether they may be > > modified (i.e. anything that is assigned to more than once or or from > > inside a loop or conditional or function is automatically volatile). [Brett Cannon] > Just to make sure I am understanding this, are you suggesting a possible > statement like ``volatile len`` to flag that len may be changed? Yeah, but not that syntax. (I'm not proposing any syntax at this point.) > That way all "volatile"-flagged globals and one that are redefined > in the module use LOAD_GLOBAL while all other built-ins either get > them set to locals or use a LOAD_BUILTIN opcode? Sort of, although I think you meant STORE_GLOBAL rather than LOAD_GLOBAL. Exactly how this would be implemented is up to the bytecode compiler; I can see that some cheap builtins (e.g. len) may be turned into special opcodes so that no lookup is used at all. > Or am I getting the use of volatile reversed (which would make more > backwards-compatible)? You're not getting it reversed. Having to flag the ones that are not volatile would indeed be more backward compatible, but it would require everybody to put lots of 'nonvolatile' statements in all the time just in the hope to get faster code. That's what I'm trying to avoid, and that's why I'm not psyched about Raymond's (or anybody's!) proposed "make this function faster" API. Given that "volatile" globals are very rare, it makes more sense to put the burden on the programmer who needs his seemingly-constant globals to be reloaded. > If we go with the former interpretation I take it we would still > need to get that warning Neil worked on a while back for warning > people when someone is injecting into a module's global namespace. Yeah, that warning is still the right thing to do, but it's implementation was a nuisance because there were some legit cases that it couldn't know about. I don't recall -- did we turn it into a silent deprecation or did we just delete the warning? --Guido van Rossum (home page: http://www.python.org/~guido/) From nbastin at opnet.com Mon Apr 19 14:37:34 2004 From: nbastin at opnet.com (Nick Bastin) Date: Mon Apr 19 14:37:56 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <40841940.4010305@ocf.berkeley.edu> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> <200404191525.i3JFPW009731@guido.python.org> <40841940.4010305@ocf.berkeley.edu> Message-ID: On Apr 19, 2004, at 2:24 PM, Brett C. wrote: > Guido van Rossum wrote: > >>> I'd rather see something like: >>> >>> from __future__ import fast_globals >>> >>> which would mean that globals and builtins could be considered >>> constants unless declared with 'global' at the module level. >> Don't you think that this should be flagged with syntax that doesn't >> permanently require the use of the word "future"? And I think that >> reusing the global statement at the global level is hardly the best >> way to do this. >> I do think that explicitly flagging "volatile" globals somehow might >> be the right way to go eventually, but it should only be required for >> those globals for which the compiler can't tell whether they may be >> modified (i.e. anything that is assigned to more than once or or from >> inside a loop or conditional or function is automatically volatile). > > Just to make sure I am understanding this, are you suggesting a > possible statement like ``volatile len`` to flag that len may be > changed? That way all "volatile"-flagged globals and one that are > redefined in the module use LOAD_GLOBAL while all other built-ins > either get them set to locals or use a LOAD_BUILTIN opcode? > > Or am I getting the use of volatile reversed (which would make more > backwards-compatible)? You could, of course, create a statement like "const len" to flag that len will NOT be changed, thus creating true backwards compatibility, but you'd like to believe that const is the 95% use case, and thus it should be "volatile foo" when you want to change foo, but that's not all that backwards compatible. Of course, you could add something like "from __past__ import volatile_globals", and "from __future__ import const_globals" for a release.. ;-) -- Nick From jim.jewett at eds.com Mon Apr 19 14:39:22 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Mon Apr 19 14:40:08 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Kevin Jacobs] #- Decimal('2.4000', precision=2, scale=1) == Decimal('2.4') #- Decimal('2.4', precision=5, scale=4) == Decimal('2.4000') #- #- Remember, these literals are frequently coming from an #- external source that must be constrained to a given schema. Facundo > I like it a lot, but not for Decimal. This is another face of "what do do with float" Just because your default context has n digits of precision doesn't mean that all your input will. If my input has 2.4000 (but the extra zeros were measured) then I want to keep that information. If my input has 2.4 (and I didn't measure beyond those digits) I want to know that for error analysis. If my input has 1.1 (only measured to two digits) but my context is 25, I don't want to assume that the machine's float-rounding is correct; the fairest "I don't know" estimate is still zero, even out to digit 100. -jJ From aahz at pythoncraft.com Mon Apr 19 14:51:49 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon Apr 19 14:51:54 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <20040419185149.GA23690@panix.com> On Mon, Apr 19, 2004, Jewett, Jim J wrote: > > [Kevin Jacobs] > #- Decimal('2.4000', precision=2, scale=1) == Decimal('2.4') > #- Decimal('2.4', precision=5, scale=4) == Decimal('2.4000') > #- > #- Remember, these literals are frequently coming from an > #- external source that must be constrained to a given schema. > > Facundo >> I like it a lot, but not for Decimal. > > This is another face of "what do do with float" This has nothing to do with binary floating point. These are strings that map to exact numbers in Decimal; if you want to do something different with them, it's your responsibility as a programmer. Decimal should provide a solid foundation, not every bell & whistle. If you disagree, go read Cowlishaw first. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From tjreedy at udel.edu Mon Apr 19 15:00:56 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Mon Apr 19 15:00:58 2004 Subject: [Python-Dev] Re: Possible modules that could use docs References: <40837822.6060906@ocf.berkeley.edu> Message-ID: "Brett C." wrote in message news:40837822.6060906@ocf.berkeley.edu... > Discovering the 'opcode' module thanks to the new PEP 329 caused me to > wonder how many modules are undocumented and yet deserve to possibly be > (I also was procrastinating from doing schoolwork and felt like doing > some light coding =). ... [list snipped] This 'bug' (and those already filed) could be minimally/temporarily disposed of by adding a page 'Modules Not Otherwise Documented' with the list and one sentence for each (plus status if discouraged/deprecated/etc) and then adding index entries pointing to this page. Also add indication/suggestion to try interactive help('modname') for the modules for which one will get something. Then people would know that they had not missed anything to be found in the Lib Ref. tjr From pje at telecommunity.com Mon Apr 19 15:25:00 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Apr 19 15:25:12 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: References: <40841940.4010305@ocf.berkeley.edu> <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> <200404191525.i3JFPW009731@guido.python.org> <40841940.4010305@ocf.berkeley.edu> Message-ID: <5.1.1.6.0.20040419150808.02b70560@telecommunity.com> At 02:37 PM 4/19/04 -0400, Nick Bastin wrote: >You could, of course, create a statement like "const len" to flag that len >will NOT be changed, thus creating true backwards compatibility, but you'd >like to believe that const is the 95% use case, and thus it should be >"volatile foo" when you want to change foo, but that's not all that >backwards compatible. For names that are never bound by a module's code (i.e. no 'foo=', 'import foo', 'global foo'), it should be safe to make them "constant", since there is nothing that can change them now except tinkering with the builtins, or assigning them from outside modules. (The big hurdle here is 'from somewhere import *' if 'somewhere' defines any names that shadow builtins. Of course, one could simply say that using 'import *' disables this optimization.) I'm more cautious wrt module globals, however. As I've been thinking about this today, I keep finding things that will not quite work as expected if too many globals become "constants". Optimizing only builtins, however, would probably give plenty of speed improvement, and not require anything more complex than: len = len to re-enable substitution of a given builtin. But I think module objects would need to change in a way that allows a loaded module to flag what builtins are changeable, so that warnings could be generated during the transition period. IOW, module objects need to know what names they should reject for __setattr__, and whether the rejection should be a warning or error. Modules with 'from __future__ import fast_builtins' would generate errors when other code tried to modify their builtins, and the rest would issue warnings indicating that you must specifically enable that by assigning to the builtin in the target module. Hm. That means, by the way, that the compiler would need to detect these issues and track modifiable builtins in *all* modules, not just those using the future statement. But only the ones using the future statement would be able to take advantage of faster access to builtins (or builtin replacements like a "len" opcode, or compiler tricks to optimize away builtins). From bac at OCF.Berkeley.EDU Mon Apr 19 15:27:54 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Apr 19 15:28:07 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <200404191835.i3JIZAm10204@guido.python.org> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> <200404191525.i3JFPW009731@guido.python.org> <40841940.4010305@ocf.berkeley.edu> <200404191835.i3JIZAm10204@guido.python.org> Message-ID: <4084283A.8050108@ocf.berkeley.edu> Guido van Rossum wrote: > [Brett Cannon] > >>Just to make sure I am understanding this, are you suggesting a possible >>statement like ``volatile len`` to flag that len may be changed? > > > Yeah, but not that syntax. (I'm not proposing any syntax at this point.) > Right. I just wanted to write it out in some form. Didn't mean to imply that is what you were suggesting. > >>That way all "volatile"-flagged globals and one that are redefined >>in the module use LOAD_GLOBAL while all other built-ins either get >>them set to locals or use a LOAD_BUILTIN opcode? > > > Sort of, although I think you meant STORE_GLOBAL rather than > LOAD_GLOBAL. Exactly how this would be implemented is up to the > bytecode compiler; I can see that some cheap builtins (e.g. len) may > be turned into special opcodes so that no lookup is used at all. > I actually was thinking of LOAD_GLOBAL; I was thinking of ``foo = len(x)`` which loads len (global), loads x (local), calls len, and then stores into foo (local). That is what Raymond's code is optimizing; storing built-ins into the local namespace to skip having to try load from the global namespace, fail, and then check the built-in namespace (which are both slower than local since they are dictionaries and not an array like locals) (I just realized how much "talking out load" text I put in my emails; I'm a talker =). As for the cheap opcodes, it seems reasonable for the frequently used built-ins. And as for implementation, while I am thinking about it, storing built-ins into the local namespace would be the fastest, but using a LOAD_BUILTIN opcode would allow for universal overriding of built-ins if so desired. That is going to be an interesting thread if this ends up coming to fruition. =) > >>Or am I getting the use of volatile reversed (which would make more >>backwards-compatible)? > > > You're not getting it reversed. Having to flag the ones that are not > volatile would indeed be more backward compatible, but it would > require everybody to put lots of 'nonvolatile' statements in all the > time just in the hope to get faster code. That's what I'm trying to > avoid, and that's why I'm not psyched about Raymond's (or anybody's!) > proposed "make this function faster" API. Given that "volatile" > globals are very rare, it makes more sense to put the burden on the > programmer who needs his seemingly-constant globals to be reloaded. > Right. I know one of the reasons I enjoy programming in Python is that I don't have to think about optimizations when I code. If I want some non-standard use of the language I have no issue of jumping through some hoops (such as defining something as volatile) but the base case (programming for "cases 90% of the time" as Neal and Jeremy pointed out to me at PyCon) is simple and does not require extra thought. > >>If we go with the former interpretation I take it we would still >>need to get that warning Neil worked on a while back for warning >>people when someone is injecting into a module's global namespace. > > > Yeah, that warning is still the right thing to do, but it's > implementation was a nuisance because there were some legit cases that > it couldn't know about. I don't recall -- did we turn it into a > silent deprecation or did we just delete the warning? > I thought we just ripped it out since it never quite worked perfectly, but I could be wrong. -Brett From mcherm at mcherm.com Mon Apr 19 15:33:10 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Apr 19 15:33:11 2004 Subject: [Python-Dev] Re: Possible modules that could use docs Message-ID: <1082403190.4084297622d8b@mcherm.com> Terry Reedy writes: > This 'bug' (and those already filed) could be minimally/temporarily > disposed of by adding a page 'Modules Not Otherwise Documented' with the > list and one sentence for each (plus status if discouraged/deprecated/etc) > and then adding index entries pointing to this page. Also add > indication/suggestion to try interactive help('modname') for the modules > for which one will get something. Then people would know that they had not > missed anything to be found in the Lib Ref. But let's not jump into doing that just yet... perhaps Brett's listing will attract some champions willing to document these orphan modules. I'd really rather not give up on documenting a module unless we're forced to... it's tantamount to giving up on the module itself! In the future, of course, no new modules should be accepted without docs. -- Michael Chermside From bac at OCF.Berkeley.EDU Mon Apr 19 15:39:31 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon Apr 19 15:39:47 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <200404191536.i3JFasX09776@guido.python.org> References: <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> <5.2.1.1.0.20040419154633.02d12be8@pop.bluewin.ch> <5.2.1.1.0.20040419161441.02d12688@pop.bluewin.ch> <200404191536.i3JFasX09776@guido.python.org> Message-ID: <40842AF3.8070600@ocf.berkeley.edu> Guido van Rossum wrote: [SNIP] > > While Samuele's words were inappropriately rude, I'm glad this was > brought up (and even to some extent I'm glad about some of the words > that Raymond used). Raymond *does* seem to have a "speed obsession", > and while I'm sure he has Python's best interests at heart, I'm not > sure that accepting everything he proposes will actually be good for > Python. > While I was thinking about this whole thread I realized that this is just a PEP and can easily be rejected by Guido. This is not like Raymond went ahead and added the module along with changing the stdlib to use it without proposing it to python-dev first (not going to get into that and the whole heapq thing here). It looks like this thread is not going to explode into a heated argument and I for one am glad it isn't since I don't think it needs to since this is just a PEP and nothing else. This needs to be viewed as just another PEP and not as "Raymond's speed obsession has acted up again!" or something of that ilk. [SNIP] > >>PS: I'm starting to be really annoyed by the fact that Jython is >>often cited good PR but a 2nd class citizen for all the rest. I'm >>aware of the practical circumstances of why that's the case in some >>occassions, OTOH we are mostly all investing our free time on Python >>activities, it would be nice if we showed respect beyond just the >>forms and politeness. I'm back from the ACCU which was fun but >>tiring and also an occasion of some related heat taking. All I was >>looking forward is restarting to finish the new-style class work >>(started in December and January) in Jython,after the stop because >>of working on the negotations for the PyPy EU proposal together with >>the others PyPy people, which we hope will conclude positively for >>the people working on it and looking forward to restart making PyPy >>a real concrete contribution to Python future. > > > I hope you don't think *everybody* sees Jython as a 2nd class citizen. > I personally see it as very important. Certainly the PSF is > considering funding it, whether via the general grants committee (just > formed) or via a special grant. > I know I appreciate Jython as well. I don't know about everyone else, but I know I view CPython as an implementation of the Python language, just like Jython is an implementation, and thus on the same playing field in that regard. While CPython might be used as the testbed and reference implementation, it is still just an implementation of the language which is the most important thing in all of this, not the specifics of how we implement it, more or less. -Brett From jim.jewett at eds.com Mon Apr 19 15:46:31 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Mon Apr 19 15:47:06 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: On Mon, Apr 19, 2004, Jewett, Jim J wrote: > [Kevin Jacobs] > #- Decimal('2.4000', precision=2, scale=1) == Decimal('2.4') > #- Decimal('2.4', precision=5, scale=4) == Decimal('2.4000') > #- > #- Remember, these literals are frequently coming from an > #- external source that must be constrained to a given schema. > > Facundo >> I like it a lot, but not for Decimal. > This is another face of "what do do with float" Aahz: > This has nothing to do with binary floating point. These are > strings that map to exact numbers in Decimal; Under the current implementation: (0, (2, 4, 0, 0, 0), -4) is not quite the same as (0, (2, 4) -1) Given this, is should be possible for the user to specify (at creation) which is desired. They do normalize to the same thing, but this requires extending the second number with zeros. (0, (1,), -17) (0, (1, 1), -1) The float discussions make it clear that "extend by zero" may not always be the most accurate, and the python docs do not yet prohibit extending by some other digit (that rounds the same). I agree that the programmer should specify if they want something odd -- and that is precisely why the constructor should take an optional context argument which allows the user to specify precision. Given that float input may well look like string (saved to a file by another program) or even an integer (C double may have more precision than C long) there is no particular reason to say "context is only allowed when constructing from a float" -jJ From mcherm at mcherm.com Mon Apr 19 15:53:08 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Mon Apr 19 15:53:13 2004 Subject: [Python-Dev] Possible modules that could use docs Message-ID: <1082404388.40842e247bc8a@mcherm.com> See http://www.python.org/cgi-bin/moinmoin/ModulesThatNeedDocs -- Michael Chermside From fdrake at acm.org Mon Apr 19 16:01:46 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon Apr 19 16:01:58 2004 Subject: [Python-Dev] Re: Possible modules that could use docs In-Reply-To: References: <40837822.6060906@ocf.berkeley.edu> Message-ID: <200404191601.46411.fdrake@acm.org> On Monday 19 April 2004 03:00 pm, Terry Reedy wrote: > This 'bug' (and those already filed) could be minimally/temporarily > disposed of by adding a page 'Modules Not Otherwise Documented' with the > list and one sentence for each (plus status if discouraged/deprecated/etc) There's already a whole chapter like that; I'd really rather the modules get documented than addiing to that, but listing them there is better than releasing them without any docs. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From martin at v.loewis.de Mon Apr 19 16:03:18 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon Apr 19 16:03:35 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: <16E1010E4581B049ABC51D4975CEDB8803060E63@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB8803060E63@UKDCX001.uk.int.atosorigin.com> Message-ID: <40843086.9000705@v.loewis.de> Moore, Paul wrote: > I don't know if anyone has stats on how many of the 3rd party > extension authors who currently provide Windows binaries have > access to MSVC7, and so can make the switch. It may be "all of > them". I don't have MSVC7, although I do have MSVC6 (my company, > or at least my group, never upgraded) but I've no idea how unusual > my situation is. A heads-up on c.l.p would probably be worth it, > but it's also likely to generate a huge amount of FUD, and anti- > Microsoft rants. Given that the Python release is still several months ahead, I'd advise against such a posting. Personally, I don't know a single VC6 user who doesn't also have a copy of VC.NET 2003 available (except for you, whom I only know via email :-) In any case, the only possible change out of this discussion is that Python 2.4 would be built with VC6. I personally know a few people which have VC.NET 2003, but not VC6, because you cannot purchase the latter one, anymore. So, standardizing on .NET 2003 is a good thing: it means that .NET (2002) gets skipped (and widely so, beyond Python); it may also mean that .NET 2005 (aka Whidbey) might get skipped, in favour of then-.NET 2006. Regards, Martin From FBatista at uniFON.com.ar Mon Apr 19 16:08:03 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Mon Apr 19 16:10:30 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Jewett, Jim J] #- Under the current implementation: #- #- (0, (2, 4, 0, 0, 0), -4) #- #- is not quite the same as #- #- (0, (2, 4) -1) #- #- Given this, is should be possible for the user to specify #- (at creation) which is desired. It *is* posible: >>> Decimal('2.4000') Decimal( (0, (2, 4, 0, 0, 0), -4) ) >>> Decimal('2.4') Decimal( (0, (2, 4), -1) ) >>> #- I agree that the programmer should specify if they want #- something odd -- and that is precisely why the constructor #- should take an optional context argument which allows the #- user to specify precision. If you construct using precision, and the precision is smaller than the quantity of digits you provide, you'll get rounded, but if the precision is greater than the quantity of digits you provide, you don't get filled with zeros. #- Given that float input may well look like string (saved to a #- file by another program) or even an integer (C double may have #- more precision than C long) there is no particular reason to #- say "context is only allowed when constructing from a float" No. Why we should do that? . Facundo From skip at pobox.com Mon Apr 19 17:46:18 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon Apr 19 17:46:26 2004 Subject: [Python-Dev] syncmail question - how to get submitter as From:? In-Reply-To: <16514.31295.926740.594739@montanaro.dyndns.org> References: <16511.65327.705618.504369@montanaro.dyndns.org> <16512.8565.21289.392919@montanaro.dyndns.org> <16512.20616.41799.327583@montanaro.dyndns.org> <200404171646.26885.fdrake@acm.org> <16514.31295.926740.594739@montanaro.dyndns.org> Message-ID: <16516.18602.256777.317163@montanaro.dyndns.org> Fred> This is something syncmail should probably support out of the box. Fred> If you'd like to submit a patch, we'd love to have it: Fred> http://sourceforge.net/projects/cvs-syncmail/ Skip> Will do when I'm back at work Monday. http://sourceforge.net/tracker/index.php?func=detail&aid=938197&group_id=47611&atid=450021 This is the simplest possible patch (no command-line flag). Just grab CVS_USER and fall back to the user running syncmail if that's not available. Skip From nas at arctrix.com Mon Apr 19 19:02:18 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Mon Apr 19 19:02:23 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <200404191835.i3JIZAm10204@guido.python.org> References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> <200404191525.i3JFPW009731@guido.python.org> <40841940.4010305@ocf.berkeley.edu> <200404191835.i3JIZAm10204@guido.python.org> Message-ID: <20040419230218.GA11314@mems-exchange.org> On Mon, Apr 19, 2004 at 11:35:10AM -0700, Guido van Rossum wrote: > Yeah, that warning is still the right thing to do, but it's > implementation was a nuisance because there were some legit cases that > it couldn't know about. I don't recall -- did we turn it into a > silent deprecation or did we just delete the warning? We ripped it out (too many spurious warnings). Neil From greg at cosc.canterbury.ac.nz Mon Apr 19 20:13:54 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon Apr 19 20:14:30 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants inthe Standard Library In-Reply-To: <5.1.1.6.0.20040419122230.02fdab40@telecommunity.com> Message-ID: <200404200013.i3K0DsrZ028600@cosc353.cosc.canterbury.ac.nz> "Phillip J. Eby" : > It's already backwards-incompatible. Currently, you can change the > behavior of a module by modifying its attributes. A module that uses this > binding approach is immune to changes in its globals. This is a > *seriously* backward incompatible change, as it essentially disallows > monkeypatching. This suggests that there should be some way of disabling the optimisation from *outside* the module. Not sure what that would be, though. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Mon Apr 19 20:23:18 2004 From: guido at python.org (Guido van Rossum) Date: Mon Apr 19 20:23:25 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants inthe Standard Library In-Reply-To: Your message of "Tue, 20 Apr 2004 12:13:54 +1200." <200404200013.i3K0DsrZ028600@cosc353.cosc.canterbury.ac.nz> References: <200404200013.i3K0DsrZ028600@cosc353.cosc.canterbury.ac.nz> Message-ID: <200404200023.i3K0NIu10904@guido.python.org> > This suggests that there should be some way of disabling > the optimisation from *outside* the module. Not sure what > that would be, though. A command line flag? --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Mon Apr 19 20:43:49 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon Apr 19 20:43:32 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants inthe Standard Library In-Reply-To: <200404200023.i3K0NIu10904@guido.python.org> References: <200404200013.i3K0DsrZ028600@cosc353.cosc.canterbury.ac.nz> Message-ID: <5.1.1.6.0.20040419203338.0339fe60@mail.telecommunity.com> At 05:23 PM 4/19/04 -0700, Guido van Rossum wrote: > > This suggests that there should be some way of disabling > > the optimisation from *outside* the module. Not sure what > > that would be, though. > >A command line flag? For that to work, there'd need to be some place to store the names of the modules to which one wished to apply this "pessimization". That being the case, one might as well expose an API to Python for that, rather than forcing the issue up to the command line and thereby reducing modularity, as well as possibly being an issue for embedders, Jython, et al. Anyway, the API would give you a function to call if you want to disable optimization for a module in order to monkeypatch it. That function would then raise an error if the module in question was already imported with optimization enabled, or if the runtime environment disallows such changes (e.g. if Jython, IronPython, or various hypothetical Python-to-static-language translators want to disallow monkeypatching). One possibly interesting use case is Pyrex, which would like to be able to disallow modifications to built-ins in order to translate e.g. 'len()' into C. It would be nice if there's a way for C modules to "play nice" with the mechanisms under discussion. Specifically, if there's a way for them to declare metadata of the same sort that's contemplated for indicating what attributes of a module are modifiable from outside. From pedronis at bluewin.ch Tue Apr 20 06:49:34 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Tue Apr 20 06:44:29 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants inthe Standard Library In-Reply-To: <006f01c426c2$56978ce0$27b2958d@oemcomputer> References: <5.2.1.1.0.20040419161441.02d12688@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20040420124302.02a7b2e0@pop.bluewin.ch> At 06:29 20.04.2004 -0400, Raymond Hettinger wrote: > > he's proposing an hack that would speed up CPython and slow down >Jython > > (because it would result in a no-op but OTOH he would like to remove >the > > code > > when some globals are bound to locals for speed reason. Of course >that's > > not nice too > > but replacing a hack with another is just sort of a win. Honestly >there > > are > > probably > > very convoluted ways to make this, in this form, a non no-op on Jython > > too, > > not that I would like to > > implement them or that Raymond cared). > >Factually false on both counts. > >I do care. Jython was a key design consideration from the beginning. >Guido and I specifically went over the non-CPython implications before >the PEP was submitted. > >The no-op is automatic and makes bind_all equal to lambda *args, **kwds: >0. It is invoked only the first time a module is imported. The time >difference is unmeasurably miniscule. from the PEP """ The library contains code such as _len=len which is intended to create fast local references instead of slower global lookups. Though necessary for performance, these constructs clutter the code and are usually incomplete (missing many opportunities). If the proposal is adopted, those constructs could be eliminated from the code base and at the same time improve upon their results in terms of performance. """ if they are eliminated (_len = len), then CPython will win because of the binding at load time and the fact that time is not lost at each function call for binding them, Jython will loose something because if they were there, there was likely a reason. From mwh at python.net Tue Apr 20 06:59:28 2004 From: mwh at python.net (Michael Hudson) Date: Tue Apr 20 06:59:31 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in the Standard Library In-Reply-To: <5.1.1.6.0.20040419103223.02a51be0@mail.telecommunity.com> (Phillip J. Eby's message of "Mon, 19 Apr 2004 10:39:06 -0400") References: <5.1.1.6.0.20040419091544.02cc6ce0@mail.telecommunity.com> <5.1.1.6.0.20040419103223.02a51be0@mail.telecommunity.com> Message-ID: <2mr7uiykxb.fsf@starship.python.net> "Phillip J. Eby" writes: > At 09:55 AM 4/19/04 -0400, Raymond Hettinger wrote: >> > I'm not sure I understand the reasoning for implementing this via a >> > bytecode hack, rather than in the compiler or loading mechanisms. >> >>The reason is that it is not always possible to know (in general) what >>things constants. Take a look at the ASPN recipe example, it ought to >>clarify for you how this works. > > I don't need clarification about the mechanism; I independently > implemented *exactly* this hack myself a couple of years ago. (I just > didn't propose it become part of standard Python.) See 'bind_func()' > in: > > http://cvs.eby-sarna.com/PEAK/src/peak/util/Code.py?rev=1.1.2.1&content-type=text/vnd.viewcvs-markup Here was me thinking I was the only one nuts enough to implement the bytecodehacks, and I find everyone's at it! Cheers, mwh -- Academic politics is the most vicious and bitter form of politics, because the stakes are so low. -- Wallace Sayre From jacobs at theopalgroup.com Tue Apr 20 08:04:16 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Tue Apr 20 08:04:20 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <408511C0.8000101@theopalgroup.com> Batista, Facundo wrote: >[Jewett, Jim J] >#- Under the current implementation: >#- >#- (0, (2, 4, 0, 0, 0), -4) >#- >#- is not quite the same as >#- >#- (0, (2, 4) -1) >#- >#- Given this, is should be possible for the user to specify >#- (at creation) which is desired. > >It *is* posible: > > >>>>Decimal('2.4000') >>>> >>>> >Decimal( (0, (2, 4, 0, 0, 0), -4) ) > > >>>>Decimal('2.4') >>>> >>>> >Decimal( (0, (2, 4), -1) ) > > Great!. One of my previous posts specifically listed that I didn't want to have to pre-parse and reformulate string literals to achieve the desired precision and scale. The "external" library solution will either have to do that, or create a Decimal from the string literal, look at the tuple length and exponent and form a new Decimal instance, rounding if necessary. Hmmm... sounds like something low-level enough that Decimal should be doing it. And remember, SQL and many financial applications will be doing this operation for every datum they load. >#- I agree that the programmer should specify if they want >#- something odd -- and that is precisely why the constructor >#- should take an optional context argument which allows the >#- user to specify precision. > >If you construct using precision, and the precision is smaller than the >quantity of digits you provide, you'll get rounded, but if the precision is >greater than the quantity of digits you provide, you don't get filled with >zeros. > > Rounding is exactly what should be done if one exceeds the desired precision. Using less that the desired precision (i.e., not filling in zeros) may be okay for many applications. This is because any operations on the value will have to be performed with the precision defined in the decimal context. Thus, the results will be identical, other than that the Decimal instance may not store the maximum precision available by the schema. Thanks, -Kevin From aleaxit at yahoo.com Tue Apr 20 08:29:37 2004 From: aleaxit at yahoo.com (Alex Martelli) Date: Tue Apr 20 08:29:44 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: <40843086.9000705@v.loewis.de> References: <16E1010E4581B049ABC51D4975CEDB8803060E63@UKDCX001.uk.int.atosorigin.com> <40843086.9000705@v.loewis.de> Message-ID: <200404201429.37605.aleaxit@yahoo.com> On Monday 19 April 2004 10:03 pm, Martin v. L?wis wrote: > Moore, Paul wrote: > > I don't know if anyone has stats on how many of the 3rd party > > extension authors who currently provide Windows binaries have > > access to MSVC7, and so can make the switch. It may be "all of > > them". I don't have MSVC7, although I do have MSVC6 (my company, > > or at least my group, never upgraded) but I've no idea how unusual > > my situation is. A heads-up on c.l.p would probably be worth it, > > but it's also likely to generate a huge amount of FUD, and anti- > > Microsoft rants. > > Given that the Python release is still several months ahead, > I'd advise against such a posting. Personally, I don't know > a single VC6 user who doesn't also have a copy of VC.NET 2003 > available (except for you, whom I only know via email :-) Add me to the set of VC6 users who don't have VC.NET 2003 (basically because the Windows I run is still Win98 -- as I can run it under Linux with cheap win4lin -- and VC.NET doesn't support that old Win version). Still, I suspect my particular reason is rare -- and from this thread, if I understand it correctly, I gather that if you do run Win/XP, as most do these days of course, it doesn't cost extra money to get a compiler able to build Python extensions. > In any case, the only possible change out of this discussion > is that Python 2.4 would be built with VC6. I personally know > a few people which have VC.NET 2003, but not VC6, because you > cannot purchase the latter one, anymore. So, standardizing > on .NET 2003 is a good thing: it means that .NET (2002) gets Yes, if VC6 cannot be purchased then it would be too backwards-looking for Python 2.4 to require it from extension-authors. Presumably, for a Python release coming out in 2004, the fact that you cannot build extensions if the only Windows version you run is win98 is acceptable (sigh). Alex From theller at python.net Tue Apr 20 08:49:20 2004 From: theller at python.net (Thomas Heller) Date: Tue Apr 20 08:49:26 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: <200404201429.37605.aleaxit@yahoo.com> (Alex Martelli's message of "Tue, 20 Apr 2004 14:29:37 +0200") References: <16E1010E4581B049ABC51D4975CEDB8803060E63@UKDCX001.uk.int.atosorigin.com> <40843086.9000705@v.loewis.de> <200404201429.37605.aleaxit@yahoo.com> Message-ID: >> In any case, the only possible change out of this discussion >> is that Python 2.4 would be built with VC6. I personally know >> a few people which have VC.NET 2003, but not VC6, because you >> cannot purchase the latter one, anymore. So, standardizing >> on .NET 2003 is a good thing: it means that .NET (2002) gets > > Yes, if VC6 cannot be purchased then it would be too backwards-looking for > Python 2.4 to require it from extension-authors. It may be that it's no longer possible to buy MSVC6 separately, but it is still on the MSDN professional DVD set (even 1.5.2, the 16-bit compiler). Thomas From theller at python.net Tue Apr 20 08:54:54 2004 From: theller at python.net (Thomas Heller) Date: Tue Apr 20 08:54:59 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: (Paul Moore's message of "Sun, 18 Apr 2004 17:13:56 +0100") References: Message-ID: Paul Moore writes: > "Tim Peters" writes: > >> Thanks to David Abrahams for passing on this info: a free optimizing >> cmdline VC 7.1 has been released by MS: >> >> http://msdn.microsoft.com/visualc/vctoolkit2003/ >> >> The 32MB download appears to contain much of the meat of Visual Studio .NET >> 2003 Professional, including the compiler, linker, static-link libraries for >> the C, C++ & STL runtimes, and the .NET Common Language Runtime. > > Sadly, a test seems to show that the /MD option links with msvcrt.dll, > and not with msvcr71.dll. Although it seems *not even* msvcrt.lib is included. The lib directory of my vctoolkit installation only contains: 10.08.2002 05:00 191.866 kernel32.lib 21.02.2003 02:58 2.707.332 libc.lib 18.03.2003 19:58 3.023.122 libcd.lib 18.03.2003 19:58 94.208 libcd.pdb 21.02.2003 03:01 2.937.240 libcmt.lib 21.02.2003 02:58 3.604.302 libcp.lib 18.03.2003 19:58 4.433.724 libcpd.lib 18.03.2003 19:58 225.280 libcpd.pdb 21.02.2003 03:02 3.654.594 libcpmt.lib 20.02.2003 19:38 18.618 mscoree.lib 21.02.2003 02:53 69.512 oldnames.lib Thomas From FBatista at uniFON.com.ar Tue Apr 20 09:28:48 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 20 09:31:13 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Kevin Jacobs] #- >[Jewett, Jim J] #- >#- Under the current implementation: #- >#- (0, (2, 4, 0, 0, 0), -4) #- >#- is not quite the same as #- >#- (0, (2, 4) -1) #- >#- Given this, is should be possible for the user to specify #- >#- (at creation) which is desired. #- > #- >It *is* posible: #- > #- >>>>Decimal('2.4000') #- >Decimal( (0, (2, 4, 0, 0, 0), -4) ) #- > #- >>>>Decimal('2.4') #- >Decimal( (0, (2, 4), -1) ) #- > #- #- Great!. One of my previous posts #- specifically listed #- that I didn't want to have to pre-parse and reformulate #- string literals to #- achieve the desired precision and scale. The "external" what? :p I still don't understand why do you want that. #- >If you construct using precision, and the precision is #- smaller than the #- >quantity of digits you provide, you'll get rounded, but if #- the precision is #- >greater than the quantity of digits you provide, you don't #- get filled with #- >zeros. #- #- Rounding is exactly what should be done if one exceeds the desired #- precision. Using #- less that the desired precision (i.e., not filling in zeros) #- may be okay #- for many applications. #- This is because any operations on the value will have to be #- performed #- with the precision #- defined in the decimal context. Thus, the results will be #- identical, #- other than that the #- Decimal instance may not store the maximum precision #- available by the #- schema. If I don't misunderstand, you're saying that store additional zeroes is important to your future operations? Let's make an example. If I have '2.4000', I go into decimal and get: >>>Decimal('2.4000') Decimal( (0, (2, 4, 0, 0, 0), -4) ) If I have '2.4', I go into decimal and get: >>>Decimal('2.4') Decimal( (0, (2, 4), -1) ) Are you trying to say that you want Decimal to fill up that number with zeroes... >>>Decimal('2.4', scale=4) # behaviour don't intended, just an example Decimal( (0, (2, 4, 0, 0, 0), -4) ) ...just to represent that you have that precision in your measurements and reflect that in future arithmetic operations? If yes, I think that: a) '2.4' and '2.4000' will behaviour identically in future operations; b) why do you need to represent in the number the precision of your measurement? . Facundo From jacobs at theopalgroup.com Tue Apr 20 10:00:40 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Tue Apr 20 10:00:48 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <40852D08.2030801@theopalgroup.com> Batista, Facundo wrote: >[Kevin Jacobs] >#- Great!. One of my previous posts >#- specifically listed >#- that I didn't want to have to pre-parse and reformulate >#- string literals to >#- achieve the desired precision and scale. The "external" > > what? :p > >I still don't understand why do you want that. > > > It seems that Jim and I want to be able to easily create Decimal instances that conform to a pre-specified (maximum) scale and (maximum) precision. The motivation for this is clearly explained in that section of the PostgeSQL manual that I sent the other day. i.e., numeric and decimal values in SQL are specified in terms of scale and precision parameters. Thus, I would like to create decimal instances that conform to those schema -- i.e., they would be rounded appropriately and overflow errors generated if they exceeded either the maximum precision or scale. e.g.: Decimal('20000.001', precision=4, scale=0) === Decimal('20000') Decimal('20000.001', precision=4, scale=0) raises an overflow exception Decimal('20000.001', precision=5, scale=3) raises an overflow exception Decimal('200.001', precision=6, scale=3) === Decimal('200.001') Decimal('200.000', precision=6, scale=3) === Decimal('200') or Decimal('200.000') (depending on if precision and scale are interpreted as absolutes or maximums) In order to be able to accomplish this behavior in an "external" library, either the literals would have to be pre-parsed and manipulated, OR an intermediate Decimal value would be created using the raw literal, which would then used to detect overflows and then apply the necessary rounding criteria based on the desired (maximum) scale. Hopefully this is somewhat clearer. >If I don't misunderstand, you're saying that store additional zeroes is >important to your future operations? > > > Not for mine. I would be content with interpreting the scale and precision parameters as maximums rather than absolutes. However, it is important to poll other users, since their applications may be less forgiving. >>>>Decimal('2.4', scale=4) # behaviour don't intended, just an example >>>> >>>> >Decimal( (0, (2, 4, 0, 0, 0), -4) ) > >...just to represent that you have that precision in your measurements and >reflect that in future arithmetic operations? > >If yes, I think that: a) '2.4' and '2.4000' will behaviour identically in >future operations; b) why do you need to represent in the number the >precision of your measurement? > > > Neither. It is well understood that operations on Decimal instances must rely on the context. The idea here is to overflow and round correctly upon instance creation without going through a great deal of additional effort. Thanks, -Kevin From aahz at pythoncraft.com Tue Apr 20 10:42:35 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Apr 20 10:42:40 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <40852D08.2030801@theopalgroup.com> References: <40852D08.2030801@theopalgroup.com> Message-ID: <20040420144235.GB8327@panix.com> On Tue, Apr 20, 2004, Kevin Jacobs wrote: > > Neither. It is well understood that operations on Decimal instances > must rely on the context. The idea here is to overflow and round > correctly upon instance creation without going through a great deal of > additional effort. Why do you think this is a "great deal" of effort? I still have some trouble understanding why you think this should go into Decimal rather than being an add-on. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From FBatista at uniFON.com.ar Tue Apr 20 11:23:05 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Tue Apr 20 11:25:27 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Kevin Jacobs] #- Thus, I would #- like to create decimal instances that conform to those #- schema -- i.e., they #- would be rounded appropriately and overflow errors generated if they #- exceeded either the maximum precision or scale. e.g.: #- #- Decimal('20000.001', precision=4, scale=0) === Decimal('20000') #- Decimal('20000.001', precision=4, scale=0) raises an #- overflow exception #- Decimal('20000.001', precision=5, scale=3) raises an #- overflow exception #- Decimal('200.001', precision=6, scale=3) === Decimal('200.001') #- Decimal('200.000', precision=6, scale=3) === Decimal('200') or #- Decimal('200.000') #- (depending on if precision and scale are interpreted as #- absolutes or #- maximums) #- #- In order to be able to accomplish this behavior in an #- "external" library, #- either the literals would have to be pre-parsed and manipulated, OR #- an intermediate Decimal value would be created using the raw literal, #- which would then used to detect overflows and then apply the #- necessary rounding criteria based on the desired (maximum) scale. I think that is better to include this functionality outside Decimal, by using it or inheritating it (the same way I'll include all money specific uses in other class). Remember that Decimal is a module to implement the Arithmetic behaviour of decimal floating point. All other specific use cases should be outside it. #- Hopefully this is somewhat clearer. Very, thank you. #- Neither. It is well understood that operations on Decimal instances #- must rely on the #- context. The idea here is to overflow and round correctly #- upon instance #- creation without #- going through a great deal of additional effort. Don't know how much effort it will be. But I prefer that before bloating Decimal. In the PEP I said: "It is not the purpose of this PEP to have a data type that can be used as Money without further effort". This is applicable to other specific uses. . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040420/557b709f/attachment.html From jacobs at theopalgroup.com Tue Apr 20 11:28:04 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Tue Apr 20 11:28:09 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <20040420144235.GB8327@panix.com> References: <40852D08.2030801@theopalgroup.com> <20040420144235.GB8327@panix.com> Message-ID: <40854184.7040009@theopalgroup.com> Aahz wrote: >On Tue, Apr 20, 2004, Kevin Jacobs wrote: > > >>Neither. It is well understood that operations on Decimal instances >>must rely on the context. The idea here is to overflow and round >>correctly upon instance creation without going through a great deal of >>additional effort. >> >> > >Why do you think this is a "great deal" of effort? I still have some >trouble understanding why you think this should go into Decimal rather >than being an add-on. > > It could be an add-on, but it seems a common and fundamental enough operation that it should be well supported by the core library. External implementations may be less efficient, as they cannot take advantage of the internal implementation details that a better integrated solution would offer. This isn't something I am willing to go to war on, but at the same time, I'm willing to expend some effort to lobby for inclusion. Either way, I will have the necessary infrastructure to accomplish my aims, though my goal is for everyone to have it without re-inventing the wheel. Silence on this topic benefits nobody. Thanks, -Kevin From guido at python.org Tue Apr 20 11:31:25 2004 From: guido at python.org (Guido van Rossum) Date: Tue Apr 20 11:31:41 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: Your message of "Tue, 20 Apr 2004 05:07:36 PDT." References: Message-ID: <200404201531.i3KFVP612202@guido.python.org> > Adopt Jack Diederich's suggested module name. I think pragma.py is a poor name for this, because (a) pragma is a candidate keyword (it has keyword status in most languages that have it) and (b) the word pragma implies compiler directives of any kind, not just the specific function proposed in this PEP. Also, a heads up: unless this PEP gets a lot more support from folks whose first name isn't Raymond, I'm going to reject it. --Guido van Rossum (home page: http://www.python.org/~guido/) From ark-mlist at att.net Tue Apr 20 11:33:48 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Tue Apr 20 11:33:42 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants inthe Standard Library In-Reply-To: Message-ID: <001901c426ec$e00e7e00$6402a8c0@arkdesktop> > You could, of course, create a statement like "const len" to flag that > len will NOT be changed, thus creating true backwards compatibility, Somehow this idea is getting tangled in my mind with the distinction between mutable and immutable objects. When you use an object as a dict key, it must not change, in order to allow the optimization that keys can be sought through hashing rather than by sequential search. Similarly, making a name such as True immutable allows the optimization that "while True:" can be determined during compilation to be unconditional. I understand that there is a difference between the kinds of immutability, but still there seems to be a strong connection here. From pje at telecommunity.com Tue Apr 20 11:58:21 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 20 11:58:32 2004 Subject: [Python-Dev] Re: No-cost optimizing VC 7.1 In-Reply-To: <200404201429.37605.aleaxit@yahoo.com> References: <40843086.9000705@v.loewis.de> <16E1010E4581B049ABC51D4975CEDB8803060E63@UKDCX001.uk.int.atosorigin.com> <40843086.9000705@v.loewis.de> Message-ID: <5.1.1.6.0.20040420115711.02463c70@telecommunity.com> At 02:29 PM 4/20/04 +0200, Alex Martelli wrote: >Yes, if VC6 cannot be purchased then it would be too backwards-looking for >Python 2.4 to require it from extension-authors. Presumably, for a Python >release coming out in 2004, the fact that you cannot build extensions if the >only Windows version you run is win98 is acceptable (sigh). I'm on Win98SE, and I built extensions with MinGW for Python 2.4, so this should not be an issue. Please see my previous posts. From jim.jewett at eds.com Tue Apr 20 11:58:44 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Tue Apr 20 11:59:04 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants in Message-ID: Brett Cannon wrote: > ``foo = len(x)`` which loads len (global), loads x (local), > calls len, and then stores into foo (local). ... > storing built-ins into the local namespace to skip having > to try load from the global namespace, fail, and then > check the built-in namespace An in-between option would be to steal an idea from Jython, and to make (some?) namespace dictionaries different from regular dictionaries. Not only do we know what sorts of keys are possible, we almost know what every key is at compile time. -jJ From pje at telecommunity.com Tue Apr 20 12:01:04 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 20 12:01:16 2004 Subject: [Python-Dev] PEP 329: Treating Builtins as Constants inthe Standard Library In-Reply-To: <001901c426ec$e00e7e00$6402a8c0@arkdesktop> References: Message-ID: <5.1.1.6.0.20040420115907.046a51f0@telecommunity.com> At 11:33 AM 4/20/04 -0400, Andrew Koenig wrote: > > You could, of course, create a statement like "const len" to flag that > > len will NOT be changed, thus creating true backwards compatibility, > >Somehow this idea is getting tangled in my mind with the distinction between >mutable and immutable objects. When you use an object as a dict key, it >must not change, in order to allow the optimization that keys can be sought >through hashing rather than by sequential search. Similarly, making a name >such as True immutable allows the optimization that "while True:" can be >determined during compilation to be unconditional. > >I understand that there is a difference between the kinds of immutability, >but still there seems to be a strong connection here. I think it's a bad idea to confuse a read-only name binding, and the concept of an immutable object. They aren't the same thing, although the former can be implemented by a namespace that is the latter. From pje at telecommunity.com Tue Apr 20 12:16:01 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 20 12:16:12 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: <200404201531.i3KFVP612202@guido.python.org> References: Message-ID: <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> At 08:31 AM 4/20/04 -0700, Guido van Rossum wrote: > > Adopt Jack Diederich's suggested module name. > >I think pragma.py is a poor name for this, because (a) pragma is a >candidate keyword (it has keyword status in most languages that have >it) and (b) the word pragma implies compiler directives of any kind, >not just the specific function proposed in this PEP. > >Also, a heads up: unless this PEP gets a lot more support from folks >whose first name isn't Raymond, I'm going to reject it. Would it be salvageable if it were changed to: * Get rid of bytecode hacking, in favor of a change to the compiler * Optimize builtins *only*, and only those that are never assigned to by the module * use a __future__ statement to enable the behavior initially, before making it the default in a future release * have module.__setattr__ warn when shadowing a previously unshadowed builtin (unless the module uses the __future__ statement, in which case it's an error) Would this be acceptable? It seems to me that this approach would allow Jython and IronPython the option in future of replacing lookups of builtins with static field accesses and/or method calls, which would give them quite a potential performance boost. From guido at python.org Tue Apr 20 12:50:19 2004 From: guido at python.org (Guido van Rossum) Date: Tue Apr 20 12:50:27 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: Your message of "Tue, 20 Apr 2004 12:16:01 EDT." <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> References: <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> Message-ID: <200404201650.i3KGoKL12450@guido.python.org> > Would it be salvageable if it were changed to: > > * Get rid of bytecode hacking, in favor of a change to the compiler > > * Optimize builtins *only*, and only those that are never assigned to by > the module > > * use a __future__ statement to enable the behavior initially, before > making it the default in a future release > > * have module.__setattr__ warn when shadowing a previously unshadowed > builtin (unless the module uses the __future__ statement, in which case > it's an error) > > Would this be acceptable? It seems to me that this approach would > allow Jython and IronPython the option in future of replacing > lookups of builtins with static field accesses and/or method calls, > which would give them quite a potential performance boost. It is quite the opposite of the PEP! The PEP proposes a quick, very visible hack that works only for one implementation; your proposal here lays the foundation for changing the language to enable the same kind of optimizations. I like that much better, but I doubt that it is doable in the timeframe for 2.4, nor do I think it is needed. Also, your 4th bullet proposes exactly (except for the __future__ statement) what was implemented in moduleobject.c in rev 2.46 and then withdrawn in rev 2.47; it is not feasible for a number of reasons (see python-dev for the gory details; I don't recall what they were, just that they were convincing). The __future__ statement sounds like an excellent idea to me, as it enables experimentation with the new feature. One thing: we need to specify the future behavior very carefully so that other Python implementations will be able to do the right thing without having to reverse-engineer CPython. --Guido van Rossum (home page: http://www.python.org/~guido/) From jim.jewett at eds.com Tue Apr 20 13:04:44 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Tue Apr 20 13:05:13 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: Kevin Jacobs: > It is well understood that operations on Decimal instances > must rely on the context. The idea here is to overflow and > round correctly upon instance creation without going through > a great deal of additional effort. Aahz: > Why do you think this is a "great deal" of effort? (1) If your data comes from an external source, it makes sense to say "create a decimal from this" instead of "create a string from this, then a decimal from that, then another decimal from that with the right rounding." (2) If this is done by inheritance, there will quickly be diamond inheritance. With a mixin, that is probably OK, but still not desirable. (3) Is there any reason *not* to honor an optional context when creating a Decimal? The machinery is already there when creating one as the result of a calculation. The machinery is even there to change context at arbitrary times -- why not allow it on input/output, which is when you are most likely to be dealing with an external data source? -jJ From mcherm at mcherm.com Tue Apr 20 13:23:26 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Tue Apr 20 13:23:31 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: <1082481806.40855c8e821a7@mcherm.com> Kevin Jacobs writes: > It could be an add-on, but it seems a common and fundamental enough > operation > that it should be well supported by the core library. External > implementations > may be less efficient, as they cannot take advantage of the internal > implementation > details that a better integrated solution would offer. > > This isn't something I am willing to go to war on, but at the same time, I'm > willing to expend some effort to lobby for inclusion. Either way, I will > have the necessary infrastructure to accomplish my aims, though my > goal is for everyone to have it without re-inventing the wheel. Silence on > this topic benefits nobody. Okay, then I'll speak up too. To me it seems that a way of specifying the precision INDEPENDENT of the string representation is an unusual enough need that it does not merit inclusion in Decimal so long as it is easy to implement as an add-on. If I receive a bunch of numbers as strings and the precision is already correct, I can use Decimal(s) to create it. If I receive a bunch of numbers as fixed position strings and need to set a fixed precision which doesn't match what the strings contain, then simple string manipulation will work: nums = ["3.14159", "2.71828", "0.57721"] # Pad with zeros high_precision = [Decimal(num + "000") for num in nums] low_precision = [Decimal(num[0:4]) for num in nums] If I receive a bunch of numbers as variable-length or variable- precision strings or I receive precision information separately, then I may need some additional logic to create the Decimal objects. I consider that to be reasonable. Again... just one opinion, and I don't mind if you disagree. -- Michael Chermside From aahz at pythoncraft.com Tue Apr 20 13:36:34 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue Apr 20 13:36:55 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <40854184.7040009@theopalgroup.com> References: <40852D08.2030801@theopalgroup.com> <20040420144235.GB8327@panix.com> <40854184.7040009@theopalgroup.com> Message-ID: <20040420173633.GB29268@panix.com> On Tue, Apr 20, 2004, Kevin Jacobs wrote: > Aahz wrote: >>On Tue, Apr 20, 2004, Kevin Jacobs wrote: >>> >>>Neither. It is well understood that operations on Decimal instances >>>must rely on the context. The idea here is to overflow and round >>>correctly upon instance creation without going through a great deal of >>>additional effort. >> >>Why do you think this is a "great deal" of effort? I still have some >>trouble understanding why you think this should go into Decimal rather >>than being an add-on. > > It could be an add-on, but it seems a common and fundamental enough > operation that it should be well supported by the core library. > External implementations may be less efficient, as they cannot take > advantage of the internal implementation details that a better > integrated solution would offer. > > This isn't something I am willing to go to war on, but at the same > time, I'm willing to expend some effort to lobby for inclusion. > Either way, I will have the necessary infrastructure to accomplish my > aims, though my goal is for everyone to have it without re-inventing > the wheel. Silence on this topic benefits nobody. How is your need here more common and fundamental than Money? I see here a repeat of the discussions around the new datetime module, where the decision was made to keep the core implementation dirt-simple, with enough hooks for people to add functionality. What makes this case different? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From pje at telecommunity.com Tue Apr 20 14:05:44 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 20 14:06:04 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: <200404201650.i3KGoKL12450@guido.python.org> References: <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> Message-ID: <5.1.1.6.0.20040420131828.04232080@telecommunity.com> At 09:50 AM 4/20/04 -0700, Guido van Rossum wrote: >It is quite the opposite of the PEP! The PEP proposes a quick, very >visible hack that works only for one implementation; your proposal >here lays the foundation for changing the language to enable the same >kind of optimizations. > >I like that much better, but I doubt that it is doable in the >timeframe for 2.4, nor do I think it is needed. Also, your 4th bullet >proposes exactly (except for the __future__ statement) what was >implemented in moduleobject.c in rev 2.46 and then withdrawn in rev >2.47; it is not feasible for a number of reasons (see python-dev for >the gory details; I don't recall what they were, just that they were >convincing). Reviewing the problems I see that the issues are with: 1) extension modules have a problem during initialization if they use setattr 2) modules and subpackages of package, that have names that shadow builtins #1 is fixable, as long as there is some kind of flag for the module's state, which is needed in order to support the __future__ statement anyway. I don't know how this would affect Jython or IronPython, but I don't think they really have "extension modules" as such. #2 is harder, because it needs sane rules. I don't think the parser should have to know about other modules. But banning modules named after builtins isn't appropriate either. OTOH, the only time this causes ambiguity is when the following conditions *all* apply: * The module is a package __init__ * The module contains functions or methods that reference the builtin name * The module does not directly import the name I'm tempted to say that this is broken code, except that it's possible for the module to import a module that then imports the module that has the conflicting name. But I believe I have a solution. See below. >The __future__ statement sounds like an excellent idea to me, as it >enables experimentation with the new feature. One thing: we need to >specify the future behavior very carefully so that other Python >implementations will be able to do the right thing without having to >reverse-engineer CPython. Here is my proposal for the semantics of "optimized builtins": * The compiler identifies names in a module that are builtin names (as defined by the language version), but are never assigned to (or otherwise declared global) in the module. It adds code at the beginning of the module that sets a module-level variable, let's say '__builtins_used__', to list those names whose use it *may* be able to optimize. (Note that this step applies to all modules compiled, not just those with the __future__ statement, and there is some additional complexity needed in the generated code to correctly handle being 'exec'-d in an existing namespace.) * If the __future__ statement is in effect, also set a '__builtins_optimized__' flag in the module dictionary, and actually implement any desired optimizations. * module.__setattr__ either warns or issues an error when setting a name listed in '__builtins_used__', depending on the status of '__builtins_optimized__'. If either is missing, the current (backward-compatible) semantics of setattr should apply. Note that if an extension module uses setattr to initialize itself, it will not break, because it does not have a '__builtins_used__' attribute. Also note that mere container packages will not break because they contain modules or packages named after builtins. Only packages which actually do something with the contained module, while also failing to bind that name, will receive warnings. Such modules can then simply add an explicit e.g. 'import .list' or 'global list' in the appropriate function(s), or use some similar approach to clarify that the named item is a module. There would be some potential pain here when new builtins are added, however, since previously-working code could break. There's a way to fix that too, but it may be a bit harsh. Issue a warning for ambiguous use of global names that are *not* builtin, but are not explicitly bound by the module. That is, if I use the name 'foo' in a function, and it is not a local, and is not declared 'global' or explicitly bound by module level code (i.e. because I am hacking globals() or because the name is a submodule), I should be warned that I should explicitly declare my intended usage. E.g. "Name 'foo' is never assigned a value: perhaps you're missing a 'global' declaration or 'import' statement?" The warning could be introduced as a PendingDeprecationWarning, upgraded to a warning for modules using the __future__ statement. This would then discourage writing such ambiguous code in future. (Oh, and speaking of ambiguity, use of 'import *' would either have to be forbidden in optimized modules, disable all optimization, or else use setattr and thus break at runtime if there's a conflict with an optimized name.) Whew. That's quite a list of things that would have to be done, but presumably we'll have to pay the piper (pyper?) sometime if we want to get to optimized builtin land someday. From kbk at shore.net Tue Apr 20 15:39:37 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Tue Apr 20 15:39:48 2004 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200404201939.i3KJdb2H017799@hydra.localdomain> Patch / Bug Summary ___________________ Patches : 265 open (+14) / 2365 closed ( +1) / 2630 total (+15) Bugs : 752 open (+14) / 3983 closed (+11) / 4735 total (+25) RFE : 134 open ( +2) / 124 closed ( +0) / 258 total ( +2) New / Reopened Patches ______________________ prefix and exec_prefix as root dir bug (2004-04-08) http://python.org/sf/931938 opened by Daniel Goertzen Implementation for PEP 318 ([as classmethod] version) (2004-04-08) http://python.org/sf/932100 opened by Mark Russell Error caused by patch #852334 (2004-04-10) http://python.org/sf/932796 opened by George Yoshida doctest: suggest the use of rawstrings for backslashes (2004-04-10) http://python.org/sf/932930 opened by Edward Loper doctest: Add Tester params to DocTestSuite (2004-04-10) http://python.org/sf/932932 opened by Edward Loper doctest: add an option to end examples w/ dedent (2004-04-10) http://python.org/sf/932933 opened by Edward Loper doctest: allow custom matchers for testing if got==want (2004-04-10) http://python.org/sf/932935 opened by Edward Loper doctest: add a special (dedented) marker for blank lines (2004-04-11) http://python.org/sf/933238 opened by Edward Loper help on re-exported names (bug 925628) (2004-04-13) http://python.org/sf/934356 opened by Jim Jewett platform-specific entropy (2004-04-13) http://python.org/sf/934711 opened by Trevor Perrin trace.py and CR at EOL (2004-04-14) CLOSED http://python.org/sf/934971 opened by Felix Wiemann sha256 module (2004-04-14) http://python.org/sf/935454 opened by Trevor Perrin CodeContext - an extension to show you where you are (2004-04-16) http://python.org/sf/936169 opened by Noam Raphael pydoc data descriptor unification (2004-04-17) http://python.org/sf/936774 opened by John Belmonte fast modular exponentiation (2004-04-17) http://python.org/sf/936813 opened by Trevor Perrin Patches Closed ______________ trace.py and CR at EOL (2004-04-14) http://python.org/sf/934971 closed by montanaro New / Reopened Bugs ___________________ re seems broken on 64-bit machines/linux (2004-04-08) http://python.org/sf/931848 opened by Mihai Ibanescu Segfault in object_reduce_ex (2004-04-08) http://python.org/sf/931877 opened by Tim Peters f.flush() fails on FreeBSD 5.2 (2004-04-08) CLOSED http://python.org/sf/931902 opened by Radim Kolar logging: need a way to discard Logger objects (2004-04-09) http://python.org/sf/932563 opened by Fred L. Drake, Jr. #!/usr/bin/python can find wrong python (2004-04-10) http://python.org/sf/932977 opened by Jack Jansen Different behaviour using super() and explicit base class. (2004-04-10) http://python.org/sf/933068 opened by Lawrence Allan Jones pty.fork() leaves slave fd's open on Solaris (2004-04-12) http://python.org/sf/933670 opened by Scott Lowrey configure not able to find ncurses/curses in Solaris (2004-04-12) http://python.org/sf/933795 opened by stefanparvu pydoc.stripid doesn't strip ID (2004-04-13) http://python.org/sf/934282 opened by Jim Jewett nametowidget throws TypeError for Tcl_Objs (2004-04-13) http://python.org/sf/934418 opened by Greg Couch configure fails at getaddrinfo() check on KAME (2004-04-14) CLOSED http://python.org/sf/934635 opened by suz@kame.net Strange behavior with '08' and '09' (2004-04-14) CLOSED http://python.org/sf/934989 opened by Pascal bdist_winist post-install script fails on non-admin Python (2004-04-14) http://python.org/sf/935091 opened by David R. Handy pkgutil doesn't understand case-senseless filesystems (2004-04-14) http://python.org/sf/935117 opened by Fred L. Drake, Jr. locale dependency of string methods undocumented (2004-04-16) http://python.org/sf/935749 opened by George Yoshida Tix not usable on windows (2004-04-15) CLOSED http://python.org/sf/935836 opened by David Stoner os.nullfilename (2004-04-16) http://python.org/sf/935915 opened by John Belmonte XMLFilterBase has bogus startElementNS (2004-04-17) http://python.org/sf/936637 opened by Magnus Lie Hetland PyNumber_And() 's description (2004-04-17) CLOSED http://python.org/sf/936827 opened by [N/A] PyNumber_InPlaceDivide()'s description (2004-04-17) http://python.org/sf/936837 opened by [N/A] Py_FilesystemDefaultEncoding leaks (2004-04-17) http://python.org/sf/936915 opened by Laszlo Toth Random.choice doesn't work with sets. (2004-04-17) http://python.org/sf/936988 opened by Jeremy Fincher configure ignores --without-pth (2004-04-17) CLOSED http://python.org/sf/937043 opened by Thomas Linden cannot find or replace umlauts (2004-04-18) CLOSED http://python.org/sf/937263 opened by Tünde Kriegl minidom.py writes TEXT_NODE wrong (2004-04-18) http://python.org/sf/937282 opened by Bernd Preusing Slicing changes item-attributes of an instance of a UserList (2004-04-19) http://python.org/sf/937700 opened by Hans Flori XMLGenerator ignores encoding in output (2004-04-19) http://python.org/sf/938076 opened by Magnus Lie Hetland Bugs Closed ___________ RecenFilesList mysteriously containing str with NULL bytes (2004-04-07) http://python.org/sf/931336 closed by kbk f.flush() fails on FreeBSD 5.2 (2004-04-08) http://python.org/sf/931902 closed by loewis Make popen raise an exception if cmd doesn't exist (2004-01-12) http://python.org/sf/875471 closed by nidoizo configure fails at getaddrinfo() check on KAME (2004-04-14) http://python.org/sf/934635 closed by perky Strange behavior with '08' and '09' (2004-04-14) http://python.org/sf/934989 closed by sjoerd Tix not usable on windows (2004-04-15) http://python.org/sf/935836 closed by loewis Malformed HTML in the FAQ Wizard (2003-07-27) http://python.org/sf/778558 closed by montanaro csv.DictReader parms inconsistent with docs (2003-08-21) http://python.org/sf/792558 closed by montanaro PyNumber_And() 's description (2004-04-17) http://python.org/sf/936827 closed by rhettinger configure ignores --without-pth (2004-04-17) http://python.org/sf/937043 closed by zarahg unicodeobjct: bad argument to internal function (2004-03-27) http://python.org/sf/924361 closed by loewis cannot find or replace umlauts (2004-04-18) http://python.org/sf/937263 closed by doerwalter New / Reopened RFE __________________ Py_INCREF/DECREF available as macros only (2004-04-19) http://python.org/sf/938302 opened by Bob Ippolito From jeremy at alum.mit.edu Tue Apr 20 16:50:35 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue Apr 20 16:51:28 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> References: <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> Message-ID: <1082494234.13726.365.camel@localhost.localdomain> On Tue, 2004-04-20 at 12:16, Phillip J. Eby wrote: > Would this be acceptable? It seems to me that this approach would allow > Jython and IronPython the option in future of replacing lookups of builtins > with static field accesses and/or method calls, which would give them quite > a potential performance boost. This is basically what IronPython does already. A sufficiently clever implementation can make the current semantics go fast enough. Thus, we'd only need changes to the compiler, not even changes to the language. This is the most attractive option to me (see PEP 267). I don't like the PEP 329 approach because it adds extra complexity to each module to work around a limitation of the current implementation that will almost surely disappear one day. Jeremy From pje at telecommunity.com Tue Apr 20 17:21:00 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue Apr 20 17:21:14 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: <1082494234.13726.365.camel@localhost.localdomain> References: <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> Message-ID: <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> At 04:50 PM 4/20/04 -0400, Jeremy Hylton wrote: >On Tue, 2004-04-20 at 12:16, Phillip J. Eby wrote: > > Would this be acceptable? It seems to me that this approach would allow > > Jython and IronPython the option in future of replacing lookups of > builtins > > with static field accesses and/or method calls, which would give them > quite > > a potential performance boost. > >This is basically what IronPython does already. A sufficiently clever >implementation can make the current semantics go fast enough. Actually, I suspect the cleverness of implementation required is what has kept anybody from actually implementing it for CPython at least. :) > Thus, >we'd only need changes to the compiler, not even changes to the >language. This is the most attractive option to me (see PEP 267). PEP 267 isn't backwards compatible either, since it requires that "Each code object would then be bound irrevocably to the module it was defined in." (And yes, there are lots of not-so-subtle problems with this, such as 'exec codeobject in dict'.) I think it would be helpful for the language spec to include the rules for potential optimizing of built-ins, and cover what happens if you attempt to replace a builtin whose dynamic nature has been optimized away. >I don't like the PEP 329 approach because it adds extra complexity to >each module to work around a limitation of the current implementation >that will almost surely disappear one day. Agreed. The thing I'm proposing really needs another PEP. Unfortunately, I need authorship or co-authorship of *another* PEP like I need another hole in my head. :) (I've been asked to help w/246 and 318 already, and have had a web container API pre-PEP on the back burner for a few months now.) From goodger at python.org Tue Apr 20 17:22:15 2004 From: goodger at python.org (David Goodger) Date: Tue Apr 20 17:22:23 2004 Subject: [Python-Dev] Re: python/nondist/peps pep-0000.txt, 1.270, 1.271 pep-0329.txt, 1.3, 1.4 In-Reply-To: References: Message-ID: <40859487.6050803@python.org> rhettinger@users.sourceforge.net wrote: > + Status > + ====== > + > + The PEP is self rejected by the author due to the unprecedented > + level of enmity it generated from developers. Oh, there's precedent. I seem to recall Guido's PEP 259, "Omit printing newline after newline", getting a lot of flak. Plenty of negative feedback from a lot of other PEPs too. So don't be so hard on the poor PEP. :-) -- David Goodger From pf_moore at yahoo.co.uk Tue Apr 20 18:49:51 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Tue Apr 20 18:49:53 2004 Subject: [Python-Dev] Re: Decimal data type issues References: <40852D08.2030801@theopalgroup.com> <20040420144235.GB8327@panix.com> <40854184.7040009@theopalgroup.com> Message-ID: Kevin Jacobs writes: > This isn't something I am willing to go to war on, but at the same > time, I'm willing to expend some effort to lobby for inclusion. > Either way, I will have the necessary infrastructure to accomplish > my aims, though my goal is for everyone to have it without > re-inventing the wheel. Silence on this topic benefits nobody. I thought I'd try to comment, based on this. After all, I do have an interest in the issue (I'm an Oracle user), and so if there is an issue, it could well affect me, so I should at least be sure I understand the point. ... and I discovered that I understand Decimal far less than I thought I did. But after some experimentation, and reading of the spec, I think that I've hit on a key point: The internal representation of a Decimal instance, and specifically the number of digits of precision stored internally, has no impact on anything, *except when the instance is converted to a string* The reason for this is that every possible operation on a Decimal instance uses context, with the sole exception of the "convert to string" operations (sections 4 and 5 of the spec). As a result of this, I'm not sure that it's valid to care about the internal representation of a Decimal instance. > It seems that Jim and I want to be able to easily create Decimal > instances that conform to a pre-specified (maximum) scale and > (maximum) precision. But here we have that same point - Decimal instances do not "conform to" a scale/precision. > The motivation for this is clearly explained in that section of the > PostgeSQL manual that I sent the other day. i.e., numeric and > decimal values in SQL are specified in terms of scale and precision > parameters. Thus, I would like to create decimal instances that > conform to those schema -- i.e., they would be rounded appropriately > and overflow errors generated if they exceeded either the maximum > precision or scale. OK, so what you are talking about is rounding during construction. Or is it? Hang on, and let's look at your examples. > e.g.: > Decimal('20000.001', precision=4, scale=0) === Decimal('20000') This works fine with the current Decimal: >>> Decimal("20000.001").round(prec=4) == Decimal("20000") True Do you dislike the need to construct an exact Decimal, and then round it? On what grounds? I got the impression that you thought it would be "hard", but I don't think the round() method is too hard to use... (Although I would say that the documentation in the PEP is currently very lacking in its coverage of how to use the type - I found the round() method after a lot of experimentation. Before the Decimal module is ready for prime time, it needs some serious documentation effort). > Decimal('20000.001', precision=4, scale=0) raises an overflow > exception Hang on - this example is the same as the previous one, but you want a different result! In any case, the General Decimal Arithmetic spec doesn't have a concept of overflow when a precision is exceeded (only when the implementation-defined maximum exponent is exceeded), so I'm not sure what you want to happen here in the context of the spec. > Decimal('20000.001', precision=5, scale=3) raises an overflow > exception A similar comment abut overflow applies here. I can imagine that you want to know if information has been lost, but that's no problem - check like this: >>> Decimal("20000.001").round(prec=5) == Decimal("20000.001") False > Decimal('200.001', precision=6, scale=3) === Decimal('200.001') Again, not an issue: >>> Decimal("200.001").round(prec=6) == Decimal("200.001") True > Decimal('200.000', precision=6, scale=3) === Decimal('200') or > Decimal('200.000') > (depending on if precision and scale are interpreted as absolutes or > maximums) This doesn't make sense, given that Decimal("200") == Decimal("200.000"). Unless your use of === is meant to imply "has the same internal representation as", in which case I don't believe that you have a right to care what the internal representation is. I've avoided considering scale too much here - Decimal has no concept of scale, only precision. But that's effectively just a matter of multiplying by the appropriate power of 10, so shouldn't be a major issue. Apologies if I've completely misunderstood or misrepresented your problem here. If it's any consolation, I've learned a lot in the process of attempting to comment. Paul. -- This signature intentionally left blank From jeremy at alum.mit.edu Tue Apr 20 21:31:13 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Tue Apr 20 21:32:02 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> References: <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> Message-ID: <1082511073.13726.538.camel@localhost.localdomain> On Tue, 2004-04-20 at 17:21, Phillip J. Eby wrote: > At 04:50 PM 4/20/04 -0400, Jeremy Hylton wrote: > >On Tue, 2004-04-20 at 12:16, Phillip J. Eby wrote: > > > Would this be acceptable? It seems to me that this approach would allow > > > Jython and IronPython the option in future of replacing lookups of > > builtins > > > with static field accesses and/or method calls, which would give them > > quite > > > a potential performance boost. > > > >This is basically what IronPython does already. A sufficiently clever > >implementation can make the current semantics go fast enough. > > Actually, I suspect the cleverness of implementation required is what has > kept anybody from actually implementing it for CPython at least. :) I'll grant you it's easier in a language like C#, but I don't think such code would be excessively complicated. The implementation needs a few levels of indirection, but not much more. > > Thus, > >we'd only need changes to the compiler, not even changes to the > >language. This is the most attractive option to me (see PEP 267). > > PEP 267 isn't backwards compatible either, since it requires that "Each > code object would then be bound irrevocably to the module it was defined > in." (And yes, there are lots of not-so-subtle problems with this, such as > 'exec codeobject in dict'.) I haven't read the text of PEP 267 in a quite a while ;-). At some point, we worked out a scheme that was completely backwards compatible. There are a couple of realistic options. One is to update the code at function definition time to insert the offsets for globals that are appropriate for the module. Another option is to initialize a table in the frame that is used for globals bindings. > I think it would be helpful for the language spec to include the rules for > potential optimizing of built-ins, and cover what happens if you attempt to > replace a builtin whose dynamic nature has been optimized away. I'm not opposed to that, but it isn't required to get good performance for globals. > >I don't like the PEP 329 approach because it adds extra complexity to > >each module to work around a limitation of the current implementation > >that will almost surely disappear one day. > > Agreed. The thing I'm proposing really needs another PEP. Unfortunately, > I need authorship or co-authorship of *another* PEP like I need another > hole in my head. :) (I've been asked to help w/246 and 318 already, and > have had a web container API pre-PEP on the back burner for a few months now.) I know how you feel, even though I've only got one PEP that I'm currently responsible for. Jeremy From tim.one at comcast.net Tue Apr 20 22:30:21 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue Apr 20 22:30:29 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <40852D08.2030801@theopalgroup.com> Message-ID: [Kevin Jacobs] ... > Hopefully this is somewhat clearer. Sorry, it really isn't to me. When you extract "a number" from one of your databases, what do you get from it, concretely? A triple of (decimal string with embedded decimal point, integer precision, integer scale)? An integer value with an integer scale? A decimal string w/o embedded decimal point and an integer scale? Etc. I gave up then after the first two examples: > ... > Thus, I would like to create decimal instances that conform to those > schema -- i.e., they would be rounded appropriately and overflow errors > generated if they exceeded either the maximum precision or scale. e.g.: > > Decimal('20000.001', precision=4, scale=0) === Decimal('20000') > Decimal('20000.001', precision=4, scale=0) raises an overflow exception The inputs on those two lines look identical to me, so I'm left more lost than before -- you can't really want Decimal('20000.001', precision=4, scale=0) *both* to return 20000 *and* raise an overflow exception. In any case, that's not what the IBM standard supports. Context must be respected in its abstract from-string operation, and maximum precision is a component of context. If context's precision is 4, then from-string('20000.001') would round to the most-significant 4 digits (according to the rounding mode specified in context), and signal both the "inexact" and "rounded" conditions. What "signal" means: if the trap-enable flags are set in context for either or both of those conditions, an exception will be raised; if the trap-enable flags for both of those conditions are clear, then the inexact-happened and rounded-happened status flags in context are set, and you can inspect them or not (as you please). That's what the standard provides. More than that would be extensions to the standard. The standard is precise about semantics, and it's plenty to implement (just) all of that at the start. From tim.one at comcast.net Tue Apr 20 23:11:13 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue Apr 20 23:11:21 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: Message-ID: [Paul Moore] > ... and I discovered that I understand Decimal far less than I thought > I did. But after some experimentation, and reading of the spec, I > think that I've hit on a key point: > > The internal representation of a Decimal instance, and > specifically the number of digits of precision stored internally, > has no impact on anything, *except when the instance is converted > to a string* > > The reason for this is that every possible operation on a Decimal > instance uses context, with the sole exception of the "convert to > string" operations (sections 4 and 5 of the spec). > > As a result of this, I'm not sure that it's valid to care about the > internal representation of a Decimal instance. It's not quite right. While this is a floating-point arithmetic, it was designed to "preserve scale" so long as precision isn't exceeded. Because you can set precision to anything, this is more powerful than it sounds at first. For example, Decimal("20.32") - Decimal("8.02") is displayed as Decimal("12.30"). That final trailing zero is effectively inherited from "the scales" of the inputs, and is entirely thanks to that the internal arithmetic is unnormalized. More info can be found here: http://www2.hursley.ibm.com/decimal/IEEE-cowlishaw-arith16.pdf However, if unnormalized floating-point is used with sufficient precision to ensure that rounding does not occur during simple calculations, then exact scale-preserving (type-preserving) arithmetic is possible, and the performance and other overheads of normalization are avoided. and in Cowlishaw's FAQ: http://www2.hursley.ibm.com/decimal/decifaq4.html#unnari Why is decimal arithmetic unnormalized? > ... > I've avoided considering scale too much here - Decimal has no concept > of scale, only precision. As above, it was carefully designed to *support* apps that need to preserve scale, but as a property that falls out of a more powerful and more general arithmetic. Note that the idea that various legacy apps *agree* on what "preserve scale" means to begin with is silly -- there's a large variety of mutually incompatible rules in use (e.g., for multiplication there's at least "the scale of the result is the sum of the scales of the multiplicands", "is the larger of the multiplicands' scales", "is the smaller of the multiplicands' scales", and "is a fixed value independent of the multiplicands' scales"). Decimal provides a single arithmetic capable of emulating all those (and more), but it's up to the app to use enough precision to begin with, and rescale according its own bizarre rules. > But that's effectively just a matter of multiplying by the > appropriate power of 10, so shouldn't be a major issue. It can also require rounding, so it's not wholly trivial. For example, what's 2.5 * 2.5? Under the "larger (or smaller) input scale" rules, it's 6.2 under "banker's rounding" or 6.3 under "European, and American tax rounding" rules. Decimal won't give either of those directly (unless precision is set to the ridiculously low 2), but it's easy to get either of them *using* Decimal (or to get 6.25 directly, which is the least surprising result to people). From pf_moore at yahoo.co.uk Wed Apr 21 03:50:12 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Wed Apr 21 03:50:20 2004 Subject: [Python-Dev] Re: Decimal data type issues References: Message-ID: "Tim Peters" writes: > [Paul Moore] >> As a result of this, I'm not sure that it's valid to care about the >> internal representation of a Decimal instance. > > It's not quite right. While this is a floating-point arithmetic, it was > designed to "preserve scale" so long as precision isn't exceeded. [...] Tim, Thanks for taking the time to clarify this. I'm going to need to think about this some more to grasp the implications, but I see where it's intended to apply. FWIW, my feeling now is that Kevin's requirement is something that can be handled by a subclass of Decimal, or a class which contains a Decimal. I'm not convinced by Kevin's suggestion that the operations needed are "hard" - code complexity can (and should) be encapsulated inside the subclass, and I don't see the need for runtime inefficiency. Specifically, I can't see why, if you can first get an (effectively, according to whatever rules you want to apply) exact Decimal representation of your "number", you can't do any further scaling and changing of precision, etc, entirely with Decimal instances, and with minimal loss of runtime efficiency. Maybe a concrete example of what Kevin is after (I have a database background, so I'm happy if it's based around SQL NUMBER datatypes) would clarify his concerns. Paul. -- This signature intentionally left blank From pf_moore at yahoo.co.uk Wed Apr 21 03:56:02 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Wed Apr 21 03:56:05 2004 Subject: [Python-Dev] Decimal - context manipulation Message-ID: <7jw9hii5.fsf@yahoo.co.uk> One thing I couldn't find an obvious way of doing with Decimal is to obtain a copy of the current context, with a different (say) precision. The Context constructor defaults from DefaultContext rather than from the current context, and I can't see a way of modifying the precision of a Context object (is the "prec" attribute intended to be publicly writeable?) I assume that copy.copy works to copy a context, although maybe an explicit copy method would be useful? Paul. -- This signature intentionally left blank From FBatista at uniFON.com.ar Wed Apr 21 09:09:54 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 21 09:12:18 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Paul Moore] #- (Although I would say that the documentation in the PEP is currently #- very lacking in its coverage of how to use the type - I found the #- round() method after a lot of experimentation. Before the Decimal #- module is ready for prime time, it needs some serious documentation #- effort). This is my fault, maybe: - Methods like round() don't need to be discussed: the Spec defines how them work, and the PEP is for implementing the PEP. - The PEP encourages to read the Spec, and round() is detailed in the Spec. - Both code and documentation will be provided when the PEP requests to be "Final". Because of these three reasons, there's a lack of definitions yet. Of course, all of that will be perfectly documented (I'll do my best) when the PEP champions to Final. . Facundo From pf_moore at yahoo.co.uk Wed Apr 21 09:56:43 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Wed Apr 21 09:56:46 2004 Subject: [Python-Dev] Re: Decimal data type issues References: Message-ID: <8ygpe8o4.fsf@yahoo.co.uk> "Batista, Facundo" writes: > This is my fault, maybe: I wasn't trying to lay blame! It was more an observation about why I suspect some of these discussions get bogged down in theory rather than providing concrete code examples. [...] > Of course, all of that will be perfectly documented (I'll do my best) when > the PEP champions to Final. If I get the time (sadly, quite unlikely) I'll do what I can to help. Paul. -- This signature intentionally left blank From FBatista at uniFON.com.ar Wed Apr 21 10:20:38 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 21 10:23:05 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Paul Moore] #- > This is my fault, maybe: #- #- I wasn't trying to lay blame! It was more an observation about why I #- suspect some of these discussions get bogged down in theory rather #- than providing concrete code examples. I wasn't trying to take the blame neither, ;) But as an unexperienced python community participant, sometimes don't know the exact steps to accomplish some things. Anyway, I'll keep asking, and asking, and asking. . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040421/6658111f/attachment.html From pje at telecommunity.com Wed Apr 21 10:50:15 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Apr 21 10:49:46 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: <1082511073.13726.538.camel@localhost.localdomain> References: <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> Message-ID: <5.1.1.6.0.20040421104017.03c8aec0@mail.telecommunity.com> At 09:31 PM 4/20/04 -0400, Jeremy Hylton wrote: >On Tue, 2004-04-20 at 17:21, Phillip J. Eby wrote: > > I think it would be helpful for the language spec to include the rules for > > potential optimizing of built-ins, and cover what happens if you > attempt to > > replace a builtin whose dynamic nature has been optimized away. > >I'm not opposed to that, but it isn't required to get good performance >for globals. I could be wrong, but it seems to me that globals shouldn't be nearly as bad for performance as builtins. A global only does one dict lookup, while builtins do two. Also, builtins can potentially be optimized away altogether (e.g. 'while True:') or converted to fast LOAD_CONST, or perhaps even a new CALL_BUILTIN opcode, assuming that adding the opcode doesn't blow the cacheability of the eval loop. From FBatista at uniFON.com.ar Wed Apr 21 11:55:00 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 21 11:57:28 2004 Subject: [Python-Dev] Decimal - context manipulation Message-ID: [Paul Moore] #- One thing I couldn't find an obvious way of doing with Decimal is to #- obtain a copy of the current context, with a different (say) #- precision. The Context constructor defaults from #- DefaultContext rather #- than from the current context, and I can't see a way of modifying the #- precision of a Context object (is the "prec" attribute intended to be #- publicly writeable?) I assume that copy.copy works to copy a context, #- although maybe an explicit copy method would be useful? >>> from Decimal import * >>> actual_context = getcontext() >>> copied_context = copy.copy(actual_context) >>> copied_context.prec = 5 >>> +Decimal('123456789') Decimal( (0, (1, 2, 3, 4, 5, 6, 7, 8, 9), 0) ) >>> actual_context.prec = 5 >>> +Decimal('123456789') Decimal( (0, (1, 2, 3, 4, 6), 4L) ) Note the + in front of the creation of Decimal: that implies an operation and that's why the context get involved. Regarding a copy() method in Context class, don't know. It's really easy to implement, but don't know if the standard preference is to get a copy method inside each data type (I'm confused about dictionaries having a copy method and lists, for example, not having it). . Facundo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ADVERTENCIA La informaci?n contenida en este mensaje y cualquier archivo anexo al mismo, son para uso exclusivo del destinatario y pueden contener informaci?n confidencial o propietaria, cuya divulgaci?n es sancionada por la ley. Si Ud. No es uno de los destinatarios consignados o la persona responsable de hacer llegar este mensaje a los destinatarios consignados, no est? autorizado a divulgar, copiar, distribuir o retener informaci?n (o parte de ella) contenida en este mensaje. Por favor notif?quenos respondiendo al remitente, borre el mensaje original y borre las copias (impresas o grabadas en cualquier medio magn?tico) que pueda haber realizado del mismo. Todas las opiniones contenidas en este mail son propias del autor del mensaje y no necesariamente coinciden con las de Telef?nica Comunicaciones Personales S.A. o alguna empresa asociada. Los mensajes electr?nicos pueden ser alterados, motivo por el cual Telef?nica Comunicaciones Personales S.A. no aceptar? ninguna obligaci?n cualquiera sea el resultante de este mensaje. Muchas Gracias. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040421/025b75b3/attachment.html From tim.one at comcast.net Wed Apr 21 12:13:16 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed Apr 21 12:13:20 2004 Subject: [Python-Dev] Decimal - context manipulation In-Reply-To: Message-ID: [Batista, Facundo] > ... > Regarding a copy() method in Context class, don't know. It's really > easy to implement, but don't know if the standard preference is to > get a copy method inside each data type (I'm confused about > dictionaries having a copy method and lists, for example, not having > it). There have always been several succinct ways to get a (shallow) copy of a list, like copied_list = list(original) and copied_list = original[:] For dicts, while you can do copied_dict = dict(original) today, at the time dict.copy() was introduced you could not do that. I usually add a .copy() method to my mutable types, like so: class MyClass: ... def __copy__(self): return build_a_shallow_copy_of(self) copy = __copy__ Note that __copy__ is a special name, and the standard copy.copy(thing) automatically invokes thing.__copy__() if thing has a __copy__() method. The "copy = __copy__" line then supplies that method directly to class users too under a conventional name. So, after the above, copy.copy(thing) and thing.copy() both work. From jim.jewett at eds.com Wed Apr 21 12:28:48 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Wed Apr 21 12:29:06 2004 Subject: [Python-Dev] peps 329, 266, 267 Message-ID: Jeremy Hylton: > I haven't read the text of PEP 267 in a quite a while ;-). > At some point, we worked out a scheme that was completely > backwards compatible. Alas, it wasn't recorded in the PEP. There are quite a few ways to change shadowing (exec object in dict, change a base class, etc). About all they have in common is that they should be rare. How about from __future__ import noshadow <==> If a name (or its immediately enclosing class) is explicitly declared dynamic, semantics are the same as today. For all other names, the compiler may assume that the nearest enclosing binding of this name will always be in the same namespace. (Names need not all be in a single namespace, but once a particular name is found, that namespace will always be the correct place to look for that name.) Results are undefined if the name is later unbound in that namespace or shadowed by a nearer enclosing namespace. Question: Should the compiler be able to assume the same *object* (=can bind as local), or only the same namespace (=can do with a single indirection, perhaps sped up with a variant of DLict). Question: Is there any reason that this should apply only to builtins, rather than to any namespace? Objection: Users can do something undefined and get "normal" results instead of a warning -- on their own platform. They can even do this strictly through changes to other modules which do not themselves import noshadow. How serious is this objection? If a warning is required, will the bookkeeping be needed even in release mode, or will it be an option depending on compiler settings? Should the compiler decline to take advantage of its freedom if it finds a namespace without noshadow somewhere in the lookup chain? -jJ From jim.jewett at eds.com Wed Apr 21 12:47:27 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Wed Apr 21 12:48:10 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Paul Moore] > I can't see why, if you can first get an (effectively, > according to whatever rules you want to apply) exact > Decimal representation of your "number", you can't do > any further scaling and changing of precision, etc, > entirely with Decimal instances, and with minimal loss > of runtime efficiency. This is effectively saying (1) Create a decimal using the default context. (2) Change the context to my custom context. (3) Perform various rounding and scaling operations. (4) Change the context back to the default. vs (1) Create a decimal using my custom context. The four-step procedure may (or may not) be done just as efficiently under the covers, but it is ugly. Is there any reason why input and output should be the only operations that do not honor an optional local context? -jJ From FBatista at uniFON.com.ar Wed Apr 21 12:54:14 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 21 12:56:43 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Jewett, Jim J] #- This is effectively saying #- #- (1) Create a decimal using the default context. #- (2) Change the context to my custom context. #- (3) Perform various rounding and scaling operations. #- (4) Change the context back to the default. #- #- vs #- #- (1) Create a decimal using my custom context. #- #- The four-step procedure may (or may not) be done just as #- efficiently under the covers, but it is ugly. #- #- Is there any reason why input and output should be the only #- operations that do not honor an optional local context? I didn't reviewed all the mails to write down the community will, but as far I recall, you *could* use context in creation time. I think that still is not clear if to use... Decimal(number, [context]) or Decimal(number) Decimal.using_context(number, [context]) ...(I prefer the laters) but I think you could use it. But: There is no such thing as "scale" in the context. . Facundo From jacobs at theopalgroup.com Wed Apr 21 12:56:35 2004 From: jacobs at theopalgroup.com (Kevin Jacobs) Date: Wed Apr 21 12:56:55 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <4086A7C3.90300@theopalgroup.com> Tim Peters wrote: >[Kevin Jacobs] >... > > >>Hopefully this is somewhat clearer. >> >> > >Sorry, it really isn't to me. When you extract "a number" from one of your >databases, what do you get from it, concretely? A triple of (decimal string >with embedded decimal point, integer precision, integer scale)? An integer >value with an integer scale? A decimal string w/o embedded decimal point >and an integer scale? Etc. > Sorry for all of the unnecessary confusion. I am in and out of meetings all of this week and have been trying to keep up several technical conversations in 5 minute breaks between sessions. As such, my examples were flawed. However, I now have _10_ minutes, to answer some of your questions, so hopefully I can explain slightly better. First, I get decimal numbers from many database adapters, flat files, XML files, in a variety of string formats, mainly. Virtually all are decimal string representations (i.e., a string of numbers with an option decimal point thrown in somewhere). Not all of them encode scale explicitly by adding trailing zeros, though most of the time do they conform to a given maximum precision. A few sources do provide decimals as an integer with an explicit decimal scale exponent. >>Thus, I would like to create decimal instances that conform to those >>schema -- i.e., they would be rounded appropriately and overflow errors >>generated if they exceeded either the maximum precision or scale. e.g.: >> >> Decimal('20000.001', precision=4, scale=0) === Decimal('20000') >> Decimal('20000.001', precision=4, scale=0) raises an overflow exception >> >> > >The inputs on those two lines look identical to me, so I'm left more lost >than before -- you can't really want Decimal('20000.001', precision=4, >scale=0) *both* to return 20000 *and* raise an overflow exception. > > Clearly not. The first example was supposed to have a precision of 5: Decimal('20000.001', precision=5, scale=0) === Decimal('20000') >In any case, that's not what the IBM standard supports. Context must be >respected in its abstract from-string operation, and maximum precision is a >component of context. If context's precision is 4, then > > from-string('20000.001') > >would round to the most-significant 4 digits (according to the rounding mode >specified in context), and signal both the "inexact" and "rounded" >conditions. What "signal" means: if the trap-enable flags are set in >context for either or both of those conditions, an exception will be raised; >if the trap-enable flags for both of those conditions are clear, then the >inexact-happened and rounded-happened status flags in context are set, and >you can inspect them or not (as you please). > > Yes -- this is what I would like to have happen, but with a short-cut to support this common operation. My previous comment about "great difficulty" was not in terms of the implementation, but rather the number of times it would have to be developed independently, if not readily available. However, I am still not aware of a trivial way to enforce a given scale when creating decimal instances. As you point out in a separate e-mail, there are many operations that in effect preserve scale due to unnormalized arithmetic operations. However, this conversation is somewhat academic since there does not seem to be a consensus that adding support for construction with scale and precision parameters are of general use. So I will create my own decimal subclass and/or utility function and be on my merry way. Thanks, -Kevin From FBatista at uniFON.com.ar Wed Apr 21 12:55:50 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 21 12:58:16 2004 Subject: [Python-Dev] Decimal - context manipulation Message-ID: [Tim Peters] #- Note that __copy__ is a special name, and the standard #- copy.copy(thing) #- automatically invokes thing.__copy__() if thing has a #- __copy__() method. #- The "copy = __copy__" line then supplies that method #- directly to class users #- too under a conventional name. So, after the above, #- copy.copy(thing) and #- thing.copy() both work. I'm +1 on introducing this behaviour to Context. . Facundo From tim.one at comcast.net Wed Apr 21 13:03:56 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed Apr 21 13:04:03 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: Message-ID: [Batista, Facundo] > ... > - Methods like round() don't need to be discussed: the Spec defines > how them work, and the PEP is for implementing the PEP. Actually, there is no round() operation in the spec. I don't remember whether there used to be, but there definitely isn't now. Doesn't mean we can't supply .round(), does mean we have to spell out what it does. I assume decimal.round(whatever) acts the same as the spec's decimal.plus() would act if "whatever" were temporarily (for the duration of plus()) folded into context. If so, that's all it needs to say. From tim.one at comcast.net Wed Apr 21 13:12:07 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed Apr 21 13:12:12 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: Message-ID: [Jewett, Jim J] > ... > Is there any reason why input and output should be the only > operations that do not honor an optional local context? While all operations in the spec except for the two to-string operations use context, no operations in the spec support an optional local context. That the Decimal() constructor ignores context by default is an extension to the spec. We must supply a context-honoring from-string operation to meet the spec. I recommend against any concept of "local context" in any operation -- it complicates the model and isn't necessary. Piles of "convenience features" should wait until people actually use this in real life, so can judge what's truly clumsy based on experience instead of speculation. From pje at telecommunity.com Wed Apr 21 13:48:33 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Apr 21 13:48:49 2004 Subject: [Python-Dev] peps 329, 266, 267 In-Reply-To: Message-ID: <5.1.1.6.0.20040421125924.03199ec0@telecommunity.com> At 12:28 PM 4/21/04 -0400, Jewett, Jim J wrote: >How about > > from __future__ import noshadow > > <==> > > If a name (or its immediately enclosing class) > is explicitly declared dynamic, semantics are > the same as today. How do you declare that something is dynamic? > For all other names, the compiler may assume that > the nearest enclosing binding of this name will > always be in the same namespace. (Names need not > all be in a single namespace, but once a particular > name is found, that namespace will always be the > correct place to look for that name.) > > Results are undefined if the name is later unbound > in that namespace or shadowed by a nearer enclosing > namespace. This actually isn't that different from my proposal for builtins, except that my proposal doesn't have "undefined results". Instead, names that are determined to be builtin are not allowed to be bound via __setattr__, and are never looked up in the globals dictionary. >Question: Should the compiler be able to assume the >same *object* (=can bind as local), or only the same >namespace (=can do with a single indirection, perhaps >sped up with a variant of DLict). > >Question: Is there any reason that this should apply >only to builtins, rather than to any namespace? Simplicity. Functions today do only three kinds of lookups: LOAD_CELL(?), LOAD_FAST and LOAD_GLOBAL. LOAD_CELL is an indirect load from a known, specific nested scope. LOAD_FAST loads from an array offset into the current frame object. LOAD_GLOBAL checks globals and then builtins. Module-level and class body code use LOAD_NAME, which looks in locals, globals, and builtins. Module and class body code is almost never important for application performance, certainly not enough to justify effort on optimization, I think. So that leaves LOAD_GLOBAL, which I suggest leaving alone in favor of adding LOAD_BUILTIN and/or CALL_BUILTIN. In other words, at that point all lookups made by functions *would* be explicit as to scope, since LOAD_GLOBAL could only fail its first lookup if the global in question was a not-yet-shadowed-but-expected-to-be builtin. >Objection: Users can do something undefined and get >"normal" results instead of a warning -- on their own >platform. They can even do this strictly through >changes to other modules which do not themselves >import noshadow. How serious is this objection? I think my proposal adequate deals with this by disallowing shadowing at runtime. And, if it was desired to disallow shadowing by modifiying globals(), then perhaps globals() and module.__dict__ could simply return a dictionary proxy that prevented modification of the disallowed values. (But the bare dictionary would still be used by the eval loop.) >If a warning is required, will the bookkeeping be >needed even in release mode, or will it be an option >depending on compiler settings? The runtime checking in my proposal only requires a bit of work at module setup, and checking when a module attribute is set. Neither of these occur frequently enough in general to be concerned about. From theller at python.net Wed Apr 21 14:52:34 2004 From: theller at python.net (Thomas Heller) Date: Wed Apr 21 14:52:43 2004 Subject: [Python-Dev] CVS problems? Message-ID: Is anyone else having CVS problems, or is it just me? The problems are: cvs update is very, very slow, or unreliable. I have to enter cvs update several times, most of the time I only get 'end of file from server: see above messages if any' without any messages. It may be related to the new network connection I have, there have been some problems with the MTU. It may be related to that I uploaded a new public ssh key to sourceforge. It may be related to that I installed plink.exe from putty additionally to the ssh.exe I have. Or is it SF? Thanks, Thomas From jim.jewett at eds.com Wed Apr 21 15:02:23 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Wed Apr 21 15:03:11 2004 Subject: [Python-Dev] peps 329, 266, 267 Message-ID: >> For all other names, the compiler may assume that >> the nearest enclosing binding of this name will >> always be in the same namespace. (Names need not >> all be in a single namespace, but once a particular >> name is found, that namespace will always be the >> correct place to look for that name.) > This actually isn't that different from my proposal > for builtins, I had been assuming that class (and instance) attribute resolution would be subject to the same speedup. If this is really only about globals and builtins, then you can just initialize each module's dictionary with a copy of builtins. (Or cache them in the module __dict__ on the first lookup, since you know where it would have gone.) This still won't catch updates to builtins, but it will eliminate the failed lookup and the second dictionary lookup. If you really want to track changes to builtin, it is still faster to echo builtin changes across each module than it would be to track every name's referrers in every module (as in PEP 266.) > Instead, names that are determined to be builtin are > not allowed to be bound via __setattr__, and > are never looked up in the globals dictionary. Some of the bugs that got the global tracking backed out involved changing __builtins__. If you only add to it, then I suppose the current method (which allows shadowing) is a reasonable fallback. It doesn't work so well if you want to remove names from builtin. In fairness, the language spec does warn that a new builtin dict may contain more than you expect, and I suppose it could be created with extra names pointed to NotImplemented instead of just raising a NameError. >>Question: Is there any reason that this should apply >>only to builtins, rather than to any namespace? > Simplicity. Functions today do only three kinds of > lookups: LOAD_CELL(?), LOAD_FAST and LOAD_GLOBAL. > LOAD_CELL is an indirect load from a known, specific > nested scope. LOAD_FAST loads from an array offset > into the current frame object. LOAD_GLOBAL checks > globals and then builtins. It could be converted to LOAD_CELL (or perhaps even LOAD_FAST) if the compiler were allowed to assume no changes in shadowing. (Including an assumption that the same dictionaries will continue to represent the globals and builtin namespaces for this code object.) > if it was desired to disallow shadowing by modifiying > globals(), then perhaps globals() and module.__dict__ > could simply return a dictionary proxy that prevented > modification of the disallowed values. (But the bare > dictionary would still be used by the eval loop.) Why not just make NameDict a subclass that has a different __setitem__? The times when eval cares should be exactly the times when you need to do extra checks. This also lets you use something like DLict (PEP 267) to move most lookup overhead to compile-time. -jJ From pje at telecommunity.com Wed Apr 21 15:27:01 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Apr 21 15:27:16 2004 Subject: [Python-Dev] peps 329, 266, 267 In-Reply-To: Message-ID: <5.1.1.6.0.20040421151250.0308db30@telecommunity.com> At 03:02 PM 4/21/04 -0400, Jewett, Jim J wrote: >I had been assuming that class (and instance) attribute >resolution would be subject to the same speedup. Nope, sorry, they're entirely unrelated lookups. >If this is really only about globals and builtins, >then you can just initialize each module's dictionary >with a copy of builtins. (Or cache them in the module >__dict__ on the first lookup, since you know where it >would have gone.) Interesting thought. The same process that currently loads the __builtins__ member could instead update the namespace directly. There's only one problem with this idea, and it's a big one: 'import *' would now include all the builtins, causing one module's builtins (or changes thereto) to propagate to other modules. Yuck. It's a pity, because I think this would otherwise be an immediately usable speedup. Indeed, if today you were to use something like: globals().update( dict([(k,v) for k,v in __builtins__.__dict__.items() if not k.startswith('_')]) ) at the top of a module, you'd get a lot of the speedup benefit. But you really better have an '__all__' definition, then. > This still won't catch updates to >builtins, but it will eliminate the failed lookup and >the second dictionary lookup. Actually, I think the language could easily live with the concept that a module's builtins are determined at module import time, given that I'm pushing for them to be determined at *compile* time. > > Instead, names that are determined to be builtin are > > not allowed to be bound via __setattr__, and > > are never looked up in the globals dictionary. > >Some of the bugs that got the global tracking backed >out involved changing __builtins__. If you only add >to it, then I suppose the current method (which allows >shadowing) is a reasonable fallback. It doesn't work >so well if you want to remove names from builtin. The idea is to simply declare that the any builtin used in a module that's known to be a builtin, is allowed to be optimized to the meaning of that builtin. This isn't just for CPython's benefit: Pyrex for example would greatly benefit from knowing whether it's safe to consider e.g. 'len()' a builtin. In effect, '__builtins__' should be considered an implementation detail, not part of the language, and screwing with it is off limits. In practice, CPython 2.x versions will need to provide backward compatibility because there is code out there that "adds new builtins". > > Simplicity. Functions today do only three kinds of > > lookups: LOAD_CELL(?), LOAD_FAST and LOAD_GLOBAL. > > LOAD_CELL is an indirect load from a known, specific > > nested scope. LOAD_FAST loads from an array offset > > into the current frame object. LOAD_GLOBAL checks > > globals and then builtins. > >It could be converted to LOAD_CELL (or perhaps even >LOAD_FAST) if the compiler were allowed to assume no >changes in shadowing. (Including an assumption that >the same dictionaries will continue to represent the >globals and builtin namespaces for this code object.) Not without also changing the dictionary type as per PEP 267 or PEP 280. LOAD_CELL and LOAD_FAST don't use dictionaries today. From jack at performancedrivers.com Wed Apr 21 15:29:41 2004 From: jack at performancedrivers.com (Jack Diederich) Date: Wed Apr 21 15:29:50 2004 Subject: [Python-Dev] CVS problems? In-Reply-To: References: Message-ID: <20040421192941.GB9062@performancedrivers.com> On Wed, Apr 21, 2004 at 08:52:34PM +0200, Thomas Heller wrote: > Is anyone else having CVS problems, or is it just me? > > The problems are: cvs update is very, very slow, or unreliable. I have > to enter cvs update several times, most of the time I only get 'end of > file from server: see above messages if any' without any messages. > [snip reasons why it might not be SF] > Or is it SF? Its SF. I get slow & intermittent failures for any project via SSH, I assume it is worse for pserver but I'm used to that. -jackdied From FBatista at uniFON.com.ar Wed Apr 21 16:03:35 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Wed Apr 21 16:06:00 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Tim Peters] #- [Batista, Facundo] #- > ... #- > - Methods like round() don't need to be discussed: the #- Spec defines #- > how them work, and the PEP is for implementing the PEP. #- #- Actually, there is no round() operation in the spec. I #- don't remember #- whether there used to be, but there definitely isn't now. You're right. My fault. #- Doesn't mean we #- can't supply .round(), does mean we have to spell out what #- it does. I #- assume decimal.round(whatever) acts the same as the spec's #- decimal.plus() #- would act if "whatever" were temporarily (for the duration #- of plus()) folded #- into context. If so, that's all it needs to say. Well, I think we must decide how it works With: >>> d = Decimal('12345.678') >>> d Decimal( (0, (1, 2, 3, 4, 5, 6, 7, 8), -3) ) >>> str(d) '12345.678' And being the syntax Decimal.round(n), we have the following options: a) n is the quantity of relevant digits of the final number (must be non negative). >>> d.round(4) Decimal( (0, (1, 2, 3, 5), 1L) ) >>> str(d.round(4)) '1.235E+4' b) n has the same behaviour that in the built in round(). >>> d.round(1) Decimal( (0, (1, 2, 3, 4, 5, 7), -1L) ) >>> str(d.round(1)) '12345.7' >>> d.round(-1) Decimal( (0, (1, 2, 3, 5), 1L) ) >>> str(d.round(-1)) '1.235E+4' What option do you all like more? . Facundo From barry at python.org Wed Apr 21 16:14:00 2004 From: barry at python.org (Barry Warsaw) Date: Wed Apr 21 16:14:05 2004 Subject: [Python-Dev] CVS problems? In-Reply-To: References: Message-ID: <1082578440.17274.104.camel@geddy.wooz.org> On Wed, 2004-04-21 at 14:52, Thomas Heller wrote: > It may be related to the new network connection I have, there have been > some problems with the MTU. It may be related to that I uploaded a new > public ssh key to sourceforge. It may be related to that I installed > plink.exe from putty additionally to the ssh.exe I have. Or is it SF? It's pretty much SF. -Barry From skip at pobox.com Wed Apr 21 15:59:24 2004 From: skip at pobox.com (Skip Montanaro) Date: Wed Apr 21 16:34:58 2004 Subject: [Python-Dev] peps 329, 266, 267 In-Reply-To: References: Message-ID: <16518.53916.22531.235406@montanaro.dyndns.org> Jim> If you really want to track changes to builtin, it is still faster Jim> to echo builtin changes across each module than it would be to Jim> track every name's referrers in every module (as in PEP 266.) This is an interesting idea but still has a problem (which I think can be worked around). Most python apps will rarely create module-level shadows of builtins, however if they do you have to recognize that fact when propagating new values out to the modules. Updating an object in builtins will look something like: def update_builtin_object(name, val): "update builtins with new value and update shadows" old = getattr(__builtins__, name) for modname in sys.modules: mod = sys.modules[modname] if hasattr(mod, name): modglob = getattr(mod, name) if modglob != old: # module has a distinct shadow - don't mess w/ it continue setattr(mod, name, val) else: # module doesn't have a copy of name - give it one setattr(mod, name, val) This guarantees that a module's globals will have a copy of each builtin. If the programmer wants to shadow the original value of a builtin object and remain immune to changes to builtins, it suffices to copy it into a different name and use that: mylen = __builtins__.len This gets you halfway there. The globals dict now has a copy of builtins. It's necessary to implement something like Jeremy's dlict object to get full performance boost. Skip From jim.jewett at eds.com Wed Apr 21 16:39:10 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Wed Apr 21 16:40:09 2004 Subject: [Python-Dev] peps 329, 266, 267 Message-ID: >> If this is really only about globals and builtins, >> then you can just initialize each module's dictionary >> with a copy of builtins. (Or cache them in the module >> __dict__ on the first lookup, since you know where it >> would have gone.) Phillip J. Eby: > Interesting thought. The same process that currently > loads the __builtins__ member could instead update the > namespace directly. > There's only one problem with this idea, and it's a big > one: 'import *' would now include all the builtins, > causing one module's builtins (or changes thereto) to > propagate to other modules. Why is this bad? The reason to import * is that you intend to use the module's members as if they were your own. If the other module actually has modified a builtin, you'll need to do the same, or the imported members won't work quite right. If you're sure you don't want everything the other module changed, then you shouldn't be using *. If you absolutely have to use * but not get builtins changed, then you can reupdate from builtins after the import. > ... declare that the any builtin used in a module > that's known to be a builtin, is allowed to be > optimized to the meaning of that builtin. > In effect, '__builtins__' should be considered an > implementation detail, not part of the language, Many builtins (None, True, KeyError) are effectively keywords, and I would agree. Others, like __debug__, are really used for intermodule communication, because there isn't any other truly global namespace. (Perhaps there should be a conventional place to look, such as a settings module?) -jJ From pje at telecommunity.com Wed Apr 21 16:54:34 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Apr 21 16:54:50 2004 Subject: [Python-Dev] peps 329, 266, 267 In-Reply-To: Message-ID: <5.1.1.6.0.20040421164413.02df82e0@telecommunity.com> At 04:39 PM 4/21/04 -0400, Jewett, Jim J wrote: > >> If this is really only about globals and builtins, > >> then you can just initialize each module's dictionary > >> with a copy of builtins. (Or cache them in the module > >> __dict__ on the first lookup, since you know where it > >> would have gone.) > >Phillip J. Eby: > > > Interesting thought. The same process that currently > > loads the __builtins__ member could instead update the > > namespace directly. > > > There's only one problem with this idea, and it's a big > > one: 'import *' would now include all the builtins, > > causing one module's builtins (or changes thereto) to > > propagate to other modules. > >Why is this bad? Because some modules are examined by software, and only the expected names belong there. For example, I believe if you run 'pydoc' on such a module, it will proceed to document all the builtins. >The reason to import * is that you intend to use the >module's members as if they were your own. If the >other module actually has modified a builtin, you'll >need to do the same, or the imported members won't >work quite right. *shudder* I'm glad the language really doesn't work in the way you just described. :) No, just because one module shadows a builtin, doesn't mean you have to follow suit. > > ... declare that the any builtin used in a module > > that's known to be a builtin, is allowed to be > > optimized to the meaning of that builtin. > > > In effect, '__builtins__' should be considered an > > implementation detail, not part of the language, > >Many builtins (None, True, KeyError) are effectively >keywords, and I would agree. > >Others, like __debug__, are really used for >intermodule communication, because there isn't >any other truly global namespace. (Perhaps >there should be a conventional place to look, >such as a settings module?) __debug__ is also a builtin, in the sense of being optimizable by the compiler, so I don't see any reason to look at it differently. In fact, isn't __debug__ *already* treated as a constant by the compilers? Python 2.2.2 (#37, Oct 14 2002, 17:02:34) [MSC 32 bit (Intel)] on win32 Type "copyright", "credits" or "license" for more information. IDLE 0.8 -- press F1 for help >>> __debug__ 1 >>> __debug__=0 SyntaxError: can not assign to __debug__ (, line 1) >>> Yep, I guess so. From pje at telecommunity.com Wed Apr 21 17:04:43 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Apr 21 17:04:57 2004 Subject: [Python-Dev] peps 329, 266, 267 In-Reply-To: <16518.53916.22531.235406@montanaro.dyndns.org> References: Message-ID: <5.1.1.6.0.20040421165458.02dfdec0@telecommunity.com> At 02:59 PM 4/21/04 -0500, Skip Montanaro wrote: > Jim> If you really want to track changes to builtin, it is still faster > Jim> to echo builtin changes across each module than it would be to > Jim> track every name's referrers in every module (as in PEP 266.) > >This is an interesting idea but still has a problem (which I think can be >worked around). Most python apps will rarely create module-level shadows of >builtins, however if they do you have to recognize that fact when >propagating new values out to the modules. Please note that: 1. None of this is needed in order to optimize builtins. Names that are unambiguously builtins can and should be optimized at compile time. Names that are ambiguous can use the existing semantics, and nothing further need be done. 2. Modifying builtins is a hack, that needs to be supported for backward compatibility, but can and should disappear in 3.0. 3. Having official rules for a Python compiler to decide whether a name should be considered builtin or global is much more broadly useful than having a trick to speed up CPython. Several projects could make use of this, including Pyrex, Jython, PyPy, and IronPython at the least. From skip at pobox.com Wed Apr 21 17:36:25 2004 From: skip at pobox.com (Skip Montanaro) Date: Wed Apr 21 17:36:00 2004 Subject: [Python-Dev] peps 329, 266, 267 In-Reply-To: <5.1.1.6.0.20040421165458.02dfdec0@telecommunity.com> References: <5.1.1.6.0.20040421165458.02dfdec0@telecommunity.com> Message-ID: <16518.59737.4238.284709@montanaro.dyndns.org> Phillip> Please note that: Phillip> 1. None of this is needed in order to optimize builtins. Names Phillip> that are unambiguously builtins can and should be optimized Phillip> at compile time. Names that are ambiguous can use the Phillip> existing semantics, and nothing further need be done. Because module A can modify module B's namespace at runtime the compiler can't tell whether an access to len() will get the builtin or a shadow version. For the above to be true the semantics of modules will have to be changed. Skip From pje at telecommunity.com Wed Apr 21 18:35:39 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed Apr 21 18:36:14 2004 Subject: [Python-Dev] peps 329, 266, 267 In-Reply-To: <16518.59737.4238.284709@montanaro.dyndns.org> References: <5.1.1.6.0.20040421165458.02dfdec0@telecommunity.com> <5.1.1.6.0.20040421165458.02dfdec0@telecommunity.com> Message-ID: <5.1.1.6.0.20040421183253.02041660@telecommunity.com> At 04:36 PM 4/21/04 -0500, Skip Montanaro wrote: > Phillip> Please note that: > > Phillip> 1. None of this is needed in order to optimize builtins. Names > Phillip> that are unambiguously builtins can and should be optimized > Phillip> at compile time. Names that are ambiguous can use the > Phillip> existing semantics, and nothing further need be done. > >Because module A can modify module B's namespace at runtime the compiler >can't tell whether an access to len() will get the builtin or a shadow >version. For the above to be true the semantics of modules will have to be >changed. Yes, as I outlined earlier this week: we should disallow modifications of this kind if the text of the modified module doesn't explicitly allow it. See: http://mail.python.org/pipermail/python-dev/2004-April/044475.html for details. From bsder at allcaps.org Wed Apr 21 19:57:23 2004 From: bsder at allcaps.org (Andrew P. Lentvorski, Jr.) Date: Wed Apr 21 19:52:53 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <20040421163120.N21089@mail.allcaps.org> On Wed, 21 Apr 2004, Batista, Facundo wrote: > #- Actually, there is no round() operation in the spec. I > #- don't remember > #- whether there used to be, but there definitely isn't now. ... > Well, I think we must decide how it works Careful. Omissions from technical specifications can be as important as inclusions. Why is there no round operation in the spec? Is it that the functionality can be implemented using existing operations? Is it that there is no good definition of the operation (I find this hard to believe)? Does it have some strange side effect that causes problems? I'm betting that the issue is that the performance of round() is dependent upon the chosen internal representation (ie. rounding a digit repesentation is O(1) but rounding a binary integer with scale could be O(n)). Removing something which is in use but has this kind of subtle implication is far harder than adding a function later which got omitted. For this pass, simply staying with the spec is more than sufficient work. Work which I heartily thank Facundo for doing. -a From tim.one at comcast.net Wed Apr 21 20:11:24 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed Apr 21 20:11:30 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: Message-ID: [Batista, Facundo] > ... > Well, I think we must decide how it works > > With: > > >>> d = Decimal('12345.678') > >>> d > Decimal( (0, (1, 2, 3, 4, 5, 6, 7, 8), -3) ) > >>> str(d) > '12345.678' > > And being the syntax Decimal.round(n), we have the > following options: > > a) n is the quantity of relevant digits of the > final number (must be non negative). > > >>> d.round(4) > Decimal( (0, (1, 2, 3, 5), 1L) ) > >>> str(d.round(4)) > '1.235E+4' > > b) n has the same behaviour that in the built in round(). > > >>> d.round(1) > Decimal( (0, (1, 2, 3, 4, 5, 7), -1L) ) > >>> str(d.round(1)) > '12345.7' > >>> d.round(-1) > Decimal( (0, (1, 2, 3, 5), 1L) ) > >>> str(d.round(-1)) > '1.235E+4' > > What option do you all like more? I like (c) best: drop round(), because it's redundant -- there are other ways to do (a) and (b) already. Perhaps (b) isn't obvious, so extending your example with what already exists: >>> dimes = Decimal.Decimal('0.1') >>> print d.quantize(dimes) 12345.7 >>> print d.quantize(Decimal.Decimal('1e1')) 1.235E+4 >>> As a bonus, the spec defines quantize in all cases (including infinite and NaN inputs, the effects on the large mass of condition flags, and so on). In general, we should be very reluctant to extend the spec at the start: it's complicated, subtle and exacting, and plenty of big brains are working hard on getting the spec exactly right. Alas, it's also still a bit of a moving target. From tim.one at comcast.net Wed Apr 21 20:44:34 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed Apr 21 20:44:39 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: Message-ID: [Paul Moore] > ... > FWIW, my feeling now is that Kevin's requirement is something that can > be handled by a subclass of Decimal, or a class which contains a > Decimal. I should emphasize that Decimal has always been intended to be an implementation of IBM's proposed standard for decimal arithmetic, not to emulate some arbitrary bag of behaviors picked from a particular database, or other app (let alone to emulate the union of all schemes in current use). I'm especially keen to keep it "pure" at the start of its life. There are extensions I'd like to see too, but I'm keeping quiet about them for now (although adding a .copy() method to context objects is clearly harmless). > I'm not convinced by Kevin's suggestion that the operations needed > are "hard" - code complexity can (and should) be encapsulated > inside the subclass, and I don't see the need for runtime > inefficiency. The current implementation is so inefficient in so many ways I'm not concerned about efficiency arguments at all now. The point is much more to get the spec implemented correctly at first. > Specifically, I can't see why, if you can first get an (effectively, > according to whatever rules you want to apply) exact Decimal > representation of your "number", you can't do any further > scaling and changing of precision, etc, entirely with Decimal > instances, and with minimal loss of runtime efficiency. Indeed, the individual digits are stored as individual decimal digits right now, so picking them apart is as cheap as it gets. > Maybe a concrete example of what Kevin is after (I have a database > background, so I'm happy if it's based around SQL NUMBER datatypes) > would clarify his concerns. That's fine -- although I don't expect it to influence the first release of Decimal (which is aimed at implementing a specific standard). From tim.one at comcast.net Wed Apr 21 21:04:44 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed Apr 21 21:04:51 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <4086A7C3.90300@theopalgroup.com> Message-ID: [Kevin Jacobs] > Sorry for all of the unnecessary confusion. ... No problem. > First, I get decimal numbers from many database adapters, flat files, > XML files, in a variety of string formats, mainly. Virtually all are > decimal string representations (i.e., a string of numbers with an option > decimal point thrown in somewhere). Then what problem are you trying to address when reading these numbers in? Is it that you don't trust, e.g., that a column of a database declared with some specific (precision, scale) pair enforced its own restrictions? Using Decimal.Decimal(string) exactly as-is today, you'll get exactly whatever number a string-of-digits-possibly-with-a-decimal-point specifies. > Not all of them encode scale explicitly by adding trailing zeros, though > most of the time do they conform to a given maximum precision. A few > sources do provide decimals as an integer with an explicit decimal scale > exponent. The spec doesn't supply any shortcuts for changing the exponent, because multiplication and division by powers of 10 are exact (barring underflow and overflow). Perhaps a shortcut for that would be handy, but it's not semantically necessary. ... > Clearly not. The first example was supposed to have a precision of 5: > > Decimal('20000.001', precision=5, scale=0) === Decimal('20000') So you're really doing a data conversion step? That is, you don't really want the numbers your data source gives you, but want to transform them first on input? You *can*, of course, it just strikes me as an odd desire. >> In any case, that's not what the IBM standard supports. Context >> must be respected in its abstract from-string operation, and maximum >> precision is a component of context. If context's precision is 4, >> then >> >> from-string('20000.001') >> >> would round to the most-significant 4 digits (according to the >> rounding mode specified in context), and signal both the "inexact" >> and "rounded" conditions. What "signal" means: if the trap-enable >> flags are set in context for either or both of those conditions, an >> exception will be raised; if the trap-enable flags for both of those >> conditions are clear, then the inexact-happened and rounded-happened >> status flags in context are set, and you can inspect them or not (as >> you please). > Yes -- this is what I would like to have happen, but with a short-cut > to support this common operation. What, exactly, is "this common operation"? Everything I described in that paragraph happens automatically as a result of a single from-string operation. Is it that you're reading in 10 numbers and want a unique precision for each one? That would surprise me. For example, if you're reading a column of a database, I'd expect that a single max precision would apply to each number in that column. > My previous comment about "great difficulty" was not in terms > of the implementation, but rather the number of times it would have to > be developed independently, if not readily available. Well, I don't know what "it" is, exactly, so I'll shut up. > However, I am still not aware of a trivial way to enforce a given > scale when creating decimal instances. Sorry, I don't even know what "enforce a given scale" *means*. If, for example, you want to round every input to the nearest penny, set context to use the rounding method you mean by "the nearest", define a penny object: penny = Decimal("0.01") and then pass that to the quantize() method on each number: to_pennies = [Decimal(n).quantize(penny) for n in input_strings] Then every result will have exactly two digits after the decimal point. If for some mysterious reason you actually want to raise an exception if any information is lost during this step, set context's inexact-trap flag once at the start. If you want to raise an exception even if just a trailing 0 is lost, set context's rounding-trap flag once at the start. > As you point out in a separate e-mail, there are many operations > that in effect preserve scale due to unnormalized arithmetic > operations. Yes. > However, this conversation is somewhat academic since there does not > seem to be a consensus that adding support for construction with > scale and precision parameters are of general use. Except I'd probably oppose them at this stage even if they were of universal use: we're trying to implement a specific standard here at the start. Note that a construction method that honors context is required by the spec, and precision is part of context. > So I will create my own decimal subclass and/or utility function and > be on my merry way. Don't forget to share it! I suspect that's the only way I'll figure out what you're after . From tim.one at comcast.net Wed Apr 21 21:14:51 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed Apr 21 21:14:57 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: Message-ID: [Batista, Facundo] > I didn't reviewed all the mails to write down the community will, but > as far I recall, you *could* use context in creation time. > > I think that still is not clear if to use... > > Decimal(number, [context]) > > or > > Decimal(number) > Decimal.using_context(number, [context]) > > ...(I prefer the laters) but I think you could use it. The second way of spelling it is fine, but there's no need to cater to an optional context argument. Decimal.using_context(input) should use the current context object. It's been a general rule so far that all operations are available as methods *of* context objects too, so in the goofy case (I say "goofy" because I predict it won't be used in real life) of wanting to force use of a particular non-current context object c, the natural spelling is (or should be) c.using_context(input) From tim.one at comcast.net Wed Apr 21 21:24:58 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed Apr 21 21:25:03 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <20040421163120.N21089@mail.allcaps.org> Message-ID: [Andrew P. Lentvorski, Jr.] > Careful. Omissions from technical specifications can be as important > as inclusions. > > Why is there no round operation in the spec? Is it that the > functionality can be implemented using existing operations? Yes. In particular, what most people will want most of the time is exactly supplied by the spec's quantize() operation (although that may not be obvious at first). > Is it that there is no good definition of the operation (I find this hard > to believe)? No. > Does it have some strange side effect that causes problems? No, although the many revisions quantize()'s definition has been thru bear witness to how subtle it is to cover every endcase. > I'm betting that the issue is that the performance of round() is > dependent upon the chosen internal representation (ie. rounding a > digit repesentation is O(1) but rounding a binary integer with scale > could be O(n)). Removing something which is in use but has this kind > of subtle implication is far harder than adding a function later > which got omitted. Yes, although it's pretty easy to trust that nothing truly essential is missing from the spec (lots of numeric experts have reviewed it, and that's still in progress). > For this pass, simply staying with the spec is more than sufficient > work. Indeed, it's a near crushing amount of work! > Work which I heartily thank Facundo for doing. Hear, hear! (For non-English readers, that means "garbanzo magnifico blotski!" .) From Paul.Moore at atosorigin.com Thu Apr 22 06:00:23 2004 From: Paul.Moore at atosorigin.com (Moore, Paul) Date: Thu Apr 22 06:00:28 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: <16E1010E4581B049ABC51D4975CEDB88052CB070@UKDCX001.uk.int.atosorigin.com> From: Tim Peters >> For this pass, simply staying with the spec is more than sufficient >> work. > > Indeed, it's a near crushing amount of work! > >> Work which I heartily thank Facundo for doing. > > Hear, hear! (For non-English readers, that means "garbanzo magnifico > blotski!" .) I agree entirely. Paul. __________________________________________________________________________ This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. __________________________________________________________________________ From john at neggie.net Thu Apr 22 08:15:22 2004 From: john at neggie.net (John Belmonte) Date: Thu Apr 22 08:15:49 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <200404191510.i3JFAxX09635@guido.python.org> References: <407C28E0.7090808@neggie.net> <4080C0C7.7070602@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> <40827869.2030300@neggie.net> <200404181721.i3IHLTp07934@guido.python.org> <40833974.10800@neggie.net> <200404191510.i3JFAxX09635@guido.python.org> Message-ID: <4087B75A.3060405@neggie.net> Guido van Rossum wrote: > But that currently doesn't work. Tbe most future-proof solution would > be to put some kind of object in the dict values whose attributes can > guide various metaclasses. Perhaps: > > class slotprop(object): > def __init__(self, **kwds): > self.__dict__.update(kwds) > > class C(object): > __slots__ = {'slot1': slotprop(doc="this is the docstring"), > 'slit2': slotprop('doc="another one")} Wouldn't it be just as future-proof to make __slots__ a list of bare slot names or objects? One advantage is that slot names don't have to be carried around externally to the objects. Moreover, using standard attribute names like __name__ and __doc__ will make the solution more future-proof. The default metaclass will handle __slots__ along these lines: for item in obj.__slots__: if isinstance(item, StringTypes): slot_name, slot_doc = item, None else: slot_name = item.__name__ slot_doc = getattr(item, '__doc__', None) In that way, the default metaclass does not impose any restrictions on what the slot objects are. Example usage: class MySlot(object): def __init__(self, name, doc=None): self.__name__ = name self.__doc__ = doc class C(object): __slots__ = ['slot1', MySlot('slot2', 'this is the docstring')] -John -- http:// ift ile.org/ From goodger at python.org Thu Apr 22 08:53:36 2004 From: goodger at python.org (David Goodger) Date: Thu Apr 22 08:53:55 2004 Subject: [Python-Dev] Re: Proposal: A more powerful alternative to ConfigParser In-Reply-To: References: Message-ID: <4087C050.5000106@python.org> [Full text included for the benefit of python-dev] Dan Gass wrote: > I have written a new module 'config.py' which can be used for > similar purposes as ConfigParser. The biggest difference is that > this fetches user configured python objects. The technique this > module supports is well thought out and provides a single solution > for the simplest to the most complex configuration problems without > sacrificing ease of use and simplicity. For these reasons I think > it should be considered for inclusion into the Python distribution. > I work in the field of embedded software development and find this > module extremely useful for unit, integration, and functional > testing of our software as well as for controlling many other > utility scripts. > > To date I have: > > 1) provided documentation in the module doc string > 2) insured made sure module is well commented and follows the coding > guidelines > 3) written a test suite to insure functionality is correct (and > remains correct) > > I am willing to do further development of this module, support of > the module and take the steps necessary to see it through the PEP > process. I am looking for feedback on whether this module should be > considered for inclusion in the standard Python distribution as well > as any technical suggestions. I have placed the GNU license on the > module but would be more than happy (and legally able) to relabel it > with the license of your choice. > > Thanks in advance for your consideration! > > Dan Gass Hi Dan, Thanks for your interest. I'm not sure why, but your message (sent April 21 around 15:40 UTC) didn't make it to the python-dev list as addressed. Typically, a new module first has to prove itself in the real world before being accepted into the standard library, especially for a module replacing existing functionality. It's usually easier to accept additions/improvements to existing modules (i.e. patches) than replacement modules. I suggest you put the module (along with docs and the test suite) up on the web and make an announcement on python-announce. Then try to generate community support. In any case, if you'd like to write a PEP for it, please read through PEP 1 and PEP 2. -- David Goodger Python Enhancement Proposal (PEP) Editor (Please cc: all PEP correspondence to .) From barry at python.org Thu Apr 22 08:59:39 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 22 08:59:53 2004 Subject: [Python-Dev] Re: Proposal: A more powerful alternative to ConfigParser In-Reply-To: <4087C050.5000106@python.org> References: <4087C050.5000106@python.org> Message-ID: <1082638778.22227.27.camel@anthem.wooz.org> On Thu, 2004-04-22 at 08:53, David Goodger wrote: > I suggest you put the module (along with docs and the test suite) up > on the web and make an announcement on python-announce. Then try to > generate community support. You may also want to upload it to PyPi: http://www.python.org/pypi -Barry From john at neggie.net Thu Apr 22 09:02:30 2004 From: john at neggie.net (John Belmonte) Date: Thu Apr 22 09:02:57 2004 Subject: [Python-Dev] Re: Proposal: A more powerful alternative to ConfigParser In-Reply-To: <4087C050.5000106@python.org> References: <4087C050.5000106@python.org> Message-ID: <4087C266.8080503@neggie.net> David Goodger wrote: > Typically, a new module first has to prove itself in the real world > before being accepted into the standard library, especially for a > module replacing existing functionality. It's usually easier to > accept additions/improvements to existing modules (i.e. patches) than > replacement modules. > > I suggest you put the module (along with docs and the test suite) up > on the web and make an announcement on python-announce. Then try to > generate community support. It may also be useful to compare with other config alternatives, such as ZConfig. -John -- http:// ift ile.org/ From barry at python.org Thu Apr 22 09:10:21 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 22 09:10:31 2004 Subject: [Python-Dev] Re: Proposal: A more powerful alternative to ConfigParser In-Reply-To: <4087C266.8080503@neggie.net> References: <4087C050.5000106@python.org> <4087C266.8080503@neggie.net> Message-ID: <1082639420.22227.41.camel@anthem.wooz.org> On Thu, 2004-04-22 at 09:02, John Belmonte wrote: > It may also be useful to compare with other config alternatives, such as > ZConfig. ZConfig takes a completely different approach than ConfigParser; it rocks but you do need to do more work (define a schema first, then a configuration file, although I usually do that in the reverse order ;). I'm hoping someday ZConfig itself (or something close to it) will make it into the standard library. -Barry From lists at hlabs.spb.ru Thu Apr 22 13:14:58 2004 From: lists at hlabs.spb.ru (Dmitry Vasiliev) Date: Thu Apr 22 09:13:35 2004 Subject: [Python-Dev] Re: Proposal: A more powerful alternative to ConfigParser In-Reply-To: <4087C050.5000106@python.org> References: <4087C050.5000106@python.org> Message-ID: <4087FD92.2070005@hlabs.spb.ru> David Goodger wrote: > Dan Gass wrote: > > I have written a new module 'config.py' which can be used for > > similar purposes as ConfigParser. The biggest difference is that [SKIP] > > Typically, a new module first has to prove itself in the real world > before being accepted into the standard library, especially for a > module replacing existing functionality. It's usually easier to > accept additions/improvements to existing modules (i.e. patches) than > replacement modules. > > I suggest you put the module (along with docs and the test suite) up > on the web and make an announcement on python-announce. Then try to > generate community support. > > In any case, if you'd like to write a PEP for it, please read through > PEP 1 and PEP 2. IMO more powerful alternative to ConfigParser is ZConfig (http://www.zope.org/Members/fdrake/zconfig/) -- Dmitry Vasiliev (dima at hlabs.spb.ru) http://hlabs.spb.ru From dmgass at hotmail.com Wed Apr 21 11:41:22 2004 From: dmgass at hotmail.com (Dan Gass) Date: Thu Apr 22 10:08:00 2004 Subject: [Python-Dev] Proposal: A more powerful alternative to ConfigParser Message-ID: I have written a new module 'config.py' which can be used for similar purposes as ConfigParser. The biggest difference is that this fetches user configured python objects. The technique this module supports is well thought out and provides a single solution for the simplest to the most complex configuration problems without sacrificing ease of use and simplicity. For these reasons I think it should be considered for inclusion into the Python distribution. I work in the field of embedded software development and find this module extremely useful for unit, integration, and functional testing of our software as well as for controlling many other utility scripts. To date I have: 1) provided documentation in the module doc string 2) insured made sure module is well commented and follows the coding guidelines 3) written a test suite to insure functionality is correct (and remains correct) I am willing to do further development of this module, support of the module and take the steps necessary to see it through the PEP process. I am looking for feedback on whether this module should be considered for inclusion in the standard Python distribution as well as any technical suggestions. I have placed the GNU license on the module but would be more than happy (and legally able) to relabel it with the license of your choice. Thanks in advance for your consideration! Dan Gass _________________________________________________________________ MSN Toolbar provides one-click access to Hotmail from any Web page – FREE download! http://toolbar.msn.com/go/onm00200413ave/direct/01/ -------------- next part -------------- A non-text attachment was scrubbed... Name: config.py Type: application/octet-stream Size: 45066 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040421/511863c2/config-0001.obj From FBatista at uniFON.com.ar Thu Apr 22 10:17:58 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 22 10:21:44 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Tim Peters] #- [Batista, Facundo] #- > Well, I think we must decide how it works #- > #- > With: #- > #- > >>> d = Decimal('12345.678') #- > >>> d #- > Decimal( (0, (1, 2, 3, 4, 5, 6, 7, 8), -3) ) #- > >>> str(d) #- > '12345.678' #- > #- > And being the syntax Decimal.round(n), we have the #- > following options: #- > #- > a) n is the quantity of relevant digits of the #- > final number (must be non negative). #- ... #- > b) n has the same behaviour that in the built in round(). #- ... #- > What option do you all like more? #- #- I like (c) best: drop round(), because it's redundant -- #- there are other #- ways to do (a) and (b) already. Perhaps (b) isn't obvious, #- so extending #- your example with what already exists: #- #- >>> dimes = Decimal.Decimal('0.1') #- >>> print d.quantize(dimes) #- 12345.7 #- >>> print d.quantize(Decimal.Decimal('1e1')) #- 1.235E+4 #- >>> The round() method has the appealing that it could has the same name/meaning/function that the built in round(). But, the PEP is for implementing the Spec, so I'm +1 to drop round(), at least in this stage. #- In general, we should be very reluctant to extend the spec #- at the start: #- it's complicated, subtle and exacting, and plenty of big #- brains are working #- hard on getting the spec exactly right. Alas, it's also #- still a bit of a #- moving target. Agree, agree. . Facundo From FBatista at uniFON.com.ar Thu Apr 22 10:20:52 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 22 10:23:23 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Tim Peters] #- The second way of spelling it is fine, but there's no need #- to cater to an #- optional context argument. Decimal.using_context(input) #- should use the #- current context object. It's been a general rule so far #- that all operations #- are available as methods *of* context objects too, so in the #- goofy case (I #- say "goofy" because I predict it won't be used in real life) #- of wanting to #- force use of a particular non-current context object c, the #- natural spelling #- is (or should be) #- #- c.using_context(input) You mean adding a method to the context, to create a Decimal using itself as context and not the one from the thread? If yes, what about c.create_decimal(number) ? And with floats? c.create_decimal_from_float(number)? Or the same method that before? . Facundo From FBatista at uniFON.com.ar Thu Apr 22 10:23:02 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 22 10:25:32 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: #- >> Work which I heartily thank Facundo for doing. #- > #- > Hear, hear! (For non-English readers, that means #- "garbanzo magnifico #- > blotski!" .) #- #- I agree entirely. Thank you all. Very much. And specially to Tim, because of heart-defending the stickness of the PEP to the Spec. As I am a non native english speaker and a number-newbie, you're extremely helpful. . Facundo From dmgass at hotmail.com Thu Apr 22 10:19:12 2004 From: dmgass at hotmail.com (Dan Gass) Date: Thu Apr 22 10:26:28 2004 Subject: [Python-Dev] Re: Proposal: A more powerful alternativeto ConfigParser Message-ID: >From: Barry Warsaw >Date: Thu, 22 Apr 2004 09:10:21 -0400 > >On Thu, 2004-04-22 at 09:02, John Belmonte wrote: > > > It may also be useful to compare with other config alternatives, such as > > ZConfig. > >ZConfig takes a completely different approach than ConfigParser; it >rocks but you do need to do more work (define a schema first, then a >configuration file, although I usually do that in the reverse order ;). > >I'm hoping someday ZConfig itself (or something close to it) will make >it into the standard library. > >-Barry Anyone who has interest in ZConfig is strongly encouraged to look at my alternative method as it is alot easier to set up and use and provides more flexibility. I'm hoping to see some cross-pollinating of ideas so your feedback would be greatly appreciated. I will be making an open source project on sourceforge in the near future (unless I hear compelling arguments to not) and will make announcements to advertise it's existance (any suggestions on posting locations is appreciated as well). Dan _________________________________________________________________ Watch LIVE baseball games on your computer with MLB.TV, included with MSN Premium! http://join.msn.com/?page=features/mlb&pgmarket=en-us/go/onm00200439ave/direct/01/ From barry at python.org Thu Apr 22 10:35:12 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 22 10:44:57 2004 Subject: [Python-Dev] Re: Proposal: A more powerful alternativeto ConfigParser In-Reply-To: References: Message-ID: <1082644511.22805.7.camel@anthem.wooz.org> On Thu, 2004-04-22 at 10:19, Dan Gass wrote: > I will be making an open source project on sourceforge in the near future > (unless I hear compelling arguments to not) and will make announcements to > advertise it's existance (any suggestions on posting locations is > appreciated as well). Sounds like a good plan! I doubt anything will get into the stdlib for Python 2.4, so it's probably a good time for a config bake-off. -Barry From fdrake at acm.org Thu Apr 22 10:32:07 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu Apr 22 10:49:35 2004 Subject: [Python-Dev] Proposal: A more powerful alternative to ConfigParser In-Reply-To: <4087C266.8080503@neggie.net> References: <4087C050.5000106@python.org> <4087C266.8080503@neggie.net> Message-ID: <200404221032.07697.fdrake@acm.org> On Thursday 22 April 2004 09:02 am, John Belmonte wrote: > It may also be useful to compare with other config alternatives, such as > ZConfig. Hey, I've heard of that! (Disclaimer: I wrote most of it.) I just released a new version as well. Information about ZConfig can be found at: http://www.zope.org/Members/fdrake/zconfig/ There's a Wiki page that discusses some future plans for it at: http://dev.zope.org/Zope3/ZConfig In particular, I'd be very interested in seeing something like ZConfig in the standard library. Some of my notes in the wiki relate specifically to what's needed to make this appropriate. I'd be very interested in comments from people outside Zope Corporation who've tried the package. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From mcherm at mcherm.com Thu Apr 22 11:47:04 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Thu Apr 22 11:47:08 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: <1082648824.4087e8f856292@mcherm.com> Tim writes: > But, the PEP is for implementing the Spec, so I'm +1 to drop > round(), at > least in this stage. Me too. But let's promise that the docs will include an example of how to use quantize() to do rounding, since the name will not be obvious to the uninitiated. -- Michael Chermside From mcherm at mcherm.com Thu Apr 22 11:50:18 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Thu Apr 22 11:50:24 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: <1082649018.4087e9ba25502@mcherm.com> Facundo writes: > > #- c.using_context(input) > > You mean adding a method to the context, to create a Decimal > using itself as > context and not the one from the thread? > > If yes, what about c.create_decimal(number) ? > > And with floats? c.create_decimal_from_float(number)? Or the > same method > that before? +1 on Context.create_decimal(float) and Context.create_decimal(string) (okay, so it's really just one method that takes two input types). The name just fits my brain. The fact that it uses the context is obvious from the fact that it's Context method. -- Michael Chermside From jeremy at alum.mit.edu Thu Apr 22 11:56:57 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu Apr 22 11:58:12 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: <5.1.1.6.0.20040421104017.03c8aec0@mail.telecommunity.com> References: <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> <5.1.1.6.0.20040421104017.03c8aec0@mail.telecommunity.com> Message-ID: <1082649417.9092.13.camel@localhost.localdomain> On Wed, 2004-04-21 at 10:50, Phillip J. Eby wrote: > I could be wrong, but it seems to me that globals shouldn't be nearly as > bad for performance as builtins. A global only does one dict lookup, while > builtins do two. Also, builtins can potentially be optimized away > altogether (e.g. 'while True:') or converted to fast LOAD_CONST, or perhaps > even a new CALL_BUILTIN opcode, assuming that adding the opcode doesn't > blow the cacheability of the eval loop. The coarse measurements I made a couple of years ago suggest that LOAD_GLOBAL is still substantially slower than LOAD_FAST. Less than 100 cycles for LOAD_FAST and about 400 cycles for LOAD_GLOBAL. http://zope.org/Members/jeremy/CurrentAndFutureProjects/PerformanceMeasurements It would be good to repeat the measurements with current Python. I suspect it's a lot harder to figure out where to measure the start and stop times. The timestamps would need to be integrated with PREDICT and fast_next_opcode, for example. Jeremy From aahz at pythoncraft.com Thu Apr 22 11:59:09 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Apr 22 11:59:15 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: References: Message-ID: <20040422155909.GA13900@panix.com> On Thu, Apr 22, 2004, Batista, Facundo wrote: > > If yes, what about c.create_decimal(number) ? Sounds good. If you're going this route, it seems to me that Decimal() would work better if it ends up being a proxy class for an internal implementation that's more closely linked to context, and that most of the Decimal methods simply delegate to the internal implementation through context. (If you look at my original implementation, that's similar to the direction I was going -- haven't actually looked at your code to see what it's like currently.) Make sure that Decimal has-a internal implementation rather than is-a (i.e. use composition instead of subclassing). > And with floats? c.create_decimal_from_float(number)? Or the same method > that before? ``from_float()`` probably is better. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From FBatista at uniFON.com.ar Thu Apr 22 12:12:47 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 22 12:16:09 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Michael Chermside] #- Me too. But let's promise that the docs will include an example #- of how to use quantize() to do rounding, since the name will #- not be obvious to the uninitiated. I'll do my best with the docs. I have in mind a concept explanation about Decimal and Context, and syntax and examples of each method. But let's promise that you'll give me feedback about them, ;) . Facundo From pje at telecommunity.com Thu Apr 22 12:18:08 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Apr 22 12:18:31 2004 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0329.txt, 1.2, 1.3 In-Reply-To: <1082649417.9092.13.camel@localhost.localdomain> References: <5.1.1.6.0.20040421104017.03c8aec0@mail.telecommunity.com> <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420120134.028b2a60@telecommunity.com> <5.1.1.6.0.20040420170833.02781dd0@telecommunity.com> <5.1.1.6.0.20040421104017.03c8aec0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040422120308.02aa39a0@telecommunity.com> At 11:56 AM 4/22/04 -0400, Jeremy Hylton wrote: >On Wed, 2004-04-21 at 10:50, Phillip J. Eby wrote: > > I could be wrong, but it seems to me that globals shouldn't be nearly as > > bad for performance as builtins. A global only does one dict lookup, > while > > builtins do two. Also, builtins can potentially be optimized away > > altogether (e.g. 'while True:') or converted to fast LOAD_CONST, or > perhaps > > even a new CALL_BUILTIN opcode, assuming that adding the opcode doesn't > > blow the cacheability of the eval loop. > >The coarse measurements I made a couple of years ago suggest that >LOAD_GLOBAL is still substantially slower than LOAD_FAST. Less than 100 >cycles for LOAD_FAST and about 400 cycles for LOAD_GLOBAL. > >http://zope.org/Members/jeremy/CurrentAndFutureProjects/PerformanceMeasurements I notice the page says 400 cycles "on average" for LOAD_GLOBAL doing "one or two dictionary lookups", so I'm curious how many of those were for builtins, which in the current scheme are always two lookups. If it was half globals and half builtins, and the dictionary lookup is half the time, then having opcodes that know whether to look in globals or builtins would drop the time to 266 cycles, which isn't spectacular but is still good at only about 3.5 times the bytecode fetch overhead. If builtins are used more frequently than globals, the picture improves still further. Still, it's very interesting to see that loading a global takes almost as much time as calling a function! That's pretty surprising to me. I guess that's why doing e.g. '_len=len' for code that does a tight loop makes such a big difference to performance. I tend to do that with attribute lookups before a tight loop, e.g. 'bar = foo.bar', but I didn't realize that global and builtin lookups were almost as slow. From jim.jewett at eds.com Thu Apr 22 12:22:22 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Thu Apr 22 12:23:06 2004 Subject: [Python-Dev] peps 329, 266, 267 Message-ID: At 04:39 PM 4/21/04 -0400, Jewett, Jim J wrote: >>>> If this is really only about globals and builtins, >>>> then you can just initialize each module's dictionary >>>> with a copy of builtins. (Note that Raymond's original proposal was much stronger; instead of just moving builtins to globals, it moved both builtins and globals into locals.) >>Phillip J. Eby: >>> There's only one problem with this idea, and it's a big >>> one: 'import *' would now include all the builtins, >> Why is this bad? > Because some modules are examined by software, and only > the expected names belong there. For example, I believe > if you run 'pydoc' on such a module, it will proceed to > document all the builtins. Fair enough. But that software could compare against builtins, and do a set-minus. >> ... If the other module actually has modified a >> builtin, you'll need to do the same ... > ... just because one module shadows a builtin, doesn't > mean you have to follow suit. I did phrase that carelessly. The objects you import will still use their original module's globals. You only need to copy these when you want your own code to act consistently with the other module. But note that import * _already_ imports names which shadow builtins, so the only real change would be when the imported module does *not* shadow a builtin, but your own module does (and does so before the import, perhaps because of an earlier import *). If you want to protect even this obscure case, import could be changed to special-case builtins. By the way, what are people benchmarking to decide that builtin lookups are frequent enough to matter? It seems intuitively true (and probably for the same handful of builtins throughout an entire program), but pystone is contrived enough not to show much difference compared to its per-run variability. -jJ From FBatista at uniFON.com.ar Thu Apr 22 12:21:29 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Thu Apr 22 12:23:57 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: [Aahz] #- Sounds good. If you're going this route, it seems to me #- that Decimal() #- would work better if it ends up being a proxy class for an internal #- implementation that's more closely linked to context, and #- that most of #- the Decimal methods simply delegate to the internal implementation I'm ok with that. But I'm kinda OO newbie, so: do you have a link to "formal theory" about this? Some example of something you've done using this design? #- through context. (If you look at my original implementation, that's #- similar to the direction I was going -- haven't actually #- looked at your #- code to see what it's like currently.) Make sure that Decimal has-a That's what is WorkRep for? Actually, I didn't messed up a lot with Decimal.py. I writed a lot of code in test_Decimal.py, and then added from_float to Deicmal and fixed a lot of small details: but just to comply with the test cases. A fundamental redesign of Decimal will be delayed until have all the test cases finished and working. But I think that will be needed in the seek of performance. #- internal implementation rather than is-a (i.e. use #- composition instead #- of subclassing). #- #- > And with floats? c.create_decimal_from_float(number)? Or #- the same method #- > that before? #- #- ``from_float()`` probably is better. Michael proposed to use the same method. This is different from Decimal (with all that that implies), but can you trust that the user that uses the method from context is aware of binary float traps? You propose another method. Its name is not clear to me: context.from_float() ? . Facundo From pje at telecommunity.com Thu Apr 22 13:55:07 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu Apr 22 13:56:44 2004 Subject: [Python-Dev] peps 329, 266, 267 In-Reply-To: Message-ID: <5.1.1.6.0.20040422124756.02bb52c0@telecommunity.com> At 12:22 PM 4/22/04 -0400, Jewett, Jim J wrote: >At 04:39 PM 4/21/04 -0400, Jewett, Jim J wrote: > >>>> If this is really only about globals and builtins, > >>>> then you can just initialize each module's dictionary > >>>> with a copy of builtins. > >(Note that Raymond's original proposal was much stronger; >instead of just moving builtins to globals, it moved both >builtins and globals into locals.) But at least it was still backward-compatible with respect to what actually is found in the module's globals. > >>Phillip J. Eby: > >>> There's only one problem with this idea, and it's a big > >>> one: 'import *' would now include all the builtins, > > >> Why is this bad? > > > Because some modules are examined by software, and only > > the expected names belong there. For example, I believe > > if you run 'pydoc' on such a module, it will proceed to > > document all the builtins. > >Fair enough. But that software could compare against >builtins, and do a set-minus. It *could*, but it's wasteful to break such programs without necessity. >But note that import * _already_ imports names which >shadow builtins, so the only real change would be when >the imported module does *not* shadow a builtin, but >your own module does (and does so before the import, >perhaps because of an earlier import *). > >If you want to protect even this obscure case, import >could be changed to special-case builtins. I think you're missing the part where builtins change from one release to the next. I might have had a global named 'zip' or 'enumerate' in a Python 1.5 program, and when I upgrade to 2.4 I am now "shadowing a builtin". From jim.jewett at eds.com Thu Apr 22 14:42:59 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Thu Apr 22 14:44:07 2004 Subject: [Python-Dev] peps 329, 266, 267 Message-ID: Jim: >> note that import * _already_ imports names which >> shadow builtins, so the only real change would be >> when the imported module does *not* shadow a >> builtin, but your own module does (and does so >> before the import, perhaps because of an earlier >> import *). Phillip J. Eby: > I think you're missing the part where builtins > change from one release to the next. I might have > had a global named 'zip' or 'enumerate' in a Python > 1.5 program, and when I upgrade to 2.4 I am now > "shadowing a builtin". How did you get that global? You either defined it or imported it (or had it added by another module). Any of these actions will still work, and will still replace the cached builtin. The only problem is if you then do an import * _after_ that definition. Doing that was already risking a break; this change just makes the break a bit more likely. -jJ From tcdelaney at optusnet.com.au Thu Apr 22 17:20:46 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Thu Apr 22 17:20:59 2004 Subject: [Python-Dev] Re: Decimal data type issues References: Message-ID: <009f01c428af$ad5abf90$a100a8c0@ryoko> Batista, Facundo wrote: > The round() method has the appealing that it could has the same > name/meaning/function that the built in round(). Perhaps the built-in round should be modified to work with Decimal ... Tim Delaney From tim.one at comcast.net Thu Apr 22 19:12:07 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu Apr 22 19:12:17 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: Message-ID: [Batista, Facundo] > The round() method has the appealing that it could has the same > name/meaning/function that the built in round(). As Tim Delaney said later, the best way to do that would be to change the implementation of Python's builtin round(). I don't think changing Python (beyond adding the new module) is in scope for the first release, though. From tim.one at comcast.net Thu Apr 22 19:20:22 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu Apr 22 19:20:32 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: Message-ID: [Batista, Facundo] > You mean adding a method to the context, to create a Decimal using itself > as context and not the one from the thread? Yes. This isn't novel: all the operations are like that already. For example, Decimal.__abs__() is also available as Context.abs(decimal). The semantic difference is that the Decimal methods implicitly use the current thread's current context object, while the Context methods (trivially) use an explicitly specified context object. > If yes, what about c.create_decimal(number) ? I think the name's kinda wordy. What else would a Decimal.Context object want to create, other than a Decimal.Decimal? > And with floats? c.create_decimal_from_float(number)? Or the same method > that before? I'd be happy with c.from_string(a_string) c.from_int(an_int_or_long) # and works identically to # c.from_string(str(an_int_or_long)) c.from_float(a_float) I have no intention of creating Decimals from floats, so I didn't pay attention to those arguments (other than to silently agree that the issues involved are ugly enough that float->Decimal should have its own spelling -- if we have to have it at all). From tim.one at comcast.net Thu Apr 22 19:32:39 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu Apr 22 19:32:51 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <1082648824.4087e8f856292@mcherm.com> Message-ID: [Michael Chermside] > Me too. But let's promise that the docs will include an example > of how to use quantize() to do rounding, since the name will > not be obvious to the uninitiated. I agree it won't. In some ways it's nicer than round(), because it works by providing an example of the scale you want, not by making you count positions from the decimal point. BTW, note that the spec has nothing to say about how an implementation spells anything, except for the results of the to-string operations. That is, there's no requirement to *call* the spec's quantize operation "quantize"; the requirement is to provide something with the semantics of the spec's quantize operation. For example, round_like() or rescale() might be friendlier names (in fact, the current quantize used to be called rescale in the spec). From tcdelaney at optusnet.com.au Thu Apr 22 19:33:01 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Thu Apr 22 19:33:15 2004 Subject: [Python-Dev] Re: Decimal data type issues References: Message-ID: <00be01c428c2$29463550$a100a8c0@ryoko> Tim Peters wrote: > [Batista, Facundo] >> The round() method has the appealing that it could has the same >> name/meaning/function that the built in round(). > > As Tim Delaney said later, the best way to do that would be to change > the > implementation of Python's builtin round(). I don't think changing > Python (beyond adding the new module) is in scope for the first > release, though. I think we probably want a supplemental PEP for upgrading the `math` module plus appropriate builtins (like round) to handle Decimal. A quick scan through the docs reveals - All the `math` module (thin wrappers for float, hand-written code for Decimal I guess). - abs - divmod - pow - round which look like candidates. Tim Delaney From tim.one at comcast.net Thu Apr 22 19:42:43 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu Apr 22 19:42:53 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <20040422155909.GA13900@panix.com> Message-ID: [Aahz] > Sounds good. If you're going this route, it seems to me that Decimal() > would work better if it ends up being a proxy class for an internal > implementation that's more closely linked to context, and that most of > the Decimal methods simply delegate to the internal implementation > through context. The current implementation is approximately the opposite: most of the Decimal methods have an optional "context" argument, and most the Context methods delegate to Decimal methods (passing context=self). That's fine by me, and I don't see a compelling reason to turn it inside out at this stage. Both classes derive from (only) object. From tim.one at comcast.net Thu Apr 22 19:52:50 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu Apr 22 19:53:00 2004 Subject: [Python-Dev] Re: Decimal data type issues In-Reply-To: <009f01c428af$ad5abf90$a100a8c0@ryoko> Message-ID: [Tim Delaney] > Perhaps the built-in round should be modified to work with Decimal ... Eventually, it should -- and many other changes throughout Python, outside of Decimal, should also be made. That won't happen for the first release, though, and possibly never if Decimal doesn't become popular. Examples: + Float format codes, like %20.6g, make fine sense for Decimal, but feeding them Decimals won't do any good. + A Decimal version of the math module doesn't exist. + Looking up context all the time is expensive, and should eventually become part of CPython's C-level thread state (much as your FPU's context register(s) are part of your C's runtime thread state). From edloper at gradient.cis.upenn.edu Thu Apr 22 21:35:49 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Thu Apr 22 21:34:17 2004 Subject: [Python-Dev] __docformat__ scope Message-ID: <408872F5.6030000@gradient.cis.upenn.edu> Pep 258 [1] specifies that the __docformat__ variable should be used to define the markup language used by docstrings in a module. Recently, I got an RFE for epydoc requesting that a __docformat__ in a package's __init__.py apply to the entire package [2]. My first reaction was that it was a good idea, but then I remembered a recent discussion on python-dev, where there seemed to be a strong sentiment that "from __future__ import absolute_import" should *not* have package scope [3]; and I wondered if some of the same arguments would go against applying __docformat__ to the entire package. So... Should __docformat__ have package scope (with submodules/ subpackages possibly overriding it), or should it only apply to the actual module that defines it? Either way, PEP 258 should be updated to reflect the decision. -Edward [1] pep 258 (Docutils Design Specification) http://www.python.org/peps/pep-0258.html [2] epydoc RFE: __docformat__ for entire packages http://sourceforge.net/tracker/index.php?func=detail&aid=928058&group_id=32455&atid=405621 [3] http://mail.python.org/pipermail/python-dev/2004-April/044201.html From guido at python.org Thu Apr 22 22:15:13 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 22 22:15:56 2004 Subject: [Python-Dev] __docformat__ scope In-Reply-To: Your message of "Thu, 22 Apr 2004 21:35:49 EDT." <408872F5.6030000@gradient.cis.upenn.edu> References: <408872F5.6030000@gradient.cis.upenn.edu> Message-ID: <200404230215.i3N2FDB17450@guido.python.org> > Pep 258 [1] specifies that the __docformat__ variable should be used to > define the markup language used by docstrings in a module. > > Recently, I got an RFE for epydoc requesting that a __docformat__ in a > package's __init__.py apply to the entire package [2]. My first > reaction was that it was a good idea, but then I remembered a recent > discussion on python-dev, where there seemed to be a strong sentiment > that "from __future__ import absolute_import" should *not* have package > scope [3]; and I wondered if some of the same arguments would go against > applying __docformat__ to the entire package. > > So... Should __docformat__ have package scope (with submodules/ > subpackages possibly overriding it), or should it only apply to the > actual module that defines it? > > Either way, PEP 258 should be updated to reflect the decision. The case isn't quite the same, but still, I think it would be better to be required to repeat this info in each module -- more explicit, and we know that explicit is better than implicit. --Guido van Rossum (home page: http://www.python.org/~guido/) From Paul.Moore at atosorigin.com Fri Apr 23 05:42:31 2004 From: Paul.Moore at atosorigin.com (Moore, Paul) Date: Fri Apr 23 05:42:33 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: <16E1010E4581B049ABC51D4975CEDB8803060E72@UKDCX001.uk.int.atosorigin.com> From: Tim Delaney > I think we probably want a supplemental PEP for upgrading the `math` > module plus appropriate builtins (like round) to handle Decimal. A > quick scan through the docs reveals While I'm sure the numerical properties are iffy, just how bad is using binary FP as an intermediate when calculating transcendental functions for Decimal values? I'm thinking of something like d = Decimal(...) f = math.sin(float(d)) # Now convert back using exact float->Decimal, followed by # context-respecting "+" operation result = +Decimal.from_float(f) I suspect that this is adequate for "casual" users. And I'm not at all sure that the target audience for a Decimal type really cares about transcendental functions... (I think that boils down to claiming YAGNI, at least in the first instance). Paul. __________________________________________________________________________ This e-mail and the documents attached are confidential and intended solely for the addressee; it may also be privileged. If you receive this e-mail in error, please notify the sender immediately and destroy it. As its integrity cannot be secured on the Internet, the Atos Origin group liability cannot be triggered for the message content. Although the sender endeavours to maintain a computer virus-free network, the sender does not warrant that this transmission is virus-free and will not be liable for any damages resulting from any virus transmitted. __________________________________________________________________________ From mcherm at mcherm.com Fri Apr 23 09:10:10 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Fri Apr 23 09:10:13 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: <1082725810.408915b28649c@mcherm.com> Tim Peters writes: > BTW, note that the spec has nothing to say about how an implementation > spells anything, except for the results of the to-string operations. That > is, there's no requirement to *call* the spec's quantize operation > "quantize"; the requirement is to provide something with the semantics of > the spec's quantize operation. Without having given it a LOT of thought, I'm in favor of calling it "quantize" just to match the name in the spec. Without deciding it yet, there are good reasons to think we MIGHT want the behavior of "round" to be somewhat different (eg: match the behavior of Python's round()) if that were supported in the future. > For example, round_like() or rescale() might > be friendlier names (in fact, the current quantize used to be called rescale > in the spec). Yes, but there are ALSO notes in the spec explaining that they changed the name from rescale to quantize because people had misleading preconceptions about what "rescale" would do. (Partly because it worked differently in previous versions of the spec.) -- Michael Chermside From tcdelaney at optusnet.com.au Fri Apr 23 09:27:37 2004 From: tcdelaney at optusnet.com.au (Tim Delaney) Date: Fri Apr 23 09:27:46 2004 Subject: [Python-Dev] Re: Decimal data type issues References: <16E1010E4581B049ABC51D4975CEDB8803060E72@UKDCX001.uk.int.atosorigin.com> Message-ID: <002c01c42936$beaf2a60$a100a8c0@ryoko> Moore, Paul wrote: > While I'm sure the numerical properties are iffy, just how bad is > using binary FP as an intermediate when calculating transcendental > functions for Decimal values? Ignoring other issues, what happens when you exceed the limits for float? Throw an exception, or change to another code path (int/long)? If the latter, might as well just use it in all cases - makes it more accurate and less complicated. > I suspect that this is adequate for "casual" users. And I'm not at all > sure that the target audience for a Decimal type really cares about > transcendental functions... Might be surprised, but there's time to work on it. See next point. > (I think that boils down to claiming YAGNI, at least in the first > instance). Most definitely not going to be in the first instance, as has been stated before. Python 2.5+ - and then only after a PEP that's gone through as rigorous a process as this one IMO. Tim Delaney From jim.jewett at eds.com Fri Apr 23 11:50:28 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Fri Apr 23 11:51:32 2004 Subject: [Python-Dev] Re: Decimal data type issues Message-ID: Batista, Facundo: > can you trust that the user that uses the > method from context is aware of binary float traps? If they are explicitly setting a context, I think it is reasonable to assume that. The problem case is when they just use the default. inputcontext.Decimal(1.1) has made an explicit request for specific handling. Decimal(1.1) has not indicated any awareness. The catch is that we don't want "just make another context" to become such an idiom for floats that people do it without knowing why. Perhaps the Context.Decimal.__init__ could delegate to the raise function unless/until the user had set an attribute of approximate_float_ok = True. -jJ From guido at python.org Fri Apr 23 14:26:58 2004 From: guido at python.org (Guido van Rossum) Date: Fri Apr 23 14:27:09 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward Message-ID: <200404231826.i3NIQxM18955@guido.python.org> I'd like to get generator expressions checked into CVS. Regarding the early-or-late binding issue, here's what I'd like to see happen: I'd like the late-binding (i.e. non-capture) version checked in and released with Python 2.4a1 and a2. If we find there are real problems with these semantics, we can switch to early-binding semantics in 2.4b1; this would be almost completely backwards compatible. But if it works in the alpha release, we'll stick with late binding. My reasons for (still!) preferring late binding are: (a) I find it hard to defend automatic variable capture given Python's general late-binding semantics (b) I believe that use cases needing early binding are uncommon and strained: they all involve creating a list of generator expressions, which IMO is a pretty unusual thing to do I cannot volunteer to do the final code review and checkin, and I believe Raymond is still on vacation (he's asked me to relieve him from the responsibility of shepherding this patch) so we need someone to volunteer. The patch is here: http://sf.net/tracker/?func=detail&aid=872326&group_id=5470&atid=305470 --Guido van Rossum (home page: http://www.python.org/~guido/) From judson at mcs.anl.gov Fri Apr 23 14:43:10 2004 From: judson at mcs.anl.gov (Ivan R. Judson) Date: Fri Apr 23 14:43:16 2004 Subject: [Python-Dev] PEP 754 Message-ID: <200404231843.i3NIhC0109140@mcs.anl.gov> What's needs to be done to possibly get PEP754 (fpconst) to move ahead and perhaps go into 2.4? --Ivan From mcherm at mcherm.com Fri Apr 23 15:02:49 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Fri Apr 23 15:02:50 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward Message-ID: <1082746969.40896859bcf09@mcherm.com> > I'd like to get generator expressions checked into CVS. > > Regarding the early-or-late binding issue, here's what I'd like to see > happen: I'd like the late-binding (i.e. non-capture) version checked > in and released with Python 2.4a1 and a2. If we find there are real > problems with these semantics, we can switch to early-binding > semantics in 2.4b1 Sounds like a good plan, but I have one question: Given the extent to which alpha releases are tested (and subtracting out those who use them only to check old code for compatibility and don't try out the new features) are we going to be able to tell if there are problems with the semantics? My best idea is that those who currently favor the early-binding approach (and I'm sure you've heard from plenty of them) should be encouraged to try it out in the alpha and report any issues, along with actual use cases. Someone collects these responses, then you (Guido) look over the results (with a grain of salt since it's mostly being tested by folks who object) and decide if it's a problem. Will that work? Anyone have a better idea? How do we go about asking people to try it out (is a mention by Guido on comp.lang.python sufficient)? -- Michael Chermside From jeremy at alum.mit.edu Fri Apr 23 15:41:32 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Apr 23 15:42:35 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <1082746969.40896859bcf09@mcherm.com> References: <1082746969.40896859bcf09@mcherm.com> Message-ID: <1082749292.9092.40.camel@localhost.localdomain> On Fri, 2004-04-23 at 15:02, Michael Chermside wrote: > > I'd like to get generator expressions checked into CVS. > > > > Regarding the early-or-late binding issue, here's what I'd like to see > > happen: I'd like the late-binding (i.e. non-capture) version checked > > in and released with Python 2.4a1 and a2. If we find there are real > > problems with these semantics, we can switch to early-binding > > semantics in 2.4b1 > > Sounds like a good plan, but I have one question: Given the extent to > which alpha releases are tested (and subtracting out those who use > them only to check old code for compatibility and don't try out the > new features) are we going to be able to tell if there are problems > with the semantics? We need reports from people writing real code with generator expressions. It's hard to guess if we'll get enough substantial feedback in the alpha releases. The kind of issues we're dealing with usually manifest themselves in real programs rather than toy examples; not sure that an alpha will get that kind of use. On the other hand, a generator expression is nearly equivalent to a list comprehension. So it should be easy for people to experiment with generator expressions, because they can swap them with list comprehensions in many cases. That leads me to wonder what exactly the rationale for generator expressions is. The PEP says that "time, clarity, and memory are conserved by using an generator expression" but I can only see how memory is conserved. That is, I don't find them any easier to read than list comprehensions and I don't understand the performance implications very well. It's not obvious to me that their faster. > My best idea is that those who currently favor the early-binding > approach (and I'm sure you've heard from plenty of them) should be > encouraged to try it out in the alpha and report any issues, along I'm not sure what you mean by "it." If you mean try the alpha with late-binding semantics and see how it goes, I think that's a good plan. > with actual use cases. Someone collects these responses, then you > (Guido) look over the results (with a grain of salt since it's > mostly being tested by folks who object) and decide if it's a > problem. Jeremy From pje at telecommunity.com Fri Apr 23 15:59:26 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri Apr 23 15:59:47 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <1082749292.9092.40.camel@localhost.localdomain> References: <1082746969.40896859bcf09@mcherm.com> <1082746969.40896859bcf09@mcherm.com> Message-ID: <5.1.1.6.0.20040423155826.03026480@telecommunity.com> At 03:41 PM 4/23/04 -0400, Jeremy Hylton wrote: >That leads me to wonder what exactly the rationale for generator >expressions is. The PEP says that "time, clarity, and memory are >conserved by using an generator expression" but I can only see how >memory is conserved. That is, I don't find them any easier to read than >list comprehensions and I don't understand the performance implications >very well. It's not obvious to me that their faster. Alex Martelli previously posted some tests that showed them to be quite a bit faster for large lists. From aahz at pythoncraft.com Fri Apr 23 16:05:16 2004 From: aahz at pythoncraft.com (Aahz) Date: Fri Apr 23 16:06:03 2004 Subject: [Python-Dev] PEP 754 In-Reply-To: <200404231843.i3NIhC0109140@mcs.anl.gov> References: <200404231843.i3NIhC0109140@mcs.anl.gov> Message-ID: <20040423200516.GA28707@panix.com> On Fri, Apr 23, 2004, Ivan R. Judson wrote: > > What's needs to be done to possibly get PEP754 (fpconst) to move ahead and > perhaps go into 2.4? Nothing, because it ain't gonna happen. In order for PEP754 to work, we need to have a bunch of people volunteer to maintain the IEEE-754 semantics for a bunch of difference platforms. Feel free to try to drum up interest, but the PEP won't be accepted until there's evidence that people are making long-term commitments. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From jeremy at alum.mit.edu Fri Apr 23 16:10:39 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Apr 23 16:11:42 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <5.1.1.6.0.20040423155826.03026480@telecommunity.com> References: <1082746969.40896859bcf09@mcherm.com> <1082746969.40896859bcf09@mcherm.com> <5.1.1.6.0.20040423155826.03026480@telecommunity.com> Message-ID: <1082751039.9092.56.camel@localhost.localdomain> On Fri, 2004-04-23 at 15:59, Phillip J. Eby wrote: > At 03:41 PM 4/23/04 -0400, Jeremy Hylton wrote: > > >That leads me to wonder what exactly the rationale for generator > >expressions is. The PEP says that "time, clarity, and memory are > >conserved by using an generator expression" but I can only see how > >memory is conserved. That is, I don't find them any easier to read than > >list comprehensions and I don't understand the performance implications > >very well. It's not obvious to me that their faster. > > Alex Martelli previously posted some tests that showed them to be quite a > bit faster for large lists. Anyone know where to find these numbers. I've done some searching, but I can't find them. It would be good to include something concrete in the PEP. Jeremy From tim.one at comcast.net Fri Apr 23 16:32:27 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri Apr 23 16:32:28 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <1082749292.9092.40.camel@localhost.localdomain> Message-ID: [Jeremy] > ... > That leads me to wonder what exactly the rationale for generator > expressions is. The PEP says that "time, clarity, and memory are > conserved by using an generator expression" but I can only see how > memory is conserved. That is, I don't find them any easier to read > than list comprehensions They're not, although they can be more clear than code that defines helper generating functions to get some of the same memory benefits. > and I don't understand the performance implications very well. It's > not obvious to me that their faster. When you've got an iterator producing a billion elements, it becomes obvious at once . Really, *when* they're faster than listcomps, it's mostly a consequence of not creating in whole, then crawling over, a giant memory object. For short sequences, I expect listcomps are faster (and earlier timings have shown that). genexps require an additional frame suspend/resume per element, and while cheap (esp. compared to a function call) it's not free. For long sequences, avoiding the creation of a giant list becomes an overwhelming advantage. > Anyone know where to find these numbers. I've done some searching, but > I can't find them. It would be good to include something concrete in > the PEP. This varies so wildly across platforms, timing procedure, and test case, that I expect concrete numbers would do more harm than good. The qualitative argument is easy to grasp. BTW, the most recent round of this was in the "genexps slow?" thread, started last month and spilling into April. Here's the start of it: http://mail.python.org/pipermail/python-dev/2004-March/043777.html From pf_moore at yahoo.co.uk Fri Apr 23 16:46:54 2004 From: pf_moore at yahoo.co.uk (Paul Moore) Date: Fri Apr 23 16:51:01 2004 Subject: [Python-Dev] Re: PEP 754 References: <200404231843.i3NIhC0109140@mcs.anl.gov> <20040423200516.GA28707@panix.com> Message-ID: Aahz writes: > On Fri, Apr 23, 2004, Ivan R. Judson wrote: >> >> What's needs to be done to possibly get PEP754 (fpconst) to move ahead and >> perhaps go into 2.4? > > Nothing, because it ain't gonna happen. In order for PEP754 to work, we > need to have a bunch of people volunteer to maintain the IEEE-754 > semantics for a bunch of difference platforms. The PEP offers a reference implementation in pure Python. (Although the link given is broken). The implementation is just about creating and manipulating specific bit patterns stored in floats. How useful the module is (given that Python doesn't mandate that floats are stored using IEEE-754 format) I couldn't comment. But I'll note that the standard library already contains modules which only work on specific types of platform. As to the inclusion of the module, I have no opinion... Paul. -- This signature intentionally left blank From tim.one at comcast.net Fri Apr 23 16:51:56 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri Apr 23 16:52:04 2004 Subject: [Python-Dev] PEP 754 In-Reply-To: <200404231843.i3NIhC0109140@mcs.anl.gov> Message-ID: [Ivan Judson] > What's needs to be done to possibly get PEP754 (fpconst) to move > ahead and perhaps go into 2.4? + The implementation URL in the PEP no longer resolves, so that needs to be fixed. + Last time I saw it, fpconst.py had a Pfizer copyright and license, which makes headaches for everyone. + It needs docs. + It needs a test suite. + Guido has to Pronounce on it. [Aahz] > In order for PEP754 to work, we need to have a bunch of people > volunteer to maintain the IEEE-754 semantics for a bunch of > difference platforms. Feel free to try to drum up interest, but > the PEP won't be accepted until there's evidence that people > are making long-term commitments. That isn't a good objection for this specific patch -- it's pure Python, and picks apart bits using the struct module. IIRC, it wasn't correctly written for most 64-bit boxes, but that looked like a shallow flaw (I think it assumed that a struct "l" format code, in native mode, always yields a 32-bit thingie). From goodger at python.org Fri Apr 23 19:43:09 2004 From: goodger at python.org (David Goodger) Date: Fri Apr 23 19:43:19 2004 Subject: [Python-Dev] Re: PEP 754 In-Reply-To: References: <200404231843.i3NIhC0109140@mcs.anl.gov> Message-ID: <4089AA0D.4030205@python.org> > [Ivan Judson] >> What's needs to be done to possibly get PEP754 (fpconst) to move >> ahead and perhaps go into 2.4? This one seems to have slipped through the cracks. I haven't heard from the author lately; hopefully he hasn't lost interest. In any case, an active champion is needed for progress. Tim Peters wrote: > + The implementation URL in the PEP no longer resolves, so that > needs to be fixed. A Google search resulted in: I've updated the PEP. > + Last time I saw it, fpconst.py had a Pfizer copyright and > license, which makes headaches for everyone. > > + It needs docs. > > + It needs a test suite. > > + Guido has to Pronounce on it. Is Guido's waiting for your (Tim's) recommendation? Guido wrote, "David, I'll leave the review of this PEP entirely to Tim, if you don't mind". At the time (2003-10-05) Tim wrote: Guido and I have discussed this in email today. We're both favorable. It may get buried under feature creep, though (that is, if the PEP is good so far as it goes, it may be necessary to make it go farther, addressing some of the other problems mentioned-- but not solved --by the PEP). Gregory Warnes (the PEP author) had a few questions the next day (2003-10-06); I wasn't copied on replies if any. Shall I forward Gregory's messages to you? I'm copying Gregory on this message. -- David Goodger Python Enhancement Proposal (PEP) Editor (Please cc: all PEP correspondence to .) From paul at prescod.net Sat Apr 24 05:51:36 2004 From: paul at prescod.net (Paul Prescod) Date: Sat Apr 24 07:39:44 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <1082749292.9092.40.camel@localhost.localdomain> References: <1082746969.40896859bcf09@mcherm.com> <1082749292.9092.40.camel@localhost.localdomain> Message-ID: <408A38A8.1030702@prescod.net> Jeremy Hylton wrote: >... > > That leads me to wonder what exactly the rationale for generator > expressions is. The PEP says that "time, clarity, and memory are > conserved by using an generator expression" but I can only see how > memory is conserved. That is, I don't find them any easier to read than > list comprehensions and I don't understand the performance implications > very well. It's not obvious to me that their faster. I think that there is a robustness argument to be made as well. It is very common to run into programs that work really well with small data sets and then completely run out of steam with large ones. Sometimes the algorithm is at fault. But it could also be that some Python programs fail to scale primarily because of the wastefulness of listcomps. Whether or not this is true in practice, it is certainly true that when you write programs that are meant to scale, you must consider every listcomp and think whether the dataset is going to get large or not. If so, you must switch to some more obfuscated syntax. Paul Prescod From scav at blueyonder.co.uk Sat Apr 24 10:58:14 2004 From: scav at blueyonder.co.uk (Peter Harris) Date: Sat Apr 24 10:49:46 2004 Subject: [Python-Dev] Re: generator expressions In-Reply-To: References: Message-ID: <408A8086.3010005@blueyonder.co.uk> Michael Chermside wrote: >How do we go about asking >people to try it out (is a mention by Guido on comp.lang.python >sufficient)? > >-- Michael Chermside > > > I'll play with generator expressions gladly. I have been using list comprehensions like mad, even in cases where I suspect a generator would be better. Also, I have a few uses of generators that might look clearer as generator expressions. So I'd be more than happy to make a test version of my current projects using generator expressions, just to see when they can make the code clearer, and whether there is anything about the semantics that trips me up too easily. I don't think my data sets are going to be big enough to test performance, just usability. Now, if someone checks in the implementation of PEP309 too , I would be as happy as a biscuit... Peter Harris From bob at redivi.com Sat Apr 24 11:02:55 2004 From: bob at redivi.com (Bob Ippolito) Date: Sat Apr 24 10:58:46 2004 Subject: [Python-Dev] Re: generator expressions In-Reply-To: <408A8086.3010005@blueyonder.co.uk> References: <408A8086.3010005@blueyonder.co.uk> Message-ID: <777FCEB3-9600-11D8-BEAF-000A95686CD8@redivi.com> On Apr 24, 2004, at 10:58 AM, Peter Harris wrote: > Michael Chermside wrote: > >> How do we go about asking >> people to try it out (is a mention by Guido on comp.lang.python >> sufficient)? >> >> -- Michael Chermside >> >> > I'll play with generator expressions gladly. I have been using list > comprehensions like mad, even in cases where I suspect a generator > would be better. Also, I have a few uses of generators that might look > clearer as generator expressions. So I'd be more than happy to make a > test version of my current projects using generator expressions, just > to see when they can make the code clearer, and whether there is > anything about the semantics that trips me up too easily. > > I don't think my data sets are going to be big enough to test > performance, just usability. > > Now, if someone checks in the implementation of PEP309 too , I > would be as happy as a biscuit... Well, I'm +1 on PEP309. I regularly wish I had partial function application in Python. I also plan to experiment with generator expressions sooner than later. -bob From niemeyer at conectiva.com Sun Apr 25 19:40:19 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Sun Apr 25 19:39:34 2004 Subject: [Python-Dev] Small issues in gettext support Message-ID: <20040425234019.GA6436@burma.localdomain> Hello folks, I've been working with gettext support in Python, and found some issues I'd like to discuss with you. First, I've noticed that there's a difference in the internal implementation of gettext and GNU gettext regarding the returned encoding on non-unicode strings. Notice the difference in the result of this code: import gettext import locale locale.setlocale(locale.LC_ALL, "") locale.textdomain("apt-cdrom-registry") gettext.textdomain("apt-cdrom-registry") print locale.gettext("Choose the available CDROMs from the list below") print gettext.gettext("Choose the available CDROMs from the list below") This has shown the following: Escolha os CDROMs dispon?ves na lista abaixo Escolha os CDROMs dispon??ves na lista abaixo The reason for this difference is clear: GNU gettext defaults to the current locale when returning encoded strings, while gettext.py returns strings in the encoding used in the .mo file. The fix is simply changing the following code # Encode the Unicode tmsg back to an 8-bit string, if possible if self._charset: return tmsg.encode(self._charset) to use the system encoding (sys.getdefaultencoding()) instead of self._charset. Regarding a similar issue, I've also noticed that we're currently missing bind_textdomain_codeset() support. This function changes the codeset used to return the translated strings. So, I'd like to implement the following changes: - Change the default codeset used by gettext.py in functions returning an encoded string to match the system encoding. - Introduce bind_textdomain_codeset() in locale. - Introduce bind_textdomain_codeset() in gettext.py implementing an equivalent functionality. Comments? -- Gustavo Niemeyer http://niemeyer.net From niemeyer at conectiva.com Mon Apr 26 08:38:26 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Mon Apr 26 08:37:44 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <1082749292.9092.40.camel@localhost.localdomain> References: <1082746969.40896859bcf09@mcherm.com> <1082749292.9092.40.camel@localhost.localdomain> Message-ID: <20040426123826.GA1922@burma.localdomain> > We need reports from people writing real code with generator > expressions. It's hard to guess if we'll get enough substantial > feedback in the alpha releases. The kind of issues we're dealing with > usually manifest themselves in real programs rather than toy examples; > not sure that an alpha will get that kind of use. An interesting idea might be to hack the standard library replacing every occurrence of list comprehensions with generator expressions, and check the respective test cases and/or applications using them. -- Gustavo Niemeyer http://niemeyer.net From barry at python.org Mon Apr 26 10:13:14 2004 From: barry at python.org (Barry Warsaw) Date: Mon Apr 26 10:13:20 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <20040425234019.GA6436@burma.localdomain> References: <20040425234019.GA6436@burma.localdomain> Message-ID: <1082988793.17933.59.camel@anthem.wooz.org> On Sun, 2004-04-25 at 19:40, Gustavo Niemeyer wrote: > Hello folks, > > I've been working with gettext support in Python, and found some > issues I'd like to discuss with you. > > First, I've noticed that there's a difference in the internal > implementation of gettext and GNU gettext regarding the returned > encoding on non-unicode strings. Notice the difference in the > result of this code: > > import gettext > import locale > locale.setlocale(locale.LC_ALL, "") > locale.textdomain("apt-cdrom-registry") > gettext.textdomain("apt-cdrom-registry") > print locale.gettext("Choose the available CDROMs from the list below") > print gettext.gettext("Choose the available CDROMs from the list below") > > This has shown the following: > > Escolha os CDROMs dispon?ves na lista abaixo > Escolha os CDROMs dispon??ves na lista abaixo > > The reason for this difference is clear: GNU gettext defaults to the > current locale when returning encoded strings, while gettext.py > returns strings in the encoding used in the .mo file. The fix is > simply changing the following code > > # Encode the Unicode tmsg back to an 8-bit string, if possible > if self._charset: > return tmsg.encode(self._charset) > > to use the system encoding (sys.getdefaultencoding()) instead of > self._charset. I'd be worried most about backwards compatibility, since the module has worked this way since its early days. Also, wouldn't this be an opportunity for getting lots of UnicodeErrors? E.g. my system encoding is 'ascii' so gettext() would fail for catalogs containing non-ascii characters. I shouldn't have to change my system encoding just to avoid errors, but with your suggestion, wouldn't that make many catalogs basically unusable for me? > Regarding a similar issue, I've also noticed that we're currently > missing bind_textdomain_codeset() support. This function changes > the codeset used to return the translated strings. > > So, I'd like to implement the following changes: > > - Change the default codeset used by gettext.py in functions > returning an encoded string to match the system encoding. > - Introduce bind_textdomain_codeset() in locale. > - Introduce bind_textdomain_codeset() in gettext.py implementing > an equivalent functionality. Would adding bind_textdomain_codeset() would provide a way for the application to change the default encoding? If so, I'd be in favor of adding bind_textdomain_codeset() but not changing the default encoding for returned strings. Then update the documentation to describe current behavior and how to change it via that function call. -Barry From niemeyer at conectiva.com Mon Apr 26 11:16:07 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Mon Apr 26 11:16:15 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <1082988793.17933.59.camel@anthem.wooz.org> References: <20040425234019.GA6436@burma.localdomain> <1082988793.17933.59.camel@anthem.wooz.org> Message-ID: <20040426151607.GA2361@burma.localdomain> > I'd be worried most about backwards compatibility, since the module has > worked this way since its early days. Also, wouldn't this be an IMO, the current behavior is wrong, so breaking backwards compatibility in that case would be fixing something important. > opportunity for getting lots of UnicodeErrors? E.g. my system encoding > is 'ascii' so gettext() would fail for catalogs containing non-ascii > characters. I shouldn't have to change my system encoding just to avoid > errors, but with your suggestion, wouldn't that make many catalogs > basically unusable for me? There are a few extra points to notice here: - Different .mo files may have different encodings. - The translation system is made in a way that the programmer should not have to worry about the encoding used by the translators. - The current scheme may introduce a wrong practice: forcing translators to use some specific encoding to avoid breaking the program. - We already have support for getting the "unicode" version of the string. This is currently the right way to get the translation in some specific encoding, since it uncouples the translation encoding from the expected encoding. - In cases where you'd get the "UnicodeError", you'd see a mangled string which would be unreadable. To avoid the UnicodeError, we may also return the original string in cases where the UnicodeError is raised. > Would adding bind_textdomain_codeset() would provide a way for the > application to change the default encoding? Yes, it changes the default encoding. > If so, I'd be in favor of adding bind_textdomain_codeset() but not > changing the default encoding for returned strings. Then update the > documentation to describe current behavior and how to change it via > that function call. Thanks for your suggestion! -- Gustavo Niemeyer http://niemeyer.net From niemeyer at conectiva.com Mon Apr 26 11:33:03 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Mon Apr 26 11:33:09 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <20040426151607.GA2361@burma.localdomain> References: <20040425234019.GA6436@burma.localdomain> <1082988793.17933.59.camel@anthem.wooz.org> <20040426151607.GA2361@burma.localdomain> Message-ID: <20040426153303.GA2676@burma.localdomain> > > opportunity for getting lots of UnicodeErrors? E.g. my system encoding > > is 'ascii' so gettext() would fail for catalogs containing non-ascii > > characters. I shouldn't have to change my system encoding just to avoid > > errors, but with your suggestion, wouldn't that make many catalogs > > basically unusable for me? Another interesting point, from the gettext documentation: """ The output character set is, by default, the value of `nl_langinfo (CODESET)', which depends on the `LC_CTYPE' part of the current locale. But programs which store strings in a locale independent way (e.g. UTF-8) can request that `gettext' and related functions return the translation in that encoding, by use of the `bind_textdomain_codeset' function. """ So, we should use some variant of locale.nl_langinfo(locale.CODESET) to get the default encoding. Since this should be changed together with the language used when 'setlocale' is issued, the UnicodeError problems you're afraid of would also go away. -- Gustavo Niemeyer http://niemeyer.net From anthony at interlink.com.au Mon Apr 26 11:59:37 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon Apr 26 12:01:21 2004 Subject: [Python-Dev] today's pointless, yet slightly terrifying, factoid(s) Message-ID: <408D31E9.80205@interlink.com.au> I received some email about some _very_ old pages (no longer existing) I put up about Python's internals. In 1996. This made me stop and think 'bloody hell, how long _have_ I been doing this?' cvs ann Misc/ACKS is slightly terrifying, saying at least 1994. A teeny tiny sort script later, and this list follows. Note that it's obviously not representative of anything at all, but may serve to terrify others, as it terrified me, as to the sheer amount of their life that they've wasted on Python disclaimer: absolutely representative of nothing at all, not to be taken internally, if pain persists see a doctor Anthony 1.1 (guido 26-Jan-94): Mark Anacker 1.1 (guido 26-Jan-94): Jan-Hein B"uhrman 1.1 (guido 26-Jan-94): Anthony Baxter 1.1 (guido 26-Jan-94): Donald Beaudry 1.1 (guido 26-Jan-94): Eric Beser 1.1 (guido 26-Jan-94): Stephen Bevan 1.1 (guido 26-Jan-94): Peter Bosch 1.1 (guido 26-Jan-94): Terrence Brannon 1.1 (guido 26-Jan-94): Erik de Bueger 1.1 (guido 26-Jan-94): Dick Bulterman 1.1 (guido 26-Jan-94): David Chaum 1.1 (guido 26-Jan-94): Jonathan Dasteel 1.1 (guido 26-Jan-94): John DeGood 1.1 (guido 26-Jan-94): Roger Dev 1.1 (guido 26-Jan-94): Stoffel Erasmus 1.1 (guido 26-Jan-94): Niels Ferguson 1.1 (guido 26-Jan-94): Michael Guravage 1.1 (guido 26-Jan-94): Paul ten Hagen 1.1 (guido 26-Jan-94): Lynda Hardman 1.1 (guido 26-Jan-94): Ivan Herman 1.1 (guido 26-Jan-94): Chris Hoffman 1.1 (guido 26-Jan-94): Philip Homburg 1.1 (guido 26-Jan-94): Jack Jansen 1.1 (guido 26-Jan-94): Bill Janssen 1.1 (guido 26-Jan-94): Drew Jenkins 1.1 (guido 26-Jan-94): Lou Kates 1.1 (guido 26-Jan-94): Robert van Liere 1.1 (guido 26-Jan-94): Steve Majewski 1.1 (guido 26-Jan-94): Lambert Meertens 1.1 (guido 26-Jan-94): Steven Miale 1.1 (guido 26-Jan-94): Doug Moen 1.1 (guido 26-Jan-94): Sape Mullender 1.1 (guido 26-Jan-94): Sjoerd Mullender 1.1 (guido 26-Jan-94): George Neville-Neil 1.1 (guido 26-Jan-94): Randy Pausch 1.1 (guido 26-Jan-94): Marcel van der Peijl 1.1 (guido 26-Jan-94): Steven Pemberton 1.1 (guido 26-Jan-94): Tim Peters 1.1 (guido 26-Jan-94): John Redford 1.1 (guido 26-Jan-94): Timothy Roscoe 1.1 (guido 26-Jan-94): Kevin Samborn 1.1 (guido 26-Jan-94): Fred Sells 1.1 (guido 26-Jan-94): Denis Severson 1.1 (guido 26-Jan-94): Michael Shiplett 1.1 (guido 26-Jan-94): Paul Sijben 1.1 (guido 26-Jan-94): Dirk Soede 1.1 (guido 26-Jan-94): Per Spilling 1.1 (guido 26-Jan-94): Quentin Stafford-Fraser 1.1 (guido 26-Jan-94): Tracy Tims 1.1 (guido 26-Jan-94): Bennett Todd 1.1 (guido 26-Jan-94): Jaap Vermeulen 1.1 (guido 26-Jan-94): Dik Winter 1.2 (guido 26-Jan-94): Amrit Prem 1.3 (guido 14-Apr-94): Jurjen Bos 1.3 (guido 14-Apr-94): Stefan Esser 1.3 (guido 14-Apr-94): Rycharde Hawkes 1.3 (guido 14-Apr-94): Steve Kirsch 1.3 (guido 14-Apr-94): Mark Lutz 1.3 (guido 14-Apr-94): Jim Roskind 1.4 (guido 04-May-94): Adrian Phillips 1.5 (guido 04-May-94): Andy Bensky 1.6 (guido 23-May-94): Rickard Westman 1.7 (guido 14-Jul-94): Tommy Burnette 1.7 (guido 14-Jul-94): Simon Johnston 1.7 (guido 14-Jul-94): William Lewis 1.7 (guido 14-Jul-94): Richard Walker 1.8 (guido 05-Aug-94): Andrew Kuchling 1.8 (guido 05-Aug-94): John Tromp 1.9 (guido 30-Aug-94): Ulf Bartelt 1.10 (guido 14-Sep-94): Michael Scharf 1.11 (guido 06-Oct-94): Anne Lord 1.11 (guido 06-Oct-94): Skip Montanaro 1.12 (guido 10-Nov-94): Eric Bouck 1.12 (guido 10-Nov-94): Dave Brennan 1.12 (guido 10-Nov-94): Mike Carlton 1.12 (guido 10-Nov-94): John Cugini 1.12 (guido 10-Nov-94): Lance Ellinghaus 1.12 (guido 10-Nov-94): Paul Everitt 1.12 (guido 10-Nov-94): Robin Friedrich 1.12 (guido 10-Nov-94): Jim Fulton 1.12 (guido 10-Nov-94): Peter Funk 1.12 (guido 10-Nov-94): Mark Hammond 1.12 (guido 10-Nov-94): Barry Hantman 1.12 (guido 10-Nov-94): Jim Lynch 1.12 (guido 10-Nov-94): Ken Manheimer 1.12 (guido 10-Nov-94): Roger Masse 1.12 (guido 10-Nov-94): Michael McLay 1.12 (guido 10-Nov-94): Jim St. Pierre 1.12 (guido 10-Nov-94): Nick Seidenman 1.12 (guido 10-Nov-94): George Sipe 1.12 (guido 10-Nov-94): Richard Stoakley 1.12 (guido 10-Nov-94): Casper Stoel 1.12 (guido 10-Nov-94): Frank Visser 1.12 (guido 10-Nov-94): Barry Warsaw 1.12 (guido 10-Nov-94): Steve Waterbury 1.13 (guido 04-Jan-95): Matt Conway 1.13 (guido 04-Jan-95): Tom Culliton 1.13 (guido 04-Jan-95): Bob Watson 1.14 (guido 17-Jan-95): Dag Gruneau 1.14 (guido 17-Jan-95): Jun Hamano 1.14 (guido 17-Jan-95): Ken Howard 1.14 (guido 17-Jan-95): Eric Siegerman 1.14 (guido 17-Jan-95): R Lindsay Todd 1.15 (guido 14-Feb-95): Steve Clift 1.15 (guido 14-Feb-95): David Hobley 1.15 (guido 14-Feb-95): John Interrante 1.15 (guido 14-Feb-95): Bob Kras 1.16 (guido 09-Mar-95): Stig Bakken 1.16 (guido 09-Mar-95): Monty Brandenberg 1.16 (guido 09-Mar-95): Eric Daniel 1.16 (guido 09-Mar-95): Kevan Heydon 1.16 (guido 09-Mar-95): Soren Larsen 1.16 (guido 09-Mar-95): Steven Reiz 1.16 (guido 09-Mar-95): Ken Stox 1.17 (guido 07-Apr-95): Graham Matthews 1.18 (guido 28-Aug-95): David Ely 1.18 (guido 28-Aug-95): Gregor Schmid 1.19 (guido 11-Oct-95): Joel Shprentz 1.20 (guido 26-Jun-96): Mitch Chapman 1.20 (guido 26-Jun-96): Steve Cousins 1.20 (guido 26-Jun-96): Hans de Graaff 1.20 (guido 26-Jun-96): Rasmus Hahn 1.20 (guido 26-Jun-96): Manus Hand 1.20 (guido 26-Jun-96): Richard Jones 1.20 (guido 26-Jun-96): Harri Pasanen 1.20 (guido 26-Jun-96): Masazumi Yoshikawa 1.21 (guido 21-Jul-96): Jim Ahlstrom 1.21 (guido 21-Jul-96): Oliver Andrich 1.21 (guido 21-Jul-96): Ross Andrus 1.21 (guido 21-Jul-96): David Ascher 1.21 (guido 21-Jul-96): Neal Becker 1.21 (guido 21-Jul-96): Tom Christiansen 1.21 (guido 21-Jul-96): Sebastian Fernandez 1.21 (guido 21-Jul-96): Jim Hugunin 1.21 (guido 21-Jul-96): Just van Rossum 1.21 (guido 21-Jul-96): Amy Taylor 1.21 (guido 21-Jul-96): John Viega 1.21 (guido 21-Jul-96): Aaron Watters 1.21 (guido 21-Jul-96): Siebren van der Zee 1.22 (guido 30-Jul-96): Bill Tutt 1.23 (guido 20-Aug-96): Vladimir Marangozov 1.24 (guido 08-Oct-96): Nils Fischbeck 1.24 (guido 08-Oct-96): Tim Hochberg 1.24 (guido 08-Oct-96): Terry Reedy 1.24 (guido 08-Oct-96): Ka-Ping Yee 1.25 (guido 09-May-97): Jyrki Alakuijala 1.25 (guido 09-May-97): Andy Eskilsson 1.25 (guido 09-May-97): Lele Gaifax 1.25 (guido 09-May-97): Jeremy Hylton 1.25 (guido 09-May-97): Per Lindqvist 1.26 (guido 19-Jul-97): Thierry Bousch 1.26 (guido 19-Jul-97): Thomas Gellekum 1.26 (guido 19-Jul-97): Konrad Hinsen 1.26 (guido 19-Jul-97): Fredrik Lundh 1.26 (guido 19-Jul-97): Neil Schemenauer 1.27 (guido 15-Aug-97): Lars Wirzenius 1.28 (guido 07-Oct-97): David Beazley 1.28 (guido 07-Oct-97): Reimer Behrends 1.28 (guido 07-Oct-97): Donn Cave 1.28 (guido 07-Oct-97): Paul Dubois 1.28 (guido 07-Oct-97): Tim Everett 1.28 (guido 07-Oct-97): Brad Howes 1.28 (guido 07-Oct-97): Bob Kahn 1.28 (guido 07-Oct-97): Hannu Krosing 1.28 (guido 07-Oct-97): Chris Lawrence 1.28 (guido 07-Oct-97): Marc-Andre Lemburg 1.28 (guido 07-Oct-97): Craig McPheeters 1.28 (guido 07-Oct-97): Geoff Philbrick 1.28 (guido 07-Oct-97): Case Roole 1.28 (guido 07-Oct-97): Hugo van Rossum 1.28 (guido 07-Oct-97): Saskia van Rossum 1.28 (guido 07-Oct-97): Greg Stein 1.28 (guido 07-Oct-97): Al Vezza 1.28 (guido 07-Oct-97): Larry Wall 1.28 (guido 07-Oct-97): Truida Wiedijk 1.28 (guido 07-Oct-97): Sue Williams 1.28 (guido 07-Oct-97): Frank Willison 1.29 (guido 24-Nov-97): Matthias Klose 1.29 (guido 24-Nov-97): Bill Noon 1.29 (guido 24-Nov-97): Charles Waldman 1.30 (guido 26-Nov-97): Lee Busby 1.30 (guido 26-Nov-97): Ben Escoto 1.30 (guido 26-Nov-97): Jeff Rush 1.31 (guido 10-Dec-97): Anders Andersen 1.31 (guido 10-Dec-97): Raymund Galvin 1.31 (guido 10-Dec-97): Chris Herborth 1.32 (guido 11-Dec-97): Doug Marien 1.33 (guido 30-Dec-97): Roy Bixler 1.33 (guido 30-Dec-97): Drew Csillag 1.33 (guido 30-Dec-97): Joseph Koshy 1.33 (guido 30-Dec-97): Roman Milner 1.33 (guido 30-Dec-97): Piet van Oostrum 1.34 (guido 09-Apr-98): Des Barry 1.34 (guido 09-Apr-98): Paul Boddie 1.34 (guido 09-Apr-98): Roger Burnham 1.34 (guido 09-Apr-98): Jaromir Dolecek 1.34 (guido 09-Apr-98): Hans Eckardt 1.34 (guido 09-Apr-98): Rob Hooft 1.34 (guido 09-Apr-98): Ben Jackson 1.34 (guido 09-Apr-98): Tamito Kajiyama 1.34 (guido 09-Apr-98): Magnus Kessler 1.34 (guido 09-Apr-98): Ron Klatchko 1.34 (guido 09-Apr-98): Piers Lauder 1.34 (guido 09-Apr-98): Kip Lehman 1.34 (guido 09-Apr-98): Christopher Lindblad 1.34 (guido 09-Apr-98): Anthony Martin 1.34 (guido 09-Apr-98): Bill van Melle 1.34 (guido 09-Apr-98): The Dragon De Monsyne 1.34 (guido 09-Apr-98): Chad Netzer 1.34 (guido 09-Apr-98): Oscar Nierstrasz 1.34 (guido 09-Apr-98): Tim O'Malley 1.34 (guido 09-Apr-98): Jason Orendorff 1.34 (guido 09-Apr-98): Ty Sarna 1.34 (guido 09-Apr-98): Kurt Vile 1.35 (guido 10-Apr-98): Jeff Epler 1.36 (guido 13-Apr-98): Peter Haight 1.36 (guido 13-Apr-98): Gary Herron 1.37 (guido 30-Jun-98): Dominic Binks 1.37 (guido 30-Jun-98): Jesse Hallio 1.37 (guido 30-Jun-98): Jonathan Hendry 1.37 (guido 30-Jun-98): John Jorgensen 1.37 (guido 30-Jun-98): Ray Loyzaga 1.37 (guido 30-Jun-98): Nigel O'Brian 1.37 (guido 30-Jun-98): Eric Raymond 1.37 (guido 30-Jun-98): Sean Reifschneider 1.37 (guido 30-Jun-98): Ben Sayer 1.38 (guido 04-Aug-98): Ralph Butler 1.38 (guido 04-Aug-98): Nicolas Chauvat 1.38 (guido 04-Aug-98): Vadim Chugunov 1.38 (guido 04-Aug-98): Carey Evans 1.38 (guido 04-Aug-98): Dan Stromberg 1.38 (guido 04-Aug-98): Gerry Wiener 1.39 (guido 11-Aug-98): Jason Harper 1.40 (guido 02-Oct-98): Robin Dunn 1.40 (guido 02-Oct-98): Jonathan Giddy 1.40 (guido 02-Oct-98): Vladimir Kushnir 1.40 (guido 02-Oct-98): Richard Wolff 1.41 (guido 21-Dec-98): Jeff Bauer 1.41 (guido 21-Dec-98): Thomas Bellman 1.41 (guido 21-Dec-98): Jochen Hayek 1.41 (guido 21-Dec-98): Detlef Lannert 1.41 (guido 21-Dec-98): Fredrik Nehr 1.41 (guido 21-Dec-98): Chris Petrilli 1.41 (guido 21-Dec-98): RajGopal Srinivasan 1.41 (guido 21-Dec-98): Greg Ward 1.41 (guido 21-Dec-98): Stefan Witzel 1.42 (guido 25-Jan-99): Greg Couch 1.42 (guido 25-Jan-99): Nathan Sullivan 1.42 (guido 25-Jan-99): Blake Winton 1.43 (guido 08-Feb-99): Uwe Zessin 1.44 (guido 16-Feb-99): Finn Bock 1.45 (guido 18-Feb-99): Scott Cotton 1.45 (guido 18-Feb-99): Toby Dickenson 1.45 (guido 18-Feb-99): Larry Hastings 1.45 (guido 18-Feb-99): Grzegorz Makarewicz 1.45 (guido 18-Feb-99): Dieter Maurer 1.45 (guido 18-Feb-99): Mike Meyer 1.45 (guido 18-Feb-99): Rafal Smotrzyk 1.45 (guido 18-Feb-99): Jean-Claude Wippler 1.46 (guido 05-Apr-99): Per Cederqvist 1.46 (guido 05-Apr-99): Michael Ernst 1.46 (guido 05-Apr-99): Tadayoshi Funaba 1.46 (guido 05-Apr-99): Magnus L. Hetland 1.46 (guido 05-Apr-99): Albert Hofkamp 1.46 (guido 05-Apr-99): Jeffrey Honig 1.46 (guido 05-Apr-99): Donald Wallace Rouse II 1.46 (guido 05-Apr-99): Orjan Johansen 1.46 (guido 05-Apr-99): Kim Knapp 1.46 (guido 05-Apr-99): Nick Lockwood 1.46 (guido 05-Apr-99): Stephanie Lockwood 1.46 (guido 05-Apr-99): Takahiro Nakayama 1.46 (guido 05-Apr-99): Christopher J. Phoenix 1.46 (guido 05-Apr-99): Bernhard Reiter 1.46 (guido 05-Apr-99): Jim Robinson 1.46 (guido 05-Apr-99): Clay Spence 1.46 (guido 05-Apr-99): Tobias Thelen 1.46 (guido 05-Apr-99): Norman Vine 1.46 (guido 05-Apr-99): Edward Welbourne 1.46 (guido 05-Apr-99): Moshe Zadka 1.46 (guido 05-Apr-99): Milan Zamazal 1.47 (guido 10-Apr-99): Andrew Dalke 1.48 (guido 13-Apr-99): Jason Asbahr 1.48 (guido 13-Apr-99): Pablo Bleyer 1.48 (guido 13-Apr-99): Mark Favas 1.48 (guido 13-Apr-99): Pat Knight 1.48 (guido 13-Apr-99): Barry Scott 1.49 (guido 13-Apr-99): Andy Dustman 1.49 (guido 13-Apr-99): Cameron Laird 1.49 (guido 13-Apr-99): Michael P. Reilly 1.49 (guido 13-Apr-99): Mike Romberg 1.50 (jhylton 28-Mar-00): Greg Ewing 1.51 (guido 31-Mar-00): Jeffrey Chang 1.51 (guido 31-Mar-00): Albert Chin-A-Young 1.51 (guido 31-Mar-00): Lars Damerow 1.51 (guido 31-Mar-00): John DuBois 1.51 (guido 31-Mar-00): Ben Gertzfield 1.51 (guido 31-Mar-00): Duncan Grisby 1.51 (guido 31-Mar-00): Milton L. Hankins 1.51 (guido 31-Mar-00): Greg Humphreys 1.51 (guido 31-Mar-00): Eric Lindvall 1.51 (guido 31-Mar-00): Loren Luke 1.51 (guido 31-Mar-00): Mikael Lyngvig 1.51 (guido 31-Mar-00): Dom Mitchell 1.51 (guido 31-Mar-00): Ofir Reichenberg 1.51 (guido 31-Mar-00): Jan Pieter Riegel 1.51 (guido 31-Mar-00): Craig Rowland 1.51 (guido 31-Mar-00): Nathan Paul Simons 1.51 (guido 31-Mar-00): Paul Sokolovsky 1.51 (guido 31-Mar-00): Laurence Tratt 1.51 (guido 31-Mar-00): Kannan Vijayan 1.51 (guido 31-Mar-00): Klaus-Juergen Wolf 1.52 (guido 31-Mar-00): Ron Bickers 1.52 (guido 31-Mar-00): Gary S. Brown 1.52 (guido 31-Mar-00): Martijn Faassen 1.52 (guido 31-Mar-00): Fred Gansevles 1.52 (guido 31-Mar-00): Barry Haddow 1.52 (guido 31-Mar-00): Joerg-Cyril Hoehle 1.52 (guido 31-Mar-00): Gerrit Holl 1.52 (guido 31-Mar-00): Brian Hooper 1.52 (guido 31-Mar-00): Randall Hopper 1.52 (guido 31-Mar-00): Michael Hudson 1.52 (guido 31-Mar-00): Lucas de Jonge 1.52 (guido 31-Mar-00): Andreas Jung 1.52 (guido 31-Mar-00): Nick Maclaren 1.52 (guido 31-Mar-00): Andy Robinson 1.52 (guido 31-Mar-00): Nick Russo 1.52 (guido 31-Mar-00): Eric Tiedemann 1.52 (guido 31-Mar-00): Christian Tismer 1.52 (guido 31-Mar-00): Greg V. Wilson 1.53 (guido 31-Mar-00): Samuel L. Bayer 1.53 (guido 31-Mar-00): Gary Duzan 1.53 (guido 31-Mar-00): Eugene Dvurechenski 1.53 (guido 31-Mar-00): Andreas Faerber 1.53 (guido 31-Mar-00): Thomas Heller 1.53 (guido 31-Mar-00): Bernhard Herzog 1.53 (guido 31-Mar-00): Chih-Hao Huang 1.53 (guido 31-Mar-00): Vivek Khera 1.53 (guido 31-Mar-00): Mads Kiilerich 1.53 (guido 31-Mar-00): Greg McFarlane 1.53 (guido 31-Mar-00): Paul Prescod 1.53 (guido 31-Mar-00): Sam Rushing 1.53 (guido 31-Mar-00): John W. Shipman 1.53 (guido 31-Mar-00): Frank Stajano 1.53 (guido 31-Mar-00): Peter Stoehr 1.53 (guido 31-Mar-00): Christian Tanzer 1.53 (guido 31-Mar-00): Jason Trowbridge 1.53 (guido 31-Mar-00): Stephen Turner 1.53 (guido 31-Mar-00): Doug Wyatt 1.54 (guido 31-Mar-00): Bastian Kleineidam 1.54 (guido 31-Mar-00): Thomas Wouters 1.55 (guido 10-Apr-00): Stephen D Evans 1.55 (guido 10-Apr-00): Lars Marius Garshol 1.55 (guido 10-Apr-00): Lance Finn Helsten 1.55 (guido 10-Apr-00): Gregor Hoffleit 1.55 (guido 10-Apr-00): Nadav Horesh 1.55 (guido 10-Apr-00): Lawrence Kesteloot 1.55 (guido 10-Apr-00): Lenny Kneler 1.55 (guido 10-Apr-00): Greg Kochanski 1.55 (guido 10-Apr-00): Hrvoje Niksic 1.55 (guido 10-Apr-00): Nicholas Riley 1.55 (guido 10-Apr-00): David Scherer 1.55 (guido 10-Apr-00): Bruce Sherwood 1.55 (guido 10-Apr-00): Kragen Sitaker 1.56 (guido 11-May-00): Caolan McNamara 1.56 (guido 11-May-00): Trent Mick 1.57 (gvanross 29-Jun-00): Oleg Broytmann 1.57 (gvanross 29-Jun-00): Kalle Svensson 1.58 (gward 29-Jun-00): Harry Henry Gebel 1.59 (bwarsaw 30-Aug-00): James Henstridge 1.60 (gvanross 02-Sep-00): Martin von L?wis 1.60 (gvanross 02-Sep-00): Hajime Saitou 1.61 (gvanross 04-Sep-00): Audun S. Runde 1.62 (tim_one 16-Sep-00): Brad Chapman 1.63 (gvanross 25-Sep-00): Stefan Norberg 1.63 (gvanross 25-Sep-00): Todd R. Palmer 1.63 (gvanross 25-Sep-00): Neale Pickett 1.63 (gvanross 25-Sep-00): Dan Pierson 1.63 (gvanross 25-Sep-00): Fran?ois Pinard 1.63 (gvanross 25-Sep-00): Dan Wolfe 1.63 (gvanross 25-Sep-00): Bob Yodlowski 1.64 (fdrake 02-Oct-00): Daniel Dittmar 1.65 (jhylton 09-Oct-00): Stan Bubrouski 1.67 (gvanross 09-Oct-00): Tony Lownds 1.68 (gvanross 09-Oct-00): Jean-Claude Rimbault 1.69 (gvanross 03-Jan-01): Daniel Calvelo 1.69 (gvanross 03-Jan-01): Mike Clarkson 1.70 (tim_one 14-Jan-01): Jeffery Collins 1.71 (tim_one 18-Jan-01): Paul Jackson 1.72 (akuchlin 19-Jan-01): Jason Tishler 1.73 (gvanross 23-Jan-01): Greg Ball 1.73 (gvanross 23-Jan-01): Bill Bumgarner 1.73 (gvanross 23-Jan-01): Doug Fort 1.73 (gvanross 23-Jan-01): Luke Kenneth Casson Leighton 1.74 (tim_one 26-Jan-01): Ivan Frohne 1.74 (tim_one 26-Jan-01): Janne Sinkkonen 1.75 (tim_one 27-Jan-01): Dave Cole 1.76 (tim_one 28-Jan-01): Randall Kern 1.77 (jhylton 02-Feb-01): Grant Edwards 1.78 (jhylton 05-Feb-01): Nick Mathewson 1.79 (tim_one 09-Feb-01): Lawrence Hudson 1.80 (lemburg 09-Feb-01): Benjamin Collar 1.81 (nascheme 10-Feb-01): David M. Cooke 1.82 (akuchlin 21-Feb-01): Donovan Baarda 1.83 (gvanross 01-Mar-01): Samuele Pedroni 1.84 (tim_one 02-Mar-01): William Tanksley 1.85 (gvanross 02-Mar-01): Peter Schneider-Kamp 1.85 (gvanross 02-Mar-01): Sam Schulenburg 1.85 (gvanross 02-Mar-01): Dietmar Schwertberger 1.87 (gvanross 22-Mar-01): Gordon McMillan 1.88 (gvanross 27-Mar-01): Robin Thomas 1.89 (tim_one 08-Apr-01): Neal Norwitz 1.90 (moshez 09-Apr-01): Itamar Shtull-Trauring 1.91 (gvanross 10-Apr-01): Alastair Burt 1.91 (gvanross 10-Apr-01): Walter D?rwald 1.91 (gvanross 10-Apr-01): Christopher Lee 1.92 (gvanross 11-Apr-01): Billy G. Allie 1.93 (gvanross 14-Apr-01): Pete Shinners 1.94 (gvanross 15-Apr-01): Douglas Orr 1.94 (gvanross 15-Apr-01): Juan M. Bello Rivas 1.95 (tim_one 21-Apr-01): Stephen Hansen 1.96 (tim_one 24-Apr-01): Kevin Rodgers 1.97 (tim_one 09-Jun-01): Armin Rigo 1.98 (tim_one 17-Jun-01): Michael Chermside 1.99 (gvanross 23-Jul-01): Alex Coventry 1.100 (gvanross 31-Jul-01): John Popplewell 1.101 (tim_one 04-Aug-01): David Bolen 1.102 (tim_one 08-Aug-01): Erik van Blokland 1.103 (nascheme 09-Aug-01): Gregory P. Smith 1.104 (gvanross 09-Aug-01): Josh Cogliati 1.105 (tim_one 12-Aug-01): David Goodger 1.106 (akuchlin 13-Aug-01): Tarn Weisner Burton 1.107 (gvanross 17-Aug-01): Henrik Weber 1.109 (gvanross 02-Sep-01): Mike Zarnstorff 1.110 (gvanross 10-Sep-01): Lionel Ulmer 1.111 (gvanross 11-Sep-01): Gerhard H?ring 1.112 (jackjans 11-Sep-01): Donovan Preston 1.113 (tim_one 30-Sep-01): Tom Epperly 1.114 (gvanross 01-Oct-01): Travers Naran 1.115 (tim_one 01-Oct-01): Steven Scott 1.116 (gvanross 05-Oct-01): Doobee R. Tzeck 1.117 (fdrake 05-Oct-01): Fred L. Drake, Jr. 1.118 (loewis 09-Oct-01): Cesar Eduardo Barros 1.119 (gvanross 12-Oct-01): Jason Lowe 1.120 (gvanross 15-Oct-01): Alfonso Baciero 1.121 (gvanross 15-Oct-01): Jason Hildebrand 1.122 (gvanross 15-Oct-01): Richard Townsend 1.123 (gvanross 16-Oct-01): Shane Hathaway 1.124 (gvanross 17-Oct-01): Quinn Dunkan 1.125 (gvanross 17-Oct-01): Joe Norton 1.126 (gvanross 18-Oct-01): Roeland Rengelink 1.127 (gvanross 18-Oct-01): Max Neunh?ffer 1.128 (gvanross 19-Oct-01): Chris Gonnerman 1.129 (gvanross 20-Oct-01): Michael Piotrowski 1.130 (gvanross 23-Oct-01): Jay T. Miller 1.131 (gvanross 25-Oct-01): Dave Brueck 1.132 (gvanross 26-Oct-01): Noah Spurrier 1.133 (gvanross 30-Oct-01): David Cinege 1.134 (tim_one 30-Oct-01): Luigi Ballabio 1.135 (tim_one 31-Oct-01): Michael Muller 1.137 (tim_one 31-Oct-01): Rich Salz 1.138 (tim_one 27-Nov-01): Alex Martelli 1.139 (gvanross 03-Dec-01): Greg Chapman 1.140 (gvanross 04-Dec-01): Stefan Schwarzer 1.141 (gvanross 06-Dec-01): David Abrahams 1.142 (tim_one 07-Dec-01): Anthony Roach 1.143 (gvanross 08-Dec-01): Burton Radons 1.144 (gvanross 10-Dec-01): David Jacobs 1.145 (jhylton 13-Dec-01): Danny Yoo 1.146 (tim_one 19-Dec-01): David Costanzo 1.147 (gvanross 19-Dec-01): Dan Parisien 1.148 (gvanross 20-Dec-01): Mark Summerfield 1.149 (gvanross 20-Dec-01): Artur Zaprzala 1.150 (jackjans 27-Dec-01): Jens B. Jorgensen 1.151 (gvanross 28-Dec-01): Eduardo P?rez 1.152 (loewis 01-Jan-02): James A Morrison 1.153 (loewis 01-Jan-02): Brad Clements 1.154 (loewis 01-Jan-02): Gustavo Niemeyer 1.155 (loewis 12-Jan-02): Robert Kern 1.156 (theller 18-Jan-02): Martin Bless 1.157 (loewis 26-Jan-02): Eddy De Greef 1.158 (loewis 26-Jan-02): Bryce "Zooko" Wilcox-O'Hearn 1.159 (gvanross 25-Feb-02): Edward K. Ream 1.160 (tim_one 28-Feb-02): Gerald S. Williams 1.161 (gvanross 28-Feb-02): Derek Harland 1.162 (gvanross 01-Mar-02): Marc Recht 1.163 (gvanross 01-Mar-02): Naofumi Honda 1.164 (loewis 29-Mar-02): Ondrej Palkovsky 1.165 (loewis 08-Apr-02): Kirill Simonov 1.167 (nnorwitz 12-Apr-02): Raymond Hettinger 1.168 (gvanross 15-Apr-02): Ben Darnell 1.168 (gvanross 15-Apr-02): Vincent Fiack 1.168 (gvanross 15-Apr-02): Frank J. Tobin 1.168 (gvanross 15-Apr-02): Jacques A. Vidrine 1.169 (tim_one 19-Apr-02): Hern?n Mart?nez Foffani 1.170 (loewis 20-Apr-02): Michel Van den Bergh 1.171 (gvanross 23-Apr-02): John Williams 1.172 (loewis 08-May-02): Geert Jansen 1.173 (rhetting 16-May-02): Matthew Dixon Cowles 1.174 (gvanross 21-May-02): Christos Georgiou 1.175 (tim_one 23-May-02): John Aycock 1.176 (gvanross 30-May-02): Holger Krekel 1.177 (loewis 06-Jun-02): Peter ?strand 1.178 (gvanross 06-Jun-02): Michael Gilfix 1.179 (gvanross 07-Jun-02): Bernard Yue 1.180 (gvanross 10-Jun-02): Erik Anders?n 1.181 (loewis 11-Jun-02): Octavian Cerna 1.182 (gvanross 14-Jun-02): Oren Tirosh 1.183 (gvanross 18-Jun-02): Tim Northover 1.184 (rhetting 24-Jun-02): Kevin Jacobs 1.185 (jhylton 28-Jun-02): Bob Kline 1.186 (gvanross 12-Jul-02): Fernando P?rez 1.187 (tim_one 15-Jul-02): Jonathan Hogg 1.188 (tim_one 15-Jul-02): Andrew I MacIntyre 1.188 (tim_one 15-Jul-02): Tim MacKenzie 1.189 (nnorwitz 19-Jul-02): Brett Cannon 1.190 (gvanross 01-Aug-02): Nathan Srebro 1.191 (gvanross 02-Aug-02): Kevin O'Connor 1.192 (gvanross 05-Aug-02): Zack Weinberg 1.193 (tim_one 12-Aug-02): Christopher A. Craig 1.194 (gvanross 12-Aug-02): Cesar Douady 1.194 (gvanross 12-Aug-02): Jiba 1.195 (loewis 14-Aug-02): Hisao Suzuki 1.196 (rhetting 18-Aug-02): Steve Purcell 1.197 (jackjans 19-Aug-02): Michael J. Barber 1.197 (jackjans 19-Aug-02): Chris Barker 1.197 (jackjans 19-Aug-02): Bill Bedford 1.197 (jackjans 19-Aug-02): Erik van Blokland 1.197 (jackjans 19-Aug-02): Tom Bridgman 1.197 (jackjans 19-Aug-02): Daniel Brotsky 1.197 (jackjans 19-Aug-02): Dean Draayer 1.197 (jackjans 19-Aug-02): Russell Finn 1.197 (jackjans 19-Aug-02): Nitin Ganatra 1.197 (jackjans 19-Aug-02): Tony Ingraldi 1.197 (jackjans 19-Aug-02): Tattoo Mabonzo K. 1.197 (jackjans 19-Aug-02): Jacob Kaplan-Moss 1.197 (jackjans 19-Aug-02): Luc Lefebvre 1.197 (jackjans 19-Aug-02): Tom Loredo 1.197 (jackjans 19-Aug-02): Pascal Oberndoerfer 1.197 (jackjans 19-Aug-02): Russel Owen 1.197 (jackjans 19-Aug-02): M. Papillon 1.197 (jackjans 19-Aug-02): Alexandre Parenteau 1.197 (jackjans 19-Aug-02): Christopher Smith 1.197 (jackjans 19-Aug-02): Oliver Steele 1.197 (jackjans 19-Aug-02): Chris Stern 1.197 (jackjans 19-Aug-02): Corran Webster 1.197 (jackjans 19-Aug-02): Gordon Worley 1.198 (gvanross 20-Aug-02): Inyeol Lee 1.199 (jackjans 28-Aug-02): Bill Fancher 1.200 (jackjans 06-Sep-02): Dinu Gherman 1.201 (gvanross 11-Sep-02): Laura Creighton 1.201 (gvanross 11-Sep-02): Anders Qvist 1.202 (gvanross 11-Sep-02): Brian Quinlan 1.203 (gvanross 11-Sep-02): Mats Wichmann 1.204 (gvanross 18-Sep-02): Martijn Pieters 1.205 (gvanross 23-Sep-02): Attila Babo 1.206 (loewis 30-Sep-02): Tim Rice 1.207 (gvanross 09-Oct-02): Flemming Kj?r Jensen 1.207 (gvanross 09-Oct-02): Martin Ligr 1.207 (gvanross 09-Oct-02): Mark Perrego 1.207 (gvanross 09-Oct-02): Jody Winston 1.208 (loewis 10-Oct-02): Steve Reeves 1.209 (nnorwitz 10-Oct-02): Greg Copeland 1.210 (loewis 13-Oct-02): Martin Franklin 1.211 (nnorwitz 17-Oct-02): Maxim Dzumanenko 1.212 (loewis 26-Oct-02): Daniel Stutzbach 1.213 (loewis 26-Oct-02): Matthew Boedicker 1.214 (loewis 04-Nov-02): Ha Shao 1.215 (mwh 08-Nov-02): Richie Hindle 1.216 (loewis 21-Nov-02): Juan David Ib??ez Palomar 1.217 (loewis 22-Nov-02): J?rgen Hermann 1.218 (tim_one 25-Nov-02): Marius Gedminas 1.219 (loewis 06-Dec-02): Jean-Fran?ois Pi?ronne 1.220 (tim_one 28-Dec-02): Ben Laurie 1.221 (loewis 31-Dec-02): Paul Swartz 1.222 (nnorwitz 05-Jan-03): Lars Gust?bel 1.223 (gvanross 07-Jan-03): Michael Stone 1.224 (loewis 15-Jan-03): Gyro Funch 1.224 (loewis 15-Jan-03): Geoff Furnish 1.225 (gvanross 27-Jan-03): Geoff Talvola 1.226 (gvanross 03-Feb-03): Scott David Daniels 1.227 (nnorwitz 10-Feb-03): Jeffrey Ollie 1.227 (nnorwitz 10-Feb-03): Grant Olson 1.227 (nnorwitz 10-Feb-03): Denis S. Otkidach 1.228 (rhetting 09-Mar-03): Sebastien Keim 1.229 (gvanross 14-Mar-03): Stuart Bishop 1.230 (twouters 17-Mar-03): Hye-Shik Chang 1.231 (montanar 20-Mar-03): Kevin Altis 1.231 (montanar 20-Mar-03): Andrew McNamara 1.231 (montanar 20-Mar-03): Cliff Wells 1.232 (jackjans 25-Mar-03): Frank Vercruesse 1.233 (gvanross 19-Apr-03): Terry Carroll 1.234 (gvanross 25-Apr-03): Jp Calderone 1.235 (jhylton 05-May-03): Mihai Ibanescu 1.236 (jhylton 09-May-03): John Anderson 1.237 (bcannon 12-May-03): John J. Lee 1.238 (rhetting 20-May-03): Paul Moore 1.239 (rhetting 09-Jun-03): Stephen M. Gava 1.239 (rhetting 09-Jun-03): Kurt B. Kaiser 1.240 (rhetting 28-Jun-03): Steven Taschuk 1.241 (rhetting 29-Jun-03): Bob Halley 1.242 (jhylton 17-Jul-03): Vincent Delft 1.243 (jhylton 17-Jul-03): Gavrie Philipson 1.244 (akuchlin 13-Aug-03): Joerg Lehmann 1.245 (rhetting 01-Sep-03): George Yoshida 1.246 (rhetting 01-Sep-03): Andrew Gaul 1.247 (rhetting 02-Sep-03): Alexander Belopolsky 1.248 (gvanross 08-Oct-03): Wojtek Walczak 1.249 (bcannon 12-Oct-03): Michael Stone 1.251 (bcannon 14-Oct-03): Bjorn Pettersen 1.252 (loewis 31-Oct-03): Hector Urtubia 1.253 (gvanross 10-Nov-03): Iustin Pop 1.254 (loewis 10-Nov-03): Yves Dionne 1.257 (rhetting 05-Jan-04): Dmitry Vasiliev 1.258 (loewis 31-Jan-04): Frederik Fix 1.259 (rhetting 08-Feb-04): Mike Pall 1.260 (rhetting 04-Mar-04): Bob Ippolito 1.261 (mondrago 23-Mar-04): Nick Bastin From kbk at shore.net Mon Apr 26 13:59:29 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon Apr 26 13:59:34 2004 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200404261759.i3QHxT14013977@hydra.localdomain> Patch / Bug Summary ___________________ Patches : 270 open ( +5) / 2366 closed ( +1) / 2636 total ( +6) Bugs : 754 open ( +2) / 3993 closed (+10) / 4747 total (+12) RFE : 133 open ( -1) / 124 closed ( +0) / 257 total ( -1) New / Reopened Patches ______________________ CodeContext - an extension to show you where you are (2004-04-16) CLOSED http://python.org/sf/936169 reopened by noamr Explain 'in' keyword as it is first used (2004-04-22) http://python.org/sf/940026 opened by Gerrit Holl Replace if/elif chain with dispatch pattern in sre_compile (2004-04-23) http://python.org/sf/941071 opened by Raymond Hettinger Allow any encodings other than latin-1 in interactive interp (2004-04-24) http://python.org/sf/941229 opened by Hye-Shik Chang Minimal fix for bug 940578 (glob.glob on broken symlinks) (2004-04-24) http://python.org/sf/941486 opened by Cherniavsky Beni PEP309 Partial implementation (2004-04-26) http://python.org/sf/941881 opened by Hye-Shik Chang Patches Closed ______________ CodeContext - an extension to show you where you are (2004-04-16) http://python.org/sf/936169 closed by kbk Py_INCREF/DECREF available as macros only (2004-04-20) http://python.org/sf/938302 closed by theller New / Reopened Bugs ___________________ message.as_string hoses headers (2004-04-20) CLOSED http://python.org/sf/938890 opened by Carl Karsten cPickle.Pickler: in "list" mode, no way to set protocol (2004-04-22) http://python.org/sf/939395 opened by John Belmonte Building python 2.3.3 RPM on Mandrake 9.2. fails (2004-04-22) http://python.org/sf/939699 opened by fabien wahl Logging failes for rotating file log (during rollOver) (2004-04-22) http://python.org/sf/939836 opened by smeyers2002 xml parser bug (2004-04-22) CLOSED http://python.org/sf/940263 opened by David Tefft pydoc.Helper.help() ignores input/output init parameters (2004-04-22) http://python.org/sf/940286 opened by Maxim Krikun string.lstrip problem with '_' string (2004-04-22) CLOSED http://python.org/sf/940316 opened by Arnold Python crashes with bus error on sparc64. (2004-04-22) http://python.org/sf/940354 opened by Generic Player misbehaviour with map / lambda / isdigit() (2004-04-23) CLOSED http://python.org/sf/940522 opened by dreary glob.glob inconsistent about broken symlinks (2004-04-23) http://python.org/sf/940578 opened by Cherniavsky Beni section 5.10: 'not' returns boolean, not int (2004-04-23) CLOSED http://python.org/sf/940579 opened by Jon Ashley List with Canvas.create_line Option arrow=LAST Broke (2004-04-24) http://python.org/sf/941262 opened by Brian Brooks AIX shared library fix (2004-04-24) http://python.org/sf/941346 opened by Dick Dunbar Bugs Closed ___________ Random.choice doesn't work with sets. (2004-04-17) http://python.org/sf/936988 closed by rhettinger minidom.py writes TEXT_NODE wrong (2004-04-18) http://python.org/sf/937282 closed by loewis re.sub on u'' gives NON-unicode '' (2004-03-31) http://python.org/sf/926075 closed by perky message.as_string hoses headers (2004-04-20) http://python.org/sf/938890 closed by carlk message.as_string hoses headers (2004-04-20) http://python.org/sf/938890 closed by tim_one Shelve slow after 7/8000 key (2004-01-21) http://python.org/sf/881522 closed by tim_one Slicing changes item-attributes of an instance of a UserList (2004-04-19) http://python.org/sf/937700 closed by hflori xml parser bug (2004-04-22) http://python.org/sf/940263 closed by loewis string.lstrip problem with '_' string (2004-04-22) http://python.org/sf/940316 closed by rhettinger misbehaviour with map / lambda / isdigit() (2004-04-23) http://python.org/sf/940522 closed by tim_one section 5.10: 'not' returns boolean, not int (2004-04-23) http://python.org/sf/940579 closed by rhettinger From theller at python.net Tue Apr 27 14:59:25 2004 From: theller at python.net (Thomas Heller) Date: Tue Apr 27 14:59:34 2004 Subject: [Python-Dev] Need help with patch Message-ID: I would appreciate very much if someone can at least proofread the patch that I posted to http://python.org/sf/876278. Even better would be if users of modulefinder would try this with really complicated applications. I'm not sure if this should go into the 23-maintanance branch. Thomas From ndbecker2 at verizon.net Tue Apr 27 15:55:46 2004 From: ndbecker2 at verizon.net (Neal D. Becker) Date: Tue Apr 27 15:55:52 2004 Subject: [Python-Dev] libjit - looks interesting Message-ID: A free jit library. Might be useful for Python. Anyone looking at this? http://www.southern-storm.com.au/libjit.html From bob at redivi.com Tue Apr 27 16:20:32 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue Apr 27 16:16:14 2004 Subject: [Python-Dev] libjit - looks interesting In-Reply-To: References: Message-ID: <55C12706-9888-11D8-B57C-000A95686CD8@redivi.com> On Apr 27, 2004, at 3:55 PM, Neal D. Becker wrote: > A free jit library. Might be useful for Python. Anyone looking at > this? > > http://www.southern-storm.com.au/libjit.html I doubt it, it's GPL'ed. There are other more mature GPL'ed JIT engines out there that we aren't looking at :) -bob From martin at v.loewis.de Tue Apr 27 17:01:23 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Apr 27 17:01:52 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <20040426151607.GA2361@burma.localdomain> References: <20040425234019.GA6436@burma.localdomain> <1082988793.17933.59.camel@anthem.wooz.org> <20040426151607.GA2361@burma.localdomain> Message-ID: <408ECA23.8000505@v.loewis.de> Gustavo Niemeyer wrote: > - The current scheme may introduce a wrong practice: forcing > translators to use some specific encoding to avoid breaking the > program. That's not true. Applications should just always use ugettext. Regards, Martin From martin at v.loewis.de Tue Apr 27 17:07:27 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue Apr 27 17:07:41 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <20040425234019.GA6436@burma.localdomain> References: <20040425234019.GA6436@burma.localdomain> Message-ID: <408ECB8F.3070707@v.loewis.de> Gustavo Niemeyer wrote: > # Encode the Unicode tmsg back to an 8-bit string, if possible > if self._charset: > return tmsg.encode(self._charset) > > to use the system encoding (sys.getdefaultencoding()) instead of > self._charset. That shouldn't be sys.getdefaultencoding(), but locale.getpreferredencoding(). However, I agree with Barry that the current behaviour should not be changed. People may already rely on gettext returning byte strings as-is. > - Change the default codeset used by gettext.py in functions > returning an encoded string to match the system encoding. No. Explicit is better that implicit; users desiring that feature should write _charset = locale.getpreferredencoding() def _(msg): return dgettext("domain", msg).encode(_charset) I advocate never to use gettext.install, in which case you have a custom _ implementation *anyway*, which would then also include the textual domain. It should not be too much effort for that function to transcode if desired. > - Introduce bind_textdomain_codeset() in locale. > - Introduce bind_textdomain_codeset() in gettext.py implementing > an equivalent functionality. That is ok. You could also try to provide that feature consistently, e.g. inside .install. Regards, Martin From niemeyer at conectiva.com Tue Apr 27 17:08:56 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Apr 27 17:09:01 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <408ECA23.8000505@v.loewis.de> References: <20040425234019.GA6436@burma.localdomain> <1082988793.17933.59.camel@anthem.wooz.org> <20040426151607.GA2361@burma.localdomain> <408ECA23.8000505@v.loewis.de> Message-ID: <20040427210856.GA12923@burma.localdomain> > >- The current scheme may introduce a wrong practice: forcing > > translators to use some specific encoding to avoid breaking the > > program. > > That's not true. Applications should just always use ugettext. As I said before, I agree that ugettext is the best way to get translations in a generic way. OTOH, that's not what I'm discussing here, as you may observe by the general discussion. -- Gustavo Niemeyer http://niemeyer.net From niemeyer at conectiva.com Tue Apr 27 17:26:43 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Apr 27 17:26:44 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <408ECB8F.3070707@v.loewis.de> References: <20040425234019.GA6436@burma.localdomain> <408ECB8F.3070707@v.loewis.de> Message-ID: <20040427212643.GB12923@burma.localdomain> > That shouldn't be sys.getdefaultencoding(), but > locale.getpreferredencoding(). Agreed. > However, I agree with Barry that the current behaviour should not > be changed. People may already rely on gettext returning byte > strings as-is. Barry also said: """ Any sane app is going to use the class-based API an the ugettext() method anyway, so maybe it does make sense to simply be API compatible with GNU gettext for the old-style interface. """ > >- Change the default codeset used by gettext.py in functions > > returning an encoded string to match the system encoding. > > No. Explicit is better that implicit; users desiring that > feature should write [...] You belive that returning a string in some unpredictable encoding used by the translator is explicit? I have to disagree. We have something named 'gettext' with a different behavior than the original project, in a way that breaks the concepts used to decide the behavior of the classical implementation. > I advocate never to use gettext.install, in which case you > have a custom _ implementation *anyway*, which would then [...] "You shouldn't be using it" is not a point I'm taking into account. > >- Introduce bind_textdomain_codeset() in locale. > >- Introduce bind_textdomain_codeset() in gettext.py implementing > > an equivalent functionality. > > That is ok. You could also try to provide that feature > consistently, e.g. inside .install. Agreed. -- Gustavo Niemeyer http://niemeyer.net From niemeyer at conectiva.com Tue Apr 27 17:33:26 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue Apr 27 17:33:27 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <408ECB8F.3070707@v.loewis.de> References: <20040425234019.GA6436@burma.localdomain> <408ECB8F.3070707@v.loewis.de> Message-ID: <20040427213326.GA13163@burma.localdomain> > However, I agree with Barry that the current behaviour should not > be changed. People may already rely on gettext returning byte > strings as-is. Btw, about people relying on the current implementation, we could introduce this as a warning in 2.4, which could be disabled by using "bind_textdomain_codeset()" for the expected translation, and then turn the behavior into the default one on 2.4+1. -- Gustavo Niemeyer http://niemeyer.net From martin at v.loewis.de Wed Apr 28 14:15:50 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed Apr 28 14:16:09 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <20040427212643.GB12923@burma.localdomain> References: <20040425234019.GA6436@burma.localdomain> <408ECB8F.3070707@v.loewis.de> <20040427212643.GB12923@burma.localdomain> Message-ID: <408FF4D6.3000108@v.loewis.de> Gustavo Niemeyer wrote: > You belive that returning a string in some unpredictable > encoding used by the translator is explicit? No. What is explicit here is that we return the bytes in the .mo file; and conversion of these bytes should be requested explicitly. As for compatibility with GNU gettext: Older versions of GNU gettext did not perform any conversion, either. I think changing the behaviour of an existing function was a bad thing to do for GNU gettext, as well, and I don't think we should repeat that mistake. It would be good to find out what users of Python gettext think about a possible change. Temporarily warning about this change of behaviour is also unacceptable. Adding another function, e.g. lgettext (for local charset), along with lngettext, ldgettext, might be another solution. Regards, Martin From niemeyer at conectiva.com Wed Apr 28 14:20:36 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Wed Apr 28 14:20:46 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <408FF4D6.3000108@v.loewis.de> References: <20040425234019.GA6436@burma.localdomain> <408ECB8F.3070707@v.loewis.de> <20040427212643.GB12923@burma.localdomain> <408FF4D6.3000108@v.loewis.de> Message-ID: <20040428182036.GA8765@burma.localdomain> > >You belive that returning a string in some unpredictable > >encoding used by the translator is explicit? > > No. What is explicit here is that we return the bytes in > the .mo file; and conversion of these bytes should be > requested explicitly. > > As for compatibility with GNU gettext: Older versions > of GNU gettext did not perform any conversion, either. > I think changing the behaviour of an existing function > was a bad thing to do for GNU gettext, as well, and I > don't think we should repeat that mistake. I wasn't aware about this. > It would be good to find out what users of Python > gettext think about a possible change. I'd be glad to hear from them as well. Anyone? > Temporarily warning about this change of behaviour > is also unacceptable. Adding another function, e.g. > lgettext (for local charset), along with lngettext, > ldgettext, might be another solution. Ack. Thanks for discussing. -- Gustavo Niemeyer http://niemeyer.net From niemeyer at conectiva.com Wed Apr 28 16:43:24 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Wed Apr 28 16:43:27 2004 Subject: [Python-Dev] PEP 754 In-Reply-To: References: <200404231843.i3NIhC0109140@mcs.anl.gov> Message-ID: <20040428204324.GA11446@burma.localdomain> > That isn't a good objection for this specific patch -- it's pure > Python, and picks apart bits using the struct module. IIRC, it wasn't > correctly written for most 64-bit boxes, but that looked like a > shallow flaw (I think it assumed that a struct "l" format code, in > native mode, always yields a 32-bit thingie). I was just wondering, now that C99 supports part of IEEE754, should we really get into the job of implementing our own mechanism? As an example, the attached patch exports part of the C99 interface to Python. If we agree that this is something interesting to have in Python, I'll be willing to prepare a patch including support for this interface and hacking some bits to introduce a better IEEE754 support (e.g. checking nan comparisons, etc). -- Gustavo Niemeyer http://niemeyer.net -------------- next part -------------- #include #include #define CHECK_FLOAT(o) \ if (!PyFloat_Check(o)) { \ PyErr_SetString(PyExc_TypeError, "float argument required"); \ return NULL; \ } static PyObject * ieee754_isnan(PyObject *self, PyObject *o) { CHECK_FLOAT(o); return PyBool_FromLong(isnan(PyFloat_AsDouble(o))); } static PyObject * ieee754_isfinite(PyObject *self, PyObject *o) { CHECK_FLOAT(o); return PyBool_FromLong(isfinite(PyFloat_AsDouble(o))); } static PyMethodDef ieee754_methods[] = { {"isnan", ieee754_isnan, METH_O, ""}, {"isfinite", ieee754_isfinite, METH_O, ""}, {NULL, NULL} }; DL_EXPORT(void) initieee754(void) { PyObject *m = Py_InitModule3("ieee754", ieee754_methods, ""); PyModule_AddObject(m, "nan", PyFloat_FromDouble(NAN)); PyModule_AddObject(m, "inf", PyFloat_FromDouble(INFINITY)); } From edcjones at erols.com Thu Apr 29 00:09:41 2004 From: edcjones at erols.com (Edward C. Jones) Date: Thu Apr 29 00:14:50 2004 Subject: [Python-Dev] Subtyping mutable and immutable built-in types Message-ID: <40908005.20905@erols.com> #! /usr/bin/env python # According to # http://www.cafepy.com/articles/python_attributes_and_methods/ch03s02.html # the correct ways to subtype mutable and immutable built-in types are # Mutable class MyList(list): def __init__(self, itr): list.__init__(self, [int(x) for x in itr]) # Immutable class MyTuple(tuple): def __new__(typ, itr): return tuple.__new__(typ, [int(x) for x in itr]) # This doesn't work. class MyTuple(tuple): def __init__(self, itr): tuple.__init__(self, [int(x) for x in itr]) # A variant is class MyList(list): def __init__(self, *args): list.__init__(self, args) class MyTuple(tuple): def __new__(typ, *args): return tuple.__new__(typ, args) # This doesn't work. class MyTuple2(tuple): def __init__(self, *args): tuple.__init__(self, args) print MyList(1, 2) print MyTuple(1, 2) print MyTuple2(1, 2) # This warty stuff needs documenting. From arigo at tunes.org Thu Apr 29 07:59:11 2004 From: arigo at tunes.org (Armin Rigo) Date: Thu Apr 29 08:02:29 2004 Subject: [Python-Dev] Slides: how Psyco works Message-ID: <20040429115911.GA15802@vicky.ecs.soton.ac.uk> Hello, I have put on-line some slides from the Python UK conference at ACCU 2004, explaining how Psyco works. It is actually a Pygame application... As far as I can tell, it is the first time someone in the room actually understood something at the end :-) http://psyco.sourceforge.net/ I hope it should help to make Psyco a bit less mysterious, and also explain why it is difficult to have a general idea about what kind of speed-up you can expect for specific kinds of code: it is, after all, a pretty low-level "local" process that Psyco does, and it sometimes pays off and sometimes not. It also explains why this process is very much like the usual interpretation that CPython does. If you think about it you might see how useful it would be for Psyco to build on top of an interpreter in a better language than C (i.e. one that can be analysed, not just blindly run). A bient?t, Armin. From mchermside at ingdirect.com Thu Apr 29 08:15:27 2004 From: mchermside at ingdirect.com (Chermside, Michael) Date: Thu Apr 29 08:15:31 2004 Subject: [Python-Dev] Subtyping mutable and immutable built-in types Message-ID: <0CFFADBB825C6249A26FDF11C1772AE155096F@ingdexj1.ingdirect.com> Edward C. Jones writes: > # According to > # http://www.cafepy.com/articles/python_attributes_and_methods/ch03s02.html > # the correct ways to subtype mutable and immutable built-in types are [...] > # This warty stuff needs documenting. The use of __init__ and __new__ with immutable classes is already documented at "http://www.python.org/2.2/descrintro.html#__new__". If you have a suggestion of a another place this should be described, let us know or better yet file a bug report at SourceForge. -- Michael Chermside This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it. From barry at python.org Thu Apr 29 08:26:43 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 29 08:26:50 2004 Subject: [Python-Dev] Small issues in gettext support In-Reply-To: <408FF4D6.3000108@v.loewis.de> References: <20040425234019.GA6436@burma.localdomain> <408ECB8F.3070707@v.loewis.de> <20040427212643.GB12923@burma.localdomain> <408FF4D6.3000108@v.loewis.de> Message-ID: <1083241602.20310.82.camel@anthem.wooz.org> On Wed, 2004-04-28 at 14:15, "Martin v. L?wis" wrote: > Temporarily warning about this change of behaviour > is also unacceptable. Adding another function, e.g. > lgettext (for local charset), along with lngettext, > ldgettext, might be another solution. That would work for me. I'd much rather see a different interface to support what Gustavo wants than to change an existing interface. -Barry From aahz at pythoncraft.com Thu Apr 29 10:19:34 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Apr 29 10:19:38 2004 Subject: [Python-Dev] Subtyping mutable and immutable built-in types In-Reply-To: <0CFFADBB825C6249A26FDF11C1772AE155096F@ingdexj1.ingdirect.com> References: <0CFFADBB825C6249A26FDF11C1772AE155096F@ingdexj1.ingdirect.com> Message-ID: <20040429141934.GB29986@panix.com> On Thu, Apr 29, 2004, Chermside, Michael wrote: > > The use of __init__ and __new__ with immutable classes is already > documented at "http://www.python.org/2.2/descrintro.html#__new__". > If you have a suggestion of a another place this should be described, > let us know or better yet file a bug report at SourceForge. It's also much easier to find now under "Documentation" | "New-style classes". -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From guido at python.org Thu Apr 29 10:20:16 2004 From: guido at python.org (Guido van Rossum) Date: Thu Apr 29 10:20:22 2004 Subject: [Python-Dev] SSH problems getting into SourceForge's CVS? Message-ID: <200404291420.i3TEKGn05101@guido.python.org> When I tried "cvs update" today, I got an error from ssh: """ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: POSSIBLE DNS SPOOFING DETECTED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ The RSA1 host key for cvs.python.sourceforge.net has changed, and the key for the according IP address 66.35.250.209 is unknown. This could either mean that DNS SPOOFING is happening or the IP address for the host and its host key have changed at the same time. @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that the RSA1 host key has just been changed. The fingerprint for the RSA1 key sent by the remote host is 33:50:33:91:b4:ec:7a:b6:fa:39:99:b0:ff:65:63:a7. Please contact your system administrator. Add correct host key in /home/guido/.ssh/known_hosts to get rid of this message. Offending key in /home/guido/.ssh/known_hosts:1 RSA1 host key for cvs.python.sourceforge.net has changed and you have requested strict checking. Host key verification failed. cvs [update aborted]: end of file from server (consult above messages if any) """ After deleting the key from my known_hosts file, all I get is requests for a password, which I refuse to give (since it shouldn't be necessary and I don't want to rule out a man-in-the-middle attack yet). I've tried both ssh -1 and ssh -2. Has this happened to anybody else? The SF.net Site Status Page shows nothing relevant. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Thu Apr 29 10:26:17 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 29 10:26:33 2004 Subject: [Python-Dev] SSH problems getting into SourceForge's CVS? In-Reply-To: <200404291420.i3TEKGn05101@guido.python.org> References: <200404291420.i3TEKGn05101@guido.python.org> Message-ID: <1083248776.21368.74.camel@anthem.wooz.org> On Thu, 2004-04-29 at 10:20, Guido van Rossum wrote: > When I tried "cvs update" today, I got an error from ssh: Same thing happens to me when I use cvs.python.sf.net:/cvsroot/python but not when I use cvs.sf.net:/cvsroot/python. I think the latter is the current approved way of connecting to the cvs repository (I've been using that form with Mailman for a while and also have no problems there). I'll attach a nice little cvs_chroot script that Greg Ward sent me a long time ago. I'm sure he won't mind. cd to your python directory and type: % cvs_chroot cvs.sf.net:/cvsroot/python Then do a cvs up and all should be golden. -Barry -------------- next part -------------- #!/usr/bin/env python """cvs_chroot Change the repository directory of an existing CVS working directory (actually, a whole working tree). Very handy when you're moving repositories around and don't want to re-check-out the whole world; nearly essential when you have a heavily customized working directory (lots of uncommited changes, a forest of symlinks to make everything work, etc.). Using "cvs_chroot" is simple: just put yourself in the top of the working directory that needs to be adjusted, and run cvs_chroot For example, if you've just moved your CVS repository from /home/cvs to /cvs, then you would go to some working directory and run cvs_chroot /cvs Or if you've just uploaded a local project to a remote CVS server, so that all the net may share in its development, you might do something like this: cvs_chroot :pserver:anonymous@cvs.python.sourceforge.net:/cvsroot/python (assuming that the project in question is Python, and the remote CVS server is one of SourceForge's). Of course, this also applies if you just happen to have a working tree of the project in question -- you don't have to be the one who uploaded it to SourceForge (or wherever). If you're paranoid and/or curious, just run "chroot" with the "-n" option -- it'll tell you what it would do, without actually doing anything. """ # created 2000/05/12 (Greg Ward) # documented 2000/05/17 __revision__ = "$Id: cvs_chroot,v 2.3 2002/09/28 16:06:08 barry Exp $" import sys, os, string import getopt import errno USAGE = "usage: %s [-n] newroot" """If you're curious about how it works, here's the whole story: "cvs_chroot" walks a directory tree, finding all CVS directories under it, and adjusting the "Root" and "Repository" file in each CVS directory. The terminology is a bit wonky here (blame CVS), so pay attention: the "Root" files contain the location of the repository -- ie. the directory named in the CVSROOT environment variable or specified in the CVS "-d" option. For example, if you check something out as follows: CVSROOT=/home/cvs cvs checkout project (or cvs -d /home/cvs checkout project; the two are equivalent), then every "CVS/Root" file under "project" will contain the sole line "/home/cvs". Every "CVS/Repository" file will contain the full path to the repository directory for that particular working directory, or a path which is relative to the root (CVS supports both forms). For example, if "project" is laid out like: project --+ src ---+ module1 | | module2 | doc then the file "project/src/CVS/Repository" would contain either "/home/cvs/project/src" or "project/src", and the file "project/src/module1/CVS/Repository" would contain either "/home/cvs/project/src/module1" or "project/src/module1". If you're dealing with a remote repository, then things are a bit more complex: the "Root" files are still all the same and contain the value of CVSROOT or the "-d" option when you checked the project out, but the Repository files leave off the remote bit (everything up to the last colon). For example, if you checkout "project" from a remote server: cvs -d :pserver:anon@cvs.somewhere.net:/home/cvs checkout project then the "CVS/Root" files under "project" will all contain ":pserver:anon@cvs.somewhere.net:/home/cvs", but the "CVS/Repository" files will be the same as before -- i.e., they will start with "/home/cvs". This time, of course, that's "/home/cvs" on the host cvs.somewhere.net. Thus, cvs_chroot has to do the following for each CVS directory it finds: * replace the contents of "Root" with the new repository root * chop off the old repository root fragment (directory only, no host information) from the Repository file, and replace it with the new repository root fragment If cvs_chroot finds any inconsistencies in the Root or Repository files of your working directory, it prints a warning and skips the afflicted directory. (This can happen if you have a subtree of your working tree checked out differently, eg. from a different repository. In that case, you'll have to run "cvs_chroot" on each such oddball subtree.) """ def warn (msg): lines = string.split(msg, "\n") sys.stderr.write("warning: " + lines[0] + "\n") for l in lines[1:]: sys.stderr.write (" " + l + "\n") def find_cvs_dirs (start_dir): def callback(arg, directory, files): for file in files: if os.path.isdir(file) and file == "CVS": arg.append(os.path.join(directory, file)) dirs = [] os.path.walk(start_dir, callback, dirs) return dirs def split_root (root): lastcolon = string.rfind(root, ":") if lastcolon != -1: root_host = root[0:lastcolon] root_dir = root[lastcolon+1:] else: root_host = None root_dir = root return (root_host, root_dir) def repos_filenames (dir): return (os.path.join(dir, "Root"), os.path.join(dir, "Repository")) def read_repos (dir): (root_fn, repos_fn) = repos_filenames(dir) root = open(root_fn).readline()[0:-1] repos = open(repos_fn).readline()[0:-1] return (root, repos) def write_repos (dir, root, repos): (root_fn, repos_fn) = repos_filenames(dir) root_bk = root_fn + "~" repos_bk = repos_fn + "~" if os.path.exists(root_bk): os.remove(root_bk) if os.path.exists(repos_bk): os.remove(repos_bk) try: os.rename(root_fn, root_bk) os.rename(repos_fn, repos_bk) open(root_fn, "w").write(root + "\n") open(repos_fn, "w").write(repos + "\n") except (IOError, os.error): try: os.rename(root_bk, root_fn) os.rename(repos_bk, repos_fn) except (IOError, os.error): pass def main (prog, args): usage = USAGE % os.path.basename(prog) dry_run = None help = None try: (opts, args) = getopt.getopt(args, "nh") for (opt, _) in opts: if opt == "-n": dry_run = 1 elif opt == "-h": help = 1 except getopt.error, msg: raise SystemExit, usage + "\n" + str(msg) if help: print __doc__ print usage sys.exit(0) if len(args) != 1: raise SystemExit, \ usage + "\nWrong number of arguments" new_root = args[0] (new_root_host, new_root_dir) = split_root(new_root) (top_root, top_repos) = read_repos ("CVS") sys.stdout.write("Finding CVS directories..."); sys.stdout.flush() cvsdirs = find_cvs_dirs (".") sys.stdout.write("found %d of them\n" % len(cvsdirs)) for dir in cvsdirs: try: (root, repos) = read_repos(dir) except IOError, e: if e.errno <> errno.ENOTDIR: raise continue (root_host, root_dir) = split_root(root) orig_repos = repos # so we can tweak 'repos' but still # report the original value # The CVS/Root file must be consistent throughout the entire # working tree; skip any directory where it's not the same as # the top-level CVS/Root file. if root != top_root: warn(("root in %s (%s) doesn't match\n" "top-level root (%s) (skipping)") % (dir, root, top_root)) continue # Checking the Repository file needs to know if it, and the # top-level repository, are absolute or relative. repos_abs = (repos[0] == '/') top_repos_abs = (top_repos[0] == '/') # If the CVS/Repository file is absolute (which happens under # older versions of RCS -- at least 1.10.6 has relative # Repository files), then the prefix of the repository directory # must match the directory portion of the root. if repos_abs and repos[0:len(root_dir)] != root_dir: warn(("repository at %s (%s) not under " "root dir (%s) (skipping)") % (dir, repos, root_dir)) continue # If the top-level repository is absolute, but the current one # is not, then force the current repository into compliance with # the top-level repository -- i.e. make it absolute. if top_repos_abs and not repos_abs: repos = root_dir + '/' + repos repos_abs = 1 # Other way round: make the current repository relative to match # the top-level repository. elif repos_abs and not top_repos_abs: repos = repos[len(root_dir)+1:] repos_abs = 0 # Now we can make sure that the current repository is valid, # ie. is some descendant of the top-level repository. if repos[0:len(top_repos)] != top_repos: warn(("repository at %s (%s) not under\n" "top-level repository (%s) (skipping)") % (dir, repos, top_repos)) continue # Now, at last we can generate a new repository directory, which # is the point of the whole exercise. It will be absolute if # the top-level repository (not necessarily the current # repository) is absolute, and relative if the top-level # repository is relative. if repos[0] == '/': new_repos = new_root_dir + repos[len(root_dir):] else: new_repos = repos print dir + ":" print " root: %s -> %s" % (root, new_root) print " repos: %s -> %s" % (orig_repos, new_repos) if not dry_run: write_repos (dir, new_root, new_repos) # for dir in cvsdirs # main () if __name__ == "__main__": (prog, args) = (sys.argv[0], sys.argv[1:]) main (prog, args) From theller at python.net Thu Apr 29 10:49:55 2004 From: theller at python.net (Thomas Heller) Date: Thu Apr 29 10:50:02 2004 Subject: [Python-Dev] SSH problems getting into SourceForge's CVS? In-Reply-To: <1083248776.21368.74.camel@anthem.wooz.org> (Barry Warsaw's message of "Thu, 29 Apr 2004 10:26:17 -0400") References: <200404291420.i3TEKGn05101@guido.python.org> <1083248776.21368.74.camel@anthem.wooz.org> Message-ID: Barry Warsaw writes: > On Thu, 2004-04-29 at 10:20, Guido van Rossum wrote: >> When I tried "cvs update" today, I got an error from ssh: > > Same thing happens to me when I use cvs.python.sf.net:/cvsroot/python > but not when I use cvs.sf.net:/cvsroot/python. I think the latter is > the current approved way of connecting to the cvs repository (I've been > using that form with Mailman for a while and also have no problems > there). Yes, this is also what they mention on the CVS pages. > I'll attach a nice little cvs_chroot script that Greg Ward sent me a > long time ago. I'm sure he won't mind. cd to your python directory and > type: > > % cvs_chroot cvs.sf.net:/cvsroot/python > > Then do a cvs up and all should be golden. I had to type cvs_chroot :ext:@cvs.sourceforge.net:/cvsroot/python where is the sourceforge users name. Greg's script even works on windows, as it seems. Thomas From aahz at pythoncraft.com Thu Apr 29 10:50:38 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Apr 29 10:50:57 2004 Subject: [Python-Dev] SSH problems getting into SourceForge's CVS? In-Reply-To: <1083248776.21368.74.camel@anthem.wooz.org> References: <200404291420.i3TEKGn05101@guido.python.org> <1083248776.21368.74.camel@anthem.wooz.org> Message-ID: <20040429145038.GA20795@panix.com> On Thu, Apr 29, 2004, Barry Warsaw wrote: > > Same thing happens to me when I use cvs.python.sf.net:/cvsroot/python > but not when I use cvs.sf.net:/cvsroot/python. I think the latter is > the current approved way of connecting to the cvs repository (I've been > using that form with Mailman for a while and also have no problems > there). Mind updating the following pages? http://www.python.org/dev/devfaq.html http://www.python.org/dev/tools.html (I'd do it myself, but I think it'd be better for someone who actually knows what zie's doing to handle it. Fortunately, http://sourceforge.net/cvs/?group_id=5470 already has the correct info.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From barry at python.org Thu Apr 29 11:00:49 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 29 11:01:01 2004 Subject: [Python-Dev] SSH problems getting into SourceForge's CVS? In-Reply-To: References: <200404291420.i3TEKGn05101@guido.python.org> <1083248776.21368.74.camel@anthem.wooz.org> Message-ID: <1083250848.21368.93.camel@anthem.wooz.org> On Thu, 2004-04-29 at 10:49, Thomas Heller wrote: > cvs_chroot :ext:@cvs.sourceforge.net:/cvsroot/python Ah, I have stuff set up in my .ssh/config file so I don't need the :ext:@ prefix. -Barry From barry at python.org Thu Apr 29 11:35:38 2004 From: barry at python.org (Barry Warsaw) Date: Thu Apr 29 11:35:47 2004 Subject: [Python-Dev] SSH problems getting into SourceForge's CVS? In-Reply-To: <20040429145038.GA20795@panix.com> References: <200404291420.i3TEKGn05101@guido.python.org> <1083248776.21368.74.camel@anthem.wooz.org> <20040429145038.GA20795@panix.com> Message-ID: <1083252937.21368.111.camel@anthem.wooz.org> On Thu, 2004-04-29 at 10:50, Aahz wrote: > On Thu, Apr 29, 2004, Barry Warsaw wrote: > > > > Same thing happens to me when I use cvs.python.sf.net:/cvsroot/python > > but not when I use cvs.sf.net:/cvsroot/python. I think the latter is > > the current approved way of connecting to the cvs repository (I've been > > using that form with Mailman for a while and also have no problems > > there). > > Mind updating the following pages? > > http://www.python.org/dev/devfaq.html > http://www.python.org/dev/tools.html > > (I'd do it myself, but I think it'd be better for someone who actually > knows what zie's doing to handle it. Fortunately, > http://sourceforge.net/cvs/?group_id=5470 > already has the correct info.) Done. -Barry From tismer at stackless.com Thu Apr 29 11:45:46 2004 From: tismer at stackless.com (Christian Tismer) Date: Thu Apr 29 11:44:43 2004 Subject: [Python-Dev] Re: [pypy-dev] Slides: how Psyco works In-Reply-To: <20040429115911.GA15802@vicky.ecs.soton.ac.uk> References: <20040429115911.GA15802@vicky.ecs.soton.ac.uk> Message-ID: <4091232A.5020607@stackless.com> Armin Rigo wrote: > Hello, > > I have put on-line some slides from the Python UK conference at ACCU 2004, > explaining how Psyco works. It is actually a Pygame application... As far as > I can tell, it is the first time someone in the room actually understood > something at the end :-) > > http://psyco.sourceforge.net/ Hey, *this* is a nice animation, with my beloved Lemmings :-) > I hope it should help to make Psyco a bit less mysterious, and also explain > why it is difficult to have a general idea about what kind of speed-up you can > expect for specific kinds of code: it is, after all, a pretty low-level > "local" process that Psyco does, and it sometimes pays off and sometimes not. > It also explains why this process is very much like the usual interpretation > that CPython does. If you think about it you might see how useful it would be > for Psyco to build on top of an interpreter in a better language than C (i.e. > one that can be analysed, not just blindly run). Yes, this is very clear. For PyPy, it will be interesting to see how efficient Psyco's peephole-like optimization is compared to whole-program analysis. There are probably programs simple enough to be completely optimized through by WPA. On the other hand, the possible optimization paths can grow to very many, giving a huge code bloat. In that case, Psyco's approach is probably more efficient, by creating new code only when it is needed. In the end, I guess both strategies should be combined to get some optimum between analysis time, speed and memory consumption. Anyway, I can highly recommend to watch this nice movie from Rigo Productions (C) :-)) ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From aahz at pythoncraft.com Thu Apr 29 12:47:37 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu Apr 29 12:47:45 2004 Subject: [Python-Dev] SSH problems getting into SourceForge's CVS? In-Reply-To: <1083252937.21368.111.camel@anthem.wooz.org> References: <200404291420.i3TEKGn05101@guido.python.org> <1083248776.21368.74.camel@anthem.wooz.org> <20040429145038.GA20795@panix.com> <1083252937.21368.111.camel@anthem.wooz.org> Message-ID: <20040429164737.GA27313@panix.com> On Thu, Apr 29, 2004, Barry Warsaw wrote: > On Thu, 2004-04-29 at 10:50, Aahz wrote: >> On Thu, Apr 29, 2004, Barry Warsaw wrote: >>> >>> Same thing happens to me when I use cvs.python.sf.net:/cvsroot/python >>> but not when I use cvs.sf.net:/cvsroot/python. I think the latter is >>> the current approved way of connecting to the cvs repository (I've been >>> using that form with Mailman for a while and also have no problems >>> there). >> >> Mind updating the following pages? >> >> http://www.python.org/dev/devfaq.html >> http://www.python.org/dev/tools.html > > Done. Thanks! -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From niemeyer at conectiva.com Thu Apr 29 13:01:43 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Thu Apr 29 13:01:46 2004 Subject: [Python-Dev] Slides: how Psyco works In-Reply-To: <20040429115911.GA15802@vicky.ecs.soton.ac.uk> References: <20040429115911.GA15802@vicky.ecs.soton.ac.uk> Message-ID: <20040429170142.GA4820@burma.localdomain> > I have put on-line some slides from the Python UK conference at ACCU > 2004, explaining how Psyco works. It is actually a Pygame > application... As far as I can tell, it is the first time someone in > the room actually understood something at the end :-) Impressive presentation! Congratulations! -- Gustavo Niemeyer http://niemeyer.net From jim.jewett at eds.com Thu Apr 29 14:28:26 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Thu Apr 29 14:29:10 2004 Subject: [Python-Dev] new datatypes - should they subclasses (deque(list)?) Message-ID: The collections module will be offering new datatypes, including a double-ended queue (deque). Right now, these are new elemental types, inheriting directly from object. I believe they should be subclassed from existing classes where this makes sense. Even if no actual methods are inherited, the API is valuable. For instance, should: list([1,2,3]) == deque([1,2,3]) deque([1]) < list([1,2]) < deque([3]) If deque inherits directly from object, then the only comparison available is based on address. If deque subclasses list (or at least pretends to), then sensible comparisons can be made. All list operations make sense on a deque (and vice versa). Operations may be more (or less) efficient depending on the type, but they do make sense. Even if the concern is to avoid encouraging inefficient code, the methods should still be available. d = deque(...) d[4:11] = ('x', 'y') is inefficient, but still better than d = deque(...) temp = list(d) temp[4:11] = ('x', 'y') d = deque(temp) As an additional benefit, making deque a subclass of list with the same API gives later python implementations more freedom; in theory, the compiler could determine which internal representation made more sense for each particular case. -jJ From niemeyer at conectiva.com Thu Apr 29 15:47:35 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Thu Apr 29 15:47:35 2004 Subject: [Python-Dev] Small additions to optparser Message-ID: <20040429194735.GA7068@burma.localdomain> Hi Folks, I have two small customizations I've used in most projects I've plugged optparse into. Since they're simple and generic, I'd like to integrate them back in optparser. These features are: - Add a help formatter which uses capitalized headers. - Add a "help" keyword in OptionParser, allowing straightforward addition of a custom help messages. Comments? Also, how are we currently handling changes in optparser? Do we have an "official" maintainer (Greg?). IOW, should I: - Send a proposed patch to SourceForge - Contact the original project - Commit the patch FWIW, here is the proposed functionality: class CapitalizedHelpFormatter(optparse.IndentedHelpFormatter): def format_usage(self, usage): return optparse.IndentedHelpFormatter \ .format_usage(self, usage).capitalize() def format_heading(self, heading): return optparse.IndentedHelpFormatter \ .format_heading(self, heading).capitalize() class OptionParser(optparse.OptionParser): def __init__(self, *args, **kwargs): optparse.OptionParser.__init__(self, *args, **kwargs) self._override_help = kwargs.get("help") def format_help(self, formatter=None): if self._override_help: return self._override_help else: return optparse.OptionParser.format_help(self, formatter) -- Gustavo Niemeyer http://niemeyer.net From captainjacksparrow at indiatimes.com Thu Apr 29 23:39:41 2004 From: captainjacksparrow at indiatimes.com (captainjacksparrow) Date: Thu Apr 29 23:45:08 2004 Subject: [Python-Dev] Mozpython help Message-ID: <200404300305.IAA03360@WS0005.indiatimes.com> hi, i have tried to run Mozpython 0.1.1 on my i386 running Python 2.3.3 and Mozilla 1.6. The compilation is successful but when i try to run a python script in Mozilla it gives me a lost sys.stderr ny comments r most welcome. Jack Get Your Private, Free E-mail from Indiatimes at http://email.indiatimes.com Buy The Best In BOOKS at http://www.bestsellers.indiatimes.com Bid for for Air Tickets @ Re.1 on Air Sahara Flights. Just log on to http://airsahara.indiatimes.com and Bid Now! From tjreedy at udel.edu Fri Apr 30 01:22:25 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Fri Apr 30 01:31:15 2004 Subject: [Python-Dev] Re: Mozpython help References: <200404300305.IAA03360@WS0005.indiatimes.com> Message-ID: "captainjacksparrow" wrote in message news:200404300305.IAA03360@WS0005.indiatimes.com... > i have tried to run Mozpython 0.1.1 on my i386 running Python 2.3.3 and Mozilla 1.6. The compilation is successful but when i try to run a python script in Mozilla it gives me a lost sys.stderr > ny comments r most welcome. This list is about developing future versions of Python. Questions about using current Python are best directed to comp.lang.python or specific mailing lists when there are such. tjr From arigo at tunes.org Fri Apr 30 06:41:58 2004 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 30 06:45:10 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200404231826.i3NIQxM18955@guido.python.org> References: <200404231826.i3NIQxM18955@guido.python.org> Message-ID: <20040430104158.GA31871@vicky.ecs.soton.ac.uk> Hello Guido, I did a quick review of the stdlib, including the tests, to see which list comprehensions could be replaced with generator expressions. Thought I admit I am biased towards early binding, I ended up looking for cases with the following properties: * doesn't use the result immediately * contains free variables This is less that 10% of the cases. However, for each of them, I had to check if the free variables had a chance of being modified after the genexpr. This is pretty rare, admittedly, but there is an example in test/test_random.py (test_avg_std). There are a number of other examples that just don't happen to modify the free variables, by chance; e.g. in optparse.py: metavar = option.metavar or option.dest.upper() short_opts = [sopt + metavar for sopt in option._short_opts] long_opts = [lopt + "=" + metavar for lopt in option._long_opts] If we find out later that long_opts actually needs a slightly different value for metavar, it would be tempting to do: metavar = option.metavar or option.dest.upper() short_opts = [sopt + metavar for sopt in option._short_opts] metavar = option.metavar or option.dest.capitalize() long_opts = [lopt + "=" + metavar for lopt in option._long_opts] Replace these with genexprs and it doesn't work any more: the 2nd metavar is unexpectedly used in the first genexpr as well. In general I find it strange to have to look if a given variable could be modified much later in the same function to be sure of what a genexpr really means. Early binding is closer to the idea that turning a listcomp into a genexprs should just work if you only iterate once on the result. A bient?t, Armin. From guido at python.org Fri Apr 30 08:16:32 2004 From: guido at python.org (Guido van Rossum) Date: Fri Apr 30 08:16:39 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: Your message of "Fri, 30 Apr 2004 11:41:58 BST." <20040430104158.GA31871@vicky.ecs.soton.ac.uk> References: <200404231826.i3NIQxM18955@guido.python.org> <20040430104158.GA31871@vicky.ecs.soton.ac.uk> Message-ID: <200404301216.i3UCGXU07211@guido.python.org> > I did a quick review of the stdlib, including the tests, to see > which list comprehensions could be replaced with generator > expressions. Thought I admit I am biased towards early binding, I > ended up looking for cases with the following properties: > > * doesn't use the result immediately > * contains free variables > > This is less that 10% of the cases. However, for each of them, I > had to check if the free variables had a chance of being modified > after the genexpr. This is pretty rare, admittedly, but there is an > example in test/test_random.py (test_avg_std). There are a number > of other examples that just don't happen to modify the free > variables, by chance; e.g. in optparse.py: > > metavar = option.metavar or option.dest.upper() > short_opts = [sopt + metavar for sopt in option._short_opts] > long_opts = [lopt + "=" + metavar for lopt in option._long_opts] > > If we find out later that long_opts actually needs a slightly > different value for metavar, it would be tempting to do: > > metavar = option.metavar or option.dest.upper() > short_opts = [sopt + metavar for sopt in option._short_opts] > metavar = option.metavar or option.dest.capitalize() > long_opts = [lopt + "=" + metavar for lopt in option._long_opts] > > Replace these with genexprs and it doesn't work any more: the 2nd > metavar is unexpectedly used in the first genexpr as well. In > general I find it strange to have to look if a given variable could > be modified much later in the same function to be sure of what a > genexpr really means. Early binding is closer to the idea that > turning a listcomp into a genexprs should just work if you only > iterate once on the result. Thanks for the analysis. My comment on this: why on earth would you want to replace the perfectly sensible list comprehension with a generator comprehension in this particular case? (Note that the remainder of that function proceeds to do list concatenation of the two lists.) A more general question would be: have you found any examples of list comprehensions that weren't immediately used where there would be a compelling reason to use a generator expression instead? In this example, I can't see any of the touted advantages -- the set of command line options will never be long enough to have to worry about the memory consumption. How about the other examples you found? --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Fri Apr 30 08:48:06 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Fri Apr 30 08:42:37 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200404301216.i3UCGXU07211@guido.python.org> References: <200404231826.i3NIQxM18955@guido.python.org> <20040430104158.GA31871@vicky.ecs.soton.ac.uk> Message-ID: <5.2.1.1.0.20040430142620.030304e8@pop.bluewin.ch> At 05:16 30.04.2004 -0700, Guido van Rossum wrote: >A more general question would be: have you found any examples of list >comprehensions that weren't immediately used where there would be a >compelling reason to use a generator expression instead? In this >example, I can't see any of the touted advantages -- the set of >command line options will never be long enough to have to worry about >the memory consumption. How about the other examples you found? this assumes that people will choose either generator expressions or list comprehesions on a case-by-case basis. I'm not sure it's a reasonable assumption for the whole range of users - starting with newbies. Probably documentation and what is predicated on c.l.p will influence this. OTOH both are probably good enough default choices, list comprehesions because if one gets insufficient performance she will learn to switch, generator expressions because for small numbers of elements things are a wash (unless someone is cretating a lot of small sequences :)).So the more they are interchangeable in all other respects the better. "There should be one-- and preferably only one --obvious way to do it." of course this is the case for generator expressions and list comprehsensions because they are separated by performance characteristics and somehow by the fact that one directly produces a list. I suspect that we have to learn to live with the fact that they are distinguished by (just) that, both a nuisance (sometimes people don't want to bother to choose) and a feature. regards. From arigo at tunes.org Fri Apr 30 09:07:00 2004 From: arigo at tunes.org (Armin Rigo) Date: Fri Apr 30 09:10:07 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200404301216.i3UCGXU07211@guido.python.org> References: <200404231826.i3NIQxM18955@guido.python.org> <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200404301216.i3UCGXU07211@guido.python.org> Message-ID: <20040430130700.GA28030@vicky.ecs.soton.ac.uk> Hello, On Fri, Apr 30, 2004 at 05:16:32AM -0700, Guido van Rossum wrote: > Thanks for the analysis. My comment on this: why on earth would you > want to replace the perfectly sensible list comprehension with a > generator comprehension in this particular case? I was using this example more generally to show that if you have a late-binding genexpr, you have to worry about not changing the local variables it uses, which looks like all uses of genexprs with free variables are subtle bugs waiting to appear whenever someone modifies the function anywhere after the genexpr. The same argument applies to the other stdlib cases I mentioned (which make somewhat less than 10% of all uses). The alternative -- actively using the late-binding aspect -- would lead to code in which you have to worry about which values the local variables have at the moment the next element is fetched from the iterator. For example: x_square = (x*x for i in itertools.repeat(None)).next ... x = 5 print x_square() # 25 x = 6 print x_square() # 36 This looks (a) cool and (b) a complete hack that nobody should be allowed to do without messing with sys._getframe(). A bient?t, Armin. From barry at python.org Fri Apr 30 09:30:30 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 30 09:30:43 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <20040430130700.GA28030@vicky.ecs.soton.ac.uk> References: <200404231826.i3NIQxM18955@guido.python.org> <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200404301216.i3UCGXU07211@guido.python.org> <20040430130700.GA28030@vicky.ecs.soton.ac.uk> Message-ID: <1083331829.29603.14.camel@anthem.wooz.org> On Fri, 2004-04-30 at 09:07, Armin Rigo wrote: > This looks (a) cool and (b) a complete hack that nobody should be allowed to > do without messing with sys._getframe(). It reminds me a little bit of the dynamic binding in Emacs lisp. As incredibly useful as that is sometimes, it's a disgusting hack. :) -Barry From dmgass at hotmail.com Fri Apr 30 11:23:05 2004 From: dmgass at hotmail.com (Dan Gass) Date: Fri Apr 30 11:23:10 2004 Subject: [Python-Dev] Small additions to optparser Message-ID: >From: Gustavo Niemeyer >To: python-dev@python.org >CC: gward@python.net >Subject: [Python-Dev] Small additions to optparser >Date: Thu, 29 Apr 2004 16:47:35 -0300 > >Hi Folks, > >I have two small customizations I've used in most projects I've >plugged optparse into. Since they're simple and generic, I'd >like to integrate them back in optparser. These features are: > >- Add a help formatter which uses capitalized headers. >- Add a "help" keyword in OptionParser, allowing straightforward > addition of a custom help messages. > >Comments? > It would be nice to have a way for the white space to be cleaned up in the help string. This would be useful when using multiline triple quoted strings to pass in help text to the option parser and you want to indent the help text on each line so the source code reads well. _________________________________________________________________ Getting married? Find tips, tools and the latest trends at MSN Life Events. http://lifeevents.msn.com/category.aspx?cid=married From edloper at gradient.cis.upenn.edu Fri Apr 30 12:24:20 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Fri Apr 30 12:22:52 2004 Subject: [Python-Dev] Small additions to optparser In-Reply-To: References: Message-ID: <40927DB4.9040508@gradient.cis.upenn.edu> [Gustavo Niemeyer wrote] >> Hi Folks, >> >> I have two small customizations I've used in most projects I've >> plugged optparse into. [...] [Dan Gass wrote] > It would be nice to have a way for the white space to be cleaned up in > the help string. This would be useful when using multiline triple > quoted strings to pass in help text to the option parser and you want to > indent the help text on each line so the source code reads well. There's already code to do this type of clean-up in inspect.getdoc(). Maybe it could be copied (or moved out of getdoc() into its own function and reused). -Edward From jim.jewett at eds.com Fri Apr 30 12:26:13 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Fri Apr 30 12:27:06 2004 Subject: [Python-Dev] dict.merge()? Message-ID: At the C level, PyDict_Update(old, new) is just a shortcut for override=1 PyDict_Merge(old, new, override) C extensions can use PyDict_Merge directly with override=0, which keeps the current value in case of duplicate keys. Python code can only use update, which means that override is always true. Is there a good reason this? -jJ From edloper at gradient.cis.upenn.edu Fri Apr 30 12:29:04 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Fri Apr 30 12:27:21 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: References: Message-ID: <40927ED0.5020402@gradient.cis.upenn.edu> > Hello Guido, > > I did a quick review of the stdlib, including the tests, to see which list > comprehensions could be replaced with generator expressions. Thought I admit > I am biased towards early binding, I ended up looking for cases with the > following properties: [...] Even if we switched to early binding, won't these issues still bite you with mutable values? E.g.: some_dict = {'x': 1, 'y': 2} iter1 = (some_dict.get(v, 3) for v in input1) some_dict['z'] = 5 iter2 = (some_dict.get(v, 7) for v in input2) It seems like it would be surprising (to me, anyway) if this gave a different result than: some_dict = {'x': 1, 'y': 2} iter1 = (some_dict.get(v, 3) for v in input1) some_dict = {'x': 1, 'y': 2, 'z': 5} iter2 = (some_dict.get(v, 7) for v in input2) -Edward From tim.one at comcast.net Fri Apr 30 12:47:56 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri Apr 30 12:47:55 2004 Subject: [Python-Dev] dict.merge()? In-Reply-To: Message-ID: [Jewett, Jim J] > At the C level, > > PyDict_Update(old, new) > > is just a shortcut for > > override=1 > PyDict_Merge(old, new, override) It is today, but that's a relatively new internal ability; IIRC, Guido added it to help with the implementation of new-style classes. > C extensions can use PyDict_Merge directly with override=0, > which keeps the current value in case of duplicate keys. Yes. > Python code can only use update, which means that override > is always true. Yes. > Is there a good reason this? Dunno -- define "good" <0.5 wink>. From jcarlson at uci.edu Fri Apr 30 12:55:43 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri Apr 30 13:01:30 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <40927ED0.5020402@gradient.cis.upenn.edu> References: <40927ED0.5020402@gradient.cis.upenn.edu> Message-ID: <20040430093439.138D.JCARLSON@uci.edu> > Even if we switched to early binding, won't these issues still bite you > with mutable values? E.g.: > > some_dict = {'x': 1, 'y': 2} > iter1 = (some_dict.get(v, 3) for v in input1) > some_dict['z'] = 5 > iter2 = (some_dict.get(v, 7) for v in input2) > > It seems like it would be surprising (to me, anyway) if this gave a > different result than: > > some_dict = {'x': 1, 'y': 2} > iter1 = (some_dict.get(v, 3) for v in input1) > some_dict = {'x': 1, 'y': 2, 'z': 5} > iter2 = (some_dict.get(v, 7) for v in input2) Semantically, in the first case you are modifying your dictionary, which has consequences to anything that has a reference to the object. Mutable modification during runtime should be something we are used to by now. In the second case, you are rebinding the name some_dict to something else. Your earlier binding in the generator expression to some_dict (if we had early-binding for generator expressions) doesn't change, so the fact that the two iterators in the second case return different results, to me (and others I'm sure), is expected. Name rebinding should also be something we are used to by now. I think that it would be very convenient if generator expressions had the exact same behavior as list comprehensions (people already know how to do those). As such, early binding is essentially what list comprehensions do, though they don't suffer from the "I modified my dictionary before I used my generator expression" issue. In that case, the only way we can preserve behavior is through a copy of all the objects that are bound in early binding, which is obviously a no-no, and doesn't necessarily solve our problem. I'm happy with early binding, even given the mutable object modification issue shown above, and I think it would be relatively easy to document with a few minor examples. What does everyone else think? - Josiah From pedronis at bluewin.ch Fri Apr 30 13:10:16 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Fri Apr 30 13:05:48 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <40927ED0.5020402@gradient.cis.upenn.edu> References: Message-ID: <5.2.1.1.0.20040430190249.0313a9c0@pop.bluewin.ch> At 12:29 30.04.2004 -0400, Edward Loper wrote: >>Hello Guido, >>I did a quick review of the stdlib, including the tests, to see which list >>comprehensions could be replaced with generator expressions. Thought I admit >>I am biased towards early binding, I ended up looking for cases with the >>following properties: [...] > >Even if we switched to early binding, won't these issues still bite you >with mutable values? E.g.: > > some_dict = {'x': 1, 'y': 2} > iter1 = (some_dict.get(v, 3) for v in input1) > some_dict['z'] = 5 > iter2 = (some_dict.get(v, 7) for v in input2) > >It seems like it would be surprising (to me, anyway) if this gave a >different result than: > > some_dict = {'x': 1, 'y': 2} > iter1 = (some_dict.get(v, 3) for v in input1) > some_dict = {'x': 1, 'y': 2, 'z': 5} > iter2 = (some_dict.get(v, 7) for v in input2) well, the deep problem here is that lazinesss and side-effects (or mutability, if one wants, inclusive of bindings) do not mix well. There is so much that we can do about that. It's really a matter of, depending on our choices, how often the casual user will be bitten by this and what kind of questions we want to get. Otherwise we would have to give up about the whole idea. From barry at python.org Fri Apr 30 13:30:37 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 30 13:30:44 2004 Subject: [Python-Dev] Small additions to optparser In-Reply-To: References: Message-ID: <1083346236.1048.72.camel@anthem.wooz.org> On Fri, 2004-04-30 at 11:23, Dan Gass wrote: > It would be nice to have a way for the white space to be cleaned up in the > help string. This would be useful when using multiline triple quoted > strings to pass in help text to the option parser and you want to indent the > help text on each line so the source code reads well. I use: def parseargs(): parser = OptionParser(...) parser.add_option('-a', '--age', action='store', default='1w', help="""\ Confirmations older than this value are removed. age is a float number followed by one of the following unit specifier letters: m - minutes h - hours d - days w - weeks""") -Barry From ark-mlist at att.net Fri Apr 30 13:59:12 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Fri Apr 30 13:59:17 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <1083331829.29603.14.camel@anthem.wooz.org> Message-ID: <000601c42edc$d88e1710$6402a8c0@arkdesktop> > > This looks (a) cool and (b) a complete hack that nobody should be > allowed to > > do without messing with sys._getframe(). > > It reminds me a little bit of the dynamic binding in Emacs lisp. As > incredibly useful as that is sometimes, it's a disgusting hack. :) And, if I understand correctly, something from which the Lisp community is moving away--preferring, instead, the notion of "lexical scoping" such as in Scheme. From barry at python.org Fri Apr 30 14:01:32 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 30 14:01:44 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <000601c42edc$d88e1710$6402a8c0@arkdesktop> References: <000601c42edc$d88e1710$6402a8c0@arkdesktop> Message-ID: <1083348092.1048.91.camel@anthem.wooz.org> On Fri, 2004-04-30 at 13:59, Andrew Koenig wrote: > > > This looks (a) cool and (b) a complete hack that nobody should be > > allowed to > > > do without messing with sys._getframe(). > > > > It reminds me a little bit of the dynamic binding in Emacs lisp. As > > incredibly useful as that is sometimes, it's a disgusting hack. :) > > And, if I understand correctly, something from which the Lisp community is > moving away--preferring, instead, the notion of "lexical scoping" such as in > Scheme. That's my understanding too. Back when I followed such things, there was a lot of discussion in the XEmacs lists about implementing lexical scoping and getting rid of this ugly hack (eventually -- no doubt there's tons of old elisp that would break as a result). tim-would-have-fixed-python-mode-ly y'rs, -Barry From nbastin at opnet.com Fri Apr 30 14:01:40 2004 From: nbastin at opnet.com (Nick Bastin) Date: Fri Apr 30 14:01:56 2004 Subject: [Python-Dev] Accessing base class C struct of subclass instance Message-ID: <6E9B692A-9AD0-11D8-961D-000393CBDF94@opnet.com> Ok, strange subject, but I have what seems like a simple question that the documentation doesn't seem to cover (and a quick perusal of the code doesn't answer, either). If I define a new style class in C, and subclass it in python, how can I get the base class instance object from a subclass instance in C? -- Nick From nbastin at opnet.com Fri Apr 30 14:15:15 2004 From: nbastin at opnet.com (Nick Bastin) Date: Fri Apr 30 14:15:42 2004 Subject: [Python-Dev] Accessing base class C struct of subclass instance In-Reply-To: <6E9B692A-9AD0-11D8-961D-000393CBDF94@opnet.com> References: <6E9B692A-9AD0-11D8-961D-000393CBDF94@opnet.com> Message-ID: <5404E33E-9AD2-11D8-961D-000393CBDF94@opnet.com> On Apr 30, 2004, at 2:01 PM, Nick Bastin wrote: > Ok, strange subject, but I have what seems like a simple question that > the documentation doesn't seem to cover (and a quick perusal of the > code doesn't answer, either). > > If I define a new style class in C, and subclass it in python, how can > I get the base class instance object from a subclass instance in C? Nevermind...the obvious solution works correctly. -- Nick From cm at leetspeak.org Fri Apr 30 14:29:49 2004 From: cm at leetspeak.org (Michael Walter) Date: Fri Apr 30 14:29:55 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <000601c42edc$d88e1710$6402a8c0@arkdesktop> References: <000601c42edc$d88e1710$6402a8c0@arkdesktop> Message-ID: <40929B1D.60608@leetspeak.org> Andrew Koenig wrote: >>>This looks (a) cool and (b) a complete hack that nobody should be >> >>allowed to >> >>>do without messing with sys._getframe(). >> >>It reminds me a little bit of the dynamic binding in Emacs lisp. As >>incredibly useful as that is sometimes, it's a disgusting hack. :) > > > And, if I understand correctly, something from which the Lisp community is > moving away--preferring, instead, the notion of "lexical scoping" such as in > Scheme. Well, global variables in Common Lisp are still dynamically-scoped, which makes perfect sense IMO. Also, I think there is a SRFI almost every Scheme implements (SRFI ~ PEP) which add dynamic bindings. Cheers, Michael From jeremy at alum.mit.edu Fri Apr 30 16:38:55 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Fri Apr 30 16:40:43 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <1083331829.29603.14.camel@anthem.wooz.org> References: <200404231826.i3NIQxM18955@guido.python.org> <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200404301216.i3UCGXU07211@guido.python.org> <20040430130700.GA28030@vicky.ecs.soton.ac.uk> <1083331829.29603.14.camel@anthem.wooz.org> Message-ID: <1083357535.9092.148.camel@localhost.localdomain> On Fri, 2004-04-30 at 09:30, Barry Warsaw wrote: > On Fri, 2004-04-30 at 09:07, Armin Rigo wrote: > > > This looks (a) cool and (b) a complete hack that nobody should be allowed to > > do without messing with sys._getframe(). > > It reminds me a little bit of the dynamic binding in Emacs lisp. As > incredibly useful as that is sometimes, it's a disgusting hack. :) The funny thing is that it's the result of a static scoping discipline rather than dynamic scoping. What's funny about it has more to do with side-effects that scoping rules. If the x_square function was returned out of its defining block, there would be no way to rebind x and it would not be possible to define a variable x local to the caller that would affect it. Put another way, it's possible to reason statically about what binding x will use when x_square() is called. It's the same technique you would use to write an accumulator generator (that is, a function that returns accumulator functions). See the appendix of Paul Graham's essay: http://www.paulgraham.com/icad.html Jeremy PS I do owe everyone a PEP on the subject of re-binding. From rowen at cesmail.net Fri Apr 30 17:00:08 2004 From: rowen at cesmail.net (Russell E. Owen) Date: Fri Apr 30 17:00:22 2004 Subject: [Python-Dev] Re: PEP 289 - Generator Expressions - Let's Move Forward References: <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200404231826.i3NIQxM18955@guido.python.org> <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200404301216.i3UCGXU07211@guido.python.org> <5.2.1.1.0.20040430142620.030304e8@pop.bluewin.ch> Message-ID: In article <5.2.1.1.0.20040430142620.030304e8@pop.bluewin.ch>, Samuele Pedroni wrote: >... > this assumes that people will choose either generator expressions or list > comprehesions on a case-by-case basis. I'm not sure it's a reasonable > assumption for the whole range of users - starting with newbies. Probably > documentation and what is predicated on c.l.p will influence this. I completely agree. Generator expressions and list comprehensions are similar enough that it seems a wart to have to carefully differentiate between two in normal use. Personally, I'd only like to see generator expressions added to Python if they can allow list comprehensions to be deprecated -- to avoid language clutter. One question, though: is "if" part of generator expressions or not? I read the current version of the PEP and I'm confused. It offers the example: max(len(line) for line in file if line.strip()) But that is the only instance in the PEP that I noticed. The section "The Details" says almost nothing about "if". The only thing I found that might be relevant appeared at the end of: "2. The syntax requires...": "...where listmaker1 is almost the same as listmaker, but only allows a single test after 'for' ... 'in'." I suppose that might be saying that a single "if" condition is allowed. If so, great! -- Russell From tim.one at comcast.net Fri Apr 30 17:28:28 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri Apr 30 17:28:27 2004 Subject: [Python-Dev] Re: PEP 289 - Generator Expressions - Let's MoveForward In-Reply-To: Message-ID: [Russell E. Owen] > I completely agree. Generator expressions and list comprehensions are > similar enough that it seems a wart to have to carefully differentiate > between two in normal use. In normal use you won't have to distinguish, where "normal use" is pretty much defined by "cases where it doesn't matter" . > Personally, I'd only like to see generator expressions added to Python > if they can allow list comprehensions to be deprecated -- to avoid > language clutter. Listcomps can be deprecated, although not all existing uses of listcomps will translate directly into genexps. I think this has become a "practicality beats purity" feature. When Python had only 3 scopes, it was fine to ignore "explicit is better than implicit": the scope model was so bare that explictness would have been overkill. When multiple nested scopes got introduced, and function bodies start appearing in places other than lambda and def (which is the case for genexps), scope gets much more mysterious (and so also potentially surprising) in the absence of explicit scope declarations. But simple genexps will work fine anyway almost all the time, so it doesn't appear to matter much that devious uses will have nightmarish semantics. > One question, though: is "if" part of generator expressions or not? Yes. If the PEP isn't clear about this, it's a flaw in the PEP. I don't think the semantics of genexps are clear in the PEP either (unless that's changed recently). From eppstein at ics.uci.edu Fri Apr 30 18:09:23 2004 From: eppstein at ics.uci.edu (David Eppstein) Date: Fri Apr 30 18:09:06 2004 Subject: [Python-Dev] Re: PEP 289 - Generator Expressions - Let's MoveForward References: Message-ID: In article , "Tim Peters" wrote: > > Personally, I'd only like to see generator expressions added to Python > > if they can allow list comprehensions to be deprecated -- to avoid > > language clutter. > > Listcomps can be deprecated, although not all existing uses of listcomps > will translate directly into genexps. Not all? When would [listcomp] ever not be replaceable by list(gencomp)? -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From barry at python.org Fri Apr 30 18:11:44 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 30 18:11:52 2004 Subject: [Python-Dev] Re: PEP 289 - Generator Expressions - Let's MoveForward In-Reply-To: References: Message-ID: <1083363103.3975.123.camel@anthem.wooz.org> On Fri, 2004-04-30 at 18:09, David Eppstein wrote: > Not all? When would [listcomp] ever not be replaceable by list(gencomp)? I dunno. I happen to like [listcomp] syntax over list(genexpr). TOOWTDI-be-damned-ly y'rs, -Barry From tim.one at comcast.net Fri Apr 30 18:20:55 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri Apr 30 18:21:04 2004 Subject: [Python-Dev] Re: PEP 289 - Generator Expressions - Let's MoveForward In-Reply-To: Message-ID: >> Listcomps can be deprecated, although not all existing uses of listcomps >> will translate directly into genexps. [David Eppstein] > Not all? When would [listcomp] ever not be replaceable by list(gencomp)? The point of Armin's original msg in this thread was that direct replacement can have different semantics. Applying a transformation (via list()) is a different story, although I expect someone obsessed enough could concoct an example that worked differently, because genexp guts truly live in a different scope than listcomp guts (e.g., play games with locals()). From guido at python.org Fri Apr 30 19:12:59 2004 From: guido at python.org (Guido van Rossum) Date: Fri Apr 30 19:13:08 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: Your message of "Fri, 30 Apr 2004 14:07:00 BST." <20040430130700.GA28030@vicky.ecs.soton.ac.uk> References: <200404231826.i3NIQxM18955@guido.python.org> <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200404301216.i3UCGXU07211@guido.python.org> <20040430130700.GA28030@vicky.ecs.soton.ac.uk> Message-ID: <200404302313.i3UND0a08111@guido.python.org> > I was using this example more generally to show that if you have a > late-binding genexpr, you have to worry about not changing the local > variables it uses, which looks like all uses of genexprs with free > variables are subtle bugs waiting to appear whenever someone > modifies the function anywhere after the genexpr. The same argument > applies to the other stdlib cases I mentioned (which make somewhat > less than 10% of all uses). That's not new, we all know that that's the downside of late binding. My counter to that is that *any* use of genexps where the consumer doesn't consume (or discard) the iterator before the next line is reached is extremely advanced use. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Fri Apr 30 19:27:37 2004 From: barry at python.org (Barry Warsaw) Date: Fri Apr 30 19:27:48 2004 Subject: [Python-Dev] Re: PEP 289 - Generator Expressions - Let's MoveForward In-Reply-To: <4131456.1083338892@dhcp31-158.ics.uci.edu> References: <1083363103.3975.123.camel@anthem.wooz.org> <4131456.1083338892@dhcp31-158.ics.uci.edu> Message-ID: <1083367654.3975.135.camel@anthem.wooz.org> On Fri, 2004-04-30 at 18:28, David Eppstein wrote: > Oh, me too (and I'm getting a little irritated with writing dict([...]) so > often instead of having a proper dictcomp syntax) but semantically I think > they are the same. I'm easily convinced. pep-274-ly y'rs, -Barry From eppstein at ics.uci.edu Fri Apr 30 18:28:12 2004 From: eppstein at ics.uci.edu (David Eppstein) Date: Sat May 1 18:10:39 2004 Subject: [Python-Dev] Re: PEP 289 - Generator Expressions - Let's MoveForward In-Reply-To: <1083363103.3975.123.camel@anthem.wooz.org> References: <1083363103.3975.123.camel@anthem.wooz.org> Message-ID: <4131456.1083338892@dhcp31-158.ics.uci.edu> On 4/30/04 6:11 PM -0400 Barry Warsaw wrote: >> Not all? When would [listcomp] ever not be replaceable by list(gencomp)? > > I dunno. I happen to like [listcomp] syntax over list(genexpr). Oh, me too (and I'm getting a little irritated with writing dict([...]) so often instead of having a proper dictcomp syntax) but semantically I think they are the same. -- David Eppstein http://www.ics.uci.edu/~eppstein/ Univ. of California, Irvine, School of Information & Computer Science From peter at designtheory.org Thu Apr 29 15:01:33 2004 From: peter at designtheory.org (Peter Dobcsanyi) Date: Sat May 1 18:19:43 2004 Subject: [Python-Dev] bags in collections - PEP 320 Message-ID: <20040429190133.GA4568@designtheory.org> Dear All, I have just read PEP-320 and noticed the following comments regarding the "collections" package: ... - ? bag (only if use cases established) ... I would like to argue for such a use case. We at designtheory.org are working on a (maths / stats / comp. sci.) project in the area of Design Theory. Design theory intersects with many fields, to mention a few: combinatorics, graph theory, finite geometry, coding theory, and the design of statistical experiments (from which then name of the field comes). Most of our software development will be in Python, although the released python software so far is very modest. To show why bags are important for us and, in general, for discrete mathematicians, here is the definition of the most important type of designs. A binary block design is a multiset (that is a 'bag') of subsets of a 'base' set; the elements of the base set are called 'points' and its subsets are called 'blocks'. (Personally, I prefer the name bag but most mathematicians use multiset.) A non-binary block design is one whose blocks can also be multisets. The computations we are facing are very much combinatorial in nature so we are happy to see that Sets became a builtin and, of course, would like to see a C implementation for bags too. Our pydesign package (will) heavily use C extensions. So far we use numarray to compute statistical properties of designs, but many more combinatorial functionalities will be implemented in C. In particular, we are planning to implement a basic permutation group package whose core will eventually be a C extension. We would like to deal with automorphism groups and isomorphisms of designs on at least a basic level within the pydesign package without having to resort to specialized mathematical packages like GAP. These functionalities can be useful not only for design theorist, but for anybody using Python to deal with combinatorial structures, like graphs for example. In all of these areas the use of multisets (bags) is pervasive. Please, implement it -- so we don't have to :-) For more details on our project, please visit http://designtheory.org/ -- , Peter Dobcsanyi