From ianb at colorstudy.com Sat Apr 1 00:36:44 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 31 Mar 2006 16:36:44 -0600 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <442D3D13.7050103@benjiyork.com> References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <442BB1BC.9030804@gmail.com> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> <442C60B9.9020302@colorstudy.com> <442C73C3.2090203@colorstudy.com> <442C8B8A.40906@canterbury.ac.nz> <8541A849-BEBC-416B-AAF6-C222BEA9F51C@gmail.com> <442CB3D3.3010705@canterbury.ac.nz> <442D3D13.7050103@benjiyork.com> Message-ID: <442DAEFC.1020506@colorstudy.com> Benji York wrote: > Alex Martelli wrote: > >>If the framework consuming X requested adaptation-to-X on all objects >>it's passed, > > > That's not generally the way Zope 3 does it (and doesn't sound like a > good idea to me). > > There are three ways (as I see it) adaptation is used in Z3. First, the > traditional idea of an "adapter", in which something has the right > state, but the wrong interface. That form of adaptation is used when > plugging different components together (whether they are Z3 or come from > other projects). In that way of using adaptation a > function/method/whatever wants things that act in a particular way (duck > typing). If I, as the user of the interface, have something I want to > pass in that doesn't match I it to the appropriate interface the burden > is on me to create something that matches expectations. People do that > all the time today without an interface/adaption framework, they just > write code that takes one thing and builds another. > > Greg Ewing wrote: > >>This is the part that bothers me, I think. It >>seems like all these adaptation requests would >>be a huge burden on the framework developer. > > > Not in the above scenario, when using adapters like that the burden is > on the user. That might sound like a bad thing, but if they're > exclusively using your library, they already have objects of the > necessary type, if not they have an adaptation framework to help them do > something they'd have to do anyway. > > The second way adaptation is used is as a general lookup facility. Say > that I have a user object and I want to know their security information. > Instead of building an API for looking up security descriptions from a > user name that I have to pull out of the user object, I could instead > register and adapter from IUser to ISecurityInfo, now I don't need any > new APIs, I just so sec_info = ISecurityInfo(the_user). This form of > adaptation is good for the "I have something and want more information > about it" use case. It also adds some flexibility, the workings of the > adapter can change without having to change all the client code as you'd > have to do if you changed (for example) the parameters an API expected. > > The third way it's used is to make systems pluggable. You mentioned > PyGUI, so say you had a schema describing a data entry form. You could > use adaptation to decide which GUI widget would be used for each field. > Looping over the form fields and adapting each to IWidget and getting > back a TextField for a string, CheckBox for a boolean, etc. Then if the > user has a nice TextField object with spell checking, they could just > plug in a different adapter and all their fields would get the new > widget without PyGUI having to support a plug-in framework. This is where I start getting nervous about adaptation, that it is used for a few too many things. For instance, ISecurityInfo(the_user) is essentially providing magic-method-like introspection, except potentially you don't need the cooperation of the object, and the fetching has a namespace (a method has only a name, but ISecurityInfo actually lives someplace specific). Those are useful features, but is it adaptation? If not adaptation, what do we call it? I suspect, given the right name, some aspects of the implementation can be streamlined and it will become a much more sensible operation to people. Magic methods are workable but hardly perfect, and the typical way they usually get used is that some function knows about all "normal" types (especially the fundamental types that can't have methods added to them) and falls back on a magic method. It's not pretty, though honestly I don't know of particularly pretty techniques that other languages use. The last item feels much more comfortable to me as generic functions (like in PJE's RuleDispatch). It covers the use cases where multi-adaptation seems to pop up, and to me multi-adapatation just looks like the poor man's generic functions implemented in terms of adaptation, because adaptation systems (well, Zope interfaces) have a nice declarative system and pattern matching facilities. But leveraging that just to avoid real generic functions seems wrong to me. Adaptation-that-is-actually-adaptation feels more like views to me. You need a specific interface, but you want to be able to accept objects that are isomorphic to that interface. Finding the right view (aka adapter) is something that calls for all the kinds of mechanisms Zope interfaces have -- interfaces and declarations of views/adapters. But the promises that a view makes are much stronger than what an adapter makes in current systems. A view can only be a subset of what the original object provides, but that doesn't seem to be the norm for Zope adapters. That said... is ISecurityInfo(the_user) a view of the user? Possibly, depending on how it is implemented. If it is implemented as "return self.security_info" then no, unless the_user.security_info is read-only. If the adapter is a proxy object that forwards everything to "the_user.security_info", then yes. But something still feels weird to me about it. And I suppose ISomeInterface(obj1, obj2) could create something that is a subset of the two objects. Hrm... the whole thing makes me feel funny, though. Maybe it's that I want adaptation to be reversable. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From jimjjewett at gmail.com Sat Apr 1 01:47:45 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 31 Mar 2006 18:47:45 -0500 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] Message-ID: On 3/30/06, Alex Martelli wrote: > Consider __index__, and a user of gmpy, assuming > gmpy didn't rush out a 2.5 release with tp_index support. > The user of gmpy would be stuck -- no way he could > use a gmpy.mpz as an index into a list, because > the ad-hoc-adaptation of __index__ means that the type > itself must grow the slot. So the adaptation would be def gmpy_mpz_index(self): return long(self) gmpy.mpz.__index__=gmpy_mpz_index Monkeypatching someone else's class may not be pretty, but adaptation is basically a standing offer to (less efficiently) pretend you did. If neither the caller nor the callee makes an explicit call to the adaption machinery, then you might as well change the original class. At the moment, some classes throw an TypeError if you try to add attributes later. Are you suggesting that this behavior should be removed? (I'll grant that it isn't *usually* helpful.) What about updates, rather than inserts? Should I be able to replace dict.__getitem__ and have it affect all existing dicts? (And if this thread continues, should it move to python-dev?) -jJ From greg.ewing at canterbury.ac.nz Sat Apr 1 02:39:16 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Apr 2006 12:39:16 +1200 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <79990c6b0603310418v738bec87s5ac710a9a1aa6dab@mail.gmail.com> References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> <442C60B9.9020302@colorstudy.com> <442C73C3.2090203@colorstudy.com> <442C8B8A.40906@canterbury.ac.nz> <8541A849-BEBC-416B-AAF6-C222BEA9F51C@gmail.com> <442CB3D3.3010705@canterbury.ac.nz> <79990c6b0603310418v738bec87s5ac710a9a1aa6dab@mail.gmail.com> Message-ID: <442DCBB4.4000706@canterbury.ac.nz> Paul Moore wrote: > The "traditional" solution, __index__, requires co-operation > from all classes that want to support the new protocol. Adaptation > doesn't - it can be added externally. I don't follow that. Adaptation only works if the framework author has thought to make an adaptation call at the point where it's needed. If I, as the framework author, were the one that benefited from making adaptation calls, then I could leave them until I needed them. But that's not how it works -- it's *users* of my framework that benefit, and they're not in a position to add them to my framework. So I have to think ahead and try to anticipate what adaptations other people might want my code to make. > Rather than being a solution looking for a problem, I suspect it's > more of a chicken and egg issue. It's more than that, I think -- it's that making effective use of it requires predicting the future. Until time machines become a commodity item, that isn't going to make adoption easy. -- Greg From greg.ewing at canterbury.ac.nz Sat Apr 1 03:15:43 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Apr 2006 13:15:43 +1200 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <442D3D13.7050103@benjiyork.com> References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <442BB1BC.9030804@gmail.com> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> <442C60B9.9020302@colorstudy.com> <442C73C3.2090203@colorstudy.com> <442C8B8A.40906@canterbury.ac.nz> <8541A849-BEBC-416B-AAF6-C222BEA9F51C@gmail.com> <442CB3D3.3010705@canterbury.ac.nz> <442D3D13.7050103@benjiyork.com> Message-ID: <442DD43F.2070400@canterbury.ac.nz> Benji York wrote: > If I, as the user of the interface, have something I want to > pass in that doesn't match I it to the appropriate interface the burden > is on me to create something that matches expectations. People do that > all the time today without an interface/adaption framework, they just > write code that takes one thing and builds another. Yes, and that seems like a fine way of doing it. You know what you've got and where you want to get to, so there's no need to look anything up in a registry. > Instead of building an API for looking up security descriptions from a > user name that I have to pull out of the user object, I could instead > register and adapter from IUser to ISecurityInfo, I don't see how this is better than just calling a get_security_info_from_user() function, the implementation of which can be changed if necessary without affecting any code that calls it. > Looping over the form fields and adapting each to IWidget and getting > back a TextField for a string, CheckBox for a boolean, etc. Here I don't see how it's better than passing in a dict mapping field classes to widget classes. Worse, in fact, since it would seem to restrict you to a single global mapping for all forms. > once you have the simple tools of adaptation in mind > ... you start to recognize places where they help > you solve problems in better ways. Or they lead you into an everything-is-a-nail mode of thinking, which is what the above seem to be to me, to some extent. -- Greg From adam.deprince at gmail.com Sat Apr 1 04:34:48 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Fri, 31 Mar 2006 21:34:48 -0500 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: References: <4422FC96.2020409@zope.com> <79990c6b0603271117l4c69372h11362dd5d2d0ca32@mail.gmail.com> <1143565628.3305.82.camel@localhost.localdomain> <4429F6BD.60704@canterbury.ac.nz> <442A5550.5090804@ofai.at> <442B20C4.70309@canterbury.ac.nz> <1143744246.3204.126.camel@localhost.localdomain> <442C9224.3050601@canterbury.ac.nz> Message-ID: <1143858888.3302.66.camel@localhost.localdomain> On Thu, 2006-03-30 at 21:56 -0800, Guido van Rossum wrote: > > Adam DePrince wrote: > > > No reason we can't make other string operations views as well ... > > > concatenation is one example. If I recall, that's how snobol handles > > > strings, view upon view upon view. > > But that's irrelevant for immutable strings -- views are about > semantic links, not implementation. I confess, I was perpetuating the abuse of the term view; snobol strings have no bearing other than comic relief on what we are doing here. To treat slicing as a view does have some bearing ... but I'll get to that in a moment. I believe I need to break this PEP up into several parts. The first part needs formalize a practice that is starting to become prevalent in Python's C implementation with the introduction of iter ... the notion of a surrogate. Right now an iter can be over anything, real python or "verbal description" ... iter-values is the later. It is an iter that mutates the form of the dict prior to returning it. I think where I want to take the view proposal, which is really a broader form of the list/iter debate is to separate the mutation associated with a view from an iter, which basically embodies "the for-loop's state." There has been some confusion ... and just to address it, the vision that I see is for items to be a pointer to the parasite rather than a generator for an iter/list. For backwards compatibility dict.items would return self when called. There is to be a one-to-one relationship between an object and a view. d is d asserts d.items is d.items under the view proposal, and d.items is some sort of parasitic object that hangs off of the dict for the single purpose of providing this different perspective. d.items will not have a next method for the same reason d doesn't, that isn't the place to store our loop state :-) Instead, each will have its own .__iter__ from which fresh objects whose sole ambition is to store a loop pointer will spring into existence. There are two ways we could create views ... the fixed reference .items, again with self on call for backwards compatibility, and a proper view generator (for instances that require the acceptance of parameters). On Thu, 2006-03-30 at 12:05 +1200, Greg Ewing wrote: > Stefan Rank wrote: > > > A big question is: Should slicing also return views? and why not? > > That's been considered before, in relation to strings. > The stumbling block is the problem of a view of a > small part of the object keeping the whole thing > alive and using up memory. > > While having a separate way of getting slice-views > could be useful, I think it would be too big a > change in semantics to make it the default > behaviour of slicing notation. The conversation about slicing came from Stefan Rank's earlier question (copied above) We could very well implement a generic OrderdView -> OrderedView mutator that effectively performs slicing. Its not absurd for that would become the default means of slicing in Python, override-able when a data structure specific version has a compelling performance advantage. It is my expectation that eventually number of the things we do in python could eventual be seen as "view" mutations; operations defined between two interfaces rather than a concrete class and an interface. This is getting rather ambitious ... I'm going to sit back and rework the PEP's both address the concerns that people have had, size, and to further clarify the relationship between data-stores, views, iters and their cousins, returned lists. Cheers - Adam DePrince From adam.deprince at gmail.com Sat Apr 1 04:45:10 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Fri, 31 Mar 2006 21:45:10 -0500 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <442C951C.4010002@canterbury.ac.nz> References: <4422FC96.2020409@zope.com> <1f7befae0603241822n1280ba67i3ed87e842114f18b@mail.gmail.com> <1143306134.3186.1.camel@localhost.localdomain> <44258376.1080709@gmx.net> <1143432484.14391.67.camel@localhost.localdomain> <79990c6b0603271117l4c69372h11362dd5d2d0ca32@mail.gmail.com> <1143565628.3305.82.camel@localhost.localdomain> <4429F6BD.60704@canterbury.ac.nz> <442B207A.60305@canterbury.ac.nz> <442BAB0C.6080901@gmail.com> <1143738089.3204.41.camel@localhost.localdomain> <442C951C.4010002@canterbury.ac.nz> Message-ID: <1143859511.3302.74.camel@localhost.localdomain> On Fri, 2006-03-31 at 14:34 +1200, Greg Ewing wrote: > Adam DePrince wrote: > > Views > > are not generated, they are either directly implemented, or returned. > > If you're thinking that the object would keep a set of > pre-allocated views, there's a problem with that -- > the views need to have a reference to the base object, > thus creating a circular reference. > > The object could perhaps keep a cache of weakly- > referenced views, returning one of those if it's > available, otherwise creating a new one. Yes, we would have to do that. object->iter relationships work because the reference is only from the iter back; we don't try to reuse iters as we would views. Weak refernces work. Okay, perhaps parasite views should be handed out by a callable after all instead of saying "hey, reuse this reference here." It would actually be more compatable with what we do, and fit better with the situation where a views would be created on the fly (slices?) - Adam > > -- > Greg > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/adam.deprince%40gmail.com From adam.deprince at gmail.com Sat Apr 1 04:52:34 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Fri, 31 Mar 2006 21:52:34 -0500 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <442BB1BC.9030804@gmail.com> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> <20060331012037.GA7960@panix.com> Message-ID: <1143859955.3302.80.camel@localhost.localdomain> On Thu, 2006-03-30 at 23:37 -0800, Neal Norwitz wrote: > On 3/30/06, Terry Reedy wrote: > > > > "Aahz" wrote in message > > news:20060331012037.GA7960 at panix.com... > > > What do we want to tell people who have code like this: > > > > > > keys = d.keys() > > > keys.sort() > > > > Could a good-enough code analyzer detect such, even if separated by > > intervening lines? If so, it could suggest sorted() as a fix. I wonder if > > the pypy analyzer could be adapted for 2.x to 3.0 warning and upgrade > > purposes. Or do pylint or pychecker gather enough information? > > pychecker is supposed to have this info, but only if d is known to be > a dict. It could be extended to assume any method keys() (and > friends) should return iterators. In which case, it would say that an > iterator doesn't have a sort method. Below is the output of the > current version. With the views that we were talking about before, with d.keys() all you have is the view, neither an iter nor list have been instantiated. I'd almost say that keys = d.keys() foo = keys.sort() could fit quite well into the view framework. > > n > > ### file: tt.py > def foo(): > d = {} > keys = d.keys() > keys.sort() > keys.sort2() > ### > > $ pychecker tt.py > Processing tt... > > Warnings... > > tt.py:6: Object (keys) has no attribute (sort2) > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/adam.deprince%40gmail.com From adam.deprince at gmail.com Sat Apr 1 05:03:39 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Fri, 31 Mar 2006 22:03:39 -0500 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <442C5152.2000400@gmail.com> References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <442BB1BC.9030804@gmail.com> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> Message-ID: <1143860619.3302.93.camel@localhost.localdomain> On Fri, 2006-03-31 at 07:44 +1000, Nick Coghlan wrote: > Adam DePrince wrote: > > There seemed to be a concensus in the community on the size of the view > > proposal, and I'm reimplementing the PEP to reflect that. But what I > > can't resolve is the other anciliary issue: "To list or iter." I'm not > > yet ready to resolve that issue. The views don't resolve it either, and > > by their nature are biased towards the iter approach. They provide > > __iter__ because its light weight to do, but there is no way a light > > weight view can provide you with ordering information from an unordered > > datastore. Now, as a means of resolving this conflict, I'm open to the > > notion of a view implementing both __iter__ and an explicit .list method > > to avoid any extra overhead in generating a list from an iter instead of > > directly from the dict as we do now. > > Umm, the whole point of the views discussion is the realisation that "list or > iterator" is a false dichotomy. The correct answer is "new iterable that looks The false dichotomy is not of my construction; some of the objections that I've received have been of the form: """When I say >>>print dict.items() with views instead of iters it still won't look like a list! """ Until that debate is resolved it will still taint discussion of any non-lists that dict.items/values might return. - Adam From ncoghlan at gmail.com Sat Apr 1 06:39:54 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 01 Apr 2006 14:39:54 +1000 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <1143859955.3302.80.camel@localhost.localdomain> References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <442BB1BC.9030804@gmail.com> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> <20060331012037.GA7960@panix.com> <1143859955.3302.80.camel@localhost.localdomain> Message-ID: <442E041A.5010701@gmail.com> Adam DePrince wrote: > On Thu, 2006-03-30 at 23:37 -0800, Neal Norwitz wrote: >> On 3/30/06, Terry Reedy wrote: >>> "Aahz" wrote in message >>> news:20060331012037.GA7960 at panix.com... >>>> What do we want to tell people who have code like this: >>>> >>>> keys = d.keys() >>>> keys.sort() >>> Could a good-enough code analyzer detect such, even if separated by >>> intervening lines? If so, it could suggest sorted() as a fix. I wonder if >>> the pypy analyzer could be adapted for 2.x to 3.0 warning and upgrade >>> purposes. Or do pylint or pychecker gather enough information? >> pychecker is supposed to have this info, but only if d is known to be >> a dict. It could be extended to assume any method keys() (and >> friends) should return iterators. In which case, it would say that an >> iterator doesn't have a sort method. Below is the output of the >> current version. > > With the views that we were talking about before, with d.keys() all you > have is the view, neither an iter nor list have been instantiated. > > I'd almost say that > > keys = d.keys() > foo = keys.sort() > > could fit quite well into the view framework. Not a good idea, since the long history of "list.sort()" encourages people to think of the sort() method as an inplace operation, which it wouldn't be on a view. "sorted()", on the other hand, already creates a new object. The only downside is that Py2.x & Py3k compatible code would look like: keys = sorted(d.keys()) which is likely to create the list *twice* in Py2.x. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From aleaxit at gmail.com Sat Apr 1 07:20:37 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Fri, 31 Mar 2006 21:20:37 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: Message-ID: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> On Mar 31, 2006, at 3:47 PM, Jim Jewett wrote: > On 3/30/06, Alex Martelli wrote: >> Consider __index__, and a user of gmpy, assuming >> gmpy didn't rush out a 2.5 release with tp_index support. >> The user of gmpy would be stuck -- no way he could >> use a gmpy.mpz as an index into a list, because >> the ad-hoc-adaptation of __index__ means that the type >> itself must grow the slot. > > So the adaptation would be > > def gmpy_mpz_index(self): > return long(self) > gmpy.mpz.__index__=gmpy_mpz_index > > Monkeypatching someone else's class may not be pretty, but adaptation And may just be impossible for ``classes'' that are actually types implemented in C, as is the case for gmpy. > is basically a standing offer to (less efficiently) pretend you did. > If neither the caller nor the callee makes an explicit call to the > adaption machinery, then you might as well change the original class. > > At the moment, some classes throw an TypeError if you try to add > attributes later. Are you suggesting that this behavior should be > removed? (I'll grant that it isn't *usually* helpful.) I doubt the advantage's worth breaking compatibility with all existing C-coded extensions. And the existing way to simulate adaptation isn't most typically to add special methods to an existing class (though, experimentally, gmpy may do it to decimal.Decimal, in the current release) -- probably because in the current implementation that just wouldn't work with types (implemented in C). Rather, look (e.g.) at copy_reg for the typical kludge that's used to reimplement that crucial design pattern that is adaptation, over and over and over again. > What about updates, rather than inserts? Should I be able to replace > dict.__getitem__ and have it affect all existing dicts? If that's what you want, I suggest Ruby -- the fact that everything is always mutable is one big reason I'm wary of actually trying production-level development in Ruby (well, right now there's also the fact that Google uses Python, not Ruby, but that reason didn't apply to me until I moved to Google exactly a year ago;-). Anyway, that tangent has nothing to do with adaptation. > > (And if this thread continues, should it move to python-dev?) If it discusses making every type mutable, I think it should stay on python-3000. As far as adaptation (which has nothing to do with that idea, really) is concerned, I have no problem moving the thread -- just little hope that adaptation will actually make it into the language in any 2.* release (for no good reason). Alex From guido at python.org Sat Apr 1 07:31:51 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 31 Mar 2006 21:31:51 -0800 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <442E041A.5010701@gmail.com> References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <442BB1BC.9030804@gmail.com> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> <20060331012037.GA7960@panix.com> <1143859955.3302.80.camel@localhost.localdomain> <442E041A.5010701@gmail.com> Message-ID: On 3/31/06, Nick Coghlan wrote: > Not a good idea, since the long history of "list.sort()" encourages people to > think of the sort() method as an inplace operation, which it wouldn't be on a > view. Right. This is degenerating quickly. :-( > "sorted()", on the other hand, already creates a new object. The only > downside is that Py2.x & Py3k compatible code would look like: > > keys = sorted(d.keys()) > > which is likely to create the list *twice* in Py2.x. So write keys = sorted(d) which should work in both versions. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Sat Apr 1 08:37:51 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 01 Apr 2006 18:37:51 +1200 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> Message-ID: <442E1FBF.50801@canterbury.ac.nz> Alex Martelli wrote: > Rather, look (e.g.) at copy_reg for the typical > kludge that's used to reimplement that crucial design pattern that is > adaptation, over and over and over again. I don't see what copy_reg has to do with adaptation. What does it adapt, and to what? You seem to be talking about adaptation as if it were a general way of mapping a class to anything else at all. We already have one of those, it's called a dict. Also, design patterns don't usually have any generic implementation, and that's okay. It's the reason they're called *design* patterns. > just little hope that adaptation will actually > make it into the language in any 2.* release > (for no good reason). The fact that many people are unconvinced by the case made for it so far seems like a good reason to me. -- Greg From aleaxit at gmail.com Sat Apr 1 09:42:19 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Fri, 31 Mar 2006 23:42:19 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <442E1FBF.50801@canterbury.ac.nz> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> Message-ID: On Mar 31, 2006, at 10:37 PM, Greg Ewing wrote: > Alex Martelli wrote: >> Rather, look (e.g.) at copy_reg for the typical >> kludge that's used to reimplement that crucial design pattern >> that is >> adaptation, over and over and over again. > > I don't see what copy_reg has to do with adaptation. Funny, it seems so blindingly obvious to me, that copy_reg is nothing but an ad-hoc implementation of an adaptation registry for one special case (while all sorts of special methods perform similar tasks in a different way which has the disadvantage of being necessarily invasive on the type needing to be adapted). > What does it adapt, and to what? It adapts whatever type you're passing to copy_reg.pickle, to the protocol "being picklable". > You seem to be talking about adaptation as if it > were a general way of mapping a class to anything > else at all. We already have one of those, it's > called a dict. A dict is one fine way to implement an adaptation registry (and I believe it's what copy_reg uses for the purpose). But dicts are fine ways of implementing a huge variety of things, from sets (dicts are what we all used to implement sets before module sets.py, and later builtin type set, appeared) to classes, instances, modules, &c, via their respective __dict__s. The existence of dicts is a laughably feeble excuse to avoid having adaptation in Python's standard library, just as it would be to avoid having sets, or classes, instances, modules. > Also, design patterns don't usually have any > generic implementation, and that's okay. It's > the reason they're called *design* patterns. Sure, but as languages get higher-level, and grow powerful libraries and frameworks, the repetitive implementations of many patterns become part of the language/libraries. In machine code, circa 1950, "procedure call/return" was a design pattern, typically and repetitively implemented by placing the PC "somewhere" (that LIFO stacks were the ideal "somewheres" wasn't necessarily clear to all implementers back then) and jumping to a piece of code which ended by restoring the PC from that "somewhere". Since Fortran, languages have embodied the concept; it's not a "design pattern" any more, it's a fundamental language concept. Similarly, at a much higher level, "reactor" is a design pattern, but libraries/frameworks such as ACE and Twisted embody it as a fundamental concept. "Mock Object" is another excellent example of a DP that (in sufficiently high-level languages) can be, and is, usefully embodied in libraries. > >> just little hope that adaptation will actually >> make it into the language in any 2.* release >> (for no good reason). > > The fact that many people are unconvinced by > the case made for it so far seems like a good > reason to me. In the end, the one person that matters, regarding what gets into Python or doesn't, is Guido. When I proposed 'sum', for example, everybody else was starting to take potshots at the idea (as usual: propose *anything* on python-dev, and there will NEVER be a dearth of nay-sayers), but that mattered not a whit, because Guido on that one occasion happened to like the idea. So, the fact that (say) Eby or Martelli or the Twisted folks or the Zope folks like adaptation, and Ewing or Bicking or (etc ec) dislike it, is really a side show. The point that matters, instead, is whether GvR likes it or not. From my viewpoint, the first blocking issue is thus that GvR sees adaptation as necessarily requiring the existence of interfaces as a formal Python concept -- perhaps because that's how PyProtocols and Twisted/Zope deploy it -- and thus won't really consider it until he's blessed some specific form of interfaces. In my view of the issue, while interfaces (or supersets thereof such as protocols) are a great setting for adaptation, there is really no _need_ to formalize a specific packet of interface features to make adaptation already useful. One could have the protocol specification be a *string*, for Pete's sake -- say 'org.python.stdlib.index' to require "an integer value usable as index into a sequence", 'org.python.stdlib.pickle' to require "an object supplying methods reductor and constructor", or whatever; and even with such a minimalistic, entirely conventions-based approach, explicitly using adaptation would still be practically useful, better than the __index__ approach (because it can be noninvasive) and better than the copy_reg approach (because it does not require ad-hoc reimplementation of registries over and over again). Blocking the consideration of adaptation because the best way to formalize interfaces is not yet clear is, in my view, not a good reason. Ah well, these discussions never produce useful results; indeed, that's why I've _almost_ stopped pointing out, each and every time yet another ad-hoc (partial, and hobbled) implementation of adaptation gets adopted, how the fragmentation could be stopped and all such cases usefully unified by accepting protocol-adaptation. Once in a while, I'm moved to sing this refrain again, but thanks to the ensuing discussion I'm soon reminded that there are many more gratifying activities I could pursue instead -- repeated beating of my forehead against suitable brick walls spring to mind as being both more productive and more fun, for example. Alex From nnorwitz at gmail.com Sat Apr 1 10:17:44 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sat, 1 Apr 2006 00:17:44 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> Message-ID: On 3/31/06, Alex Martelli wrote: > > So, the fact that (say) Eby or Martelli or the Twisted folks or the > Zope folks like adaptation, and Ewing or Bicking or (etc ec) dislike > it, is really a side show. The point that matters, instead, is > whether GvR likes it or not. From my viewpoint, the first blocking > issue is thus that GvR sees adaptation as necessarily requiring the > existence of interfaces as a formal Python concept -- perhaps because > that's how PyProtocols and Twisted/Zope deploy it -- and thus won't > really consider it until he's blessed some specific form of interfaces. The best way to demonstrate a better way is through code. Compare real code with and without adaptation. n From thomas.lotze at gmx.net Sat Apr 1 14:32:19 2006 From: thomas.lotze at gmx.net (Thomas Lotze) Date: Sat, 01 Apr 2006 14:32:19 +0200 Subject: [Python-3000] Iterating over a dict Message-ID: Hi, I wonder what's the reason for iterating over a dict by keys: >>> for x in {1:"a", 2:"b"}: ... print x ... 1 2 I find it much more intuitive for the values, "a" and "b", to be accessed. This is particularly confusing as iterating over tuples, lists and sets in the same way does access the values. (It feels like iterating over a list l actually iterates over the index values, range(len(l)).) In fact, iterating over any container should access the contained values. The reason I ask this on the python-3000 list is that I wonder whether the iterating behaviour of dicts might be changed in Python 3k, so that in the above code, foo() would be applied to the dict's values. -- Thomas From benji at benjiyork.com Sat Apr 1 16:35:23 2006 From: benji at benjiyork.com (Benji York) Date: Sat, 01 Apr 2006 09:35:23 -0500 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: Message-ID: <442E8FAB.9050604@benjiyork.com> Alex Martelli wrote: > Consider __index__, and a user of gmpy, Jim Jewett wrote: > So the adaptation would be > > def gmpy_mpz_index(self): > return long(self) > gmpy.mpz.__index__=gmpy_mpz_index > > Monkeypatching someone else's class may not be pretty, but adaptation > is basically a standing offer to (less efficiently) pretend you did. No, monkey patching isn't pretty, thankfully that's not how it works. To express that use case with Zope 3 adaptation you'd want an IIndex interface that (say) promises to have a .index() method, if you want to get an index from a gmpy value, you'd adapt it like so, and call the method: IIndex(gmpy_value).index() It's a design decision as to weather a particular method wants just an index passed in, wants something that already conforms to the IIndex interface (which is really duck typing, in Z3 we don't enforce interfaces), or something that is adaptable to IInterface. So, if I wanted to use gmpy.mpz objects as indices and there isn't an appropriate adapter, I can define my own. If the gmpy project then added an .index() method to mpz objects and flagged them as providing IIndex, my adapter wouldn't be necessary any more (because adapting something to an interface that it already provides returns the original object). Alternately they could provide their own adapter, and I could stop using mine. -- Benji York From benji at benjiyork.com Sat Apr 1 16:46:09 2006 From: benji at benjiyork.com (Benji York) Date: Sat, 01 Apr 2006 09:46:09 -0500 Subject: [Python-3000] Iterating over a dict In-Reply-To: References: Message-ID: <442E9231.9020504@benjiyork.com> Thomas Lotze wrote: > Hi, > > I wonder what's the reason for iterating over a dict by keys: I suspect that at least part of the reason is that given a key you can easily get the corresponding value, but given the value it's difficult to get the key. Seems like a good choice for "bare" dict iteration. -- Benji York From benji at benjiyork.com Sat Apr 1 17:34:30 2006 From: benji at benjiyork.com (Benji York) Date: Sat, 01 Apr 2006 10:34:30 -0500 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <442DCBB4.4000706@canterbury.ac.nz> References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> <442C60B9.9020302@colorstudy.com> <442C73C3.2090203@colorstudy.com> <442C8B8A.40906@canterbury.ac.nz> <8541A849-BEBC-416B-AAF6-C222BEA9F51C@gmail.com> <442CB3D3.3010705@canterbury.ac.nz> <79990c6b0603310418v738bec87s5ac710a9a1aa6dab@mail.gmail.com> <442DCBB4.4000706@canterbury.ac.nz> Message-ID: <442E9D86.5030101@benjiyork.com> Greg Ewing wrote: > Paul Moore wrote: >>The "traditional" solution, __index__, requires co-operation >>from all classes that want to support the new protocol. Adaptation >>doesn't - it can be added externally. > > I don't follow that. Adaptation only works if the > framework author has thought to make an adaptation > call at the point where it's needed. For this use case, you're right, but Paul was talking about co-operation from the inputs, not the framework. If the framework adapted something to IIndex and the framework user wanted to pass something that didn't conform to IIndex and didn't have an adapter available, the user could provide one. > If I, as the framework author, were the one that > benefited from making adaptation calls, then I > could leave them until I needed them. But that's > not how it works -- it's *users* of my framework > that benefit, and they're not in a position to > add them to my framework. As the framework author you want to increase the benefit to your users, so you find ways to do that. Those ways include adding adaptation where it creates benefits. > So I have to think > ahead and try to anticipate what adaptations > other people might want my code to make. Just as you have to think ahead and anticipate what functionality other people might want. Adaptation isn't magic; it's still part of iterative design improvement. -- Benji York From benji at benjiyork.com Sat Apr 1 17:46:47 2006 From: benji at benjiyork.com (Benji York) Date: Sat, 01 Apr 2006 10:46:47 -0500 Subject: [Python-3000] Iterators for dict keys, values, and items == annoying :) In-Reply-To: <442DD43F.2070400@canterbury.ac.nz> References: <435DF58A933BA74397B42CDEB8145A86010CBC5E@ex9.hostedexchange.local> <442BB1BC.9030804@gmail.com> <1143744058.3204.124.camel@localhost.localdomain> <442C5152.2000400@gmail.com> <442C60B9.9020302@colorstudy.com> <442C73C3.2090203@colorstudy.com> <442C8B8A.40906@canterbury.ac.nz> <8541A849-BEBC-416B-AAF6-C222BEA9F51C@gmail.com> <442CB3D3.3010705@canterbury.ac.nz> <442D3D13.7050103@benjiyork.com> <442DD43F.2070400@canterbury.ac.nz> Message-ID: <442EA067.4080204@benjiyork.com> Greg Ewing wrote: > Benji York wrote: >>If I, as the user of the interface, have something I want to >>pass in that doesn't match I it to the appropriate interface the burden >>is on me to create something that matches expectations. People do that >>all the time today without an interface/adaption framework, they just >>write code that takes one thing and builds another. > > Yes, and that seems like a fine way of doing it. You > know what you've got and where you want to get to, so > there's no need to look anything up in a registry. There are two benefits. The first comes when you need to do it more than once, so you don't end up designing an API to go from the one thing to another. Instead you have an adaptation framework to work within. The second is that you can generalize and not have to know what, exactly, you're converting from, if it is, or has an adapter to, the correct thing, you're good. Much like duck typing. >>Instead of building an API for looking up security descriptions from a >>user name that I have to pull out of the user object, I could instead >>register and adapter from IUser to ISecurityInfo, > > I don't see how this is better than just calling > a get_security_info_from_user() function Less code. Instead of designing a special purpose API to retrieve one form of information given another, you adapt. >> Looping over the form fields and adapting each to IWidget and getting >>back a TextField for a string, CheckBox for a boolean, etc. > > Here I don't see how it's better than passing in > a dict mapping field classes to widget classes. > Worse, in fact, since it would seem to restrict > you to a single global mapping for all forms. Again, simpler API. You're freed from passing around (or building) lots of mini-registries of things and defining APIs for each. >>once you have the simple tools of adaptation in mind >> ... you start to recognize places where they help >>you solve problems in better ways. > Or they lead you into an everything-is-a-nail > mode of thinking, which is what the above seem > to be to me, to some extent. Adaptation, like anything else, can be overused. There are situations, like the above, where adaptation can improve the code. Seems pretty Pythonic to me, just as dicts and lists can be used (well) in lots of situations, likewise for adaptation. -- Benji York From aahz at pythoncraft.com Sat Apr 1 18:31:02 2006 From: aahz at pythoncraft.com (Aahz) Date: Sat, 1 Apr 2006 08:31:02 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> Message-ID: <20060401163102.GA8292@panix.com> On Fri, Mar 31, 2006, Alex Martelli wrote: > > Once in a while, I'm moved to sing this refrain again, but thanks to > the ensuing discussion I'm soon reminded that there are many more > gratifying activities I could pursue instead -- repeated beating of > my forehead against suitable brick walls spring to mind as being both > more productive and more fun, for example. That's because you're a masochist. ;-) Seriously, I can almost see why you think adaptation is a huge gain, but every time I start looking closer, I get bogged down in trying to understand adaptation registration. Do you have a simple way of explaining how that works *well* and *simply*? Because unless one can handle registration cleanly, I don't understand how adaptation can be generally useful for Python. Conversely, if adaptation registration is *NOT* required, please explain that in simple terms. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet From benji at benjiyork.com Sat Apr 1 19:42:33 2006 From: benji at benjiyork.com (Benji York) Date: Sat, 01 Apr 2006 12:42:33 -0500 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <20060401163102.GA8292@panix.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> Message-ID: <442EBB89.4020506@benjiyork.com> Aahz wrote: > On Fri, Mar 31, 2006, Alex Martelli wrote: > >>Once in a while, I'm moved to sing this refrain again, but thanks to >>the ensuing discussion I'm soon reminded that there are many more >>gratifying activities I could pursue instead -- repeated beating of >>my forehead against suitable brick walls spring to mind as being both >>more productive and more fun, for example. > > > That's because you're a masochist. ;-) > > Seriously, I can almost see why you think adaptation is a huge gain, but > every time I start looking closer, I get bogged down in trying to > understand adaptation registration. Do you have a simple way of > explaining how that works *well* and *simply*? Because unless one can > handle registration cleanly, I don't understand how adaptation can be > generally useful for Python. Conversely, if adaptation registration is > *NOT* required, please explain that in simple terms. In Zope 3 adapters are normally registered via a one line ZCML directive, but that's more for configuration management than anything else; it makes it easier to swap adapters in and out. You can also use Python to do the same thing: from zope import component component.provideAdapter(MyAdapter) MyAdapter can be any callable that takes one or more objects (the things being adapted) and provides a single interface. The class would look like this: class MyAdapter: component.adapts(ISomething) interface.implements(ISomethignElse) def __init__(self, something): ... So in this case the "provideAdapter" queries the adapter to find out what is being adapted to and from. You can also provide those to the provideAdapter call if MyAdapter implements more than one interface or you need to for some other reason. For adapters that are just functions, decorators can be used to indicate the "to" and "from" interfaces (following the "index" example from earlier): @component.adapter(IMpz) @interface.implementer(IIndex) def index(mpz): return long(mpz) The zope.component README goes into more detail (especially the "Adapters" section): http://svn.zope.org/*checkout*/Zope3/trunk/src/zope/component/README.txt?rev=39671 -- Benji York From aleaxit at gmail.com Sat Apr 1 20:50:51 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Sat, 1 Apr 2006 10:50:51 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <20060401163102.GA8292@panix.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> Message-ID: <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> On Apr 1, 2006, at 8:31 AM, Aahz wrote: ... > Seriously, I can almost see why you think adaptation is a huge > gain, but > every time I start looking closer, I get bogged down in trying to > understand adaptation registration. Do you have a simple way of > explaining how that works *well* and *simply*? Because unless one can > handle registration cleanly, I don't understand how adaptation can be > generally useful for Python. Conversely, if adaptation > registration is > *NOT* required, please explain that in simple terms. Yes, the ability to register adapters is indeed necessary to gain the full benefits of "protocol adaptation". Let's see if I'm up to explaining things a bit more clearly than I've managed in the past, by "abstracting out" just adaptation and registration from the many cognate issues (you can find more real-world approaches in Zope, Twisted or PyProtocols, but perhaps I can better show the underlying idea by sidestepping practical concerns such as performance optimization and effective error-checking). Suppose, just for simplicity and to separate this issue from issues related to "how should interfaces or protocols be formalized", that we choose, as the one and only way to identify each protocol, a unique-string, such as the half-hearted 'com.python.stdlib.index' example I mentioned earlier. It's probably an oversimplification -- it entirely leaves the issue of defining what syntactic, semantic, and pragmatic assurances a protocol makes up to human-readable documentation, and attempts no enforcement of such promises; the reason I posit this hypothesis now is to sidestep the long, interesting and difficult discussion of that issue, and focus on adaptation and registration. Second simplification: let's ignore inheritance. Again, I'm not advocating this as a pratical approach: the natural choice, when looking for an adaptation of type T to protocol P, is instead, no doubt, to walk T's MRO, looking for the adaptation of each of T's bases, and stop when the first one is met, and this has many advantages (but also some issues, partly due to the fact that sometimes inheritance is used more for implementation purposes than for Liskovian purity, etc, etc). In the following, however, I'll just assume that "adaptation is not inherited" -- just as a simplification. Similarly, I will assume "no transitivity": adaptation in the following is assume to go from a type T to a protocol P (the latter identified by a unique string), in a single step which either succeeds or fails, period. The advantages of transitivity, like those of inheritance, are no doubt many, but they can be discussed separately. I do not even consider the possibility of having an inheritance structure among protocols, even though it might be very handy; etc, etc. So, each entity which we can call a "registration of adaptation" (ROA for short) is a tuple ( (T, P), A) where: T is a type; P is a unique string identifying a protocol; A is a callable, such that, for any direct instance t of T (i.e., one such that type(t) is T), A(t) returns an object (often t itself, or a wrapper over t) which claims "I satisfy all the constraints which, together, make up protocol P". A(t) may also raise some exception, in which case the adaptation attempt fails and the exception propagates. For example, A may often be the identity function: def identity(x): return x where a ROA of ((T, P), identity) means "all direct instances of T satisfy protocol P without need for wrappers or whatever". The adaptation registry is a mapping from (T, P) to A, where each ROA is an item -- (T, P) the key, A the value. The natural implementation would of course be a simple dict. Consider for example the hypothetical protocol P 'com.python.stdlib.index'. We could define it to mean: an object o satisfies P iff int(o) is directly usable as index into a sequence, with no loss of information. So, a sequence type's __getitem__ would be...: def __getitem__(self, index): adapted_index = adapt(index, 'com.python.stdlib.index') i = int(adapted_index) # continue indexing using the int 'i', which is asserted to be the correct index equivalent of argument 'index' instead of today's (roughly): def __getitem__(self, index): i = index.__index__() # etc, as above Whoever (python-dev, in this case;-) invents the protocol P, besides using it in methods such as __getitem__, might also pre-register adapters (often, identity) for the existing types which are known to satisfy protocol P. I.e., during startup, it would execute: register_adapter(int, 'com.python.stdlib.index', identity) register_adapter(long, 'com.python.stdlib.index', identity) This is similar to what the inventor would do today (inserting an __index__, at Python level, or tp_index, at C level, into the types) but not "invasive" of the types (which doesn't matter here, since the protocol's inventor also owns the types in question). Once the protocol P is published, the author of a type T which does respect/satisfy P would also execute register_adapter calls (instead of modifying T by adding __index__/tp_index). Say that instances of T do already satisfy the protocol, then identity is the correct adapter: register_adapter(T, 'com.python.stdlib.index', identity) But say that some T1 doesn't quite satisfy the protocol but can easily be adapted to it. For example, instances t of T1 might not support int(t), or support it in a way that's not correct for the protocol, but (again for example) T1 might offer an existing as_index method that would do what's required. Then, a non-identity adapter is needed: class wrapT1(object): def __init__(self, t): self.t = t def __int__(self): return self.t.as_index() register_adapter(T1, 'com.python.stdlib.index', wrapT1) The big deal here is that protocol P and type T1 may have been developed independently and separately, and yet a third-party author can still author and register such a wrapT1 adapter, and as long as that startup code has executed, the application will be able to use T1 instances as indexes into sequences "transparently". The author of the application code need not even be aware of the details: he or she may just choose to import a third-party module "implement_T1_adaptations.py" just as he or she imports the framework supplying T1 and the one(s) using protocol P. This separation of concerns into up to 4 groups of developers (authors, respectively, of: a framework defining/using protocol P; a framework supplying type T1; a framework adapting T1 to P; an application using all of the above) always seems overblown for "toy" examples that are as simple as this one, of course -- and even in the real world in many cases a developer will be wearing more than one of these four hats. But adaptation *allows* the separation of "ownership" concerns by affording *non-invasive* operation. Here is a simple reference implementation of adaptation under all of these simplifying assumptions: _global_registry = {} def register_adapter(T, P, A, registry=_global_registry): registry[T, P] = A def adapt(t, P, registry=_global_registry): return registry[type(t), P] Now, a million enrichments and optimizations can obviously be imagined and discussed: the many simplifying assumptions I've been making to try to isolate adaptation and registration down to its simplest core leave _ample_ space for a lot of that;-). But I hope that instead of immediately focusing on (premature?-) optimizations, and addition of functionality of many kinds, we can focus on the salient points I've been trying to make: a. protocols may be identified quite arbitrarily (e.g. by unique- strings), though specific formalizations are also perfectly possible and no doubt offer many advantages (partial error-checking, less indirectness, ...): there is no strict need to formalize interfaces or protocols in order to add protocol-adaptation to Python b. the possibility of supplying, consuming and adapting-to protocols becomes non-invasive thanks to registration In this simplified outline I've supported *only* registration as the one and only way to obtain adaptation -- conceptually, through the identity function, registration can indeed serve the purpose, although optimizations are quite obviously possible. I should probably also mention what I mean by "protocol": it's a very general term, potentially encompassing interfaces (sets of methods with given signatures -- a "syntactic" level), design-by-contract kinds of constraints (a "semantic" level), and also the fuzzier but important kinds of constraints that linguists call "pragmatic" (for example, the concept of "integer usable as an index into a sequence without significant loss of information" is definitely a pragmatic constraint, not formalizable as semantics). Alex From tim.hochberg at ieee.org Sat Apr 1 20:52:14 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Sat, 01 Apr 2006 11:52:14 -0700 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <442EBB89.4020506@benjiyork.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <442EBB89.4020506@benjiyork.com> Message-ID: Benji York wrote: > Aahz wrote: > >>On Fri, Mar 31, 2006, Alex Martelli wrote: >> >> >>>Once in a while, I'm moved to sing this refrain again, but thanks to >>>the ensuing discussion I'm soon reminded that there are many more >>>gratifying activities I could pursue instead -- repeated beating of >>>my forehead against suitable brick walls spring to mind as being both >>>more productive and more fun, for example. >> >> >>That's because you're a masochist. ;-) >> >>Seriously, I can almost see why you think adaptation is a huge gain, but >>every time I start looking closer, I get bogged down in trying to >>understand adaptation registration. Do you have a simple way of >>explaining how that works *well* and *simply*? Because unless one can >>handle registration cleanly, I don't understand how adaptation can be >>generally useful for Python. Conversely, if adaptation registration is >>*NOT* required, please explain that in simple terms. > > > In Zope 3 adapters are normally registered via a one line ZCML > directive, but that's more for configuration management than anything > else; it makes it easier to swap adapters in and out. You can also use > Python to do the same thing: > > from zope import component > component.provideAdapter(MyAdapter) > > MyAdapter can be any callable that takes one or more objects (the things > being adapted) and provides a single interface. The class would look > like this: > > class MyAdapter: > component.adapts(ISomething) > interface.implements(ISomethignElse) > > def __init__(self, something): > ... > > So in this case the "provideAdapter" queries the adapter to find out > what is being adapted to and from. You can also provide those to the > provideAdapter call if MyAdapter implements more than one interface or > you need to for some other reason. Bah! This is why adapters never get anywhere. My eyes glazed over 10 lines ago. At their core, adapters seem like a simple enough concept, but whenever anyone starts talking about them we get what seems like overengineered examples that make my brain lock up. Someone who knows something about adapters needs to go off and come up with the worlds simplest, most light weight, dumb as a post, adapter proposal. It may not be what you eventually want, but until people can grok the idea in a few simple lines of code it's not going to get that groundswell of support that Alex keeps looking for. Couldn't an adapter protocol be as simple as a module with a dictionary in it and two methods? # adapters.py registry = {} def register(adapter, resultdescr, *targetdescrs): for x in targetdescrs: registry[(resultdescr, x)] = adapter def find(resultdescr, targetdescr): return registry[(resultdescr, target)] The convention being that adapter(target) = result, where target and result satisfy targetdescr and resultdescr in some sense. What these descriptions are doesn't matter much as long as they're hashable and the users can come to some agreement. Here's a trivial example: adapters.register(list, 'sequence', generatortype, dictkeyitertype,...) #... def f(obj): seq = adapters.find('sequence', type(obj))(obj) # do stuff with seq I'm assuming that sticking the above six lines in a file somewhere wouldn't be sufficient for some reason, but if not, why not? And what's the simplest possible proposal that would work? Regards, -tim > > For adapters that are just functions, decorators can be used to indicate > the "to" and "from" interfaces (following the "index" example from earlier): > > @component.adapter(IMpz) > @interface.implementer(IIndex) > def index(mpz): > return long(mpz) > > The zope.component README goes into more detail (especially the > "Adapters" section): > http://svn.zope.org/*checkout*/Zope3/trunk/src/zope/component/README.txt?rev=39671 > -- > From tim.hochberg at ieee.org Sat Apr 1 22:02:09 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Sat, 01 Apr 2006 13:02:09 -0700 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> Message-ID: Alex Martelli wrote: [SNIP a bunch or erudite stuff.] > > > Here is a simple reference implementation of adaptation under all of > these simplifying assumptions: > > _global_registry = {} > > def register_adapter(T, P, A, registry=_global_registry): > registry[T, P] = A > > def adapt(t, P, registry=_global_registry): > return registry[type(t), P] This is more like it. I suspect that last line has a typo: I think it needs a "(t)" appended to it, but that's minor. It's great to see someone who knows something about adaption put forward a reference implementation that's grockable. Remarkably, it looks a lot like my implementation that I posted in parallel with this. Let's compare: > # adapters.py > registry = {} > def register(adapter, resultdescr, *targetdescrs): > for x in targetdescrs: > registry[(resultdescr, x)] = adapter > def find(resultdescr, targetdescr): > return registry[(resultdescr, targetdescr)] # fixed type here There are some superficial differences, but the only substantial difference is that I chose to use 'find' to get an adapter, rather than to actually do the adaption using adapt. This was a conscious choice, driven by my desire to avoid tying targetdescr to concrete types. My proposal is dumber, and thus more flexible. Let me give an example where this would matter based, more or less, on real code. I load pairs of numpy arrays from a file. These pairs represent imaginary numbers but they may be in several different formats: real-imag, mag-angle or db-angle. The format is specified in the file and the arrays gets tagged with this format when they are loaded. Now lets suppose I want to use adaption to handle this. Trying to use concrete types is a mess here. However, in the find proposal, I can simply punt and use strings: def register(z_from_ri, "complex_array", ('real_array', 'imag_array')) def register(z_from_ma, "complex_array", ('mag_array', 'angle_array')) def register(z_from_dba, "complex_array", ('db_array', 'angle_array')) #... def f(a1, a2): adapter = adaption.fine("complex_array, (a1.descr, a2.descr)) z = adapter(a1, a2) It's also very easy to register a new type if I need to do so. Suppose that they add mag, radians. No problem: def register(z_from_mrad, "complex_array", ('db_array', 'radian_array')) I've completely punted on any attempt at avoiding collisions. All of these names would want to be prefixed, or some other method found, to avoid collisions, as you (Alex) already mentioned. That's my take, FWIW, which is probably not much considering I've never tried to use adaption, or at least not that I've noticed, in real life. [SNIP] Regards, -tim From p.f.moore at gmail.com Sat Apr 1 22:04:12 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 1 Apr 2006 21:04:12 +0100 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <442EBB89.4020506@benjiyork.com> Message-ID: <79990c6b0604011204r33a616d7i981d27a2aab4a219@mail.gmail.com> On 4/1/06, Tim Hochberg wrote: > Bah! This is why adapters never get anywhere. My eyes glazed over 10 > lines ago. At their core, adapters seem like a simple enough concept, > but whenever anyone starts talking about them we get what seems like > overengineered examples that make my brain lock up. Absolutely. On the other side of the coin, though, whenever I see frameworks which use adaptation (for example, Twisted, or PEAK), my immediate reaction is "wow, that's a really useful idea - why isn't it used more?". Go figure. > Couldn't an adapter protocol be as simple as a module with a dictionary > in it and two methods? Yes - see Alex's posting. > # adapters.py > registry = {} > def register(adapter, resultdescr, *targetdescrs): > for x in targetdescrs: > registry[(resultdescr, x)] = adapter > def find(resultdescr, targetdescr): > return registry[(resultdescr, target)] You've pretty much rewritten Alex's code here, so I'd say you have understood the core of adapation pretty well. Maybe we need someone to explain, in practical but not overcomplicated terms, just why this isn't enough in itself, and why "real world" adaptation systems seem so much more overengineered. (I'm sure there are good reasons, but they need a bit of explaining!) For example, all the "real life" adaptation systems that I am aware of use interfaces. OK, Alex wants to divorce adaptation from interfaces, because Guido isn't ready to bless an interface system yet, and Alex doesn't want that to hold up adaptation. But maybe someone could clarify: - why all the adaptation systems use interfaces (what problems require them) - how a non-interface based system would address these problems, or how it would leave room to develop in a backward-compatible way to address them. To put it another way, we don't want something too simple in the stdlib, and real-world implementations seem to imply that there's something important lacking in the simple code you (and Alex) have suggested. > I'm assuming that sticking the above six lines in a file somewhere > wouldn't be sufficient for some reason, but if not, why not? And what's > the simplest possible proposal that would work? Exactly. Alex seems to be saying that it would - or if he isn't, he needs to develop his earlier post a bit more... Paul. From aleaxit at gmail.com Sat Apr 1 22:05:10 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Sat, 1 Apr 2006 12:05:10 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <442EBB89.4020506@benjiyork.com> Message-ID: <7D3C71FD-9F69-4077-8952-788D4C429C3C@gmail.com> On Apr 1, 2006, at 10:52 AM, Tim Hochberg wrote: ... > overengineered examples that make my brain lock up. Someone who knows > something about adapters needs to go off and come up with the worlds > simplest, most light weight, dumb as a post, adapter proposal. It may Heh, funny -- that's basically what I tried to do in my post, composed no doubt at the same time you were composing this; and my "proposal" (more of an "explanation", really) shares many aspect with yours, such as the concept of protocol/descriptions being somewhat arbitrary (but hashable) "tags" relying entirely on conventions (human readable descriptions of constraints) without enforcement. > I'm assuming that sticking the above six lines in a file somewhere > wouldn't be sufficient for some reason, but if not, why not? And > what's > the simplest possible proposal that would work? The simplest possible proposal that would do some good would look very similar to yours, and mine, net of minor factoring issues. If adaptation was available in Python's standard library and used reasonably widely within it, that would kickstart its wide adoption. I'm quite ready to discuss the various reasons which make these levels of proposals/explanatioms not really optimal for real-world adoption; I mentioned such reasons repeatedly in my post. For example, we could start with a fundamental inheritance issue: if there's an adapter registered from type T to protocol P, should I really register it again if I subclass T into S -- shouldn't S just inherit the adaptation from T (since that's what's wanted most often, when inheritance is used in a properly Liskovian way to assert that each S is-a T), at least as the default behavior, perhaps with the possibility to override when needed -- by registering a different adapter specifically from S to P, of course -- perhaps a cannot_adapt one, such as: def cannot_adapt(*whocares): raise CannotAdaptError If "inheritance of adaptation" is desired, it can be done in at least two ways: by having adapt loop on the MRO and checking each, or by having the metaclass look for all adapters of bases and registering what it finds. The first strategy might be augmented by caching/ memoizing: if an adapter is found for some type that's in the MRO but not in first place, before returning it, adapt could explicitly register that so that future lookups will be much faster. But then we need to decide what happens to this 'cache' when some different adapter gets registered for a base type... Such need to dwell on practical details is not special to adaptation, of course: if Python didn't have the concept of a mapping and we were trying to develop one, the development from "cool abstract idea with a simple-as-dirt reference implementation" to "real-world precious tool" would be even more torturous;-). We could start with "a list of key/value pairs" (a ``property list'' kind of thing), then have to deal with performance issues, uniqueness, handy auxiliary methods, ...;-). But the usefulness of adaptation doesn't entirely depend on the ways all such issues are brought up and resolved... even a dirt-simple approach would be better than nothing (except perhaps for standing in the way of future improvements along the various axes -- error- checking, performance, handiness, ... -- so, the detailed behavior of such a first-cut, extra-simple approach should probably be carefully left underspecified, if it were to be adopted, to allow for future improvements;-). Alex From walter at livinglogic.de Sat Apr 1 22:25:29 2006 From: walter at livinglogic.de (=?iso-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 1 Apr 2006 22:25:29 +0200 (CEST) Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> Message-ID: <61104.89.54.38.161.1143923129.squirrel@isar.livinglogic.de> Alex Martelli wrote: > [...] > So, each entity which we can call a "registration of adaptation" (ROA for short) is a tuple ( (T, P), A) where: > T is a type; > P is a unique string identifying a protocol; > A is a callable, such that, for any direct instance t of T > (i.e., one such that type(t) is T), A(t) returns an object (often t itself, or a wrapper over t) which claims "I satisfy > all the > constraints which, together, make up protocol P". A(t) may also raise some exception, in which case the adaptation attempt > fails and the exception propagates. > > [...] > Here is a simple reference implementation of adaptation under all of these simplifying assumptions: > > _global_registry = {} > > def register_adapter(T, P, A, registry=_global_registry): > registry[T, P] = A > > def adapt(t, P, registry=_global_registry): > return registry[type(t), P] > > [...] > > a. protocols may be identified quite arbitrarily (e.g. by unique- strings), though specific formalizations are also > perfectly possible and no doubt offer many advantages (partial error-checking, less indirectness, ...): there is no > strict need to formalize interfaces or protocols in order to add protocol-adaptation to Python > > b. the possibility of supplying, consuming and adapting-to protocols becomes non-invasive thanks to registration > > In this simplified outline I've supported *only* registration as the one and only way to obtain adaptation -- conceptually, > through the identity function, registration can indeed serve the purpose, > although optimizations are quite obviously possible. > > [...] I'm beginning to see your point. Suddenly several things in the stdlib look like they use (or could benefit from) some kind of adaption: The pickle protocol uses copy_reg.dispatch_table which adapts an object to the pickle protocol. The copy module uses __copy__ and __deepcopy__ methods to adapt an object to the copy protocol. The pprint module uses simple type checks (isinstance(foo, dict), isinstance(foo, tuple), isinstance(foo, list)) and a fall back to the __repr__() method to adapt an object to some kind of pretty printing protocol. There are probably other ad-hoc adaptions lurking in the stdlib. IMHO there are two possible routes: 1) Identify all cases of adaption in the stdlib, implement a version of those modules that use the "real" adaption and see if the result looks cleaner and more extensible. 2) Implement a self contained adaption module that only does adaption and nothing else and publish it in the cheeseshop. This should be so simple that every other project that wants to use it, but doesn't want to depend on another package can simply incorporate it. (I immediate recognized two spots in my own code, where I would use this adaption module). At the minimum this module should support inheritance of the adapted type, so a simple dict won't work. Defining an adaptor for the copy protocol might look like this: @adaptor(Foo, "org.python.copy") class CopyFoo: def __init__(self, foo): self.foo = foo def copy(self): ... def deepcopy(self, memo=None): ... Would this be a useful application of class decorators? Bye, Walter D?rwald From brett at python.org Sun Apr 2 00:40:00 2006 From: brett at python.org (Brett Cannon) Date: Sat, 1 Apr 2006 14:40:00 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> Message-ID: Woohoo! I get it, finally! Some comments below, but I suddenly feel a little less stupid since I get the whole process now! =) On 4/1/06, Alex Martelli wrote: > > On Apr 1, 2006, at 8:31 AM, Aahz wrote: > ... > > Seriously, I can almost see why you think adaptation is a huge > > gain, but > > every time I start looking closer, I get bogged down in trying to > > understand adaptation registration. Do you have a simple way of > > explaining how that works *well* and *simply*? Because unless one can > > handle registration cleanly, I don't understand how adaptation can be > > generally useful for Python. Conversely, if adaptation > > registration is > > *NOT* required, please explain that in simple terms. > > Yes, the ability to register adapters is indeed necessary to gain the > full benefits of "protocol adaptation". Let's see if I'm up to > explaining things a bit more clearly than I've managed in the past, > by "abstracting out" just adaptation and registration from the many > cognate issues (you can find more real-world approaches in Zope, > Twisted or PyProtocols, but perhaps I can better show the underlying > idea by sidestepping practical concerns such as performance > optimization and effective error-checking). > > Suppose, just for simplicity and to separate this issue from issues > related to "how should interfaces or protocols be formalized", that > we choose, as the one and only way to identify each protocol, a > unique-string, such as the half-hearted 'com.python.stdlib.index' > example I mentioned earlier. It's probably an oversimplification -- > it entirely leaves the issue of defining what syntactic, semantic, > and pragmatic assurances a protocol makes up to human-readable > documentation, and attempts no enforcement of such promises; the > reason I posit this hypothesis now is to sidestep the long, > interesting and difficult discussion of that issue, and focus on > adaptation and registration. > > Second simplification: let's ignore inheritance. Again, I'm not > advocating this as a pratical approach: the natural choice, when > looking for an adaptation of type T to protocol P, is instead, no > doubt, to walk T's MRO, looking for the adaptation of each of T's > bases, and stop when the first one is met, and this has many > advantages (but also some issues, partly due to the fact that > sometimes inheritance is used more for implementation purposes than > for Liskovian purity, etc, etc). In the following, however, I'll > just assume that "adaptation is not inherited" -- just as a > simplification. Similarly, I will assume "no transitivity": > adaptation in the following is assume to go from a type T to a > protocol P (the latter identified by a unique string), in a single > step which either succeeds or fails, period. The advantages of > transitivity, like those of inheritance, are no doubt many, but they > can be discussed separately. I do not even consider the possibility > of having an inheritance structure among protocols, even though it > might be very handy; etc, etc. > > So, each entity which we can call a "registration of adaptation" (ROA > for short) is a tuple ( (T, P), A) where: > T is a type; > P is a unique string identifying a protocol; > A is a callable, such that, for any direct instance t of T > (i.e., one such that type(t) is T), A(t) returns an object (often t > itself, or a wrapper over t) which claims "I satisfy all the > constraints which, together, make up protocol P". A(t) may also > raise some exception, in which case the adaptation attempt fails and > the exception propagates. > > For example, A may often be the identity function: > > def identity(x): > return x > > where a ROA of ((T, P), identity) means "all direct instances of T > satisfy protocol P without need for wrappers or whatever". > > The adaptation registry is a mapping from (T, P) to A, where each ROA > is an item -- (T, P) the key, A the value. The natural > implementation would of course be a simple dict. > > > Consider for example the hypothetical protocol P > 'com.python.stdlib.index'. We could define it to mean: an object o > satisfies P iff int(o) is directly usable as index into a sequence, > with no loss of information. > > So, a sequence type's __getitem__ would be...: > def __getitem__(self, index): > adapted_index = adapt(index, 'com.python.stdlib.index') > i = int(adapted_index) > # continue indexing using the int 'i', which is asserted to > be the correct index equivalent of argument 'index' > > instead of today's (roughly): > > def __getitem__(self, index): > i = index.__index__() > # etc, as above > > Whoever (python-dev, in this case;-) invents the protocol P, besides > using it in methods such as __getitem__, might also pre-register > adapters (often, identity) for the existing types which are known to > satisfy protocol P. I.e., during startup, it would execute: > > register_adapter(int, 'com.python.stdlib.index', identity) > register_adapter(long, 'com.python.stdlib.index', identity) > > This is similar to what the inventor would do today (inserting an > __index__, at Python level, or tp_index, at C level, into the types) > but not "invasive" of the types (which doesn't matter here, since the > protocol's inventor also owns the types in question). > I am going to assume an optimization is possible where if an object meets a protocol it doesn't need it's contract explicitly stated. Is that reasonable? The reason I ask is I could see an explosion of registration calls for objects trying to cover every protocol they match and then new protocols that have been defined doing for the objects, etc., and ending up still with some protocols missed since it seems to require some knowledge of what types will work. Take our __index__ example. I might want to use an object that I think can be used for indexing a sequence but I don't know about any specific protocols required and neither the __index__ protocol creator nor the object designer knew of each other and thus didn't bother with registering. Is it still going to work, or am I going to get an exception saying that the object didn't register for some protocol I wasn't aware of? Also, if defaults are not implied, then a good way to handle registration of classes will need to be developed. This might be another place where class decorators come in handy over metaclasses since if inheritance comes into play then registering every subclass would be overkill. > > Once the protocol P is published, the author of a type T which does > respect/satisfy P would also execute register_adapter calls (instead > of modifying T by adding __index__/tp_index). Say that instances of > T do already satisfy the protocol, then identity is the correct adapter: > > register_adapter(T, 'com.python.stdlib.index', identity) > > But say that some T1 doesn't quite satisfy the protocol but can > easily be adapted to it. For example, instances t of T1 might not > support int(t), or support it in a way that's not correct for the > protocol, but (again for example) T1 might offer an existing as_index > method that would do what's required. Then, a non-identity adapter > is needed: > > class wrapT1(object): > def __init__(self, t): > self.t = t > def __int__(self): > return self.t.as_index() > > register_adapter(T1, 'com.python.stdlib.index', wrapT1) > > > The big deal here is that protocol P and type T1 may have been > developed independently and separately, and yet a third-party author > can still author and register such a wrapT1 adapter, and as long as > that startup code has executed, the application will be able to use > T1 instances as indexes into sequences "transparently". The author > of the application code need not even be aware of the details: he or > she may just choose to import a third-party module > "implement_T1_adaptations.py" just as he or she imports the framework > supplying T1 and the one(s) using protocol P. This separation of > concerns into up to 4 groups of developers (authors, respectively, > of: a framework defining/using protocol P; a framework supplying type > T1; a framework adapting T1 to P; an application using all of the > above) always seems overblown for "toy" examples that are as simple > as this one, of course -- and even in the real world in many cases a > developer will be wearing more than one of these four hats. But > adaptation *allows* the separation of "ownership" concerns by > affording *non-invasive* operation. > > > Here is a simple reference implementation of adaptation under all of > these simplifying assumptions: > > _global_registry = {} > > def register_adapter(T, P, A, registry=_global_registry): > registry[T, P] = A > > def adapt(t, P, registry=_global_registry): > return registry[type(t), P] > > > Now, a million enrichments and optimizations can obviously be > imagined and discussed: the many simplifying assumptions I've been > making to try to isolate adaptation and registration down to its > simplest core leave _ample_ space for a lot of that;-). But I hope > that instead of immediately focusing on (premature?-) optimizations, > and addition of functionality of many kinds, we can focus on the > salient points I've been trying to make: > > a. protocols may be identified quite arbitrarily (e.g. by unique- > strings), though specific formalizations are also perfectly possible > and no doubt offer many advantages (partial error-checking, less > indirectness, ...): there is no strict need to formalize interfaces > or protocols in order to add protocol-adaptation to Python > > b. the possibility of supplying, consuming and adapting-to protocols > becomes non-invasive thanks to registration > > In this simplified outline I've supported *only* registration as the > one and only way to obtain adaptation -- conceptually, through the > identity function, registration can indeed serve the purpose, > although optimizations are quite obviously possible. > > I should probably also mention what I mean by "protocol": it's a very > general term, potentially encompassing interfaces (sets of methods > with given signatures -- a "syntactic" level), design-by-contract > kinds of constraints (a "semantic" level), and also the fuzzier but > important kinds of constraints that linguists call "pragmatic" (for > example, the concept of "integer usable as an index into a sequence > without significant loss of information" is definitely a pragmatic > constraint, not formalizable as semantics). > If we can make the default case for when an object implements a protocol dead-simple (if not automatic) in terms of registering or doing the right thing, then I can see this being really helpful. -Brett From tjreedy at udel.edu Sat Apr 1 23:26:44 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 1 Apr 2006 16:26:44 -0500 Subject: [Python-3000] Iterating over a dict References: Message-ID: "Thomas Lotze" wrote in message news:pan.2006.04.01.12.32.19.172601 at gmx.net... > The reason I ask this on the python-3000 list is that I wonder whether > the > iterating behaviour of dicts might be changed in Python 3k, so that in > the > above code, foo() would be applied to the dict's values. Dicts give you the choice of iterating by keys, values, or items (key,value pairs). Keys was chosen as the default after much discussion because it was most intuitive to more people and was expected to be the most common in usage. I believe the choice has worked well enough for most people, so that there is no reason to gratuitously break existing code. So I strongly suspect the answer is 'no'. Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sun Apr 2 02:56:12 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 02 Apr 2006 12:56:12 +1200 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> Message-ID: <442F212C.9050602@canterbury.ac.nz> Alex Martelli wrote: > The existence of dicts is > a laughably feeble excuse to avoid having adaptation in Python's > standard library, The point is whether it buys you anything important over using a dict. If you use it in such a wide-open way that it's just a mapping from anything to anything, I don't see that it does. Another thing that makes me way of adaptation is that it relies on a global registry. I'm leery about global registries in general. One problem is that, because they're global, you only get one of them per program. I actually think the very existince of copy_reg is wrongheaded, because it assumes that in any given program there will be one correct way to copy any given type of object. On the extremely rare occasions when I want to deep-copy something, I have very specific ideas on how deeply I want to copy it, and that could vary from one situation to another. I wouldn't trust anything found in a global registry to do the right thing. Another problem is that, because they're global, any part of the program can put stuff in them that gets used by any other part, without its explicit knowledge. This can make it hard to tell what any given piece of code is going to do without searching the whole program. -- Greg From greg.ewing at canterbury.ac.nz Sun Apr 2 03:09:24 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 02 Apr 2006 13:09:24 +1200 Subject: [Python-3000] Iterating over a dict In-Reply-To: References: Message-ID: <442F2444.4010107@canterbury.ac.nz> Thomas Lotze wrote: > The reason I ask this on the python-3000 list is that I wonder whether the > iterating behaviour of dicts might be changed in Python 3k, I doubt it. This issue was considered very carefully when support for iteration was added to dicts, and I'm not aware that the reasons for choosing the current behaviour have changed. -- Greg From talin at acm.org Sun Apr 2 02:52:11 2006 From: talin at acm.org (Talin) Date: Sun, 2 Apr 2006 00:52:11 +0000 (UTC) Subject: [Python-3000] It's a statement! It's a function! It's BOTH! Message-ID: This is about the print / writeln debate. Let me say up front that I don't expect this posting in its original form to be widely accepted (In fact, I'll be disappointed if I don't get at least a sum total of -5000 on responses to this.) At the same time, however, I think that the issue that I am raising is legit, even if my solution is not. Part of my issue is that I like both solutions. That is, the "professional programmer" part of me likes the stream.writeln, as seen in many other languages, in particular Java and C#. On the other hand, the "recreational doodler" part of me remembers with fondness the simplicity and ease of learning of my first experiences in BASIC, where it was completely intuitive and natural to just say "print x" and not have to understand about the complexities of line endings, function argument lists, and so on. And my notion of "pythonic" includes the eschewing of needless delimiters and other non-word characters when possible. For the most part, the print statement isn't that much different from a function, except that it doesn't have parentheses. (I'll discuss the exceptions to this later.) That is, it takes an argument list which is separated by commas, and each of those arguments is treated pretty much the same. However, Python doesn't recognize the general concept of a paren-less function call, so that's why print has to be a special case, that's built into the parser. I've read a number of posts which all sort of lead up to the idea - why can't we allow the interpreter to recognized paren-less functions? Now obviously we wouldn't want every function to act this way, as it would lead to a nightmare of ambiguities. So you'd have to have some means of telling the parsers which functions were to be parsed as "statements" and which should not. The difficult with that notion is that now you are asking the Python parser to do something that it never did before, which is to make parsing decisions based on semantics rather than just syntax. While there's no technical difficulty with this, it goes against the Python tradition in a fairly fundamental way. But what the heck, lets forget tradition for a moment and see what would happen if we were to go ahead and do it anyway. We would start by defining a simple metalanguage for giving instructions to the parser. I won't even try to suggest a syntax, but in essence this would be something that has the same role as a macro language or preprocessor, in that it is not part of the programming language itself, but instead describes how the subsequent text is to be interpreted. (There's already some precedent for this with the __future__ syntax.) Specifically, one of the commands of this metalanguage would be a command to parse an expression beginning with a specific keyword (such as "print") as a statement rather than as an expression. A comma-separated list of arguments would follow the initial identifier, and these would be treated as result function arguments. Thus you could have: print a, b, c send a, b, c read a, b, c ...and so on. Yes, its true - there is vast potential for abuse here, I don't deny it. (With great power...etc.) Now, about those exceptions: One that's fairly easy to handle is the "print >> stream" syntax. We can tweak the syntax for the "function as statement" so that instead of the rule being: identifier [ arg, ... ] it can be: expression [ arg, ... ] Thus, the parser sees the word "print" which tells it that we're going to have a paren-less function, but it still parses the text "print >> stream" as an expression. Then its simply a matter of overloading the ">>" operator to return a closure function that prints to the given stream. The semantics are equivalent to: (print >> stream)( arg, arg, arg ) OK, so suppose you find this just too wide open for abuse. An alternative is to dump the ">>" syntax and use a keyword argument: print stream=fh, arg, arg, arg Since print is being executed as a normal function call, except without parens, we would expect the normal keyword argument syntax to work, as well as *args and **args. (Note that in this case the keyword argument is coming before the non-keyword arguments, which is something that I hope will be addressed in Python 3000.) Now, what about the semantics of the "spaces between args"? There's a couple of ideas: 1) Define "print" as putting spaces between args, and "write" as not doing so. 2) Have a keyword arg that allows specifying a separator char, where space is the default: # No space between args print sep="", arg, arg, arg Heck, why not even make it a function: # Insert enough spaces between args to align to 8-char boundaries print sep=tabToNext( 8 ), arg, arg, arg Finally, there is the issue of the trailing comma to suppress the final newline. I must confess that I don't have a clever solution in this case (I can think of lots of hacky solutions...) I suspect that the best compromise is to have distinct "print" and "println" functions to cover this case. -- Talin From aleaxit at gmail.com Sun Apr 2 04:45:11 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Sat, 1 Apr 2006 18:45:11 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> Message-ID: On Apr 1, 2006, at 2:40 PM, Brett Cannon wrote: > Woohoo! I get it, finally! Some comments below, but I suddenly feel > a little less stupid since I get the whole process now! =) Well, I guess this may be the first time I've presented the thing properly, after all these many years -- in which case, the blame definitely falls mostly on me ("mostly" only because SOME people got it despite my inability to present it properly;-)... > I am going to assume an optimization is possible where if an object > meets a protocol it doesn't need it's contract explicitly stated. Is OTOH, the blame for spellng "its" as "it's" -- assuming that's what you're doing -- IS all on YOUR shoulders, Brett!!! > that reasonable? The reason I ask is I could see an explosion of > registration calls for objects trying to cover every protocol they > match and then new protocols that have been defined doing for the > objects, etc., and ending up still with some protocols missed since it > seems to require some knowledge of what types will work. Many optimizations are definitely possible, but I'm not sure which ones are worthwhile. > Take our __index__ example. I might want to use an object that I > think can be used for indexing a sequence but I don't know about any > specific protocols required and neither the __index__ protocol creator > nor the object designer knew of each other and thus didn't bother with > registering. Is it still going to work, or am I going to get an > exception saying that the object didn't register for some protocol I > wasn't aware of? Well, *SOME*body must be aware of the type/protocol compliance: it could be the protocol's author, the type's author, the application's author, or a 4th party whose adaptations-module the application imports -- but it cannot just happen "like black magic". Any one of the four parties may perform the registration (or whatever other action, equivalent e.g. to registering the identify function but faster, gets picked as an optimization) -- but defaulting to "everything satisfies every protocol" is not a real option. Consider indexing: we want someseq[x] to fail (with a TypeError or thereabouts) if x is an instance of float, decimal.Decimal, gmpy.mpf, gmpy.mpq, str, unicode, ... -- a huge host of types that DO supply an __int__ (so that int(x) would work), but whose __int__ has "significant loss of information" and thus does NOT meet the pragmatics of the protocol "being suitable as a sequence index". A Python beginner might easily think that a float "can be used for indexing a sequence" (with an automatic truncation to int), or even that a string can (with an implicit str->int translation) -- but such a beginner would be dead wrong. If you defaulted adaptation to "everything satisfies every protocol unless otherwise stated", you'd end up with a LOT of things which "happen to work"... but in fact can silently fail, e.g. because a float used to index a sequence is sometimes computed as 6.9999, truncated to 6, rather than rounded to 7 as the poor beginner expected. This being Python, *NOT* Perl, I most definitely think we do NOT want to go there. If you think that, most often, types will be written in full knowledge of many existing protocols that they want to support, one optimization might be to explicitly mark the types with an optional __protocols__ attribute which is a set of directly supported protocols. Istinctively I'm not enthusiastic about this optimization -- I'm not sure it would buy you all that much. Rather, we might want to supply an (e.g.) functools.identity built-in function, and specialcase it when supplied as "the adapter" in registration (such a functools.identity would have other uses too -- sure, 'lambda x: x' is easy to write, but having a singleton identity-function might allow other optimizations even quite apart from adaptation). Semantically, registration suffices -- how best to optimize frequent cases, I'm not certain. > Also, if defaults are not implied, then a good way to handle > registration of classes will need to be developed. This might be > another place where class decorators come in handy over metaclasses > since if inheritance comes into play then registering every subclass > would be overkill. One way to implement __protocols__ would be to have the metaclass deal with it, of course -- with or without inheritance (I do believe that it would be handier if inheritance of adaptation was in force by default, but that's debatable, of course). But perhaps class decorators are even nicer - I'm on the fence on the subject of class decorators in general, particularly given the issue of whether they should come before the class statement (like function decorators do) or inside it (with @class or whatever) - I can see excellent arguments on both sides. > If we can make the default case for when an object implements a > protocol dead-simple (if not automatic) in terms of registering or > doing the right thing, then I can see this being really helpful. Automatic is way too much, because you'd then have to state all the protocols an object does NOT satisfy, which would be many, MANY more than those it does meet. But surely, even without class decorators, using: class blah(bloh): ... body of class omitted ... satisfies_protocols(blah, 'zip', 'zap', 'zop', 'balup') isn't TOO complicated -- with from functools import identity def satisties_protocols(klass, *protocols): for protocol in protocols: register_adapter(klass, protocol, identity) return klass # currently innocuous, support future decorator- like use;-) Similarly, if we supported protocol-satisfaction-inheritance by default, a simple cannot_satisfy_protocols function could be used to REMOVE from blah any protocols that are supported by bloh but blah itself cannot support, maybe with a function cannot_support to be registered as the adapter. I would suggest not focusing too exclusively on these black-and-white cases, where a protocol is supported "as is" or is definitely unsupported. Adaptation is at its best where protocols are ALMOST supported, and just need some little tweaking/wrapping to become fully supported. For example, renaming methods or prebinding some arguments, etc. These cases are unlikely to matter when somebody is coding a new type intended to support existing protocols X, Y and Z (it's then simplest for them to just code the new type according to those protocols' constraints!) -- but they're going to be common where adaptation shines.... for removal of impedance mismatches among separately developed types and protocols. With Python's power, it's child's play to make support for such cases into a simple callable, too (left as an exercise for the reader, since pasta's just been dropped and dinner's impending;-). Alex From brett at python.org Sun Apr 2 06:23:37 2006 From: brett at python.org (Brett Cannon) Date: Sat, 1 Apr 2006 20:23:37 -0800 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> Message-ID: On 4/1/06, Alex Martelli wrote: > > On Apr 1, 2006, at 2:40 PM, Brett Cannon wrote: > > > Woohoo! I get it, finally! Some comments below, but I suddenly feel > > a little less stupid since I get the whole process now! =) > > Well, I guess this may be the first time I've presented the thing > properly, after all these many years -- in which case, the blame > definitely falls mostly on me ("mostly" only because SOME people got > it despite my inability to present it properly;-)... > > > I am going to assume an optimization is possible where if an object > > meets a protocol it doesn't need it's contract explicitly stated. Is > > OTOH, the blame for spellng "its" as "it's" -- assuming that's what > you're doing -- IS all on YOUR shoulders, Brett!!! > =) I blame my homework for my grammatical slip. > > that reasonable? The reason I ask is I could see an explosion of > > registration calls for objects trying to cover every protocol they > > match and then new protocols that have been defined doing for the > > objects, etc., and ending up still with some protocols missed since it > > seems to require some knowledge of what types will work. > > Many optimizations are definitely possible, but I'm not sure which > ones are worthwhile. > > > Take our __index__ example. I might want to use an object that I > > think can be used for indexing a sequence but I don't know about any > > specific protocols required and neither the __index__ protocol creator > > nor the object designer knew of each other and thus didn't bother with > > registering. Is it still going to work, or am I going to get an > > exception saying that the object didn't register for some protocol I > > wasn't aware of? > > Well, *SOME*body must be aware of the type/protocol compliance: it > could be the protocol's author, the type's author, the application's > author, or a 4th party whose adaptations-module the application > imports -- but it cannot just happen "like black magic". Any one of > the four parties may perform the registration (or whatever other > action, equivalent e.g. to registering the identify function but > faster, gets picked as an optimization) -- but defaulting to > "everything satisfies every protocol" is not a real option. > I am not suggesting the default be that everything satisfies a protocol. I am thinking about situations like our __index__ situation; will someone have to explicitly somewhere say that a type meets the index protocol even if it does implement the __index__ method? > Consider indexing: we want someseq[x] to fail (with a TypeError or > thereabouts) if x is an instance of float, decimal.Decimal, gmpy.mpf, > gmpy.mpq, str, unicode, ... -- a huge host of types that DO supply an > __int__ (so that int(x) would work), but whose __int__ has > "significant loss of information" and thus does NOT meet the > pragmatics of the protocol "being suitable as a sequence index". > > A Python beginner might easily think that a float "can be used for > indexing a sequence" (with an automatic truncation to int), or even > that a string can (with an implicit str->int translation) -- but such > a beginner would be dead wrong. If you defaulted adaptation to > "everything satisfies every protocol unless otherwise stated", you'd > end up with a LOT of things which "happen to work"... but in fact can > silently fail, e.g. because a float used to index a sequence is > sometimes computed as 6.9999, truncated to 6, rather than rounded to > 7 as the poor beginner expected. This being Python, *NOT* Perl, I > most definitely think we do NOT want to go there. > > If you think that, most often, types will be written in full > knowledge of many existing protocols that they want to support, one > optimization might be to explicitly mark the types with an optional > __protocols__ attribute which is a set of directly supported > protocols. Istinctively I'm not enthusiastic about this optimization > -- I'm not sure it would buy you all that much. Rather, we might > want to supply an (e.g.) functools.identity built-in function, and > specialcase it when supplied as "the adapter" in registration (such a > functools.identity would have other uses too -- sure, 'lambda x: x' > is easy to write, but having a singleton identity-function might > allow other optimizations even quite apart from adaptation). > > Semantically, registration suffices -- how best to optimize frequent > cases, I'm not certain. > Yeah, that is mostly what I am wondering about. How to make the registration process simple or unneeded in cases where the protocol is based on an interface and an object implements the interface but has not bothered to registered its compliance with the protocol. > > > Also, if defaults are not implied, then a good way to handle > > registration of classes will need to be developed. This might be > > another place where class decorators come in handy over metaclasses > > since if inheritance comes into play then registering every subclass > > would be overkill. > > One way to implement __protocols__ would be to have the metaclass > deal with it, of course -- with or without inheritance (I do believe > that it would be handier if inheritance of adaptation was in force by > default, but that's debatable, of course). But perhaps class > decorators are even nicer - I'm on the fence on the subject of class > decorators in general, particularly given the issue of whether they > should come before the class statement (like function decorators do) > or inside it (with @class or whatever) - I can see excellent > arguments on both sides. > > > > If we can make the default case for when an object implements a > > protocol dead-simple (if not automatic) in terms of registering or > > doing the right thing, then I can see this being really helpful. > > Automatic is way too much, because you'd then have to state all the > protocols an object does NOT satisfy, which would be many, MANY more > than those it does meet. But surely, even without class decorators, > using: > > class blah(bloh): > ... body of class omitted ... > > satisfies_protocols(blah, 'zip', 'zap', 'zop', 'balup') > > isn't TOO complicated -- with > > from functools import identity > def satisties_protocols(klass, *protocols): > for protocol in protocols: > register_adapter(klass, protocol, identity) > return klass # currently innocuous, support future decorator- > like use;-) > > > Similarly, if we supported protocol-satisfaction-inheritance by > default, a simple cannot_satisfy_protocols function could be used to > REMOVE from blah any protocols that are supported by bloh but blah > itself cannot support, maybe with a function cannot_support to be > registered as the adapter. > > I would suggest not focusing too exclusively on these black-and-white > cases, where a protocol is supported "as is" or is definitely > unsupported. Adaptation is at its best where protocols are ALMOST > supported, and just need some little tweaking/wrapping to become > fully supported. For example, renaming methods or prebinding some > arguments, etc. These cases are unlikely to matter when somebody is > coding a new type intended to support existing protocols X, Y and Z > (it's then simplest for them to just code the new type according to > those protocols' constraints!) -- but they're going to be common > where adaptation shines.... for removal of impedance mismatches among > separately developed types and protocols. With Python's power, it's > child's play to make support for such cases into a simple callable, > too (left as an exercise for the reader, since pasta's just been > dropped and dinner's impending;-). > Right, the grey area cases will be adaption shows its usefulness, but the black-and-white cases are what I am used to and thus what I am going to think about while evaluating this stuff initially. And I suspect I am not the only one. Basically, as long as I don't have to put much effort into making this work for the common case where I just inherently implement a protocol (such as iterators) then I am definitely interested in this. -Brett From ncoghlan at gmail.com Sun Apr 2 06:29:27 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 02 Apr 2006 14:29:27 +1000 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <442F212C.9050602@canterbury.ac.nz> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> Message-ID: <442F5327.1040204@gmail.com> Greg Ewing wrote: > Alex Martelli wrote: > >> The existence of dicts is >> a laughably feeble excuse to avoid having adaptation in Python's >> standard library, > > The point is whether it buys you anything important > over using a dict. If you use it in such a wide-open > way that it's just a mapping from anything to anything, > I don't see that it does. > > Another thing that makes me way of adaptation is that > it relies on a global registry. I'm leery about global > registries in general. > > One problem is that, because they're global, you only > get one of them per program. I actually think the very > existince of copy_reg is wrongheaded, because it assumes > that in any given program there will be one correct way > to copy any given type of object. On the extremely > rare occasions when I want to deep-copy something, I > have very specific ideas on how deeply I want to copy > it, and that could vary from one situation to another. > I wouldn't trust anything found in a global registry > to do the right thing. > > Another problem is that, because they're global, > any part of the program can put stuff in them that > gets used by any other part, without its explicit > knowledge. This can make it hard to tell what any > given piece of code is going to do without searching > the whole program. This is exactly what bothers me about the concept. Adaptation strikes me as very easily becoming an attractive nuisance, analogous to implicit type conversions in C++ and VB6. This is where I've always come unstuck in thinking about adaptation - actually using C++ and VB6 has persuaded me that implicit type conversions are generally evil, and there doesn't seem to be anything in adaptation that makes it the exception. OTOH, there may be a hidden assumption among the fans of adaptation that adaptation to a mutable interface should never add state to, nor copy the state of, an adapted object. Any mutation made via an adaptor would be reflected as a mutation of the original object. Adaptation to immutable interfaces would always be fine, naturally. If that's an unwritten rule of adaptation, then: 1. It addresses the main evil of implicit type conversion (hidden state) 2. It needs to become a *written* rule, so that anyone writing a stateful adapter can be duly admonished by their peers I seem to recall PJE making a point along those lines during the last big PEP 263 discussion. . . The other thing is that it makes more sense to me for there to be a per-protocol type->adapter registry, rather than a global registry with tuples of source type/target protocol pairs. Secondly, given that each framework is likely to be defining the protocols that it consumes, I don't see the problem with each one defining its *own* adaptation registry, rather than having one mega-registry that adapts everything to everything. In its own registry, a library/framework would preregister: 1. Its own types that can be adapted to the defined protocols 2. Types from any libraries/frameworks it uses that can be adapted A user of the framework would then: 1. Register any external types that can be adapted 2. Use the framework's registry when consuming an interface defined there Concerns about naming conflicts go away, because that is resolved by having each framework store its protocols in different registries (i.e. the Python module namespace is used to disambiguate protocol names). Whether protocols are identified via string names, interface classes, or what have you is then also a per-framework decision. Decisions about default behaviour, transitivity, and so on and so forth are also up to the framework. Then the role of an adaptation module in the standard library would be to provide a standard API for per-framework registries, without also providing a mega-registry for adapting everything to everything. For example: # ---- adapt.py ---- # sans error handling. . . class AdapterRegistry(object): def __init__(self, protocol): self.adapters = {} self.protocol = protocol def register_adapter(self, source_type, adapter): self.adapters[source_type] = adapter def find_adapter(self, source_type): return self.adapters[source_type] def adapt(self, obj): return self.adapters[type(obj)](obj) class ProtocolRegistry(object): def __init__(self, *protocols): self._protocols = {} for protocol in protocols: self._protocols[protocol] = AdapterRegistry(protocol) def register_adapter(self, source_type, target_protocol, adapter): self._protocols[target_protocol].register_adapter(source_type, adapter) def find_adapter(self, source_type, target_protocol): return self._protocols[target_protocol].find_adapter(source_type) def adapt(self, obj): return self._protocols[target_protocol].adapt(obj) # Used like: copy_registry = ProtocolRegistry("copy", "deepcopy") copy_adapters = copy_registry["copy"] copy_adapters.register_adapter(int, functools.ident) copy_adapters.register_adapter(float, functools.ident) etc. . . deepcopy_adapters = copy_registry["deepcopy"] copy_adapters.register_adapter(int, functools.ident) copy_adapters.register_adapter(float, functools.ident) etc. . . Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From p.f.moore at gmail.com Sun Apr 2 15:44:40 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 2 Apr 2006 14:44:40 +0100 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <20060401163102.GA8292@panix.com> <2A082C0E-F0DE-405C-87CC-68B930415245@gmail.com> Message-ID: <79990c6b0604020644l1f6d665en6b821aff558366a4@mail.gmail.com> On 4/2/06, Brett Cannon wrote: > I am not suggesting the default be that everything satisfies a > protocol. I am thinking about situations like our __index__ > situation; will someone have to explicitly somewhere say that a type > meets the index protocol even if it does implement the __index__ > method? I guess that should be possible, in some way. You'd need to: 1. Register the fact that the "org.python.index" protocol is equivalent to "has a __index__ method" 2. Make the adapt() function consider (1) above, as well as the usual registry. You might need to consider issues like types that implement an inappropriate method called __index__, but I guess that's not new - you have that with the current __index__ protocol (but you wouldn't have it with an explicit registration scheme, so maybe it's relevant anyway). So yes, it's possible, but it may not be a gain in practice. I suppose this comes into the category of "potential optimisations" Alex talks about. "Real life" implementations of adaptation certainly include convenience stuff like this - PyProtocols includes a number of ways of saying that a class provides an interface, or an interface is directly supported by certain classes. You could probably make PyProtocols understand that "having a method called __index__" implies "supports IIndex" (although it may take some black magic). This is all in the "real life baggage" area that makes adaptation seem harder than Alex's basic example. Is it useful? Maybe. Useful enough to add complexity to the adaptation spec? Again, maybe. How much complexity? Enough to make the thing incomprehensible again? Who knows...? > Basically, as long as I don't have to put much effort into making this > work for the common case where I just inherently implement a protocol > (such as iterators) then I am definitely interested in this. Does adding something like "implements(IIterator)" to your class count as "not much effort"? That's pretty normal for most adaptation implementations I've seen... Paul. From p.f.moore at gmail.com Sun Apr 2 16:11:19 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 2 Apr 2006 15:11:19 +0100 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <442F5327.1040204@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> Message-ID: <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> On 4/2/06, Nick Coghlan wrote: > This is where I've always come unstuck in thinking about adaptation - actually > using C++ and VB6 has persuaded me that implicit type conversions are > generally evil, and there doesn't seem to be anything in adaptation that makes > it the exception. On the face of it, there's nothing implicit going on. There has to be an explicit adaptation call. However, I agree that systems that make extensive use of adaptation can seem to end up in a situation where data seems to magically appear without anybody providing it. Black magic of that level seems to me to be a clear abuse, though. The problem may be that there are no "good" uses of adaptation to compare against, just because it's easy to hand-code your own alternative in straightforward cases, so by the time you're ready to add a dependency on someone's external adaptation library, you're already part way to the dark side... If, for example, basic ideas like copy_reg, __index__ and __iter__ were framed in terms of adaptation, people would maybe be less likely to see it as deep magic, where it becomes a pervasive theme of your framework, and triggers the "everything is a nail" reaction... > OTOH, there may be a hidden assumption among the fans of adaptation that > adaptation to a mutable interface should never add state to, nor copy the > state of, an adapted object. Any mutation made via an adaptor would be > reflected as a mutation of the original object. Adaptation to immutable > interfaces would always be fine, naturally. If that's an unwritten rule of > adaptation, then: > 1. It addresses the main evil of implicit type conversion (hidden state) > 2. It needs to become a *written* rule, so that anyone writing a stateful > adapter can be duly admonished by their peers I don't know if that's an "unwritten rule" as such - but I can barely imagine what you're describing as unacceptable (adaptation to a mutable interface which adds or copies state). It just seems like a stupid thing to do (or at least, not at all what adaptation is about). But maybe that's what you mean by a "hidden assumption". Regardless, I'd have no problem with a style guide, or good practice document, stating that this is what adaptation is about, and stateful adapters are bad practice. (That's just my opinion - better check this with people who make heavy use of adaptation). But to me it feels like labouring the obvious - along the lines of explicitly prohibiting metaclass or decorator abuse. > The other thing is that it makes more sense to me for there to be a > per-protocol type->adapter registry, rather than a global registry with tuples > of source type/target protocol pairs. What difference would that make in practice? > Secondly, given that each framework is likely to be defining the protocols > that it consumes, I don't see the problem with each one defining its *own* > adaptation registry, rather than having one mega-registry that adapts > everything to everything. [...] > Then the role of an adaptation module in the standard library would be to > provide a standard API for per-framework registries, without also providing a > mega-registry for adapting everything to everything. Not an unreasonable idea, but how valuable would it be in practice? Alex's proposal allowed for explicitly specifying a registry, while still having a default "central" registry. For 99% of use, I'd suspect that people would not bother with a special registry. And if protocols were defined via some "interface" approach (like zope.interfaces and PyProtocols do) then encapsulation is taken care of by uniqueness of types/interfaces. I know interfaces are outside the scope of what's being proposed right now, but one of their benefits is that they *do* solve this problem. Structured strings naming protocols ("org.python.std.index" or whatever) do this as well, but without language support. Paul. From guido at python.org Sun Apr 2 18:29:13 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 2 Apr 2006 09:29:13 -0700 Subject: [Python-3000] Iterating over a dict In-Reply-To: References: Message-ID: On 4/1/06, Thomas Lotze wrote: > Hi, > > I wonder what's the reason for iterating over a dict by keys: > > >>> for x in {1:"a", 2:"b"}: > ... print x > ... > 1 > 2 > > I find it much more intuitive for the values, "a" and "b", to be accessed. > This is particularly confusing as iterating over tuples, lists and sets in > the same way does access the values. (It feels like iterating over a list > l actually iterates over the index values, range(len(l)).) In fact, > iterating over any container should access the contained values. > > The reason I ask this on the python-3000 list is that I wonder whether the > iterating behaviour of dicts might be changed in Python 3k, so that in the > above code, foo() would be applied to the dict's values. Just to confirm what's already been said, this was considered very carefully and won't change. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aleaxit at gmail.com Sun Apr 2 18:54:30 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Sun, 2 Apr 2006 09:54:30 -0700 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> Message-ID: <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> On Apr 2, 2006, at 7:11 AM, Paul Moore wrote: ... > On the face of it, there's nothing implicit going on. There has to be > an explicit adaptation call. However, I agree that systems that make > extensive use of adaptation can seem to end up in a situation where > data seems to magically appear without anybody providing it. Black > magic of that level seems to me to be a clear abuse, though. The I believe that such "magically appearing" does not depend on adaptation, per se, but on the mix of "convenience" approaches to adaptation and registration that one chooses to provide alongside it. With the simplistic scheme I illustrated (protocols denoted by unique strings, adaptation occurring only from an object's exact [leafmost] type to a specific protocol depending on registration, not even support for inheritance of types, much less of protocols, no transitivity whatsoever, ...) there is exactly zero risk of anything "magically appearing". Of course, the flip side of that is the inconvenience of having to register everything. I do believe that inconvenience can be reduced by supporting mechanisms that Python programmers are already very familiar to, such as inheritance of types: nobody finds anything strange if, after declaring 'class X(Y):...', they find that instances of X "magically acquire" attributes supply by Y -- after all, that IS roughly the whole POINT of doing inheritance;-). I believe that, similarly, nothing will be found surprising if, along with attributes, X's instances inherit the adaptations enjoyed by Y (of course, inherited adaptations can be overridden, just like inherited attributes can). But supplying something like the "duck adaptation" that Brett is so keen on might easily destroy this kind of transparency: essentially, that would be saying something like """forget the twaddle about syntax, semantics and pragmatics: we won't even check half of SYNTAX compatibility [[the fact that methods need to be signature- compatible]], just the possibly-accidental coincidence of method NAMES, and we'll have a hearty laugh at the expense of anybody foolish enough to think that "T conforms to protocol P" should mean "I have studied the documentation about all the constraints of P, and the implementation of T, and I certify that the latter meets the former with no need for any wrapping"...quack, quack!""". After all, we can all plainly see that "class GraphicArtist" supplies a method call "draw", and since that's what protocol 'wild.west.gunslinger' syntactically requires (net of signature- compatibility), we can obviously claim that all graphic artists are gunslingers, quack quack. Now if THAT was part of one's adaptation/registration machinery, I can well see it as problematic enough for the poor artist who finds himself cast in a duel at the OK Corrall. To a lesser extent, "inheritance of protocols" might sometimes cause surprises (though Philip Eby has extensive experience with it and claims otherwise, I do not know how much of his complete lack of surprises and confusion in the matter might depend on his personal skills and not be applicable to many other programmers). That would be a feature like: a protocol P1 might be declared as "inheriting from" (or "adaptable to") another protocol P2. Then, when type T is adaptable to P1, and we're looking for an adaptation of T to P2, we'd walk a path T -> P1 -> P2 -- possibly with two wrappers in play (unless protocol inheritance does not support any true wrapping, just identity; in the case the one and only wrapper, is any, is the one for the T -> P1 adaptation). This seems attractive, but at least potentially it does mean there might be multiple ways to get from T to P2, and it's possible that not all ways would be obvious to an observer. This also applies to inheritance of types, but in that case we're all perfectly used to Python solving such "ambiguities" by walking T's MRO, so I contend there would arise no surprise nor ambiguity; for adaptation of protocol to protocol, we don't have a similar intuition based on solid experience to see us through. >> OTOH, there may be a hidden assumption among the fans of >> adaptation that >> adaptation to a mutable interface should never add state to, nor >> copy the >> state of, an adapted object. Any mutation made via an adaptor >> would be >> reflected as a mutation of the original object. Adaptation to >> immutable >> interfaces would always be fine, naturally. If that's an unwritten >> rule of >> adaptation, then: >> 1. It addresses the main evil of implicit type conversion >> (hidden state) >> 2. It needs to become a *written* rule, so that anyone writing >> a stateful >> adapter can be duly admonished by their peers > > I don't know if that's an "unwritten rule" as such - but I can barely > imagine what you're describing as unacceptable (adaptation to a > mutable interface which adds or copies state). It just seems like a > stupid thing to do (or at least, not at all what adaptation is about). > But maybe that's what you mean by a "hidden assumption". Uh? Consider iteration -- that's a prime example of an adaptation which adds "hidden state", and I don't see it as particularly problematic to frame as an adaptation. When I adapt an instance L of list to 'org.python.stdlib.iterator', I add one bit of state -- the "current index". I mutate that state each and every time I call .next() on the resulting object, and that state is not at all reflected on the original list, which in fact is totally unaware of whether there are iterators outstanding on it, and if so, how many, and in which states. Why is this seen as a problem? > Regardless, I'd have no problem with a style guide, or good practice > document, stating that this is what adaptation is about, and stateful > adapters are bad practice. (That's just my opinion - better check this > with people who make heavy use of adaptation). But to me it feels like > labouring the obvious - along the lines of explicitly prohibiting > metaclass or decorator abuse. Do we have a "good practice document" about what you should or shouldn't do with metaclasses, or decorators, or, for that matter, with inheritance, operator overloading, and other powerful constructs and tools that have been with Python a long, long time? >> The other thing is that it makes more sense to me for there to be a >> per-protocol type->adapter registry, rather than a global registry >> with tuples >> of source type/target protocol pairs. > > What difference would that make in practice? Not all that much: I see this as a mere implementation detail. You can implement a mapping from (A,B) to C in two ways (and, no doubt, many others yet): -- the most direct one: a dict with (A,B) as the key, C as the value -- an indirect one: a dict with A as the key, whose value is a dict with B as the key and C as the value If all you do is lookups from (A,B) to get C, the former is simpler; the latter may be faster when you need to look up all B's corresponding to an A, since it makes that operation O(1) rather than O(N). So, if you add to the adapt and register primitives other primitives for "give me all protocols to which T can be adapted", or "give me all types which can adapted to P", choosing the "right" nested-dict implementation might make one of these lookups (not both) faster. Probably worth doing only if such lookup are indeed frequent; since I believe their frequency of use will be tiny (as will the number of calls to registration) compared with the number of calls to adapt, I'd go all out to optimize the latter, first and foremost -- after which, one can pick tradeoffs between cost of registration and costs of other kinds of lookups (after all, one might also want to see "all (A,B)s for a given C", no?-). For example, keep the "most direct one" dict, and add auxiliary ones to support other lookups -- this makes registration slower (but it's a very rare operation anyway) and takes up a bit more memory (but, I believe we're talking pennies), but can speed up all kinds of lookups. Exactly why we're so prematurely discussing fine-tuning-level optimization concerns, at this stage, escapes me a bit, though. >> Secondly, given that each framework is likely to be defining the >> protocols >> that it consumes, I don't see the problem with each one defining >> its *own* >> adaptation registry, rather than having one mega-registry that adapts >> everything to everything. > [...] >> Then the role of an adaptation module in the standard library >> would be to >> provide a standard API for per-framework registries, without also >> providing a >> mega-registry for adapting everything to everything. > > Not an unreasonable idea, but how valuable would it be in practice? > Alex's proposal allowed for explicitly specifying a registry, while > still having a default "central" registry. For 99% of use, I'd suspect > that people would not bother with a special registry. And if protocols I don't think this other tweak would be a _big_ "bother", but neither would it be at all useful, just a medium-level useless bother. Say, for example, that protocols are identified (as in my strawman proposal) by unique strings anyway. E.g., if I were to invent a protocol, I could name it 'it.aleax.myprot' -- since I own the aleax.it domain, nobody else could create a name conflict. Saying that each framework has a separate registry is just the same as saying that each protocol "lives" in one specific registry, so that any registration or lookup regarding protocol P *must* also specify registryof(P). Hopefully, rather than having to keep this correspondence in our human memory, we're allowed to have a registry of registries which remembers this correspondence for you: we can register_protocol(P, registry) and we can lookup the registry for a given protocol with function registryof. E.g.: _reg_of_regs = {} def register_protocol(P, registry): _reg_of_regs[P] = registry def registryof(P): return _reg_of_regs[P] So now, all calls which, in my proposal, would be (e.g.) adapt(x, P), must instead become adapt(x, P, registryof(P)). Not a big bother, just an amount of totally useless boilerplate that's just sufficient to be annoying, it seems to me. Of course, if the registry of registries was somehow forbidden, then the bother WOULD become bigger, since in that case the poor programmer would have to mentally memorize, or continually look up (with grep, Google search, or similar means) the total equivalent of the registryof(P) function result. I may be missing something here, I guess, because I just don't see the point. > were defined via some "interface" approach (like zope.interfaces and > PyProtocols do) then encapsulation is taken care of by uniqueness of > types/interfaces. I know interfaces are outside the scope of what's > being proposed right now, but one of their benefits is that they *do* > solve this problem. Structured strings naming protocols > ("org.python.std.index" or whatever) do this as well, but without > language support. I did mention that one issue with my "strawman proposal" was exactly that it performs no error checking: it entirely relies on programers respecting some simple and reasonable conventions, rather than piling up machinery to provide enforcement. Much like, oh, say, Python. Isn't it just wonderful, how the foes of adaptation switch horses on you? First they request a simple-as-dirt, bare-bones "example system" -- then as soon as you provide one they come back at you with all sort of "cruft" to be piled on top. Ah well, having tried to evangelize for adaptation for years, I've grown wearily accustomed to this kind of response; it sometimes looks more like such foes are feeling defensive, and ready to pull any trick to stop adaptation from getting in the language, rather than interested in the various technica aspect of the issue. To be fair, this isn't all that different from the average reaction one always gets from python-dev as a whole to any proposal whatsoever. Anyway, I hope it's clearer now why, each and every time, I end up giving up, and deciding that beating my head against a wall is more productive and fun;-). I guess I'll just adopt a signature of "Praeterea censeo adaptatio esse adoptanda!", for all the good that all the detailed discussions appear to have been doing;-). Alex From crutcher at gmail.com Sun Apr 2 20:15:15 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Sun, 2 Apr 2006 11:15:15 -0700 Subject: [Python-3000] String formating operations in python 3k Message-ID: Python currently supports 'S % X', where S is a strinng, and X is one of: * a sequence * a map * treated as (X,) But I have some questions about this for python 3000. 1. Shouldn't there be a format method, like S.format(), or S.fmt()? 2. What about using __call__ instead of / in addition to __rmod__? * "time: %s"(time.ctime()) == "time: %s" % time.ctime() * "%(a)s %(b)s"(a=1, b=2) == "%(a)s %(b)s" % {'a'=1, 'b'=2} -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From guido at python.org Sun Apr 2 20:30:47 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 2 Apr 2006 11:30:47 -0700 Subject: [Python-3000] It's a statement! It's a function! It's BOTH! In-Reply-To: References: Message-ID: On 4/1/06, Talin wrote: > This is about the print / writeln debate. No, it isn't. :-) Let me focus on the part that you skirt, to explain that that's what makes anything like this pretty much impossible. > We would start by defining a simple metalanguage for giving instructions to > the parser. I won't even try to suggest a syntax, but in essence this would > be something that has the same role as a macro language or preprocessor, in > that it is not part of the programming language itself, but instead describes > how the subsequent text is to be interpreted. (There's already some precedent > for this with the __future__ syntax.) So here's the rub. Assume we have nice simple meta-syntax for making "print" a special form. For the sake of having a concrete (if silly) example, let's say you do this by writing (-:print:-) The problem is, where do you put this so that the parser knows that 'print' is a special form? You'd like to be able to define a whole library of these, and import it; you don't want to have to write (-:print:-) at the top of each file where you want to use this (this would pretty much destroy the newbie-friendliness of the feature). But the Python parser doesn't read the modules it imports! When it sees "import foo" it just generates a few bytecodes that cause foo to be imported at run time, and notes that foo is now a local variable (yes, local!). If there are any (-:print:-) declarations in foo.py, the parser won't know about them. This is different from import in Java and #include in C/C++, where the contents of what is imported/included can inform the parser. Based on my experience with past wild proposals, you're probably going to counter by proposing that the Python interpreter somehow follows imports at compile time; I could then object that it's totally reasonable that the module to be imported is unavailable to the parser (either due to import hooks to be installed later, or simple separate compilation as done by compileall.py). You could then counter by proposing an alternative syntax for compile-time imports (-:import foo:-) that must be followed and interpreted by the parser. And I could call that ugly, hard to implement, etc. But please save your breath. Programmable syntax is not in Python's future -- or at least it's not for Python 3000. The problem IMO is that everybody will abuse it to define their own language. And the problem with that is that it will fracture the Python community because nobody can read each other's code any more. It's one thing to read code that calls an external function whose meaning you have to guess from its name. It's quite another thing to realize that you can't even know for sure where the function calls are. Let's not go there. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.hochberg at ieee.org Sun Apr 2 20:52:16 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Sun, 02 Apr 2006 11:52:16 -0700 Subject: [Python-3000] Adaptation In-Reply-To: <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> Message-ID: Just in case anyone wants to play with a slightly more fleshed out, but still simple minded (in a good way) approach to adaption, I've put up an implementation here: http://members.cox.net/~tim.hochberg/adaption2.py It's adaption2, because I decided I didn't like my first try. No guarantees that it's not full of bugs, wrongheaded, etc, but it does give something concrete to look at. It's quite similar to the Alex's super-simple proposal, with two main differences. 1. Protocols are indicated by Protocol objects, not strings. These objects just have a name, and no other features, so they aren't any more meaningful than strings, but by dint of being objects they can't collide. This also makes cancelling protocols a bit simpler. 2. The adaption is P->P, that is from one protocol to another protocol, rather than from T->P, that is from type to protocol. This makes things marginally more complicated since you need to associate types with protocols, but seems better for reasons I can't really articulate right now. As Alex suggested protocols are inherited by walking the mro for objects that have one. I'll include the module docstring below. Regards, -tim """adaption -- the worlds fourth most naive implementation of adaption Based on some post one python-dev-3000 on April 1st, 2006. Alex Martelli's being one of the main ones. Idiocies are all atributable to me (Tim Hochberg), however. The strategy is that objects have associated with them a set of protocols. These protocols are just tags that represent some interface or behaviour of the object. These tags are associated with the object by registering them using register_type. As an example, let's define a sequence type that starts it's indexing at 1: >>> class BaseOneSeq(object): ... def __init__(self, values): ... self._values = values ... def __getitem__(self, i): ... return self._values[i-1] We need a protocol to describe this class and we then need to register the class so the that the adaption machinery knows that the class and protocol are connected. >>> base1seq = Protocol("base1seq") >>> register_type(BaseOneSeq, base1seq) Now we want an adapter to adapt this to a normal sequence. >>> class BaseOneSeqToSeq(object): ... def __init__(self, target): ... self._target = target ... def __getitem__(self, i): ... return self._target[i+1] Finally, we need to register this adapter: >>> register_adapter(base1seq, protocols.sequence, BaseOneSeqToSeq) Phew. All done, now we can try it out: >>> b1 = BaseOneSeq([2,3,5,7,11,13,17,19,23]) >>> b1[1], b1[3], b1[5] (2, 5, 11) So far, so good. Now to adapt it to a normal, base-0 sequence: >>> seq = adapt(b1, protocols.sequence) >>> seq[0], seq[2], seq[4] (2, 5, 11) Well, that's cool, but it seemed like a lot of work. Now let's try out a cool, if possibly useless feature. AntiProtocols. This time, we repeat the same problem, but we derive from list to get __len__ and stuff for free: >>> class BaseOneSeq2(list): ... def __getitem__(self, i): ... return list.__getitem__(self, i-1) >>> register_type(BaseOneSeq2, base1seq) Notice that this time around is much easier -- we've already done a lot of the work. >>> b2 = BaseOneSeq2([2,3,5,7,11,13,17,19,23]) >>> b2[1], b2[3], b2[5] (2, 5, 11) >>> len(b2) 9 >>> seq2 = adapt(b2, protocols.sequence) >>> seq2[0], seq2[2], seq2[4] (23, 3, 7) Huh! The problem here is that we're inheriting from list and list is a sequence. If an object already satisfies a protocol, it gets returned from adapt unchanged. To fix that we need to nullify the sequence protocol that BaseOneSeq2 gets from list. AntiProtocols to the rescue: >>> register_type(BaseOneSeq2, ~protocols.sequence) >>> seq3 = adapt(b2, protocols.sequence) >>> seq3[0], seq3[2], seq3[4] (2, 5, 11) That's all folks. """ From g.brandl at gmx.net Sun Apr 2 21:01:04 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 02 Apr 2006 21:01:04 +0200 Subject: [Python-3000] Iterating over a dict In-Reply-To: References: Message-ID: Guido van Rossum wrote: > On 4/1/06, Thomas Lotze wrote: >> Hi, >> >> I wonder what's the reason for iterating over a dict by keys: >> >> >>> for x in {1:"a", 2:"b"}: >> ... print x >> ... >> 1 >> 2 >> >> I find it much more intuitive for the values, "a" and "b", to be accessed. >> This is particularly confusing as iterating over tuples, lists and sets in >> the same way does access the values. (It feels like iterating over a list >> l actually iterates over the index values, range(len(l)).) In fact, >> iterating over any container should access the contained values. >> >> The reason I ask this on the python-3000 list is that I wonder whether the >> iterating behaviour of dicts might be changed in Python 3k, so that in the >> above code, foo() would be applied to the dict's values. > > Just to confirm what's already been said, this was considered very > carefully and won't change. A quick idea of mine: Wouldn't it be useful to maintain a list of what will not change, collected in the discussions here? That way, people eager to suggest braces and whatnot can be referred to it. (I'd be volunteering to maintain such a list/PEP/whatever). Georg From p.f.moore at gmail.com Sun Apr 2 21:47:48 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 2 Apr 2006 20:47:48 +0100 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> Message-ID: <79990c6b0604021247t5b79b3k16030a7b1eca6cc0@mail.gmail.com> On 4/2/06, Alex Martelli wrote: > I believe that such "magically appearing" does not depend on > adaptation, per se, but on the mix of "convenience" approaches to > adaptation and registration that one chooses to provide alongside it. Apologies if I didn't explain myself well - that's what I was trying to say, that it does happen, but it's to some extent an abuse of adaptation. How much of an abuse is a style issue, but *I* don't like it. > > I don't know if that's an "unwritten rule" as such - but I can barely > > imagine what you're describing as unacceptable (adaptation to a > > mutable interface which adds or copies state). It just seems like a > > stupid thing to do (or at least, not at all what adaptation is about). > > But maybe that's what you mean by a "hidden assumption". > > Uh? Consider iteration -- that's a prime example of an adaptation > which adds "hidden state", and I don't see it as particularly > problematic to frame as an adaptation. Again, I'm not disagreeing with you - I see iteration as a perfectly acceptable use of adaptation, but I don't see it as related to Nick's "hidden assumption". That's what I was trying to say when I said I can't imagine what Nick's getting at - in the same way that Nick is failing to "get" your explanation of why adaptation is good, I (and you?) am failing to see what he's getting at with his concerns. It's a communication issue more than anything, I suspect. > Do we have a "good practice document" about what you should or > shouldn't do with metaclasses, or decorators, or, for that matter, > with inheritance, operator overloading, and other powerful constructs > and tools that have been with Python a long, long time? Again, that was my point - I don't think there's much value in such a document for adaptation, just as there isn't any for metaclasses, etc. > Isn't it just wonderful, how the foes of adaptation switch horses on > you? First they request a simple-as-dirt, bare-bones "example > system" -- then as soon as you provide one they come back at you with > all sort of "cruft" to be piled on top. Ah well, having tried to > evangelize for adaptation for years, I've grown wearily accustomed to > this kind of response; it sometimes looks more like such foes are > feeling defensive, and ready to pull any trick to stop adaptation > from getting in the language, rather than interested in the various > technica aspect of the issue. To be fair, this isn't all that > different from the average reaction one always gets from python-dev > as a whole to any proposal whatsoever. Anyway, I hope it's clearer > now why, each and every time, I end up giving up, and deciding that > beating my head against a wall is more productive and fun;-). :-) To be fair, your simple-as-dirt strawman *does* have limitations. That's fine, given what it's designed to do. But it's also so simple and self-evident that it's easy to see the holes, and start trying to fill them. Whether that's a good thing or not, I couldn't say... Anyway, I'm getting sucked into writing more and more, and i doubt I'm coherent by now. So I'll stop. Paul. From guido at python.org Sun Apr 2 22:06:57 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 2 Apr 2006 13:06:57 -0700 Subject: [Python-3000] Iterating over a dict In-Reply-To: References: Message-ID: On 4/2/06, Georg Brandl wrote: > A quick idea of mine: Wouldn't it be useful to maintain a list of what will > not change, collected in the discussions here? That way, people eager to suggest > braces and whatnot can be referred to it. > > (I'd be volunteering to maintain such a list/PEP/whatever). Excellent idea! It should be one of the meta-peps (in the 3001-3099 range). Ideally the list should either point to the discusussion explaining the decision, or provide a capsule explanation. (Braces are exempted from this policy; if someone doesn't understand why we're not doing braces they should probably look elsewhere... :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Sun Apr 2 22:27:08 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 02 Apr 2006 22:27:08 +0200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: Message-ID: Crutcher Dunnavant wrote: > Python currently supports 'S % X', where S is a strinng, and X is one of: > * a sequence > * a map > * treated as (X,) > > But I have some questions about this for python 3000. > > 1. Shouldn't there be a format method, like S.format(), or S.fmt()? Possible, but why? "%" works quite fine. "Abusing" the modulus operator is okay in this context since the symbol gives a nonoverseeable clue to what the construct is doing. > 2. What about using __call__ instead of / in addition to __rmod__? > * "time: %s"(time.ctime()) == "time: %s" % time.ctime() > * "%(a)s %(b)s"(a=1, b=2) == "%(a)s %(b)s" % {'a'=1, 'b'=2} Damn ugly. How would you explain to a newbie that you can _call_ a string? And that _calling_ a string does a formatting operation? For me, string formatting with "%" is fine as it is. BTW, has anyone seen string.Template being used somewhere? Georg From talin at acm.org Sun Apr 2 23:19:22 2006 From: talin at acm.org (Talin) Date: Sun, 2 Apr 2006 21:19:22 +0000 (UTC) Subject: [Python-3000] It's a statement! It's a function! It's BOTH! References: Message-ID: Guido van Rossum python.org> writes: > Based on my experience with past wild proposals, you're probably going > to counter by proposing that the Python interpreter somehow follows > imports at compile time; I could then object that it's totally > reasonable that the module to be imported is unavailable to the parser > (either due to import hooks to be installed later, or simple separate > compilation as done by compileall.py). You could then counter by > proposing an alternative syntax for compile-time imports import > foo that must be followed and interpreted by the parser. And I > could call that ugly, hard to implement, etc. No worries. My purpose here is to "think the unthinkable", not to argue that I'm right. I often throw out wild ideas at my work; occasionally I hit a bullseye. In the mean time, criticism is welcome and enjoyed :) And thanks for the explanation. -- Talin From g.brandl at gmx.net Sun Apr 2 23:36:38 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 02 Apr 2006 23:36:38 +0200 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 In-Reply-To: References: Message-ID: Guido van Rossum wrote: > On 4/2/06, Georg Brandl wrote: >> A quick idea of mine: Wouldn't it be useful to maintain a list of what will >> not change, collected in the discussions here? That way, people eager to suggest >> braces and whatnot can be referred to it. >> >> (I'd be volunteering to maintain such a list/PEP/whatever). > > Excellent idea! It should be one of the meta-peps (in the 3001-3099 range). > > Ideally the list should either point to the discusussion explaining > the decision, or provide a capsule explanation. (Braces are exempted > from this policy; if someone doesn't understand why we're not doing > braces they should probably look elsewhere... :-) Here's a draft (please provide me with whatever you can remember, I've been following the list on gmane and my history's therefore lost): PEP: XXX Title: Things that will Not Change in Python 3000 Version: $Revision$ Last-Modified: $Date$ Author: Georg Brandl Status: Draft Type: Informational Content-Type: text/x-rst Created: 02-Apr-2006 Abstract ======== This PEP tries to list all BDFL pronouncements on Python 3000 that refer to changes that will not happen and new features that will not be introduced, sorted by topics, along with a short explanation or a reference to the relevant thread on the python-3000 mailing list. Core language ============= * Python will not have programmable syntax. Thread: "It's a statement! It's a function! It's BOTH!" http://mail.python.org/pipermail/python-3000/2006-April/000286.html * There won't be a syntax for ``zip()``-style parallel iteration. Thread: "Parallel iteration syntax", http://mail.python.org/pipermail/python-3000/2006-March/000210.html Standard types ============== * Iterating over a dictionary will yield the keys. Thread: "Iterating over a dict", http://mail.python.org/pipermail/python-3000/2006-April/000283.html Coding style ============ * The maximum line width will be 80 characters. Thread: "C style guide", http://mail.python.org/pipermail/python-3000/2006-March/000131.html Copyright ========= This document has been placed in the public domain. From talin at acm.org Sun Apr 2 23:39:56 2006 From: talin at acm.org (Talin) Date: Sun, 2 Apr 2006 21:39:56 +0000 (UTC) Subject: [Python-3000] A few small py3k wishes Message-ID: Just a few things that have wanted while doing python programming. Maybe some of these are already doable or are already on someone's wish list... -- The __main__ module should contain a file path like imported modules. in other words, every module should know where it came from. In fact, I see no reason why the __main__ module should have any different attributes than an imported module other than the name. -- An easy way to make a case-insensitive, case-preserving dict that works with regular string keys. -- An easy way to iterate over key, value pairs in a dict in sorted order. Currently you have to get the list of keys, sort them, and then lookup each value, OR you have to get the list of tuples and call sorted() with a key= arg containing a lambda function that extracts the first tuple element. (Another one of my wild ideas was an "order by" clause for list comprehensions, but let's not go there.) -- A simple way to import all modules in a directory (this would be used for plugins) -- A mechanism whereby imported modules can import symbols from the module that imported them. (Specifically, I want to take a bunch of variables in my __main__ module and make them accessible to the imported module.) -- The module class should have a method to iterate over child modules. Currently you can iterator through all of its attributes, but you have to filter out which ones are modules. -- A path-globbing function that supports regex-style captures. -- A path-globbing function that supports the perforce-style syntax "..." (three dots) to mean "all descendants". So for example, "foo/.../*.cpp" matches any cpp files in foo or any of its subdirectories. That should be enough for the moment. -- Talin From ncoghlan at gmail.com Sun Apr 2 23:56:39 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 03 Apr 2006 07:56:39 +1000 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: Message-ID: <44304897.80307@gmail.com> Georg Brandl wrote: > Crutcher Dunnavant wrote: >> Python currently supports 'S % X', where S is a strinng, and X is one of: >> * a sequence >> * a map >> * treated as (X,) >> >> But I have some questions about this for python 3000. >> >> 1. Shouldn't there be a format method, like S.format(), or S.fmt()? > > Possible, but why? "%" works quite fine. "Abusing" the modulus operator > is okay in this context since the symbol gives a nonoverseeable clue to > what the construct is doing. A "format" function that accepted the pattern as the first argument and the values to be interpolated as the remaining arguments would be even clearer though. Instead of giving you a clue, it would tell you exactly what's going on. Basing this on string.Template gives very readable syntax when using positional arguments, as well as allowing each positional argument to be used more than once: >>> x = 3 >>> format("$1 * $1 = $2", x, x*x) '3 * 3 = 9' Tweaking the string.Template regex a bit even allows formatting: >>> x = 3.14159 >>> format("$[.2f]1 * $[.2f]1 = $[.4f]2", x, x*x) '3.14 * 3.14 = 9.8696' >> 2. What about using __call__ instead of / in addition to __rmod__? >> * "time: %s"(time.ctime()) == "time: %s" % time.ctime() >> * "%(a)s %(b)s"(a=1, b=2) == "%(a)s %(b)s" % {'a'=1, 'b'=2} > > Damn ugly. How would you explain to a newbie that you can _call_ a string? > And that _calling_ a string does a formatting operation? > For me, string formatting with "%" is fine as it is. I think there is merit in having a higher level interpolation syntax underpinned by the raw mod-formatting syntax. > BTW, has anyone seen string.Template being used somewhere? I imagine Barry, at least, (and probably others) are using it for its primary intended use case - localisation. As more applications move to requiring 2.4, then its use is also likely to increase. Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2005-September/056231.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From aleaxit at gmail.com Mon Apr 3 00:08:10 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Sun, 2 Apr 2006 15:08:10 -0700 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: Message-ID: On Apr 2, 2006, at 2:39 PM, Talin wrote: ... > -- An easy way to iterate over key, value pairs in a dict in > sorted order. Currently you have to get the list of keys, > sort them, and then lookup each value, OR you have to > get the list of tuples and call sorted() with a key= arg > containing a lambda function that extracts the first > tuple element. I believe this assertion is severely mistaken. Please give one example of a dictionary d where your proposed, contorted idea: for k, v in sorted(d.items(), key=lambda kv: kv[0]): ... makes ANY difference compared to the streamlined and effective "one and only obvious way to do it": for k, v in sorted(d.iteritems()): considering that: a. keys are unique (no two distinct keys can ever compare equal), therefore that key=... clause is totally unneeded; b. from sorted's viewpoint, it makes no difference whether its argument is a list or any other iterable, therefore it's a pure waste of memory (in Python 2.*) to pass it d.items() ["the list of tuples", as you call it] where d.iteritems() will do just fine. Personally, despite having just claimed that the above is "the one and only obvious way", I find the following quite fine too: for k in sorted(d): v = d[k] ... Yes, that's your dreaded "lookup each value", but it's nowhere as dreadful as you make it sound -- just about equivalent, in both lexical conciseness and conceptual clarity, to "the one and only obvious way". > -- A mechanism whereby imported modules can import > symbols from the module that imported them. (Specifically, > I want to take a bunch of variables in my __main__ module > and make them accessible to the imported module.) A module can be imported several times, from many other modules, although the "import machinery" only LOADS each module once (save for explicit reload(...) calls). Are you proposing that the symbols being imported by module X should depend on the half-accident of what other module happens to import X _FIRST_?! If you want to have X import from __main__, doing so by "from __main__ import ..." is simple, clear, and obvious -- and needs absolutely no change whatsoever in Python, since it works just fine today, of course. I don't like any of these proposals, but these two are particularly egregious, particularly the first one I've quoted, as it rests on an obviously erroneous assertion. Alex From tim.hochberg at ieee.org Mon Apr 3 00:11:09 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Sun, 02 Apr 2006 15:11:09 -0700 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: Message-ID: Talin wrote: > Just a few things that have wanted while doing python > programming. Maybe some of these are already doable > or are already on someone's wish list... Some of these are already pretty easy to do. Perhaps you should explain what's inadequate about the current solutions. For example: > -- The __main__ module should contain a file path like imported > modules. in other words, every module should know where it > came from. In fact, I see no reason why the __main__ module > should have any different attributes than an imported module > other than the name. > > -- An easy way to make a case-insensitive, case-preserving > dict that works with regular string keys. > > -- An easy way to iterate over key, value pairs in a dict in > sorted order. Currently you have to get the list of keys, > sort them, and then lookup each value, OR you have to > get the list of tuples and call sorted() with a key= arg > containing a lambda function that extracts the first > tuple element. for key in sorted(adict.keys()): value = adict[key] # ... It would also be trivial to define your own generator function to do this if you are doing it a lot: def keysorted_items(obj): return ((x, obj[x]) for x in sorted(obj.keys())) for (key, value) in keysorted_items(adict): # ... > > (Another one of my wild ideas was an "order by" > clause for list comprehensions, but let's not go there.) > > -- A simple way to import all modules in a directory > (this would be used for plugins) > > -- A mechanism whereby imported modules can import > symbols from the module that imported them. (Specifically, > I want to take a bunch of variables in my __main__ module > and make them accessible to the imported module.) For this specific use, can't you just use "import __main__" and look at the symbols that way? > > -- The module class should have a method to iterate > over child modules. Currently you can iterator through > all of its attributes, but you have to filter out which ones > are modules. Why? In other words, what wrong with: for key, value in module.__dict__.items(): if is_module(value): # do something? Or again, define your own generator if you do this a lot: def submodules(obj): for key, value in obj.__dict__.items(): if is_module(value): yield key, value for name, module in submodules(mymodule): # do something. > > -- A path-globbing function that supports regex-style > captures. > > -- A path-globbing function that supports the > perforce-style syntax "..." (three dots) to mean "all descendants". > So for example, "foo/.../*.cpp" matches any cpp files in > foo or any of its subdirectories. > > That should be enough for the moment. Regards, -tim From msoulier at digitaltorque.ca Mon Apr 3 00:21:30 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Sun, 2 Apr 2006 18:21:30 -0400 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: Message-ID: <20060402222130.GA12075@tigger.digitaltorque.ca> On 02/04/06 Talin said: > Just a few things that have wanted while doing python > programming. Maybe some of these are already doable > or are already on someone's wish list... It's a minor thing, but it'd be nice to make a method alias for append() on mutable sequences called push(), to go with pop(). To save memory, it would also be nice if docstrings were only made available in memory with an interpreter switch. Beyond that, I'd love to see distribution to platforms without python installed get easier. Kinda like a py2exe for each platform. I have a hard time convincing people to install Python just for my little program at times. For some silly reason people seem to already have the JRE installed. I can't imagine why. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060402/a50b01ad/attachment.pgp From thomas at python.org Mon Apr 3 00:30:19 2006 From: thomas at python.org (Thomas Wouters) Date: Mon, 3 Apr 2006 00:30:19 +0200 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: Message-ID: <9e804ac0604021530r3437e99dv9e221b070ed3f30e@mail.gmail.com> On 4/2/06, Talin wrote: > -- The __main__ module should contain a file path like imported > modules. in other words, every module should know where it > came from. In fact, I see no reason why the __main__ module > should have any different attributes than an imported module > other than the name. I'm not sure what you're missing. The __main__ module has __file__: centurion:~ > cat tmp.py print "__file__:", __file__ import __main__ print "__main__.__file__:", __main__.__file__ centurion:~ > python tmp.py __file__: tmp.py __main__.__file__: tmp.py -- A path-globbing function that supports regex-style > captures. Improvements like that happen because someone writes the module and people end up using it. It isn't part of the language, it's just stdlib evolution. -- A path-globbing function that supports the > perforce-style syntax "..." (three dots) to mean "all descendants". > So for example, "foo/.../*.cpp" matches any cpp files in > foo or any of its subdirectories. I prefer ksh's '**' myself. It's trivial to add, see for instance http://www.python.org/sf/628075 Again, this isn't something for py3k. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060403/ff6cbefa/attachment.htm From rasky at develer.com Mon Apr 3 00:44:47 2006 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 3 Apr 2006 00:44:47 +0200 Subject: [Python-3000] String formating operations in python 3k References: Message-ID: <132501c656a7$0b7d8360$f44c2597@bagio> Georg Brandl wrote: > BTW, has anyone seen string.Template being used somewhere? Not me. Surely it'd help if its documentation explained why on earth it has been introduced. There are no apparent advantages over the builtin %s operator, at first sight. Nothing worth using a non-standard non-builtin substitution engine, at least. Giovanni Bajo From talin at acm.org Mon Apr 3 00:51:53 2006 From: talin at acm.org (Talin) Date: Sun, 2 Apr 2006 22:51:53 +0000 (UTC) Subject: [Python-3000] A few small py3k wishes References: Message-ID: Alex Martelli gmail.com> writes: > for k, v in sorted(d.iteritems()): Ah, thanks! I didn't know that worked. (The missing piece was that I wasn't aware that relative comparisons worked on tuples. I even looked in the library reference manual under "sequence types") > A module can be imported several times, from many other modules, > although the "import machinery" only LOADS each module once (save for > explicit reload(...) calls). Are you proposing that the symbols > being imported by module X should depend on the half-accident of what > other module happens to import X _FIRST_?! No, that's not what I am proposing :) Sorry if my suggestion was badly worded. It really has to do more, I think, with how to manage interdependencies within a large python app, consisting of dozens of source files. Since that's not a Py3K question, let's agree to discuss that elsewhere... From crutcher at gmail.com Mon Apr 3 00:54:46 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Sun, 2 Apr 2006 15:54:46 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: Message-ID: On 4/2/06, Georg Brandl wrote: > Crutcher Dunnavant wrote: > > Python currently supports 'S % X', where S is a strinng, and X is one of: > > * a sequence > > * a map > > * treated as (X,) > > > > But I have some questions about this for python 3000. > > > > 1. Shouldn't there be a format method, like S.format(), or S.fmt()? > > Possible, but why? "%" works quite fine. "Abusing" the modulus operator > is okay in this context since the symbol gives a nonoverseeable clue to > what the construct is doing. Why? Because: 1 It is trivially cheap, format() would be the same function as __rmod__ 2 It adds consistency with lower(), strip(), and other methods which produce new strings. 3 I am not arguing _against_ syntactic support, I am arguing _for_ a method; we can keep the syntactic support. > > > 2. What about using __call__ instead of / in addition to __rmod__? > > * "time: %s"(time.ctime()) == "time: %s" % time.ctime() > > * "%(a)s %(b)s"(a=1, b=2) == "%(a)s %(b)s" % {'a'=1, 'b'=2} > > Damn ugly. How would you explain to a newbie that you can _call_ a string? > And that _calling_ a string does a formatting operation? > For me, string formatting with "%" is fine as it is. This is a non-argument. I'd do it the same way that we currently explain that you can _modulo_ or _multiply_ a string. The basic argument goes something like this: While addition and multiplication are binary operations, string formatting usually has more elements involved, and as such, treating it as a binary operation is a bit strange. Especially when keyword formatting is involved. It so happens that python has a well developed syntax for exactly this sort of relationship, and it is the _call_ syntax. So this reduces the complexity of the language for the learner. > > BTW, has anyone seen string.Template being used somewhere? > > Georg > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/crutcher%40gmail.com > -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From g.brandl at gmx.net Mon Apr 3 00:56:27 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 03 Apr 2006 00:56:27 +0200 Subject: [Python-3000] pre-PEP: Process for reviewing/improving stdlib modules in 3.0 Message-ID: PEP: XXX Title: Procedure for changing standard library modules Version: $Revision$ Last-Modified: $Date$ Author: Georg Brandl Status: Draft Type: Informational Content-Type: text/x-rst Created: 02-Apr-2006 Abstract ======== This PEP describes the procedure for reviewing and improving standard library modules, especially those written in Python, making them ready for Python 3000. There can be different steps of refurbishing, each of which is described in a section below. Step 1: Removal of obsolete modules =================================== All modules marked as deprecated in 2.x versions should be removed for Python 3000. The same applies to modules which are seen as obsolete today, but are too widely used to be deprecated or removed. Python 3000 is the big occasion to get rid of them. There will have to be a document listing all removed modules, together with information on possible substitutes or alternatives. This infor- mation will also have to be provided by the python3warn.py porting helper script mentioned in PEP XXX. Step 2: Renaming modules ======================== Aside from a "great stdlib renaming" introducing a hierarchic library namespace or a top-level package from which to import standard modules, some modules' names are known to have been chosen unwisely, a mistake which could never be corrected in the 2.x series. Examples are names like "StringIO" or "Cookie". For Python 3000, there will be the possi- bility to rename those modules to less confusing and more conforming names. Of course, each rename will have to be stated in the documentation of the respective module and perhaps in the global document of Step 1. Additionally, the python3warn.py script will recognize the old module names and notify the user accordingly. Step 3: Code cleanup ==================== As most library modules written in Python have not been touched except for bug fixes, following the policy of never changing a running system, many of them may contain code that is not up to the newest language features and could be rewritten in a more concise, modern Python. As long as these changes don't change the module's interface and behavior, no documentation updates are necessary. Step 4: Enhancement of test coverage ==================================== Code coverage by unit tests varies greatly between modules. Each test suite should be checked for completeness, and the remaining classic tests should be converted to PyUnit (or whatever new shiny testing framework comes with Python 3000, perhaps py.test?). No documentation changes are necessary for this step. Step 5: Unification of module metadata ====================================== This is a small and probably not very important step. There have been various attempts at providing author, version and similar metadata in modules (such as a "__version__" global). Those could be standardized and used throughout the library. No documentation changes are necessary for this step, too. Step 6: Backwards incompatible bug fixes ======================================== Over the years, many bug reports have been filed which complained about bugs in standard library modules, but have subsequently been closed as "Won't fix" since a fix would have introduced a major incompatibility which was not acceptable in the Python 2.x series. In Python 3000, the fix can be applied if the interface per se is still acceptable. Each slight behavioral change caused by such fixes must be mentioned in the documentation, perhaps in a "Changed in Version 3.0" paragraph. Step 7: Interface changes ========================= The last and most disruptive change is the overhaul of a module's public interface. If a module's interface is to be changed, a justification should be made beforehand, or a PEP should be written. The change must be fully documented as "New in Version 3.0", and the python3warn.py script must know about it. References ========== TBD Copyright ========= This document has been placed in the public domain. From talin at acm.org Mon Apr 3 01:08:13 2006 From: talin at acm.org (Talin) Date: Sun, 2 Apr 2006 23:08:13 +0000 (UTC) Subject: [Python-3000] A few small py3k wishes References: <9e804ac0604021530r3437e99dv9e221b070ed3f30e@mail.gmail.com> Message-ID: Thomas Wouters python.org> writes: > I'm not sure what you're missing. The __main__ module has __file__: > centurion:~ > cat tmp.py > print "__file__:", __file__ > import __main__ > print "__main__.__file__:", __main__.__file__ > > centurion:~ > python tmp.py > __file__: tmp.py > __main__.__file__: tmp.py Except that they are not the same! When I print __file__ from my __main__ module, I get the name of the file only, no path. However, when I print __file__ from my imported module, I get the complete, absolute path to the module. In fact, if I do a "print dir()" for the main module and the imported module, I get something like this: main: ['__builtins__', '__file__', '__name__', '__pymate', 'input', 'raw_input'] imported: ['__builtins__', '__doc__', '__file__', '__name__', '__path__', 'fnmatch', 'os'] Note that the impored module has a "__path__" attribute, while the main module does not. The reason I bring this up is that it is a common usage pattern for a particular source file to be run as both a main and as an imported module. Suppose you have a bunch of test data that is in a subdir of your source file. You'd like to be able to have a single, uniform way to locate your module's data, regardless if you are __main__ or not. Feel free to let me know if I am missing something... -- Talin From brett at python.org Mon Apr 3 01:09:32 2006 From: brett at python.org (Brett Cannon) Date: Sun, 2 Apr 2006 16:09:32 -0700 Subject: [Python-3000] Iterating over a dict In-Reply-To: References: Message-ID: On 4/2/06, Guido van Rossum wrote: > On 4/2/06, Georg Brandl wrote: > > A quick idea of mine: Wouldn't it be useful to maintain a list of what will > > not change, collected in the discussions here? That way, people eager to suggest > > braces and whatnot can be referred to it. > > > > (I'd be volunteering to maintain such a list/PEP/whatever). > > Excellent idea! It should be one of the meta-peps (in the 3001-3099 range). > > Ideally the list should either point to the discusussion explaining > the decision, or provide a capsule explanation. (Braces are exempted > from this policy; if someone doesn't understand why we're not doing > braces they should probably look elsewhere... :-) There is PEP 13 of which there is a draft that I wrote at PyCon and I am waiting to have edited by Thomas and Tim. That lists ideas rejected for Python regardless of version. If that PEP does go in the Py3K version could reference PEP 13 and get a good list of things that are not changing. -Brett From thomas at python.org Mon Apr 3 01:19:39 2006 From: thomas at python.org (Thomas Wouters) Date: Mon, 3 Apr 2006 01:19:39 +0200 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: <9e804ac0604021530r3437e99dv9e221b070ed3f30e@mail.gmail.com> Message-ID: <9e804ac0604021619n7c64db00n6cf549a72fdaca85@mail.gmail.com> On 4/3/06, Talin wrote: > > Thomas Wouters python.org> writes: > > > I'm not sure what you're missing. The __main__ module has __file__: > > Except that they are not the same! > > When I print __file__ from my __main__ module, I get the name of the > file only, no path. You get the path that was used to run the file: centurion:~ > python ~thomas/tmp.py __file__: /home/thomas/tmp.py __main__.__file__: /home/thomas/tmp.py > However, when I print __file__ from my imported module, I get the > complete, absolute path to the module. Incorrect: you get the path that was used to import the module. If a relative path is in sys.path (like its first element usually is), the module's __file__ will be a relative path: centurion:~ > python -c 'import tmp' __file__: tmp.py In fact, if I do a "print dir()" for the main module and the imported > module, I get something like this: > > main: ['__builtins__', '__file__', '__name__', '__pymate', 'input', > 'raw_input'] > imported: ['__builtins__', '__doc__', '__file__', '__name__', > '__path__', 'fnmatch', 'os'] > > Note that the impored module has a "__path__" attribute, while the > main module does not. Only packages have a __path__ attribute. Try checking on a module that is a .py file. Likewise, not all modules have a __file__ attribute: >>> posix.__file__ Traceback (most recent call last): File "", line 1, in ? AttributeError: 'module' object has no attribute '__file__' The reason I bring this up is that it is a common usage pattern for > a particular source file to be run as both a main and as an imported > module. Suppose you have a bunch of test data that is in a subdir > of your source file. You'd like to be able to have a single, uniform > way to locate your module's data, regardless if you are __main__ > or not. __file__ is it. os.path.abspath() it with os.getcwd(). -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060403/c0898dd5/attachment-0001.htm From thomas.lotze at gmx.net Mon Apr 3 01:25:33 2006 From: thomas.lotze at gmx.net (Thomas Lotze) Date: Mon, 03 Apr 2006 01:25:33 +0200 Subject: [Python-3000] Iterating over a dict References: Message-ID: Am Sun, 02 Apr 2006 09:29:13 -0700 schrieb Guido van Rossum: > Just to confirm what's already been said, this was considered very > carefully and won't change. OK, thanks. -- Thomas From walter at livinglogic.de Mon Apr 3 01:39:06 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 03 Apr 2006 01:39:06 +0200 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> Message-ID: <4430609A.3060302@livinglogic.de> Alex Martelli wrote: > On Apr 2, 2006, at 7:11 AM, Paul Moore wrote: > ... > [...] > > Say, for example, that protocols are identified (as in my strawman > proposal) by unique strings anyway. E.g., if I were to invent a > protocol, I could name it 'it.aleax.myprot' -- since I own the > aleax.it domain, nobody else could create a name conflict. Saying > that each framework has a separate registry is just the same as > saying that each protocol "lives" in one specific registry, so that > any registration or lookup regarding protocol P *must* also specify > registryof(P). Why not make the registry identical to the protocol? The protocol is just a convention anyway: class AdaptionError(Exception): def __init__(self, adaptor, type): self.adaptor = adaptor self.type = type def __str__(self): return "can adapt %r to %r" % (self.type, self.adaptor) class Adaptor(object): def __init__(self): self.registry = {} def register(self, adaptor, *types): for type in types: self.registry[type] = adaptor def __call__(self, obj, *args, **kwargs): for bases in type(obj).__mro__: try: return self.registry[bases](obj, *args, **kwargs) except KeyError: pass raise AdaptionError(self, type(obj)) With this, code would look like: import adapt formatter = adapt.Adaptor() def formatnumber(x): return hex(x) formatter.register(formatnumber, int, long) def formatstring(x): return repr(x) formatter.register(formatstring, basestring) print formatter(42) print formatter(42L) print formatter("foo") print formatter(u"bar") print formatter(None) (The last call would raise an AdaptionError). > Hopefully, rather than having to keep this > correspondence in our human memory, we're allowed to have a registry > of registries which remembers this correspondence for you: we can > register_protocol(P, registry) and we can lookup the registry for a > given protocol with function registryof. No need for that if each registry is self-contained. > [...] >> were defined via some "interface" approach (like zope.interfaces and >> PyProtocols do) then encapsulation is taken care of by uniqueness of >> types/interfaces. I know interfaces are outside the scope of what's >> being proposed right now, but one of their benefits is that they *do* >> solve this problem. Structured strings naming protocols >> ("org.python.std.index" or whatever) do this as well, but without >> language support. > > I did mention that one issue with my "strawman proposal" was exactly > that it performs no error checking: it entirely relies on programers > respecting some simple and reasonable conventions, rather than piling > up machinery to provide enforcement. Much like, oh, say, Python. > > Isn't it just wonderful, how the foes of adaptation switch horses on > you? First they request a simple-as-dirt, bare-bones "example > system" -- then as soon as you provide one they come back at you with > all sort of "cruft" to be piled on top. I think you might be misinterpreting reactions. If the initial reaction was "I don't understand it. Nobody needs this." (at least that was my reaction), you're "strawman proposal" has put us past this. (At least you got two "I finally got it, this seems useful" from me and Brett.) So now lets answer the questions: How do we implement adaption of subtypes? What is a protocol? How can we make registration as painless as possible? etc. Bye, Walter D?rwald From walter at livinglogic.de Mon Apr 3 01:48:40 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 03 Apr 2006 01:48:40 +0200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: Message-ID: <443062D8.50701@livinglogic.de> Crutcher Dunnavant wrote: > On 4/2/06, Georg Brandl wrote: >> Crutcher Dunnavant wrote: >>> Python currently supports 'S % X', where S is a strinng, and X is one of: >>> * a sequence >>> * a map >>> * treated as (X,) >>> >>> But I have some questions about this for python 3000. >>> >>> 1. Shouldn't there be a format method, like S.format(), or S.fmt()? >> Possible, but why? "%" works quite fine. "Abusing" the modulus operator >> is okay in this context since the symbol gives a nonoverseeable clue to >> what the construct is doing. > > Why? Because: > 1 It is trivially cheap, format() would be the same function as __rmod__ > 2 It adds consistency with lower(), strip(), and other methods which > produce new strings. > 3 I am not arguing _against_ syntactic support, I am arguing _for_ a method; > we can keep the syntactic support. and it avoids one problem you might run into with %: If you have only one argument, writing ``s % (x,)`` as ``s % x`` will break when the argument x happens to be a tuple. You won't have this problem with s.format(x). Bye, Walter D?rwald From greg.ewing at canterbury.ac.nz Mon Apr 3 02:33:59 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Apr 2006 12:33:59 +1200 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 In-Reply-To: References: Message-ID: <44306D77.2090409@canterbury.ac.nz> Georg Brandl wrote: > PEP: XXX > Title: Things that will Not Change in Python 3000 > > ... > > Copyright > ========= I was expecting to see something like "The Python licence will not change" under this heading. :-) -- Greg From greg.ewing at canterbury.ac.nz Mon Apr 3 02:58:58 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Apr 2006 12:58:58 +1200 Subject: [Python-3000] A few small py3k wishes In-Reply-To: <20060402222130.GA12075@tigger.digitaltorque.ca> References: <20060402222130.GA12075@tigger.digitaltorque.ca> Message-ID: <44307352.1080305@canterbury.ac.nz> Michael P. Soulier wrote: > It's a minor thing, but it'd be nice to make a method alias for append() on > mutable sequences called push(), to go with pop(). That depends on what value you have in mind for "nice". There would then be two methods which do exactly the same thing, and everyone would have to keep that in mind, or forever have anxiety when coming across code that used "push" and wondering whether there was some subtle difference. -1 from me, I think. -- Greg From greg.ewing at canterbury.ac.nz Mon Apr 3 03:27:00 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Apr 2006 13:27:00 +1200 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: Message-ID: <443079E4.9040108@canterbury.ac.nz> Talin wrote: > -- A mechanism whereby imported modules can import > symbols from the module that imported them. (Specifically, > I want to take a bunch of variables in my __main__ module > and make them accessible to the imported module.) In my_imported_module.py: import __main__ def my_stuff(): # do what you want with __main__.x etc. But be careful not to use from __main__ import foo at the top, because it's highly likely that foo won't have been defined yet when the module is imported. -- Greg From aleaxit at gmail.com Mon Apr 3 03:27:31 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Sun, 2 Apr 2006 18:27:31 -0700 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <4430609A.3060302@livinglogic.de> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> <4430609A.3060302@livinglogic.de> Message-ID: <92B6430C-65DE-4CC0-80F8-1159E9C81421@gmail.com> On Apr 2, 2006, at 4:39 PM, Walter D?rwald wrote: ... > Why not make the registry identical to the protocol? The protocol > is just a convention anyway: Yes, a 1<->1 relationship between registries and protocols makes the 'registryof' function I was talking about simpler, at least when it's implemented as identity. What's the advantage of this approach, compared to allowing protocols to be arbitrary hashable attributes? The disadvantage, essentially, is that one gives up the degrees of freedom implied by protocols being "just a convention": a protocol must now (say) be a callable object exposing a register method (with given signature and semantics). I guess adopting that kind of constraint is OK, as long as the constraint doesn't in any way interfere with whatever else Guido wants to do with protocols and thus give him a reason to reject adaptation. I guess I can see some potential for minor optimization: a protocol that's written with some existing types already in mind might use a subclass of your 'Adaptor', such as: class FastTracker(Adaptor): def __init__(self, types): self.fastrack = set(types) Adaptor.__init__(self) def _anyof(self, types): for t in types: if t in self.fastrack: return True return False def register(self, adaptor, types): if self._anyof(types): # need better diagnostics here, of course raise RegistrationError, "Cannot override identity- adaptation" return Adaptor.register(self, adaptor, types) def __call__(self, obj, *a, **k): if self._anyof(type(obj).__mro__): if a or k: raise ...some kind of diagnostics about a/k not allowed here... return obj return Adaptor.__call__(self, obj, *a, **k) I'm not sure the advantage is big enough to warrant all the machinery, but I can't assert it isn't, either. >> Isn't it just wonderful, how the foes of adaptation switch horses >> on you? First they request a simple-as-dirt, bare-bones >> "example system" -- then as soon as you provide one they come >> back at you with all sort of "cruft" to be piled on top. > > I think you might be misinterpreting reactions. If the initial > reaction was "I don't understand it. Nobody needs this." (at least > that was my reaction), you're "strawman proposal" has put us past > this. (At least you got two "I finally got it, this seems useful" > from me and Brett.) You're quite likely right, and I apologize for the misinterpretation; I guess I may have misread Brett's desire to hammer adaptation's very reason for being down to miniscule smithereens by having the existence of some set of methods "imply" identity-adaptation, as a subconscious desire to defend against the whole idea of adaptation in the guise of sort-of-accepting it, for example. > So now lets answer the questions: How do we implement adaption of > subtypes? What is a protocol? How can we make registration as > painless as possible? etc. Looks like the "loop on __mro__" idea is sort of standard for the first of these questions. As for "what is a protocol", I'd rather minimize the machinery that goes with it, but I guess that's some kind of holdover from the "strawman proposal"; if we're making protocols into classes anyway, I'd like to provide them with a check_compliance method, taking an object and an integer that represents the amount of effort to be spent in the checking -- 0 meaning 'just check existence of methods', 1 meaning 'check methods' signature compatibility too', 2 meaning 'check some semantics at a level suitable for fast unit-tests', and so on up. The author of a protocol doesn't have to do all that much (levels 0 and 1 can be provided by helper functions/methods relying on the inspect module), but he or she CAN make the protocol into "an executable specification" at whatever level he or she desires. Other error checking (e.g., against registration of multiple adapters for one protocol/type pair) as well as some modest optimization (mostly of adaptation, the frequent case) may also be warranted. I'm not sure what's painful at all about registration -- at worst it's one call per type/protocol pair, some refactoring such as yours making it even slighter by allowing the registration of several pairs in one call. Do you mean that people are so keen on declarative styles that they'd crave, e.g., some special machinery such as class Duck(object): __implements__ = walk, talk, quack ... (with the metaclass doing the registration calls, with identity adaptation, under the cover for each protocol in __implements__), prizing it highly over explicit: class Duck(object): ... adapt.register(Duck, (walk, talk, quack), identity) or the loop or classdecorator version of the latter? Ah well, maybe, but most of the discussion sort of smacks to me of AGNI and/or premature optimization. Could we agree that simplicity and even minimalism (with a side order of future extensibility if/when warranted) are high on the list of desired properties for a protocol adaptation mechanism? After all, the more machinery, cruft, and black magic we pile on top of adaptation's simple core ideas, the more likely the whole caboodle is to provoke allergic reactions in anybody who's not being a part of this discussion... Alex From greg.ewing at canterbury.ac.nz Mon Apr 3 03:38:25 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Apr 2006 13:38:25 +1200 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: <9e804ac0604021530r3437e99dv9e221b070ed3f30e@mail.gmail.com> Message-ID: <44307C91.6080803@canterbury.ac.nz> Talin wrote: > When I print __file__ from my __main__ module, I get the name of the > file only, no path. It seems to be whatever pathname you passed to the interpreter on the command line. If that's a relative path, it will end up in the __file__ of __main__ as a relative path. But it will still work as long as the working directory of the process hasn't changed. So either leave the cwd alone, or expand the __file__ to a full path before changing it. > Note that the impored module has a "__path__" attribute, while the > main module does not. __path__ is something different -- it's a list of directories to search when importing submodules of a package. Non-package modules don't have a __path__. > You'd like to be able to have a single, uniform > way to locate your module's data, regardless if you are __main__ > or not. There already is. I do this frequently and it works fine, as long as you keep the cwd issue in mind. -- Greg From barry at python.org Mon Apr 3 03:55:38 2006 From: barry at python.org (Barry Warsaw) Date: Sun, 02 Apr 2006 21:55:38 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: Message-ID: <1144029338.11452.41.camel@resist.wooz.org> On Sun, 2006-04-02 at 22:27 +0200, Georg Brandl wrote: > BTW, has anyone seen string.Template being used somewhere? I use it, which shouldn't be a big suprise (though not in much public code). You'll probably see it get used quite a bit in a future Mailman release. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060402/ad1ea057/attachment.pgp From barry at python.org Mon Apr 3 04:00:33 2006 From: barry at python.org (Barry Warsaw) Date: Sun, 02 Apr 2006 22:00:33 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <132501c656a7$0b7d8360$f44c2597@bagio> References: <132501c656a7$0b7d8360$f44c2597@bagio> Message-ID: <1144029633.11452.43.camel@resist.wooz.org> On Mon, 2006-04-03 at 00:44 +0200, Giovanni Bajo wrote: > Not me. Surely it'd help if its documentation explained why on earth it has > been introduced. There are no apparent advantages over the builtin %s operator, > at first sight. Nothing worth using a non-standard non-builtin substitution > engine, at least. PEP 292. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060402/a6a1046e/attachment.pgp From tim.hochberg at ieee.org Mon Apr 3 04:35:24 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Sun, 02 Apr 2006 19:35:24 -0700 Subject: [Python-3000] Adaptation: T->P vs P->P In-Reply-To: <92B6430C-65DE-4CC0-80F8-1159E9C81421@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> <4430609A.3060302@livinglogic.de> <92B6430C-65DE-4CC0-80F8-1159E9C81421@gmail.com> Message-ID: In this thread, Alex has been advocating adaption where types are adapted to protocols: T->P adaption for short. By contrast, my two sample implementations have involved Protocol->Protocol adaption where objects that implement a certain protocol are adapted to another protocol: P->P adaption for short. I suppose this also could be considered T->P->P adaptation since you look up the protocol from the type, but let's stick with the terminology P->P adaption. Alex has been advocating for adaption for quite a while (I imagine it seems like forever to him), so I give his views here great weight. Still, something about T->P adaption has been bothering me, but until now I haven't been able to put my finger on it beyond a vague sense that pressing concrete types into service in this way is asking for trouble. Here's the problem I have with T->P adaption: it increases coupling between the various clients of the adaption process. Lets talk about these clients, I believe Alex said there were four: 1. The author of the type: T 2. The writer of the adapter: A 3. The person defining the destination protocol: P 3. The user of the whole shebang: U Now under a T->P regime, T needs to search out all relevant adapters and register them for their type. Similarly when adding a new adapter, A needs to search out all relevant classes and register the new adapter for them. Thus A and T become highly coupled. On the surface, the P->P regime is not much different. In this case, when creating a new class, T needs to search out all relevant protocols and register their type with them. Similarly, A needs to search out all relevant source protocols and register the adapter. [I realize that I've artificially constrained who registers what, but the fact remains that someone has to do the registrations at some point. Another issue that I'm glossing over is that in P->P adaption there is one more actor: the author of the source protocol. I don't think that this materially affects any of the conclusions.] However, there is a huge difference in scale here. A class will typically only satisfy a few protocols, whereas it could potentially be associated with many adapters. Similarly an adapter will typically be associated with only a single source protocol, but it may be associated with many, many types. Just to make this concrete, I imagine there will be many adapters that work on every sequence type; doing all of those registrations could be quite a challenge regardless of which actor is charged with it. As another way to get a sense of the coupling here, let's define A=>B to mean "A needs to know about B". For T->P adaption: T=>A or A=>T or P=>(A and T) or U=>(A and T and P) For P->P adaption: T=>Psrc or P=>Psrc or U=>Psrc Generally A can be assumed to know about Psrc, so I've left that out. Again, these look somewhat similar on the surface, but in practice it's much more feasible for one of the actors to know about the few protocols relevant to the case in question than it is for them to know about the potentially large number of types and adapters. And that's why I'm uncomfortable with T->P adaption. I hope I've managed to present that with something resembling clarity. Regards, -tim From adam.deprince at gmail.com Mon Apr 3 04:42:20 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Sun, 02 Apr 2006 22:42:20 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: Message-ID: <1144032140.4014.55.camel@localhost.localdomain> On Sun, 2006-04-02 at 22:27 +0200, Georg Brandl wrote: > Crutcher Dunnavant wrote: > > Python currently supports 'S % X', where S is a strinng, and X is one of: > > * a sequence > > * a map > > * treated as (X,) > > > > But I have some questions about this for python 3000. > > > > 1. Shouldn't there be a format method, like S.format(), or S.fmt()? > > Possible, but why? "%" works quite fine. "Abusing" the modulus operator > is okay in this context since the symbol gives a nonoverseeable clue to > what the construct is doing. > > > 2. What about using __call__ instead of / in addition to __rmod__? > > * "time: %s"(time.ctime()) == "time: %s" % time.ctime() > > * "%(a)s %(b)s"(a=1, b=2) == "%(a)s %(b)s" % {'a'=1, 'b'=2} > Damn ugly. How would you explain to a newbie that you can _call_ a string? Carefully. You can't predict how a "newbie" might react to a language feature. Even now from a distance of some 7 years I recall with fondness my first taste of Python. My thoughts were somewhat along the lines of: """What an odd use of %. Its as if the string is an expression in some quaint language and we are evaluating it. I would think / and % would better mean: "string"/n == "string"[::n] "string"%m == \ lambda s,n:[s[k*n:(k+1)*n] for k in range( 0, len( s)/n,n )]""" Of course my transcription of the semantics would have differed somewhat at the time. IIRC, the motivation for the use of dict instead of dictionary was based on a two pronged test that I propose we henceforth call the "dictionary test." dict was selected over dictionary because the full form was harder to spell than the abbreviation was to pronounce. Format/fmt doesn't pass that, therefore I'd recommend sticking to .format. Thinking back, lambda x:"somestring"%( x,) was a common pattern until list interpolation made ["somestring"%( x, ) for x ] the more common form, so the need for a callable form is somewhat reduced. As nice as that would be, are there better uses for a callable string? Eval perhaps? S % X where X is a sequence or dict is obviously necessary, but I've always thought that the automatic rewriting of a non-sequence X as (X,) to be a bit of syntactic saccharine. 1. The single parameter version breaks when a tuple is the single object. 2. Only tuples are accepted; S % X doesn't work with other data-types. I could say tuple( my-sequence ). Deprecate the string_mod operation and replace it with .format. This would eliminate yet another obscure method of parameter passing. It would also give map users a real callable they could use: "mystring".format Cheers - Adam From adam.deprince at gmail.com Mon Apr 3 05:04:55 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Sun, 02 Apr 2006 23:04:55 -0400 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: Message-ID: <1144033496.4014.68.camel@localhost.localdomain> On Sun, 2006-04-02 at 21:39 +0000, Talin wrote: > Just a few things that have wanted while doing python > programming. Maybe some of these are already doable > or are already on someone's wish list... > > -- The __main__ module should contain a file path like imported > modules. in other words, every module should know where it > came from. In fact, I see no reason why the __main__ module > should have any different attributes than an imported module > other than the name. > > -- An easy way to make a case-insensitive, case-preserving > dict that works with regular string keys. One of my more recent abominations, my views PEP, would have addressed this nicely. And so would ... class caselessdict( dict ): def __new__( self ): retval = set.__new__( caselessdict ) retval.cC = {} retval.Cc = {} return retval def __setitem__( self, key, value ): lkey = key.lower() self.cC[lkey] = key # need for deletion self.Cc[key] = lkey dict.__setitem__( self, key, value ) def __getitem__( self, key ): lkey = key.lower() dict.__getitem__( self, lkey ) def myCapitalization( self, key ): return self.cC[key] def iteritems( self ): return (self.cC[key], values in dict.iteritems( self )) del __delitem__( self, key ): del( retval.Cc[ retval.cC[ key.lower()]]) del( retval.cC[key.lower()] ) dict.__delitem__( self, key.low..... ... sorry, I'm all out of coffee. > > -- An easy way to iterate over key, value pairs in a dict in > sorted order. Currently you have to get the list of keys, > sort them, and then lookup each value, OR you have to > get the list of tuples and call sorted() with a key= arg > containing a lambda function that extracts the first > tuple element. sorted( dict.items() ) Doesn't Zope have a nice tree to do this in-place? > > (Another one of my wild ideas was an "order by" > clause for list comprehensions, but let's not go there.) [x for x in sorted( l )] ?? From rrr at ronadam.com Mon Apr 3 03:19:29 2006 From: rrr at ronadam.com (Ron Adam) Date: Sun, 02 Apr 2006 20:19:29 -0500 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: Message-ID: Talin wrote: > -- An easy way to iterate over key, value pairs in a dict in > sorted order. Currently you have to get the list of keys, > sort them, and then lookup each value, OR you have to > get the list of tuples and call sorted() with a key= arg > containing a lambda function that extracts the first > tuple element. What you probably want here instead is an ordered dict, and not a sorted dict. It's easy enough to sort the keys, but maintaining an other than sorted order, such as the order the items were added to the dict, or the order in which you will need the items from a dict isn't as easy and requires keeping a duplicate list of the keys. > -- The module class should have a method to iterate > over child modules. Currently you can iterator through > all of its attributes, but you have to filter out which ones > are modules. Is this what you are looking for? >>> import inspect >>> import __main__ >>> inspect.getmembers(__main__, inspect.ismodule) [('__builtins__', ), ('__main__', ), ('inspect', )] This is how PyDoc does it. I just happen to be in the middle of attempting to rewrite it so it's more modular and easier to extend. ;-) Cheers, Ron From talin at acm.org Mon Apr 3 05:54:26 2006 From: talin at acm.org (Talin) Date: Mon, 3 Apr 2006 03:54:26 +0000 (UTC) Subject: [Python-3000] A few small py3k wishes References: <1144033496.4014.68.camel@localhost.localdomain> Message-ID: Adam DePrince gmail.com> writes: > One of my more recent abominations, my views PEP, would have addressed > this nicely. And so would ... > > class caselessdict( dict ): Interesting approach. I'm not sure I like storing two copies of every key thought. > sorted( dict.items() ) As I mentioned, I wasn't aware the relative comparisons worked on tuples (in fact, the only place I can find a reference to it is in the python performance tips page - but perhaps my Google queries are ill-formed.) > [x for x in sorted( l )] ?? How about: [x for x in l order (x.last_name, x.first_name)] Again, that's *not* a suggestion, just a hypothetical example. Anyway, the responses have been good - and I am sure you don't want this thread to go on any longer :) My only concern is this: Some of these issues may seem like newbie questions to you folks, but I've been programming Python for over 5 years now, I've read the various reference manuals many times, done Google searches on all these issues, browsed the ASPN cookbooks (although that may be partly due to the really bad search engine on ASPN), and yet failed to find an "obvious" way to do things. I will certainly use some of the suggestions that have been provided in this thread, but at the same time I feel that some of them are, well, rather tricky solutions to what I feel should be common problems. For example, the suggestion of testing each member of a module with "is_module" is fine - except that a newbie programmer's first attempt to write "is_module" is probably going to be "isinstance( value, module )" which won't work because the word "module" isn't bound to a type. At which point, they will have to go digging through the library docs to figure out where the standard types live, or if there's some other way to determine if an object is a module or not. Similarly, the need to understand how to construct the current module's path is concise, but it still requires a deeper understanding of how modules and packages are loaded -- I mean, what would be wrong with every module having an __abspath__ attribute or something? The same is true for the problem of a "plugins" dir containing modules to import. Yes, its only 3 lines of code to list the directory and import each file - but coming up with those exact 3 lines (in particular, getting the arguments to __import__ correct) is trickier than it looks, especially when you are trying not to hard-code the module name. What I came up with is this, which is probably incorrect: # Import all parsers dirname = os.path.dirname( __file__ ) for name in os.listdir( dirname ): if fnmatch.fnmatch( name, "*.py" ) and name != "__init__.py": mod = __import__( "%s.%s" % ( __name__, os.path.splitext( name )[ 0 ] ) ) Perhaps the answer is better docs? For example, there's a number of docs on the package / module loading mechanism, but they all seem like pieces of an incomplete puzzle to me. -- Talin From ianb at colorstudy.com Mon Apr 3 06:26:47 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 02 Apr 2006 23:26:47 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: Message-ID: <4430A407.5090502@colorstudy.com> Georg Brandl wrote: > BTW, has anyone seen string.Template being used somewhere? I use it from time to time, usually when formatting user-provided strings (because "%(foo)s" is not very pleasant or easy-to-explain compared to "$foo"). However, I *never* use it internally in code, because the overhead of importing string and creating the template far outweighs the inconvenience of %-based substitution. Oh, and I really like %r. I'd personally be very happy if $"$foo" worked, as well as "$foo".substitute(). $/shell-style substitution is the norm out in the world, not the %/printf style that Python uses; it'd be nice to move towards that norm. string.Template is a bit of a tease that way, since it doesn't actually provide a very convenient alternative to the bulk of what % is used for. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From tim.hochberg at ieee.org Mon Apr 3 06:29:09 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Sun, 02 Apr 2006 21:29:09 -0700 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: <1144033496.4014.68.camel@localhost.localdomain> Message-ID: Talin wrote: [...] > For example, the suggestion of testing each member of a module with > "is_module" is fine - except that a newbie programmer's first attempt to > write "is_module" is probably going to be "isinstance( value, module )" > which won't work because the word "module" isn't bound to a type. > At which point, they will have to go digging through the library docs to > figure out where the standard types live, or if there's some other way > to determine if an object is a module or not. Well, a general trick to find a type, assuming you can get an instance is, unsuprisingly, to use type(). For instance: import sys modtype = type(sys) def ismodule(obj): return isinstance(obj, modtype) That said, depending on your application, this might actually be better implemented, although it's a bit slower, as: def ismodule(obj): return (obj in sys.modules.values()) It's not unheard of for people to stuff things other than modules into sys.modules in order to pursue their devious ends. Probably the less said about that the better though. Anyway, finding out what is a module and what is not is a bit tricky, and depends a bit on what you are trying to do, but that seems a rather specialized application. For this kind of question you can always ask on c.l.python. I bet you'd get dozens of answers and no more than half of them would be wrong ;) [...] > > The same is true for the problem of a "plugins" dir containing modules > to import. Yes, its only 3 lines of code to list the directory and import > each file - but coming up with those exact 3 lines (in particular, getting > the arguments to __import__ correct) is trickier than it looks, especially > when you are trying not to hard-code the module name. What I came > up with is this, which is probably incorrect: > > # Import all parsers > dirname = os.path.dirname( __file__ ) > for name in os.listdir( dirname ): > if fnmatch.fnmatch( name, "*.py" ) and name != "__init__.py": > mod = __import__( "%s.%s" % ( __name__, os.path.splitext( name )[ 0 ] ) ) Yes, __import__ can be tricky. You might want to look at imp.load_source instead. I don't use that stuff enough to toss off the correct incantation off the top of my head, but I think that would likely be cleaner. > Perhaps the answer is better docs? For example, there's a number of > docs on the package / module loading mechanism, but they all seem > like pieces of an incomplete puzzle to me. There's always room for better docs or perhaps domain specific tutorials. I don't know if that's really a Python 3000 issue, doc improvement should be an ongoing process. Regards, -tim From fdrake at acm.org Mon Apr 3 06:34:35 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Mon, 3 Apr 2006 00:34:35 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144029338.11452.41.camel@resist.wooz.org> References: <1144029338.11452.41.camel@resist.wooz.org> Message-ID: <200604030034.35304.fdrake@acm.org> On Sunday 02 April 2006 21:55, Barry Warsaw wrote: > I use it, which shouldn't be a big suprise (though not in much public > code). You'll probably see it get used quite a bit in a future Mailman > release. We've been using it at Zope Corporation recently in some of our "buildout" framework. It's nice not to be rolling our own. :-) -Fred -- Fred L. Drake, Jr. From tjreedy at udel.edu Mon Apr 3 07:45:27 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 3 Apr 2006 01:45:27 -0400 Subject: [Python-3000] A few small py3k wishes References: <20060402222130.GA12075@tigger.digitaltorque.ca> Message-ID: "Michael P. Soulier" wrote in message news:20060402222130.GA12075 at tigger.digitaltorque.ca... >It's a minor thing, but it'd be nice to make a method alias for append() >on >mutable sequences called push(), to go with pop(). Bound methods to your rescue: _stack = [] push = _stack.append pop = _stack.pop Now push(i) and i = pop() to your heart's content ;-) What's ironic about your 'wish' is that making it possible to make a bound pop to go with the previously possible bound push was part of my rationale, several years ago, for proposing the addition of a pop method. Terry Jan Reedy From walter at livinglogic.de Mon Apr 3 11:21:26 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 03 Apr 2006 11:21:26 +0200 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <92B6430C-65DE-4CC0-80F8-1159E9C81421@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> <4430609A.3060302@livinglogic.de> <92B6430C-65DE-4CC0-80F8-1159E9C81421@gmail.com> Message-ID: <4430E916.8060707@livinglogic.de> Alex Martelli wrote: > On Apr 2, 2006, at 4:39 PM, Walter D?rwald wrote: > ... >> Why not make the registry identical to the protocol? The protocol is >> just a convention anyway: > > Yes, a 1<->1 relationship between registries and protocols makes the > 'registryof' function I was talking about simpler, at least when it's > implemented as identity. What's the advantage of this approach, > compared to allowing protocols to be arbitrary hashable attributes? The > disadvantage, essentially, is that one gives up the degrees of freedom > implied by protocols being "just a convention": a protocol must now > (say) be a callable object exposing a register method (with given > signature and semantics). Yes, but otherwise the adapter has to be, so this doesn't buy us much. > I guess adopting that kind of constraint is OK, as long as the > constraint doesn't in any way interfere with whatever else Guido wants > to do with protocols and thus give him a reason to reject adaptation. I > guess I can see some potential for minor optimization: a protocol that's > written with some existing types already in mind might use a subclass of > your 'Adaptor', such as: > > class FastTracker(Adaptor): > def __init__(self, types): > self.fastrack = set(types) > Adaptor.__init__(self) > def _anyof(self, types): > for t in types: > if t in self.fastrack: return True > return False > def register(self, adaptor, types): > if self._anyof(types): > # need better diagnostics here, of course > raise RegistrationError, "Cannot override identity-adaptation" > return Adaptor.register(self, adaptor, types) > def __call__(self, obj, *a, **k): > if self._anyof(type(obj).__mro__): > if a or k: > raise ...some kind of diagnostics about a/k not allowed > here... > return obj > return Adaptor.__call__(self, obj, *a, **k) > > I'm not sure the advantage is big enough to warrant all the machinery, > but I can't assert it isn't, either. OK, this is the version where you know that some types implement the protocol itself, but you can't register those types via an attribute on the types. >>> Isn't it just wonderful, how the foes of adaptation switch horses on >>> you? First they request a simple-as-dirt, bare-bones "example >>> system" -- then as soon as you provide one they come back at you >>> with all sort of "cruft" to be piled on top. >> >> I think you might be misinterpreting reactions. If the initial >> reaction was "I don't understand it. Nobody needs this." (at least >> that was my reaction), you're "strawman proposal" has put us past >> this. (At least you got two "I finally got it, this seems useful" from >> me and Brett.) > > You're quite likely right, and I apologize for the misinterpretation; I > guess I may have misread Brett's desire to hammer adaptation's very > reason for being down to miniscule smithereens by having the existence > of some set of methods "imply" identity-adaptation, as a subconscious > desire to defend against the whole idea of adaptation in the guise of > sort-of-accepting it, for example. I guess it's more an attempt to compare adaption to something we already know. I fact getattr() is the stupid version of adapation. There's only identity adaption and the protocol is a method or an attribute! ;) >> So now lets answer the questions: How do we implement adaption of >> subtypes? What is a protocol? How can we make registration as painless >> as possible? etc. > > Looks like the "loop on __mro__" idea is sort of standard for the first > of these questions. Yes, probably with a hand crafted version of a mro for classic classes. The big advantage of using the mro is that performance is independent from the number of registered adapter functions. > As for "what is a protocol", I'd rather minimize > the machinery that goes with it, but I guess that's some kind of > holdover from the "strawman proposal"; if we're making protocols into > classes anyway, I'd like to provide them with a check_compliance method, > taking an object and an integer that represents the amount of effort to > be spent in the checking -- 0 meaning 'just check existence of methods', > 1 meaning 'check methods' signature compatibility too', 2 meaning 'check > some semantics at a level suitable for fast unit-tests', and so on up. I don't see how that buys us much (And it looks like you're trying to solve the halting problem ;) class Foo: __implements__ = [Bar] def bar(self): if Bar.check_compliance(self, 2): # don't comply else: # comply > The author of a protocol doesn't have to do all that much (levels 0 and > 1 can be provided by helper functions/methods relying on the inspect > module), but he or she CAN make the protocol into "an executable > specification" at whatever level he or she desires. I'm really not sure how this would look in practice. > Other error checking > (e.g., against registration of multiple adapters for one protocol/type > pair) as well as some modest optimization (mostly of adaptation, the > frequent case) may also be warranted. > > I'm not sure what's painful at all about registration -- at worst it's > one call per type/protocol pair, Exactly, especially for the case where we already have this adaption machinery today (i.e. copy_reg). > some refactoring such as yours making > it even slighter by allowing the registration of several pairs in one > call. Do you mean that people are so keen on declarative styles that > they'd crave, e.g., some special machinery such as > > class Duck(object): > __implements__ = walk, talk, quack > ... > > (with the metaclass doing the registration calls, with identity > adaptation, under the cover for each protocol in __implements__), > prizing it highly over explicit: There's no need for registration calls in this case. The protocol/adapter can do it: class Adapter(object): # __init__() and register() as before. def __call__(self, obj, *args, **kwargs): objtype = type(obj) if hasattr(objtype, "__implements__") and self in objtype.__implements__: return obj # rest of the code as before > class Duck(object): > ... > adapt.register(Duck, (walk, talk, quack), identity) > > or the loop or classdecorator version of the latter? Ah well, maybe, > but most of the discussion sort of smacks to me of AGNI and/or premature > optimization. > > Could we agree that simplicity and even minimalism (with a side order of > future extensibility if/when warranted) are high on the list of desired > properties for a protocol adaptation mechanism? Absolutely! > After all, the more > machinery, cruft, and black magic we pile on top of adaptation's simple > core ideas, the more likely the whole caboodle is to provoke allergic > reactions in anybody who's not being a part of this discussion... True. Make it simple enough to cover 80% of the cases and be understandable, but extensible enough, so we can add anything that might come up in the future without having to contort the adaption protocol so much that it's no longer recognizable. From a higher point of view, this is one of the few occasions where a Design Pattern makes sense in Python. Usually this means that the language isn't high level enough (nobody implements the "Factory Function Pattern" in Python because classes are callable). If Python had multi methods we wouldn't be discussing adaption, because basically what we're discussing here is how to implement double dispatch. Bye, Walter D?rwald From ncoghlan at gmail.com Mon Apr 3 11:56:08 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 03 Apr 2006 19:56:08 +1000 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> Message-ID: <4430F138.5090205@gmail.com> Alex Martelli wrote: > Do we have a "good practice document" about what you should or shouldn't > do with metaclasses, or decorators, or, for that matter, with > inheritance, operator overloading, and other powerful constructs and > tools that have been with Python a long, long time? Fair cop - I can accept that adaptation really isn't any more dangerous than some of Python's existing magic mechanisms, and abuse can be slapped down the same way (i.e. peer review). It may not make your head hurt less, but I really think you're making cracks in that wall of incomprehension this time around ;) > I don't think this other tweak would be a _big_ "bother", but neither > would it be at all useful, just a medium-level useless bother. > > Say, for example, that protocols are identified (as in my strawman > proposal) by unique strings anyway. E.g., if I were to invent a > protocol, I could name it 'it.aleax.myprot' -- since I own the aleax.it > domain, nobody else could create a name conflict. But if everbody is using a global registry, this still creates the need for a (new) globally adopted convention for avoiding name conflicts, be it "use reverse-DNS naming" or "always use unique objects with identity based equality". My point is that instead of creating a new convention to avoid namespace conflicts for protocols, we should try to reuse an existing one - the Python import mechanism. Starting from that perspective, I *really* like Walter's suggestion of using the protocol itself to store a per-protocol adapter registry. Protocols are identified unambiguously (using the Python import namespace), and once you have access to the protocol object there's no extra work to find its registry. > I may be missing something here, I guess, because I just don't see the > point. Walter's suggestion was what I was groping towards, but hadn't reached yet (my next suggestion was going to be that a protocol should know where its registry lived). As for why I cared in the first place, when I see a suggestion to add more interpreter global state I want to see a *really*, *really* good justification for it. In this case, it makes far more sense to me to move the adapter registration out to the individual protocols. > Isn't it just wonderful, how the foes of adaptation switch horses on > you? First they request a simple-as-dirt, bare-bones "example system" > -- then as soon as you provide one they come back at you with all sort > of "cruft" to be piled on top. 99% of my resistance stemmed from my objection to the global protocol registry that added the need for an entirely new convention for avoiding name conflicts between different registered protocols. If we can pursue an approach that pushes adaptation registration out to the protocols themselves, then I am 100% in favour of the idea :) In a glorious fit of self-referentiality, one of the first things needed by such an approach would be a protocol for the protocol interface that allowed other protocol objects to register themselves as implementing it. This would then allow the ever popular generic adaptation function to be written as: def adapt(obj, target_protocol): return Protocol.adapt(target_protocol).adapt(obj) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From rasky at develer.com Mon Apr 3 12:06:32 2006 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 3 Apr 2006 12:06:32 +0200 Subject: [Python-3000] String formating operations in python 3k References: <132501c656a7$0b7d8360$f44c2597@bagio> <1144029633.11452.43.camel@resist.wooz.org> Message-ID: <13de01c65706$48d42200$f44c2597@bagio> >> Not me. Surely it'd help if its documentation explained why on earth it has >> been introduced. There are no apparent advantages over the builtin %s operator, >> at first sight. Nothing worth using a non-standard non-builtin substitution >> engine, at least. > PEP 292. Yes, I said "documentation" for a reason. Since when people have to go looking in PEPs for rationale of a module? string.Template introduces a very clear TOOWTDI conflict, I think the documentation should mention and try to explain it. For instance, the documentation for Decimal is pretty clear as to when Decimal is to be used and how it is different from float. Giovanni Bajo From ncoghlan at gmail.com Mon Apr 3 12:52:05 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 03 Apr 2006 20:52:05 +1000 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: <1144033496.4014.68.camel@localhost.localdomain> Message-ID: <4430FE55.8090407@gmail.com> Talin wrote: > The same is true for the problem of a "plugins" dir containing modules > to import. Yes, its only 3 lines of code to list the directory and import > each file - but coming up with those exact 3 lines (in particular, getting > the arguments to __import__ correct) is trickier than it looks, especially > when you are trying not to hard-code the module name. What I came > up with is this, which is probably incorrect: > > # Import all parsers > dirname = os.path.dirname( __file__ ) > for name in os.listdir( dirname ): > if fnmatch.fnmatch( name, "*.py" ) and name != "__init__.py": > mod = __import__( "%s.%s" % ( __name__, os.path.splitext( name )[ 0 ] ) ) > > Perhaps the answer is better docs? For example, there's a number of > docs on the package / module loading mechanism, but they all seem > like pieces of an incomplete puzzle to me. I think this is a fair cop, but its not really a Py3k problem. At the moment, Python's import mechanism is only properly documented in PEP 302, and the answer to most of your questions is that "assuming your module is on a real filesystem may not be a safe assumption". But the standard import mechanism doesn't provide the __loader__ attributes that would let you get around that problem in a consistent fashion. Until PEP 302 itself is completely implemented, I can't really see these kinds of niceties being added. At this point, I don't even know how thoroughly the integration of PEP 302 and PEP 328 (absolute/relative imports) has been tested. . . Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From barry at python.org Mon Apr 3 14:00:15 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 03 Apr 2006 08:00:15 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4430A407.5090502@colorstudy.com> References: <4430A407.5090502@colorstudy.com> Message-ID: <1144065615.11451.53.camel@resist.wooz.org> On Sun, 2006-04-02 at 23:26 -0500, Ian Bicking wrote: > Georg Brandl wrote: > > BTW, has anyone seen string.Template being used somewhere? > > I use it from time to time, usually when formatting user-provided > strings (because "%(foo)s" is not very pleasant or easy-to-explain > compared to "$foo"). However, I *never* use it internally in code, > because the overhead of importing string and creating the template far > outweighs the inconvenience of %-based substitution. Oh, and I really > like %r. Good point. Yes string.Template was designed primarily for i18n applications for the reasons above, and because %(foo)s is more error prone than necessary. > I'd personally be very happy if $"$foo" worked, as well as > "$foo".substitute(). $/shell-style substitution is the norm out in the > world, not the %/printf style that Python uses; it'd be nice to move > towards that norm. string.Template is a bit of a tease that way, since > it doesn't actually provide a very convenient alternative to the bulk of > what % is used for. I don't much like the $"" prefix, but I agree that it would be nicer if there were more direct support for $-strings. OTOH, I don't see a good way to marry the rich coercion of %-substitution with the simplicity of $-substition. I wouldn't want to lose that simplicity to gain that richness in $-strings, so I suppose a prefix might be necessary. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060403/9338013d/attachment.pgp From barry at python.org Mon Apr 3 15:31:34 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 03 Apr 2006 09:31:34 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <13de01c65706$48d42200$f44c2597@bagio> References: <132501c656a7$0b7d8360$f44c2597@bagio> <1144029633.11452.43.camel@resist.wooz.org> <13de01c65706$48d42200$f44c2597@bagio> Message-ID: <1144071094.11459.63.camel@resist.wooz.org> On Mon, 2006-04-03 at 12:06 +0200, Giovanni Bajo wrote: > Yes, I said "documentation" for a reason. Since when people have to go looking > in PEPs for rationale of a module? string.Template introduces a very clear > TOOWTDI conflict, I think the documentation should mention and try to explain > it. For instance, the documentation for Decimal is pretty clear as to when > Decimal is to be used and how it is different from float. There's a link to PEP 292 in the documentation, but if you feel that's not enough, feel free to submit a patch. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060403/166632b4/attachment.pgp From ncoghlan at gmail.com Mon Apr 3 15:36:53 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 03 Apr 2006 23:36:53 +1000 Subject: [Python-3000] Adaptation: T->P vs P->P In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> <4430609A.3060302@livinglogic.de> <92B6430C-65DE-4CC0-80F8-1159E9C81421@gmail.com> Message-ID: <443124F5.6070404@gmail.com> FWIW, I went back and read PEP 246. That PEP goes to great lengths to permit arbitrary objects to be used as protocols, with the process of checking for compliance being a bit of a tap dance back and forth between the object and the protocol. Great effort was also put into registering adaptability without mutating any attributes of the protocol object. The current discussion has persuaded me that it's better to require that protocols be special types, themselves providing a particular interface for adaptation purposes, and having ultimate authority over what it means to be compliant to themselves (thus allowing, e.g. an IIndex protocol to consider anything with a zero-argument __index__ method to be compliant - the act of defining a special interpreter method is a fair indication that the class considers itself compliant with that protocol). The fact that PyProtocols, Twisted and Zope all use special objects to define their interfaces is also a factor in forming that opinion. The PyProtocols docs mention the possibility of using arbitrary objects, and then points out that doing so means you miss out on most of the benefits of PyProtocols. Yet the usability of the entire API suffers for the sake of that genericity. So I'd be inclined to make the interface for a protocol reasonably fat by including things like adapter and implementation registration as methods. With such an approach (and class decorators), it would be straightforward to write: class Duck(object): """The duck class""" @class implements(IDuckWalk, IDuckQuack, IDuckLook) # ... where implements works something like: def implements(*interfaces): def register_class(cls): for interface in interfaces: interface.register_type(cls) return register_class The protocol itself can preregister known good classes in its class definition. Third parties that know a particular class implements a particular protocol can simply register that class directly. Tim Hochberg wrote: > In this thread, Alex has been advocating adaption where types are > adapted to protocols: T->P adaption for short. By contrast, my two > sample implementations have involved Protocol->Protocol adaption where > objects that implement a certain protocol are adapted to another > protocol: P->P adaption for short. I suppose this also could be > considered T->P->P adaptation since you look up the protocol from the > type, but let's stick with the terminology P->P adaption. > > Alex has been advocating for adaption for quite a while (I imagine it > seems like forever to him), so I give his views here great weight. > Still, something about T->P adaption has been bothering me, but until > now I haven't been able to put my finger on it beyond a vague sense that > pressing concrete types into service in this way is asking for trouble. > > Here's the problem I have with T->P adaption: it increases coupling > between the various clients of the adaption process. Lets talk about > these clients, I believe Alex said there were four: > > 1. The author of the type: T > 2. The writer of the adapter: A > 3. The person defining the destination protocol: P > 3. The user of the whole shebang: U > > Now under a T->P regime, T needs to search out all relevant adapters and > register them for their type. Similarly when adding a new adapter, A > needs to search out all relevant classes and register the new adapter > for them. Thus A and T become highly coupled. This misses the whole point of dynamic adaptation. T and P might define a few convenience adaptations (e.g. to or from standard library interfaces), but A and U will usually be the same person. Suppose framework X produces a Wibble, and framework Y expects an IWobble in various places. The integrator (U) needs to plug them together. If Wibble provides the right interface, U can simply write: IWobble.register_type(Wibble) Or, more commonly, U may need to write an adapter: class WibbleAsIWobble(object): def __init__(self, the_wibble): self.the_wibble = the_wibble # delegate the IWobble API to the Wibble instance IWobble.register_type_adapter(Wibble, WibbleAsIWobble) Either way, after either conformance or the adapter have been registered, a Wibble can be used seamlessly anywhere an IWobble was expected. On a completely different note, now that I've researched things a bit further, I think PyProtocols really has done a good job thrashing out the issues associated with adaptation. The *only* thing I really disagree with in PyProtocols is that I think the API usability has suffered due to the fact that state is spread out amongst not only protocol objects, but is also sometimes stored on the types being adapted. This leads to all sorts of "well sometimes you can do this, but sometimes you can't, depending on what you're adapting, so maybe you shouldn't rely on it" caveats. If the protocol API itself was richer (mandating that protocol objects support dynamic registration), then the API could be simplified significantly (as most operations would simply become method invocations on the target protocol, and the source objects could be whatever you wanted, since the state would all be on the protocol objects). Regards, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Mon Apr 3 15:43:17 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 03 Apr 2006 23:43:17 +1000 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144065615.11451.53.camel@resist.wooz.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> Message-ID: <44312675.7060702@gmail.com> Barry Warsaw wrote: > On Sun, 2006-04-02 at 23:26 -0500, Ian Bicking wrote: >> Georg Brandl wrote: >>> BTW, has anyone seen string.Template being used somewhere? >> I use it from time to time, usually when formatting user-provided >> strings (because "%(foo)s" is not very pleasant or easy-to-explain >> compared to "$foo"). However, I *never* use it internally in code, >> because the overhead of importing string and creating the template far >> outweighs the inconvenience of %-based substitution. Oh, and I really >> like %r. > > Good point. Yes string.Template was designed primarily for i18n > applications for the reasons above, and because %(foo)s is more error > prone than necessary. > >> I'd personally be very happy if $"$foo" worked, as well as >> "$foo".substitute(). $/shell-style substitution is the norm out in the >> world, not the %/printf style that Python uses; it'd be nice to move >> towards that norm. string.Template is a bit of a tease that way, since >> it doesn't actually provide a very convenient alternative to the bulk of >> what % is used for. > > I don't much like the $"" prefix, but I agree that it would be nicer if > there were more direct support for $-strings. OTOH, I don't see a good > way to marry the rich coercion of %-substitution with the simplicity of > $-substition. I wouldn't want to lose that simplicity to gain that > richness in $-strings, so I suppose a prefix might be necessary. What do you think of a "format" builtin function that accepts the format as the first argument (similar to printf). The version on my harddrive permits positional arguments via $1, $2, etc, as well as string formatting (by sticking the format code in square brackets between the $ and the identifier). Keyword arguments still work, naturally. And if you don't want formatting, you just leave out the square brackets. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From jimjjewett at gmail.com Mon Apr 3 16:12:09 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 3 Apr 2006 10:12:09 -0400 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> Message-ID: On 4/1/06, Alex Martelli wrote: > [Monkeypatching] may just be impossible for ``classes'' that > are actually types implemented in C, as is the case for gmpy. > ... Rather, look (e.g.) at copy_reg for the typical kludge > that's used to reimplement that crucial design pattern > that is adaptation, over and over and over again. Right -- as an implementation issue, monkeypatching an extension class is ugly, and requires cooperation from the caller. As I see it, there are three possibilities for adaptation: (1) The callee helps explicitly. (In which case, why not just do it directly?) (2) The caller helps explicitly. This is the equivalent of the copy_reg hack. (Except that there might be single registry for all adaptations, instead of separate ones for each piece of functionality. Is that really the key advantage you're looking for?) (3) Neither the callee nor the caller has to change; if there are problems, the system knows to check a system registry for 3rd-party adaptors. This is the (less efficient) equivalent of opening all classes to monkeypatching. (As you mention, this is what Ruby does, but would be a big change for python.) -jJ From barry at python.org Mon Apr 3 16:55:54 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 03 Apr 2006 10:55:54 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44312675.7060702@gmail.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> Message-ID: <1144076154.29376.8.camel@resist.wooz.org> On Mon, 2006-04-03 at 23:43 +1000, Nick Coghlan wrote: > What do you think of a "format" builtin function that accepts the format as > the first argument (similar to printf). > > The version on my harddrive permits positional arguments via $1, $2, etc, as > well as string formatting (by sticking the format code in square brackets > between the $ and the identifier). Keyword arguments still work, naturally. > > And if you don't want formatting, you just leave out the square brackets. I'm not totally sure I would need a builtin. If I look at the two sources of $-strings in an app like Mailman, I'll see 1) literal human readable/translatable strings in the source code, 2) human entered strings that come from a web form. In the first case, all string formatting will funnel through one bottleneck function, which will do the catalog lookup, frame variable discovery, and substitution all in one fell swoop. So there, the builtin doesn't buy you much convenience. In the second case, it's possible that it could help you a bit, but even there, I don't think a builtin would buy you much clarity or simplicity. I'm willing to believe that my uses are fairly narrow and that such a builtin could be useful for other applications, but I don't see too much of a benefit for the code I've written. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060403/dd6a5927/attachment.pgp From ian.bollinger at gmail.com Mon Apr 3 18:05:37 2006 From: ian.bollinger at gmail.com (Ian D. Bollinger) Date: Mon, 3 Apr 2006 12:05:37 -0400 Subject: [Python-3000] It's a statement! It's a function! It's BOTH! In-Reply-To: References: Message-ID: <86a6a97b0604030905if8579d8pc9216b18ab08286e@mail.gmail.com> On 4/1/06, Talin wrote: > > This is about the print / writeln debate. > A somewhat tangental point, but I hope it's been decided that any function would be named writeline rather than writeln. I know writeln is a common function name, but abbreviations irritate me. Also, it might be possible that someone would read it as writeIn (capital "eye"), though that's probably silly. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060403/8a9c971c/attachment.html From ianb at colorstudy.com Mon Apr 3 18:13:28 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 03 Apr 2006 11:13:28 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144076154.29376.8.camel@resist.wooz.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> Message-ID: <443149A8.4010401@colorstudy.com> Barry Warsaw wrote: > On Mon, 2006-04-03 at 23:43 +1000, Nick Coghlan wrote: > > >>What do you think of a "format" builtin function that accepts the format as >>the first argument (similar to printf). >> >>The version on my harddrive permits positional arguments via $1, $2, etc, as >>well as string formatting (by sticking the format code in square brackets >>between the $ and the identifier). Keyword arguments still work, naturally. >> >>And if you don't want formatting, you just leave out the square brackets. > > > I'm not totally sure I would need a builtin. If I look at the two > sources of $-strings in an app like Mailman, I'll see 1) literal human > readable/translatable strings in the source code, 2) human entered > strings that come from a web form. > > In the first case, all string formatting will funnel through one > bottleneck function, which will do the catalog lookup, frame variable > discovery, and substitution all in one fell swoop. So there, the > builtin doesn't buy you much convenience. Well, error messages are a common place I use %. So: assert path.startswith(prefix), ( "%r should start with %r" % (path, prefix)) assert path.startswith(prefix), ( $"${repr(path)} should start with ${repr(prefix)}") assert path.startswith(prefix), ( "$path should start with $prefix".substitute( path=repr(path), prefix=repr(prefix)) The second example assumes that you can include full expressions, otherwise that example would start looking really unpleasant. As it is, the first example still looks more-or-less the best, and it's too bad $-based substitution doesn't include it. I'm not sure how it would include it, unless there was something like $=path, or $, or $`path` or something. I dunno, none of those are very appealing. str.substitute isn't too bad, except that "substitute" feels a little long-winded (compared to, say, str.format). If "$path".format() worked, that'd be nice, but even though it's not that hard to implement, it breaks a lot of expectations. Anyway, that's one example. Any non-i18n'd code will likely have a fair amount of string substitution which isn't otherwise wrapped in any routines. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From tim.hochberg at ieee.org Mon Apr 3 18:14:03 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon, 03 Apr 2006 09:14:03 -0700 Subject: [Python-3000] Adaptation: T->P vs P->P In-Reply-To: <443124F5.6070404@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> <4430609A.3060302@livinglogic.de> <92B6430C-65DE-4CC0-80F8-1159E9C81421@gmail.com> <443124F5.6070404@gmail.com> Message-ID: <443149CB.7000000@ieee.org> Nick Coghlan wrote: [SNIP] > > Tim Hochberg wrote: > >>In this thread, Alex has been advocating adaption where types are >>adapted to protocols: T->P adaption for short. By contrast, my two >>sample implementations have involved Protocol->Protocol adaption where >>objects that implement a certain protocol are adapted to another >>protocol: P->P adaption for short. I suppose this also could be >>considered T->P->P adaptation since you look up the protocol from the >>type, but let's stick with the terminology P->P adaption. >> >>Alex has been advocating for adaption for quite a while (I imagine it >>seems like forever to him), so I give his views here great weight. >>Still, something about T->P adaption has been bothering me, but until >>now I haven't been able to put my finger on it beyond a vague sense that >>pressing concrete types into service in this way is asking for trouble. >> >>Here's the problem I have with T->P adaption: it increases coupling >>between the various clients of the adaption process. Lets talk about >>these clients, I believe Alex said there were four: >> >>1. The author of the type: T >>2. The writer of the adapter: A >>3. The person defining the destination protocol: P >>3. The user of the whole shebang: U >> >>Now under a T->P regime, T needs to search out all relevant adapters and >>register them for their type. Similarly when adding a new adapter, A >>needs to search out all relevant classes and register the new adapter >>for them. Thus A and T become highly coupled. > > > This misses the whole point of dynamic adaptation. You dropped this bit of context: [I realize that I've artificially constrained who registers what, but the fact remains that someone has to do the registrations at some point. ...] > T and P might define a few > convenience adaptations (e.g. to or from standard library interfaces), but A > and U will usually be the same person. I was simply attempting to copy Alex here. I more or less agree with you. Or more precisely, I agree that A will be a U. There may be other users that come after that also use A's work. But I'm muddying up the jargon here, so I'll stop. Suppose framework X produces a Wibble, > and framework Y expects an IWobble in various places. The integrator (U) needs > to plug them together. If Wibble provides the right interface, U can simply write: > > IWobble.register_type(Wibble) > > Or, more commonly, U may need to write an adapter: > > class WibbleAsIWobble(object): > def __init__(self, the_wibble): > self.the_wibble = the_wibble > # delegate the IWobble API to the Wibble instance > > IWobble.register_type_adapter(Wibble, WibbleAsIWobble) > > Either way, after either conformance or the adapter have been registered, a > Wibble can be used seamlessly anywhere an IWobble was expected. Suppose instead that the interface of framework X claims that it produces a sequence, or a file-like-object or even a WibbleLike object. Then U/A needs to dig into the guts of framework X to determine what concrete types it actually produces in order to register them for their adapter. Then, when A/U downloads the new spiffy version of framework X that now can also produce the new WibbleLike object SuperSpiffyWibble, things break until A/U figures out what new types can be produced and updates things accordingly. You can work around this if all the WibbleLike objects in framework X inherit from some base class WibbleBase (assuming that you are walking the MRO), but tying protocols to classes like that is not really the python way. And could be a pain in the neck, particularly if some of the types involved are implemented in C. With P->P addaption, the situation is better. The author of framework X can simply document that it returns something that satisfies the WibbleLike protocol. Then the most that A/U ever needs to do, regardless of how many differently implemented Wibbles the frameworks spits out is: adaption.register_adapter(WibbleLike, WobbleLike, WibbleAsWobble) Pep 246 and PyProtocols don't seem to implement either P->P or T->P adaption. That's because the __conform__ method can do pretty much anything it wants -- the other portions of PEP 246 appear to be pretty much vanilla T->P. In a sense, P->P adaption is an attempt to come closer to the power of PEP 246 without the complexity of the arcane identity/conform/adapt/registry dance. Zope adapation appears to be P->P, although it can also do T->P. In general, it should be trivial to have a P->P setup also do T->P adaption. Regards, -tim From ian.bollinger at gmail.com Mon Apr 3 18:19:45 2006 From: ian.bollinger at gmail.com (Ian D. Bollinger) Date: Mon, 3 Apr 2006 12:19:45 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144076154.29376.8.camel@resist.wooz.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> Message-ID: <86a6a97b0604030919n3ef34081s601f18084acd70f5@mail.gmail.com> Aren't a lot of the string formatting operations superfluous as %s does the work that type-specific operations (%i, etc) do? However, I guess I don't see how to merge the other formatting operations with $-string substitution. Could the other formatting operations be added to the str() function as an additional argument? Although I don't particularly like that idea, and I think I remember someone else making that suggestion and it not panning out very well. -- - Ian D. Bollinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060403/d0cb8ab3/attachment.htm From barry at python.org Mon Apr 3 18:59:56 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 03 Apr 2006 12:59:56 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443149A8.4010401@colorstudy.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> Message-ID: <1144083596.29355.40.camel@resist.wooz.org> On Mon, 2006-04-03 at 11:13 -0500, Ian Bicking wrote: > assert path.startswith(prefix), ( > "%r should start with %r" % (path, prefix)) > assert path.startswith(prefix), ( > $"${repr(path)} should start with ${repr(prefix)}") > assert path.startswith(prefix), ( > "$path should start with $prefix".substitute( > path=repr(path), prefix=repr(prefix)) The second example is why I added sys._getframe(). In my i18n code, I wanted to do things like this (rewritten for $-string style): print _("$user is not a member of the $listname mailing list") and it would Just Work. Now, the trick is that in the same function, just before the print line, you'd have code like this (pseudocoded): user = utils.websafe(form.get('user')) listname = mlist.get_name() So _() does a sys._getframe() to dig the locals out of the calling function, and does the substitutions from there. So, if your applications isn't i18n, I could see the use for a format() global, but while I'd probably want the behavior I just described, you might not, so I'm not sure we could write a format() builtin that would be useful to us both. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060403/84dbfb43/attachment.pgp From mcherm at mcherm.com Mon Apr 3 19:18:14 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Mon, 03 Apr 2006 10:18:14 -0700 Subject: [Python-3000] Adaptation: T->P vs P->P Message-ID: <20060403101814.4vme5g95gogskckw@login.werra.lunarpages.com> Tim Hochberg writes: > In this thread, Alex has been advocating adaption where types are > adapted to protocols: T->P adaption for short. By contrast, my two > sample implementations have involved Protocol->Protocol adaption > [...] P->P adaption for short. > However, there is a huge difference in scale here. A class will > typically only satisfy a few protocols, whereas it could potentially be > associated with many adapters. Similarly an adapter will typically be > associated with only a single source protocol, but it may be associated > with many, many types. Just to make this concrete, I imagine there will > be many adapters that work on every sequence type I may be misinterpreting what Tim is saying here, but it sounds like this touches on the idea of *transitivity* in adaptation. That is, Tim is proposing that there are a small number of "protocols", but I can't tell the difference between a protocol and an adaption target so I am taking them to be synonomous. I am (and have been for nearly 2 years... Alex: your previous efforts at evangalism have not been *totally* useless) a fan of adaption. But I am *deeply* concerned about introducing transitive adaption where it is assumed that registering ((A B) f) and ((B C) g) implies ((A C) lambda x: f(g(x)) I have heard it argued previously (I apologize, but I forget who said this) that if f *really* adapts A to B and g *really* adapts B to C, then their composition must really adapt A to C, and I understand the mathematical inevitability behind this statement. But I fear that in the real world there are often subtle distinctions and corner cases that are ignored for the sake of practicality which make such transitive adaption dangerous. After all, a substantial portion of Python classes fail true to achieve true Liskov substitutability despite this being a widely-understood concept. Okay... time for me to stop attacking a position (pro transitivity) that no one has explicitly taken in this discussion and return to what Tim was saying. He claims: > let's define A=>B to > mean "A needs to know about B". For T->P adaption: > T=>A or A=>T or P=>(A and T) or U=>(A and T and P) > For P->P adaption: > T=>Psrc or P=>Psrc or U=>Psrc > Generally A can be assumed to know about Psrc, so I've left that out. Tim's proposal (if I understand it correctly) depends on transitivity -- his idea is to have the class authors register as satisfying a small number of widely-known protocols (Psrc in his case), then use transitivity to infer ways to satisfy less well-known protocols. This is a tempting game... it avoids having to have even a third party know about both the type and the protocol to be satisfied. But I would rather FORCE someone (even a third party) to at least glance at the specific case and reassure us that it works. If the individual wants to do a quick and sloppy job rather than a careful one, then it is reduced to a question of whose libraries I trust, which I a problem I understand how to handle. To return to the example that we used previously in this thread, it seems that creating an __index__ special method would not be necessary if adaption were in place... we could instead ask classes to register an adapter to the "python.as_index" protocol. But if we were to infer that anything satisfying "python.as_int" necessarily satisfied "python.as_index", then we would have defeated the purpose of the feature. One last point: it is entirely possible that I have misunderstood completely. If so, please gently explain why. -- Michael Chermside From ianb at colorstudy.com Mon Apr 3 19:25:29 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 03 Apr 2006 12:25:29 -0500 Subject: [Python-3000] Generic functions Message-ID: <44315A89.9000104@colorstudy.com> As an alternative to adaptation, I'd like to propose generic functions. I think they play much the same role, except they are much simpler to use and think about. Though RuleDispatch offers considerably more features through predicate dispatch, it would probably be best to just consider type based dispatch, as that's more equivalent to adaptation. So, the copy_reg module is one example where adaptation has been proposed. The idea is that you adapt an object to a pickleable object, or something along those lines. It's a little vague, because while you typically adapt an instance coming in, you "adapt" a string to a new instance on the way out. Or, I dunno, it's not clear to me. In fact, though that's the typical example, I'm going to bail on that because the pickling protocol is an aside to this and too complex for me to digest right now. pprint is a lot easier, and conveniently is much nicer with generic functions than adaptation ;) Anyway, pprint could work like: class PrettyPrinter: @generic def pformat(self, object): """Return the pretty string representation of object""" return repr(object) # pformat is now "more" than just a function, it's a callable # object that does type-based dispatch using an internal registery # of implementations, with the implementation above as the fallback. # It also now can be used as a decorator that registers implementations: @PrettyPrinter.pformat.when(object=list) def pformat_list(self, object): s = '[' for item in object: s += (' '*self.indent) + self.pformat(item) + ',\n' return s + (' '*self.indent) + ']' Some things to note: * There's no interface created here. There's no hidden interface lurking in the background either. * It requires cooperation from the original function (pformat -- I'm using "function" and "method" interchangably). It does not require any cooperation from classes like list, similar to adaptation and dissimilar to magic methods. Adaptation also requires cooperation from the caller, as the adaptation would be applied inside pformat. * The function is mostly self-describing. If it has certain return values, then you state what those values are in documentation; there's no need to be formal about it. In contrast you have to come up with a whole collection of interfaces to start using adaptation. * The function is the hub of all the registration, which seems very natural, since you are extending the function. * Like adaptation, you must import a module that defines extra specializations of the generic function before those are active (just like you have to import adapter declarations). This strikes me as a significant problem. I assume ZCML addresses this, but I also assume that's not a reasonable solution for core Python. * Magic methods do *not* have this import problem, because once you have an object you have all its methods, including magic methods. RuleDispatch has a bunch more features than just simple type-based generic functions. But I think that type-based generic functions would be an easier or more comfortable place to start, and wouldn't preclude a more featureful implementation later. Type-based generic functions and adaptation are more-or-less equivalent. That is, you can express one in terms of the other, at least functionally if not syntactically. If you really wanted adaptation, then the interface becomes a things-obeying-this-interface factory -- i.e., a generic function. Generic functions are similer to multi-adaptaters, where you adapt a tuple of objects, similar to the tuple of arguments to a function. This is technically like generic functions, but syntactically rather awkward. [Predicate-based dispatching goes considerably further, allowing real duck typing, e.g., you could implement a pformat method for everything that has an "__iter__" method and no "next" method (i.e., all iterables, but not iterators which could lead to unintentionally consuming the iterator).] Anyway, I think generic functions are very compatible with Python syntax and style, and Python's greater emphasis on what an object or function can *do*, as opposed to what an object *is*, as well as the use of functions instead of methods for many operations. People sometimes see the use of functions instead of methods in Python as a weakness; I think generic functions turns that into a real strength. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From crutcher at gmail.com Mon Apr 3 19:44:38 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Mon, 3 Apr 2006 10:44:38 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144083596.29355.40.camel@resist.wooz.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> Message-ID: On 4/3/06, Barry Warsaw wrote > On Mon, 2006-04-03 at 11:13 -0500, Ian Bicking wrote: > > > assert path.startswith(prefix), ( > > "%r should start with %r" % (path, prefix)) > > assert path.startswith(prefix), ( > > $"${repr(path)} should start with ${repr(prefix)}") > > assert path.startswith(prefix), ( > > "$path should start with $prefix".substitute( > > path=repr(path), prefix=repr(prefix)) > > The second example is why I added sys._getframe(). In my i18n code, I > wanted to do things like this (rewritten for $-string style): > > print _("$user is not a member of the $listname mailing list") > > and it would Just Work. Now, the trick is that in the same function, > just before the print line, you'd have code like this (pseudocoded): > > user = utils.websafe(form.get('user')) > listname = mlist.get_name() > > So _() does a sys._getframe() to dig the locals out of the calling > function, and does the substitutions from there. So, if your > applications isn't i18n, I could see the use for a format() global, but > while I'd probably want the behavior I just described, you might not, so > I'm not sure we could write a format() builtin that would be useful to > us both. > Well, what if we added '%{expression}s' as a formating type? This would make your example: print _("%{user}s is not a member of the %{listname}s mailing list") or even: print _("%{utils.websafe(form.get('user'))}s is not a member of the %{mlist.get_name()}s mailing list") > -Barry > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.2.2 (GNU/Linux) > > iQCVAwUARDFUjHEjvBPtnXfVAQJsWQP/Rk7WNMv3aduLhjVbRApgcpjv78Cv0gXG > M1096pL9J2jvj29D/pb9SeHTlGG/cZTNoRNsFHUYJ8yMHT57VUiPcuz7VIWCeqPP > 4RaMP9AvdGufFULfGmT13mKTSZ0EG33f8aq74lf4GEUnmyWa/p2goUoe/lBxKwSP > u9fjNwBxRrg= > =mUlb > -----END PGP SIGNATURE----- > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/crutcher%40gmail.com > > > -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From jimjjewett at gmail.com Mon Apr 3 19:48:03 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 3 Apr 2006 13:48:03 -0400 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> Message-ID: Based on later discussion, it sounds like Alex sees adaptation as (what I called case 2) -- The caller must make an explict call to adapt. The advantage of adaption is that somebody other than the caller or callee can fill in the appropriate code __adapt__ code. Is this much correct? If so, I start to see why it hasn't caught on -- it is too wordy, and arguably too inefficient for the common case. When I'm writing a new class, I'm already annoyed by translation functions such as: def __index__(self): return long(self).__index__() but the payoff is that I can write seq[myinst] in calling classes. At this point, adaptation sounds like I would have to write (or count on a 3rd-party to write a more cumbersome version of) @implements(IIndex) def __index__(self): return long(self).__index__() Meanwhile, callers would have to write seq[IIndex(myinst)] If we were talking only a few large or obscure protocols, this would be reasonable. If it gets used very often, it will start polluting code with boilerpate. If the calling code were moved beneath the covers (so that __getitem__ made the IIndex cast, instead of my code), that would help. If direct protocol implementations could be auto-discovered, that would also help, but then we're back to the original question of what adaptors really add over the present ad-hoc duck typing. Perhaps the default metaclass could autodiscover a few well-known or legacy protocols, such as __index__, __iter__, and copy_reg? (And then you're back to having trouble finding a motivating example...) -jJ From barry at python.org Mon Apr 3 20:01:07 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 03 Apr 2006 14:01:07 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> Message-ID: <1144087267.29376.57.camel@resist.wooz.org> On Mon, 2006-04-03 at 10:44 -0700, Crutcher Dunnavant wrote: > Well, what if we added '%{expression}s' as a formating type? > This would make your example: > > print _("%{user}s is not a member of the %{listname}s mailing list") > > or even: > > print _("%{utils.websafe(form.get('user'))}s is not a member of the > %{mlist.get_name()}s mailing list") I wouldn't use it because it would be way too easy for translators to make catastrophic mistakes. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060403/3a7c4c90/attachment.pgp From tim.hochberg at cox.net Mon Apr 3 19:47:47 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Mon, 03 Apr 2006 10:47:47 -0700 Subject: [Python-3000] Adaptation: T->P vs P->P In-Reply-To: <20060403101814.4vme5g95gogskckw@login.werra.lunarpages.com> References: <20060403101814.4vme5g95gogskckw@login.werra.lunarpages.com> Message-ID: <44315FC3.9030007@cox.net> Michael Chermside wrote: >Tim Hochberg writes: > > >>In this thread, Alex has been advocating adaption where types are >>adapted to protocols: T->P adaption for short. By contrast, my two >>sample implementations have involved Protocol->Protocol adaption >>[...] P->P adaption for short. >> >> > > > >>However, there is a huge difference in scale here. A class will >>typically only satisfy a few protocols, whereas it could potentially be >>associated with many adapters. Similarly an adapter will typically be >>associated with only a single source protocol, but it may be associated >>with many, many types. Just to make this concrete, I imagine there will >>be many adapters that work on every sequence type >> >> > >I may be misinterpreting what Tim is saying here, but it sounds like >this touches on the idea of *transitivity* in adaptation. > Your are mispresenting what I'm *trying* to say. That may well be my fault. > That is, Tim >is proposing that there are a small number of "protocols", but I can't >tell the difference between a protocol and an adaption target so I am >taking them to be synonomous. I am (and have been for nearly 2 years... >Alex: your previous efforts at evangalism have not been *totally* useless) >a fan of adaption. But I am *deeply* concerned about introducing >transitive adaption where it is assumed that registering ((A B) f) and >((B C) g) implies ((A C) lambda x: f(g(x)) > >I have heard it argued previously (I apologize, but I forget who said this) >that if f *really* adapts A to B and g *really* adapts B to C, then their >composition must really adapt A to C, and I understand the mathematical >inevitability behind this statement. But I fear that in the real world >there are often subtle distinctions and corner cases that are ignored >for the sake of practicality which make such transitive adaption dangerous. >After all, a substantial portion of Python classes fail true to achieve >true Liskov substitutability despite this being a widely-understood >concept. > > I'm completely in agreement with that actually. >Okay... time for me to stop attacking a position (pro transitivity) that >no one has explicitly taken in this discussion and return to what Tim was >saying. He claims: > > >>let's define A=>B to >>mean "A needs to know about B". For T->P adaption: >> T=>A or A=>T or P=>(A and T) or U=>(A and T and P) >>For P->P adaption: >> T=>Psrc or P=>Psrc or U=>Psrc >> Generally A can be assumed to know about Psrc, so I've left that out. >> >> > >Tim's proposal (if I understand it correctly) depends on transitivity -- >his idea is to have the class authors register as satisfying a small >number of widely-known protocols (Psrc in his case), > Yes. > then use transitivity >to infer ways to satisfy less well-known protocols. > No. All I'm saying is that every object is associated with zero or more protocols. The details of this association are TBD, but look here for one simple/simplistic approach: http://members.cox.net/~tim.hochberg/adaption2.py Adapters are always between two protocols. There is not attempt at transitivity at all. So adapt looks like this: def adapt(obj, Pdest, registry=_global_registry): protocols = find_protocols(obj) if Pdest in protocols: # Always return the original object if it satisfies Pdest return obj for Psrc in protocols: adapter = registry.get((Psrc,Pdest), None) if adapter is not None: return adapter(obj) raise ValueError('adapter not found') >This is a tempting >game... it avoids having to have even a third party know about both the >type and the protocol to be satisfied. But I would rather FORCE someone >(even a third party) to at least glance at the specific case and >reassure us that it works. If the individual wants to do a quick and >sloppy job rather than a careful one, then it is reduced to a question >of whose libraries I trust, which I a problem I understand how to handle. > > The difference in T->P and P->P is that in the first the third party needs to know about every type they want to adapt, while in the second the third party needs to know about every protocol that they want to adapt. The second should generally be a much smaller burden. It's true that you could get into trouble if someone lies about conformance to a protocol. If a class is associated with a protocol it doesn't really satisfy or an adapter relies on some information outside the protocol, then this will break. That's a general problem though and probably not made appreciably worse by P->P adaption versus T->P adaption. Note that T-P adaption is really a subset of P->P adaption as long as you let types be used as protocols. >To return to the example that we used previously in this thread, it seems >that creating an __index__ special method would not be necessary if >adaption were in place... we could instead ask classes to register an >adapter to the "python.as_index" protocol. But if we were to infer that >anything satisfying "python.as_int" necessarily satisfied "python.as_index", >then we would have defeated the purpose of the feature. > >One last point: it is entirely possible that I have misunderstood >completely. If so, please gently explain why. > > I've tried. Now whether that explanation is comprehensible or not is another matter. Regards, -tim From ianb at colorstudy.com Mon Apr 3 20:12:52 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 03 Apr 2006 13:12:52 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> Message-ID: <443165A4.6070807@colorstudy.com> Crutcher Dunnavant wrote: > Well, what if we added '%{expression}s' as a formating type? > This would make your example: > > print _("%{user}s is not a member of the %{listname}s mailing list") > > or even: > > print _("%{utils.websafe(form.get('user'))}s is not a member of the > %{mlist.get_name()}s mailing list") I don't think evaluation should be allowed except with syntactic support. So $"%{expression}s" is safe, because it is a string literal that the programmer specifically wanted substituted. _("%{expression}s") is less safe, because _() doesn't know where that string came from, and maybe it came from an unsafe source (like a web form submission). And _() is also less safe, because it is not obvious that it looks in the calling frame for information. Even what Mailman does is potentially slightly unsafe if they were to accept input to _() from untrusted sources, though exploiting str() is rather hard, and Mailman presumably has at least a moderate amoung of trust for translators. It's not actually unreasonable that translation strings could contain expressions, though it's unlikely that Python expressions are really called for. Like with pluralization: "Displaying $count ${'user' if count==1 else 'users'}" is reasonable, though a more constrained syntax would probably be more usable for the translators. It seems there's a continuum of use cases. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From guido at python.org Mon Apr 3 20:14:17 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Apr 2006 11:14:17 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443062D8.50701@livinglogic.de> References: <443062D8.50701@livinglogic.de> Message-ID: > Crutcher Dunnavant wrote: > >>> 1. Shouldn't there be a format method, like S.format(), or S.fmt()? > > Why? Because: > > 1 It is trivially cheap, format() would be the same function as __rmod__ No it shouldn't be. format() should be a varargs function; __rmod__ takes a single argument which may be a tuple. Also, format() could take keyword args in case the string contains named format, so I can write e.g. "%(foo)s".format(foo=123). > > 2 It adds consistency with lower(), strip(), and other methods which > > produce new strings. I'm not sure that's much of an argument. > > 3 I am not arguing _against_ syntactic support, I am arguing _for_ a method; > > we can keep the syntactic support. But remember TOOWTDI from the Zen of Python. On 4/2/06, Walter D?rwald wrote: > and it avoids one problem you might run into with %: If you have only > one argument, writing ``s % (x,)`` as ``s % x`` will break when the > argument x happens to be a tuple. You won't have this problem with > s.format(x). In fact, now that I think of it, if s.format() were available, I'd use it in preference over s%x, just like I already use repr(x) in favor of `x`. And just like `x` is slated for removal in Python 3000, we might consider removing using % for formatting. The main reason probably being the problem Walter points out (which is very real). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 3 20:40:42 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Apr 2006 11:40:42 -0700 Subject: [Python-3000] [Python-Dev] SF:1463370 add .format() method to str and unicode In-Reply-To: References: Message-ID: On 4/3/06, Crutcher Dunnavant wrote: > >From discussion on python-3000, it occured to me that this shouldn't > break anything. > This patch adds a .format() method to the string and unicode types. > > SF:1463370 Hmm... Let's not jump to conclusions. While I like your patch, we need to have community consensus that s.format(x) is better than s%x, and we need to discuss alternatives such as a different format syntax. I guess I have to amend my process proposals (and yes, I know it's high time for me to get back on the wagon and start spending quality time with Python 3000). While I still believe that new features which can be introduced without backwards incompatibility are fair game for introduction in Python 2.x rather than waiting for 3.0 (and in fact, introduction in 2.x is perhaps preferable over waiting), the realities of community opinion and proposal history need to be taken into account. We also, in particular, need to be really careful that we don't introduce things into 2.x that we *think* we'll want in Py3k but which might turn out later to require more tweaks. For example, in the case of the formatting method, it would be tragic if Python 3000 switched to a different format syntax but we had already introduced s.format(x) in Python 2.x as an alias to s%x -- then the meaning of s.format(x) would change in Python 3000, while we might have had the opportunity of a 10)% *compatible* change if we had waited until the Python 3000 version of the feature had settled before rushing it into Python 2.x. Concluding, perhaps the right time to include certain features in Python 2.x is only *after* the feature has been discussed, specified, agreed upon, and implemented in Python 3000. Of course, this doesn't mean we shouldn't plan to add anything new to Python 2.x (even though that would greatly reduce merge problems with the Py3k branch ;-). I guess changes to 2.x should follow the established, 100% backwards compatible, evolutionary path: if python-dev agrees it's a good idea, it should probably go in. OTOH if it's a potentially disruptive change, or if it could benefit from synchronicity with other Py3k features, or perhaps even if it just adds a new way of saying something that might eventually mean the old way should be deprecated, it's better to refine the idea in a Python 3000 context first. It's going to be inevitable that we'll get the occasional idea first brought up on python-dev that makes more sense to move to Python 3000, or vice versa; let's all be mindful of such cases. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Mon Apr 3 20:51:38 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 03 Apr 2006 13:51:38 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <443062D8.50701@livinglogic.de> Message-ID: <44316EBA.3020302@colorstudy.com> Guido van Rossum wrote: >>Crutcher Dunnavant wrote: >> >>>>>1. Shouldn't there be a format method, like S.format(), or S.fmt()? >>> >>>Why? Because: >>>1 It is trivially cheap, format() would be the same function as __rmod__ > > > No it shouldn't be. format() should be a varargs function; __rmod__ > takes a single argument which may be a tuple. Also, format() could > take keyword args in case the string contains named format, so I can > write e.g. "%(foo)s".format(foo=123). Would "%(foo)s".format({'foo': 123}) work? Or would you need **{...}? FWIW, I suspect I'd be much more likely to use named %'s with .format() than with %; which is probably good, since named markers are more flexible. E.g., I never do: "%(path)s: path %(path)r does not exist" % {'path': path}, but I often do "%s: path %r does not exist" % (path, path). But the first form is really better in several ways. If .substitute() (or .sub()?) was available to do $-based substitution alongside .format() for %-based substitution, that would both have a nice symmetry and make it more comfortable to move between the two. >>>3 I am not arguing _against_ syntactic support, I am arguing _for_ a method; >>> we can keep the syntactic support. > > > But remember TOOWTDI from the Zen of Python. Syntactic support does allow for a level of flexibility (evaluation) plus security (special casing literals) that no method can provide. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon Apr 3 20:52:01 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 03 Apr 2006 13:52:01 -0500 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: Message-ID: <44316ED1.8070409@colorstudy.com> Talin wrote: > -- An easy way to make a case-insensitive, case-preserving > dict that works with regular string keys. Adam gave one possible implementation. Another would be a keyed dictionary, e.g.,: KeyedDict(key=lambda s: s.lower()). A keyed dictionary would internally call that function on keys to get the "true" key, but things like .keys() would return the keys that were passed in. So a minimal implementation... class KeyedDict(DictMixin): def __init__(self, key): self._data = {} self._key_func = key def __getitem__(self, key): return self._data[self._key_func(key)][1] def __setitem__(self, key, value): self._data[self._key_func(key)] = (key, value) def __delitem__(self, key): del self._data[self._key_func(key)] def keys(self): return [t[0] for t in self._data.values()] > (Another one of my wild ideas was an "order by" > clause for list comprehensions, but let's not go there.) Yeah, I'd like that too. > -- A simple way to import all modules in a directory > (this would be used for plugins) I think setuptools entry points (and other things that pkg_resources has) offer a better model for this. It's not as light as just importing all modules; but it also works better ;) I think any system sophisticated enough to use plugins should also start using proper packages with setup.py and all, not a directory of files. > -- A mechanism whereby imported modules can import > symbols from the module that imported them. (Specifically, > I want to take a bunch of variables in my __main__ module > and make them accessible to the imported module.) > > -- The module class should have a method to iterate > over child modules. Currently you can iterator through > all of its attributes, but you have to filter out which ones > are modules. These also seem plugin-related, and I think there are other, better ways to handle these issues. That's not to say it's obvious what those better ways are, but they are out there somewhere ;) > -- A path-globbing function that supports regex-style > captures. I'm not sure what you'd propose; like (*).txt? > -- A path-globbing function that supports the > perforce-style syntax "..." (three dots) to mean "all descendants". > So for example, "foo/.../*.cpp" matches any cpp files in > foo or any of its subdirectories. I think this would be best to implement as a separate module, and try it out. I don't feel like the conventions here are well agreed upon; people agree on what simple file globbing looks like, but I don't really see any agreed upon conventions for things like this. If you implement the same API as fnmatch and glob, then your richer globbing will be a mostly drop-in replacement. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From barry at python.org Mon Apr 3 21:07:02 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 03 Apr 2006 15:07:02 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443165A4.6070807@colorstudy.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> <443165A4.6070807@colorstudy.com> Message-ID: <1144091223.29355.80.camel@resist.wooz.org> On Mon, 2006-04-03 at 13:12 -0500, Ian Bicking wrote: > Even what Mailman > does is potentially slightly unsafe if they were to accept input to _() > from untrusted sources, though exploiting str() is rather hard, and > Mailman presumably has at least a moderate amoung of trust for translators. Right, the attack vector would be through a broken translation (either maliciously or inadvertently) accessing a local unescaped string causing an XSS exploit. > It's not actually unreasonable that translation strings could contain > expressions, though it's unlikely that Python expressions are really > called for. Like with pluralization: "Displaying $count ${'user' if > count==1 else 'users'}" is reasonable, though a more constrained syntax > would probably be more usable for the translators. It seems there's a > continuum of use cases. Except with some language's plural forms (e.g. Polish IIUC) simple expressions like that won't cut it. OTOH, gettext has facilities for supporting all those bizarre plural forms so I don't think we have to reinvent them in Python (though we may need to do more to support them). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060403/d764277e/attachment.pgp From crutcher at gmail.com Mon Apr 3 21:27:45 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Mon, 3 Apr 2006 11:27:45 -0800 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <443062D8.50701@livinglogic.de> Message-ID: 1463370 On 4/3/06, Guido van Rossum wrote: > > Crutcher Dunnavant wrote: > > >>> 1. Shouldn't there be a format method, like S.format(), or S.fmt()? > > > Why? Because: > > > 1 It is trivially cheap, format() would be the same function as __rmod__ > > No it shouldn't be. format() should be a varargs function; __rmod__ > takes a single argument which may be a tuple. Also, format() could > take keyword args in case the string contains named format, so I can > write e.g. "%(foo)s".format(foo=123). Yes, In doing a patch for this against 2.5 (sf:1463370), I saw the issue there. I tossed it to python-dev for comment, and was smacked down for lack of a PEP. I suppose I could write up a PEP for this, if you think it's worthwhile. > > > 2 It adds consistency with lower(), strip(), and other methods which > > > produce new strings. > > I'm not sure that's much of an argument. I think it is for newibes, but that isn't a deal maker. > > > > 3 I am not arguing _against_ syntactic support, I am arguing _for_ a method; > > > we can keep the syntactic support. > > But remember TOOWTDI from the Zen of Python. which is why we have d.get(k) and d[k]; k in d and d.has_key(k), etc. > On 4/2/06, Walter D?rwald wrote: > > and it avoids one problem you might run into with %: If you have only > > one argument, writing ``s % (x,)`` as ``s % x`` will break when the > > argument x happens to be a tuple. You won't have this problem with > > s.format(x). > > In fact, now that I think of it, if s.format() were available, I'd use > it in preference over s%x, just like I already use repr(x) in favor of > `x`. And just like `x` is slated for removal in Python 3000, we might > consider removing using % for formatting. The main reason probably > being the problem Walter points out (which is very real). > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From brett at python.org Mon Apr 3 21:50:14 2006 From: brett at python.org (Brett Cannon) Date: Mon, 3 Apr 2006 12:50:14 -0700 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <4430E916.8060707@livinglogic.de> References: <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> <4430609A.3060302@livinglogic.de> <92B6430C-65DE-4CC0-80F8-1159E9C81421@gmail.com> <4430E916.8060707@livinglogic.de> Message-ID: On 4/3/06, Walter D?rwald wrote: > Alex Martelli wrote: > > > On Apr 2, 2006, at 4:39 PM, Walter D?rwald wrote: > > ... > >> Why not make the registry identical to the protocol? The protocol is > >> just a convention anyway: > > > > Yes, a 1<->1 relationship between registries and protocols makes the > > 'registryof' function I was talking about simpler, at least when it's > > implemented as identity. What's the advantage of this approach, > > compared to allowing protocols to be arbitrary hashable attributes? The > > disadvantage, essentially, is that one gives up the degrees of freedom > > implied by protocols being "just a convention": a protocol must now > > (say) be a callable object exposing a register method (with given > > signature and semantics). > > Yes, but otherwise the adapter has to be, so this doesn't buy us much. > > > I guess adopting that kind of constraint is OK, as long as the > > constraint doesn't in any way interfere with whatever else Guido wants > > to do with protocols and thus give him a reason to reject adaptation. I > > guess I can see some potential for minor optimization: a protocol that's > > written with some existing types already in mind might use a subclass of > > your 'Adaptor', such as: > > > > class FastTracker(Adaptor): > > def __init__(self, types): > > self.fastrack = set(types) > > Adaptor.__init__(self) > > def _anyof(self, types): > > for t in types: > > if t in self.fastrack: return True > > return False > > def register(self, adaptor, types): > > if self._anyof(types): > > # need better diagnostics here, of course > > raise RegistrationError, "Cannot override identity-adaptation" > > return Adaptor.register(self, adaptor, types) > > def __call__(self, obj, *a, **k): > > if self._anyof(type(obj).__mro__): > > if a or k: > > raise ...some kind of diagnostics about a/k not allowed > > here... > > return obj > > return Adaptor.__call__(self, obj, *a, **k) > > > > I'm not sure the advantage is big enough to warrant all the machinery, > > but I can't assert it isn't, either. > > OK, this is the version where you know that some types implement the > protocol itself, but you can't register those types via an attribute on > the types. > > >>> Isn't it just wonderful, how the foes of adaptation switch horses on > >>> you? First they request a simple-as-dirt, bare-bones "example > >>> system" -- then as soon as you provide one they come back at you > >>> with all sort of "cruft" to be piled on top. > >> > >> I think you might be misinterpreting reactions. If the initial > >> reaction was "I don't understand it. Nobody needs this." (at least > >> that was my reaction), you're "strawman proposal" has put us past > >> this. (At least you got two "I finally got it, this seems useful" from > >> me and Brett.) > > > > You're quite likely right, and I apologize for the misinterpretation; I > > guess I may have misread Brett's desire to hammer adaptation's very > > reason for being down to miniscule smithereens by having the existence > > of some set of methods "imply" identity-adaptation, as a subconscious > > desire to defend against the whole idea of adaptation in the guise of > > sort-of-accepting it, for example. > > I guess it's more an attempt to compare adaption to something we already > know. I fact getattr() is the stupid version of adapation. There's only > identity adaption and the protocol is a method or an attribute! ;) > That's exactly what I am doing. It might be the wrong approach, but I am sure I won't be the only one who will try to see how adaption directly relates to how we do things now. -Brett > >> So now lets answer the questions: How do we implement adaption of > >> subtypes? What is a protocol? How can we make registration as painless > >> as possible? etc. > > > > Looks like the "loop on __mro__" idea is sort of standard for the first > > of these questions. > > Yes, probably with a hand crafted version of a mro for classic classes. > > The big advantage of using the mro is that performance is independent > from the number of registered adapter functions. > > > As for "what is a protocol", I'd rather minimize > > the machinery that goes with it, but I guess that's some kind of > > holdover from the "strawman proposal"; if we're making protocols into > > classes anyway, I'd like to provide them with a check_compliance method, > > taking an object and an integer that represents the amount of effort to > > be spent in the checking -- 0 meaning 'just check existence of methods', > > 1 meaning 'check methods' signature compatibility too', 2 meaning 'check > > some semantics at a level suitable for fast unit-tests', and so on up. > > I don't see how that buys us much (And it looks like you're trying to > solve the halting problem ;) > > class Foo: > __implements__ = [Bar] > > def bar(self): > if Bar.check_compliance(self, 2): > # don't comply > else: > # comply > > > > The author of a protocol doesn't have to do all that much (levels 0 and > > 1 can be provided by helper functions/methods relying on the inspect > > module), but he or she CAN make the protocol into "an executable > > specification" at whatever level he or she desires. > > I'm really not sure how this would look in practice. > > > Other error checking > > (e.g., against registration of multiple adapters for one protocol/type > > pair) as well as some modest optimization (mostly of adaptation, the > > frequent case) may also be warranted. > > > > I'm not sure what's painful at all about registration -- at worst it's > > one call per type/protocol pair, > > Exactly, especially for the case where we already have this adaption > machinery today (i.e. copy_reg). > > > some refactoring such as yours making > > it even slighter by allowing the registration of several pairs in one > > call. Do you mean that people are so keen on declarative styles that > > they'd crave, e.g., some special machinery such as > > > > class Duck(object): > > __implements__ = walk, talk, quack > > ... > > > > (with the metaclass doing the registration calls, with identity > > adaptation, under the cover for each protocol in __implements__), > > prizing it highly over explicit: > > There's no need for registration calls in this case. The > protocol/adapter can do it: > > class Adapter(object): > # __init__() and register() as before. > > def __call__(self, obj, *args, **kwargs): > objtype = type(obj) > if hasattr(objtype, "__implements__") and self in > objtype.__implements__: > return obj > # rest of the code as before > > > class Duck(object): > > ... > > adapt.register(Duck, (walk, talk, quack), identity) > > > > or the loop or classdecorator version of the latter? Ah well, maybe, > > but most of the discussion sort of smacks to me of AGNI and/or premature > > optimization. > > > > Could we agree that simplicity and even minimalism (with a side order of > > future extensibility if/when warranted) are high on the list of desired > > properties for a protocol adaptation mechanism? > > Absolutely! > > > After all, the more > > machinery, cruft, and black magic we pile on top of adaptation's simple > > core ideas, the more likely the whole caboodle is to provoke allergic > > reactions in anybody who's not being a part of this discussion... > > True. Make it simple enough to cover 80% of the cases and be > understandable, but extensible enough, so we can add anything that might > come up in the future without having to contort the adaption protocol so > much that it's no longer recognizable. > > From a higher point of view, this is one of the few occasions where a > Design Pattern makes sense in Python. Usually this means that the > language isn't high level enough (nobody implements the "Factory > Function Pattern" in Python because classes are callable). If Python had > multi methods we wouldn't be discussing adaption, because basically what > we're discussing here is how to implement double dispatch. > > Bye, > Walter D?rwald > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/brett%40python.org > From adam.deprince at gmail.com Mon Apr 3 22:00:24 2006 From: adam.deprince at gmail.com (adam deprince) Date: Mon, 3 Apr 2006 16:00:24 -0400 Subject: [Python-3000] A few small py3k wishes In-Reply-To: <44316ED1.8070409@colorstudy.com> References: <44316ED1.8070409@colorstudy.com> Message-ID: On 4/3/06, Ian Bicking wrote: [snip] > Adam gave one possible implementation. Another would be a keyed > dictionary, e.g.,: KeyedDict(key=lambda s: s.lower()). A keyed > dictionary would internally call that function on keys to get the "true" > key, but things like .keys() would return the keys that were passed in. I'm sorry, I should have been more clear. My "implementation," if you can even call it that, was intended as a counter-example to Talin's suggestion. Talin's concern was addressed long ago with the ability to subclass built in types. Its easy to flavor a dict to act however you want. My point wasn't "look how easy this is, lets add it to the core language" but rather "look how easy the core language makes it for you to do this yourself." IMHO, if you want a flavor of dict, extend it yourself. Case insensitivity has no place in the core language. Cheers - Adam DePrince From barry at python.org Mon Apr 3 22:05:46 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 03 Apr 2006 16:05:46 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44316EBA.3020302@colorstudy.com> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> Message-ID: <1144094746.29376.100.camel@resist.wooz.org> On Mon, 2006-04-03 at 13:51 -0500, Ian Bicking wrote: > > No it shouldn't be. format() should be a varargs function; __rmod__ > > takes a single argument which may be a tuple. Also, format() could > > take keyword args in case the string contains named format, so I can > > write e.g. "%(foo)s".format(foo=123). > > Would "%(foo)s".format({'foo': 123}) work? Or would you need **{...}? I do think you'd want **{}. > FWIW, I suspect I'd be much more likely to use named %'s with .format() > than with %; which is probably good, since named markers are more > flexible. Especially if .format() takes keywords as the substitution variables. > E.g., I never do: "%(path)s: path %(path)r does not exist" % > {'path': path}, but I often do "%s: path %r does not exist" % (path, > path). But the first form is really better in several ways. > > If .substitute() (or .sub()?) was available to do $-based substitution > alongside .format() for %-based substitution, that would both have a > nice symmetry and make it more comfortable to move between the two. It does feel that way. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060403/8a692fed/attachment.pgp From ianb at colorstudy.com Mon Apr 3 22:15:59 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 03 Apr 2006 15:15:59 -0500 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: <44316ED1.8070409@colorstudy.com> Message-ID: <4431827F.6070609@colorstudy.com> adam deprince wrote: > On 4/3/06, Ian Bicking wrote: > [snip] > >>Adam gave one possible implementation. Another would be a keyed >>dictionary, e.g.,: KeyedDict(key=lambda s: s.lower()). A keyed >>dictionary would internally call that function on keys to get the "true" >>key, but things like .keys() would return the keys that were passed in. > > > I'm sorry, I should have been more clear. My "implementation," if > you can even call it that, was intended as a counter-example to > Talin's suggestion. > > Talin's concern was addressed long ago with the ability to subclass > built in types. Its easy to flavor a dict to act however you want. > My point wasn't "look how easy this is, lets add it to the core > language" but rather "look how easy the core language makes it for you > to do this yourself." > > IMHO, if you want a flavor of dict, extend it yourself. Case > insensitivity has no place in the core language. I think a keyed dictionary is just as useful a case as defaultdict, though like defaultdict I don't think it needs to be part of dict itself. I strongly favor a greater number of collection patterns being in the standard library, including things like bags and multidicts and ordered dictionaries and all that. People are reinventing these things all the time, with varying levels of robustness, performance, and different APIs. For example, many people implement case insensitive dictionaries with key normalization, but I think the keyed form is generally better (though less obvious). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon Apr 3 22:23:55 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 03 Apr 2006 15:23:55 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144091223.29355.80.camel@resist.wooz.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> <443165A4.6070807@colorstudy.com> <1144091223.29355.80.camel@resist.wooz.org> Message-ID: <4431845B.2030601@colorstudy.com> Barry Warsaw wrote: >>Even what Mailman >>does is potentially slightly unsafe if they were to accept input to _() >>from untrusted sources, though exploiting str() is rather hard, and >>Mailman presumably has at least a moderate amoung of trust for translators. > > > Right, the attack vector would be through a broken translation (either > maliciously or inadvertently) accessing a local unescaped string causing > an XSS exploit. I hadn't even thought of that one; XSS opens up a whole new batch of security errors related to string substitution. Ideally in this case, then, you'd actually do HTML escaping on the extracted locals before string substitution. You could do this in _(), but you'd have to pass something in to indicate if you were creating HTML/XML or plain text. >>It's not actually unreasonable that translation strings could contain >>expressions, though it's unlikely that Python expressions are really >>called for. Like with pluralization: "Displaying $count ${'user' if >>count==1 else 'users'}" is reasonable, though a more constrained syntax >>would probably be more usable for the translators. It seems there's a >>continuum of use cases. > > > Except with some language's plural forms (e.g. Polish IIUC) simple > expressions like that won't cut it. "Simple", sure, but with the full power of Python expressions you can manage any pluralization, even if the string degrades into one big chunk of code squeezed into an expression. Though a DSL will also be more appropriate for these rules than Python syntax. > OTOH, gettext has facilities for > supporting all those bizarre plural forms so I don't think we have to > reinvent them in Python (though we may need to do more to support them). It's not magic, it's just code, be that code in gettext or directly in the translation strings. E.g., "%{user}s es %{'bonita' if user.gender == 'f' else 'guapo'}". You can't tell me gettext also has support for gender-appropriate adjectives! This is all wandering off-topic, except that all these cases make me think that different kinds of wrapping are very useful. For instance, if you want to make sure everything is quoted before being inserted: class EscapingWrapper: def __init__(self, d): self.d = d def __getitem__(self, item): return cgi.escape(str(self.d[item]), 1) Or if you want expressions: class EvalingWrapper: def __init__(self, d): self.d = d def __getitem__(self, item): return eval(item, d) Then you do: string.Template(pattern).substitute(EscapingWrapper(EvalingWrapper(locals())) Probably wrapping that in a function of some sort, of course, because it's no longer something you just whip out on a whim. In this case Template.substitute works nicely, but str.format would not work well if it required **kw for named arguments (since these wrappers can't be turned into actual dictionaries). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From skip at pobox.com Mon Apr 3 22:37:21 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 3 Apr 2006 15:37:21 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144094746.29376.100.camel@resist.wooz.org> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <1144094746.29376.100.camel@resist.wooz.org> Message-ID: <17457.34689.5787.114875@montanaro.dyndns.org> I sort of lost track of the thread, then noticed a recent message where Guido seems to be cozying up to the idea, so I thought maybe I ought to think about it for a few minutes. As far as I can tell, the proposal is: usage example s.format(x, y) simple % substition, e.g. "my %s has %s".format("dog", "fleas") s.format(**x) named substitution using dictionaries, e.g. pet = "kangaroo" pest = "marmots" "my %(pet)s has %(pest)s".format(**locals()) s.format(key=val) named substitution using keyword args, e.g. "my %(pet)s has %(pest)s".format(pet="armadillo", pest="texans") One thing that's always been a bit cumbersome for me with the current mod operator syntax is that it's all or nothing. I either have to cram everything into a single dictionary, hack up some sort of multimap, or fall back to simple substitution. With str.format(), I could presumably do something like pest = current_plague() if pest: print "my %(pet)s has %(pest)s".format(pet="dog", **locals()) That is, mix keyword-style and dict-style parameter passing. The above is a lame example, but I occasionally find myself creating single-use local variables to hold arithmetic expressions I want to display. Computing the expression to be expanded in the function call would be cleaner. If I have my keyword mappings in multiple dictionaries might something like "my %(pet)s has %(pest)s".format(**pets, **pests) be supported? That reminds me of the debate about adding to dictionaries: how are duplicate keys handled? Multiple occurrences of **dict aren't supported in Python 2.4. I suppose it probably doesn't typically make a lot of sense, but for this it seems like it might be reasonable. Skip From g.brandl at gmx.net Mon Apr 3 22:40:49 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 03 Apr 2006 22:40:49 +0200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <17457.34689.5787.114875@montanaro.dyndns.org> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <1144094746.29376.100.camel@resist.wooz.org> <17457.34689.5787.114875@montanaro.dyndns.org> Message-ID: skip at pobox.com wrote: > I sort of lost track of the thread, then noticed a recent message where > Guido seems to be cozying up to the idea, so I thought maybe I ought to > think about it for a few minutes. As far as I can tell, the proposal is: > > usage example > s.format(x, y) simple % substition, e.g. > "my %s has %s".format("dog", "fleas") > s.format(**x) named substitution using dictionaries, e.g. > pet = "kangaroo" > pest = "marmots" > "my %(pet)s has %(pest)s".format(**locals()) > s.format(key=val) named substitution using keyword args, e.g. > "my %(pet)s has %(pest)s".format(pet="armadillo", > pest="texans") > > One thing that's always been a bit cumbersome for me with the current mod > operator syntax is that it's all or nothing. I either have to cram > everything into a single dictionary, hack up some sort of multimap, or fall > back to simple substitution. With str.format(), I could presumably do > something like > > pest = current_plague() > if pest: > print "my %(pet)s has %(pest)s".format(pet="dog", **locals()) You can do that right now, using print "my %(pet)s has %(pest)s" % dict(pet="dog", **locals()) Georg From ncoghlan at gmail.com Mon Apr 3 23:40:21 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 04 Apr 2006 07:40:21 +1000 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443149A8.4010401@colorstudy.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> Message-ID: <44319645.1020306@gmail.com> Ian Bicking wrote: > Barry Warsaw wrote: >> On Mon, 2006-04-03 at 23:43 +1000, Nick Coghlan wrote: >> >> >>> What do you think of a "format" builtin function that accepts the >>> format as the first argument (similar to printf). >>> >>> The version on my harddrive permits positional arguments via $1, $2, >>> etc, as well as string formatting (by sticking the format code in >>> square brackets between the $ and the identifier). Keyword arguments >>> still work, naturally. >>> >>> And if you don't want formatting, you just leave out the square >>> brackets. >> >> >> I'm not totally sure I would need a builtin. If I look at the two >> sources of $-strings in an app like Mailman, I'll see 1) literal human >> readable/translatable strings in the source code, 2) human entered >> strings that come from a web form. >> >> In the first case, all string formatting will funnel through one >> bottleneck function, which will do the catalog lookup, frame variable >> discovery, and substitution all in one fell swoop. So there, the >> builtin doesn't buy you much convenience. > > Well, error messages are a common place I use %. So: > > assert path.startswith(prefix), ( > "%r should start with %r" % (path, prefix)) > assert path.startswith(prefix), ( > $"${repr(path)} should start with ${repr(prefix)}") > assert path.startswith(prefix), ( > "$path should start with $prefix".substitute( > path=repr(path), prefix=repr(prefix)) > > > The second example assumes that you can include full expressions, > otherwise that example would start looking really unpleasant. As it is, > the first example still looks more-or-less the best, and it's too bad > $-based substitution doesn't include it. I'm not sure how it would > include it, unless there was something like $=path, or $, or > $`path` or something. I dunno, none of those are very appealing. Given a 'format' (or 'string.format') function that permits format specifiers and positional arguments*, the following would all be options: # My favourite assert path.startswith(prefix), ( format("$[r]1 should start with $[r]2", path, prefix)) # Most self-explanatory assert path.startswith(prefix), ( format("$1 should start with $2", repr(path), repr(prefix))) # A couple of wordier options assert path.startswith(prefix), ( format("$path should start with $prefix", path=repr(path), prefix=repr(prefix))) assert path.startswith(prefix), ( format("$[r]path should start with $[r]prefix", **locals()) Cheers, Nick. * (such a beast simply needs to take a string as the first argument, and convert it to a string.Template variant which has an extra optional field in the regex for a %-style format specifier, as well as using enumerate() on the arglist if no keywords are given) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From tdelaney at avaya.com Tue Apr 4 00:43:34 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Tue, 4 Apr 2006 08:43:34 +1000 Subject: [Python-3000] hash as attribute/property Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> I've been thinking that `hash` could be an attribute (or property if it needs to be calculated on-the-fly) rather than the current method call. Or it could be an easy thing to add to the "won't change" PEP ... Tim Delaney From greg.ewing at canterbury.ac.nz Tue Apr 4 03:27:41 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Apr 2006 13:27:41 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144065615.11451.53.camel@resist.wooz.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> Message-ID: <4431CB8D.9040004@canterbury.ac.nz> Barry Warsaw wrote: > I don't much like the $"" prefix This was discussed during the last round of formatting wars, and the conclusion was that having $ both inside and outside the string would be too visually confusing. > I don't see a good > way to marry the rich coercion of %-substitution with the simplicity of > $-substition. If I were designing a formatting system from scratch, I think I'd separate the issue of formatting numbers into strings from the issue of inserting strings into other strings. So instead of "Answer no. %5d is %8.3f" % (n, x) you would say something like subst("Answer no. {1} is {2}", format(i, 'd', 5), format(x, 'f', 8, 3)) -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From greg.ewing at canterbury.ac.nz Tue Apr 4 04:00:49 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Apr 2006 14:00:49 +1200 Subject: [Python-3000] Adaptation: T->P vs P->P In-Reply-To: <20060403101814.4vme5g95gogskckw@login.werra.lunarpages.com> References: <20060403101814.4vme5g95gogskckw@login.werra.lunarpages.com> Message-ID: <4431D351.6010106@canterbury.ac.nz> Michael Chermside wrote: > But if we were to infer that > anything satisfying "python.as_int" necessarily satisfied "python.as_index", > then we would have defeated the purpose of the feature. We just need to keep a clear distinction between "is an integer" and "convertible to an integer". Python ints, longs, gmpy integers, etc. would satisfy "is an integer" and be usable as indexes. Floats, decimals, etc. would satisfy "convertible to an integer" but not "is an integer" and would therefore not be usable directly as indexes. So this isn't an argument against transitivity, just for being careful about what implies what. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From greg.ewing at canterbury.ac.nz Tue Apr 4 04:06:10 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Apr 2006 14:06:10 +1200 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <4430F138.5090205@gmail.com> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> <4430F138.5090205@gmail.com> Message-ID: <4431D492.9030409@canterbury.ac.nz> Nick Coghlan wrote: > In this case, it makes far more sense to me to move the adapter > registration out to the individual protocols. That would alleviate some of my concerns as well. This seems more Pythonic: Namespaces are one honking... etc. > In a glorious fit of self-referentiality, one of the first things needed by > such an approach would be a protocol for the protocol interface that allowed > other protocol objects to register themselves as implementing it. This would > then allow the ever popular generic adaptation function to be written as: > > def adapt(obj, target_protocol): > return Protocol.adapt(target_protocol).adapt(obj) Surely the protocol protocol would be well-known enough that other protocols would simply be duck-typed to it, rather than having to be adapted to it? Also I prefer the idea of the protocol object being callable, so you just do target_protocol(obj) to do an adaptation. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From adam.deprince at gmail.com Tue Apr 4 04:18:20 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Mon, 03 Apr 2006 22:18:20 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144087267.29376.57.camel@resist.wooz.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> <1144087267.29376.57.camel@resist.wooz.org> Message-ID: <1144117100.14473.76.camel@localhost.localdomain> On Mon, 2006-04-03 at 14:01 -0400, Barry Warsaw wrote: > On Mon, 2006-04-03 at 10:44 -0700, Crutcher Dunnavant wrote: > > > Well, what if we added '%{expression}s' as a formating type? > > This would make your example: > > > > print _("%{user}s is not a member of the %{listname}s mailing list") > > > > or even: > > > > print _("%{utils.websafe(form.get('user'))}s is not a member of the > > %{mlist.get_name()}s mailing list") > > I wouldn't use it because it would be way too easy for translators to > make catastrophic mistakes. Worse are the maintance problems when you have to change your expression ... inside every translation of your string. > > -Barry > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/adam.deprince%40gmail.com From eric+python-dev at trueblade.com Tue Apr 4 03:55:46 2006 From: eric+python-dev at trueblade.com (Eric V. Smith) Date: Mon, 03 Apr 2006 21:55:46 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4431CB8D.9040004@canterbury.ac.nz> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> Message-ID: <4431D222.5090306@trueblade.com> > If I were designing a formatting system from scratch, > I think I'd separate the issue of formatting numbers into > strings from the issue of inserting strings into other > strings. So instead of > > "Answer no. %5d is %8.3f" % (n, x) > > you would say something like > > subst("Answer no. {1} is {2}", format(i, 'd', 5), format(x, 'f', 8, 3)) Doesn't this violate Guido's "don't have a parameter that is always a constant that changes how a function operates" principle (or whatever its formal name is)? In this case, that's the type identifier 'd' or 'f'. I think format_int(i, 5) and format_float(x, 8, 3) would be better. Or maybe a format() member of int and float, which take different parameters. It's unlikely you'd need to specify the type identifier at runtime. But I agree with separating string insertion from number formatting. You'd probably need to support locales with the number formatting. Eric. From greg.ewing at canterbury.ac.nz Tue Apr 4 04:28:39 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Apr 2006 14:28:39 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44316EBA.3020302@colorstudy.com> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> Message-ID: <4431D9D7.2050609@canterbury.ac.nz> Ian Bicking wrote: > FWIW, I suspect I'd be much more likely to use named %'s with .format() > than with %; which is probably good, since named markers are more > flexible. If we're going to encourage use of named arguments (which I think we should) I think we also need to get rid of the need for %(foo)s constructs, which are extremely error-prone and hard to read. I'm -0.7 on having two different formatting styles (one using % and the other using $) with partially-overlapping functionality. In Py3k there should be OOWTDI. I'm also not all that keen on $, either inside or outside the string. It seems to me that something like "User {user} has printed {n} pages" sets off the parameters from the rest of the string more readably than "User $user has printed $n pages" There are various ways that the traditional %-formatting parameters could be incorporated, e.g. "Answer {num:d,5} is {result:f,8,3}" -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From greg.ewing at canterbury.ac.nz Tue Apr 4 04:46:42 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Apr 2006 14:46:42 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <17457.34689.5787.114875@montanaro.dyndns.org> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <1144094746.29376.100.camel@resist.wooz.org> <17457.34689.5787.114875@montanaro.dyndns.org> Message-ID: <4431DE12.2040908@canterbury.ac.nz> skip at pobox.com wrote: > Multiple occurrences of **dict > aren't supported in Python 2.4. I suppose it probably doesn't typically > make a lot of sense, but for this it seems like it might be reasonable. Maybe it would be better to have a general way of combining dicts, e.g give dicts a '+' operator. Duplicate keys should probably raise an exception in that case. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From aleaxit at gmail.com Tue Apr 4 04:57:08 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Mon, 3 Apr 2006 19:57:08 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <443062D8.50701@livinglogic.de> Message-ID: <3F514FCC-1AA3-4F26-B72B-C09DFAA76C94@gmail.com> On Apr 3, 2006, at 11:14 AM, Guido van Rossum wrote: ... > In fact, now that I think of it, if s.format() were available, I'd use > it in preference over s%x, just like I already use repr(x) in favor of > `x`. And just like `x` is slated for removal in Python 3000, we might > consider removing using % for formatting. The main reason probably Same here, hence, +1 for removing the % operator for strings in Py3k. Alex From guido at python.org Tue Apr 4 05:29:00 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Apr 2006 20:29:00 -0700 Subject: [Python-3000] pre-PEP: Process for reviewing/improving stdlib modules in 3.0 In-Reply-To: References: Message-ID: I didn't see any comments on this PEP. Is there consensus that this is the way we should do it? I'm not sure that the order in which the steps are to be carried out is all that important, nor that it's necessary to do this in the same order for all modules, but otherwise the only thing that bugs me is the reference to the great stdlib renaming (which I'm not sure is such a great idea). I expect that some of the goals (especially test coverage) are too ambitious, but it's worth at least aspiring to great works! BTW we need a procedure for assigning PEP numbers > 3000. Once it's clear whether a PEP is a meta-pep or a feature proposal (usually pretty clear) IMO the author can just pick the next available number in the appropriate range (3001-3099 for meta-PEPs, 3100-3999 for feature-PEPs) and check it in. Perhaps that process is an issue for the meta-meta-PEP, PEP 3000? (Also, since the current PEP 3000 mostly focuses on features, perhaps it ought to be torm apart into pieces that are each given their own feature PEP?) --Guido On 4/2/06, Georg Brandl wrote: > PEP: XXX > Title: Procedure for changing standard library modules > Version: $Revision$ > Last-Modified: $Date$ > Author: Georg Brandl > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 02-Apr-2006 > > > Abstract > ======== > > This PEP describes the procedure for reviewing and improving standard > library modules, especially those written in Python, making them ready > for Python 3000. There can be different steps of refurbishing, each > of which is described in a section below. > > > Step 1: Removal of obsolete modules > =================================== > > All modules marked as deprecated in 2.x versions should be removed for > Python 3000. The same applies to modules which are seen as obsolete today, > but are too widely used to be deprecated or removed. Python 3000 is the > big occasion to get rid of them. > > There will have to be a document listing all removed modules, together > with information on possible substitutes or alternatives. This infor- > mation will also have to be provided by the python3warn.py porting > helper script mentioned in PEP XXX. > > > Step 2: Renaming modules > ======================== > > Aside from a "great stdlib renaming" introducing a hierarchic library > namespace or a top-level package from which to import standard modules, > some modules' names are known to have been chosen unwisely, a mistake > which could never be corrected in the 2.x series. Examples are names > like "StringIO" or "Cookie". For Python 3000, there will be the possi- > bility to rename those modules to less confusing and more conforming > names. > > Of course, each rename will have to be stated in the documentation of > the respective module and perhaps in the global document of Step 1. > Additionally, the python3warn.py script will recognize the old module > names and notify the user accordingly. > > > Step 3: Code cleanup > ==================== > > As most library modules written in Python have not been touched except > for bug fixes, following the policy of never changing a running system, > many of them may contain code that is not up to the newest language > features and could be rewritten in a more concise, modern Python. > > As long as these changes don't change the module's interface and behavior, > no documentation updates are necessary. > > > Step 4: Enhancement of test coverage > ==================================== > > Code coverage by unit tests varies greatly between modules. Each test > suite should be checked for completeness, and the remaining classic tests > should be converted to PyUnit (or whatever new shiny testing framework > comes with Python 3000, perhaps py.test?). > > No documentation changes are necessary for this step. > > > Step 5: Unification of module metadata > ====================================== > > This is a small and probably not very important step. There have been > various attempts at providing author, version and similar metadata in > modules (such as a "__version__" global). Those could be standardized > and used throughout the library. > > No documentation changes are necessary for this step, too. > > > Step 6: Backwards incompatible bug fixes > ======================================== > > Over the years, many bug reports have been filed which complained about > bugs in standard library modules, but have subsequently been closed as > "Won't fix" since a fix would have introduced a major incompatibility > which was not acceptable in the Python 2.x series. In Python 3000, the > fix can be applied if the interface per se is still acceptable. > > Each slight behavioral change caused by such fixes must be mentioned in > the documentation, perhaps in a "Changed in Version 3.0" paragraph. > > > Step 7: Interface changes > ========================= > > The last and most disruptive change is the overhaul of a module's public > interface. If a module's interface is to be changed, a justification > should be made beforehand, or a PEP should be written. > > The change must be fully documented as "New in Version 3.0", and the > python3warn.py script must know about it. > > > References > ========== > > TBD > > Copyright > ========= > > This document has been placed in the public domain. > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 4 05:32:44 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Apr 2006 20:32:44 -0700 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 In-Reply-To: References: Message-ID: I was going to comment about this one "check it in, we'll add to it later", but then I realized it's not 100% clear whether this is a feature PEP or a meta-PEP? It focuses on features so by that yardstick it's a feature PEP. But in its list-of-miscellany nature it approximates a meta-PEP. Hmm, perhaps it ought to be PEP 3999 (which can be seen as minus one in the numbering scheme for Python-3000 PEPs :-)? In any case let's not let it longer for long. Realistically, I think it can be a meta-PEP. --Guido On 4/2/06, Georg Brandl wrote: > Guido van Rossum wrote: > > On 4/2/06, Georg Brandl wrote: > >> A quick idea of mine: Wouldn't it be useful to maintain a list of what will > >> not change, collected in the discussions here? That way, people eager to suggest > >> braces and whatnot can be referred to it. > >> > >> (I'd be volunteering to maintain such a list/PEP/whatever). > > > > Excellent idea! It should be one of the meta-peps (in the 3001-3099 range). > > > > Ideally the list should either point to the discusussion explaining > > the decision, or provide a capsule explanation. (Braces are exempted > > from this policy; if someone doesn't understand why we're not doing > > braces they should probably look elsewhere... :-) > > Here's a draft (please provide me with whatever you can remember, I've been > following the list on gmane and my history's therefore lost): > > PEP: XXX > Title: Things that will Not Change in Python 3000 > Version: $Revision$ > Last-Modified: $Date$ > Author: Georg Brandl > Status: Draft > Type: Informational > Content-Type: text/x-rst > Created: 02-Apr-2006 > > > Abstract > ======== > > This PEP tries to list all BDFL pronouncements on Python 3000 that > refer to changes that will not happen and new features that will not > be introduced, sorted by topics, along with a short explanation or a > reference to the relevant thread on the python-3000 mailing list. > > > Core language > ============= > > * Python will not have programmable syntax. > > Thread: "It's a statement! It's a function! It's BOTH!" > http://mail.python.org/pipermail/python-3000/2006-April/000286.html > > * There won't be a syntax for ``zip()``-style parallel iteration. > > Thread: "Parallel iteration syntax", > http://mail.python.org/pipermail/python-3000/2006-March/000210.html > > > Standard types > ============== > > * Iterating over a dictionary will yield the keys. > > Thread: "Iterating over a dict", > http://mail.python.org/pipermail/python-3000/2006-April/000283.html > > > Coding style > ============ > > * The maximum line width will be 80 characters. > > Thread: "C style guide", > http://mail.python.org/pipermail/python-3000/2006-March/000131.html > > > Copyright > ========= > > This document has been placed in the public domain. > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Tue Apr 4 05:34:22 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Apr 2006 15:34:22 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4431D222.5090306@trueblade.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> <4431D222.5090306@trueblade.com> Message-ID: <4431E93E.40002@canterbury.ac.nz> Eric V. Smith wrote: > Doesn't this violate Guido's "don't have a parameter that is always a > constant that changes how a function operates" principle (or whatever > its formal name is)? It's not inconceivable that a function might want to accept formatting parameters and pass them on to another function which used format(). If there were separate functions for the different formats, this would be awkward. Guido's principle applies when the parameter in question alters the meaning so radically that it seems inconceivable to want to vary it from one call to another. I don't think that quite applies here, although that's obviously a matter of opinion. I wouldn't object to having separate functions available as an alternative for the cases where you don't need the flexibility, which would be most of the time. > Or maybe a format() member of int and float, which take > different parameters. That would make it awkward to take a number which might be an int or float and format it as a float -- you'd have to be careful to cast it to float first. I don't see it as a matter of formatting different types of object, but of having one type of object (a number) and formatting it in different ways. You could even consider 'd' to be a special case of 'f' with 0 decimal places (although it isn't quite). -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From guido at python.org Tue Apr 4 06:03:53 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 3 Apr 2006 21:03:53 -0700 Subject: [Python-3000] Generic functions In-Reply-To: <44315A89.9000104@colorstudy.com> References: <44315A89.9000104@colorstudy.com> Message-ID: On 4/3/06, Ian Bicking wrote: > As an alternative to adaptation, I'd like to propose generic functions. > I think they play much the same role, except they are much simpler to > use and think about. Given that Phillip Eby is another proponent of generic functions I seriously doubt the latter. Certainly your post causes me to produce an endless list of questions (some of which I'll interject below). > Though RuleDispatch offers considerably more features through predicate > dispatch, it would probably be best to just consider type based > dispatch, as that's more equivalent to adaptation. I may have missed some context (I have so far skipped all threads mentioning adaptation), but since you're starting a new thread here, could you perhaps explain what RuleDispatch refers to? > So, the copy_reg module is one example where adaptation has been > proposed. The idea is that you adapt an object to a pickleable object, > or something along those lines. That definition doesn't sound right; but since you are waving your hands and I haven't yet read the thread about adaptation and copy_reg I'll leave it at that rather than give me own version of how copy_reg could be seen as adaptation (actually I'm not sure yet). > It's a little vague, because while you > typically adapt an instance coming in, you "adapt" a string to a new > instance on the way out. You only use adaptation for pickling; for unpickling all the information is in the pickle data. Pickling is a thoroughly asymmetric API. > Or, I dunno, it's not clear to me. In fact, > though that's the typical example, I'm going to bail on that because the > pickling protocol is an aside to this and too complex for me to digest > right now. pprint is a lot easier, and conveniently is much nicer with > generic functions than adaptation ;) Watch it though. it may be a great example to explain generic functions. But it may be the only example, and its existence may not be enough of a use case to motivate the introduction of gneric functions. > Anyway, pprint could work like: > > class PrettyPrinter: > @generic > def pformat(self, object): > """Return the pretty string representation of object""" > return repr(object) > # pformat is now "more" than just a function, it's a callable > # object that does type-based dispatch using an internal registery > # of implementations, with the implementation above as the fallback. Whoa! First of all, my gut reaction is already the same as for adaptation: having a single global registry somehow feels wrong. (Or is it not global? "internal" certainly sounds like that's what you meant; but for methods this seems wrong, one would expect a registry per class, or something like that.) Second, I'm curious if the fact that the argument name is 'object' (which is also a built-in type) is an accident, or has significance. Next, I wonder what the purpose of the PrettyPrinter class is. Is it just there because the real pprint module defines a class by that name? Or does it have some special significance? Are generic functions really methods? Can they be either? > # It also now can be used as a decorator that registers implementations: > @PrettyPrinter.pformat.when(object=list) > def pformat_list(self, object): > s = '[' > for item in object: > s += (' '*self.indent) + self.pformat(item) + ',\n' > return s + (' '*self.indent) + ']' Ah, the infamous "when" syntax again, which has an infinite number of alternative calling conventions, each of which is designed to address some "but what if...?" objection that might be raised. If pformat is a method of the quite ordinary class PrettyPrinter, why isn't the pformat_list() method/function declared inside that class? Sooner or later the name conflict between your argument and the built-in type is going to cause problems, either because you need to access the built-in type or because the reader is confused. What does when(object=list) mean? Does it do an isinstance() check? Is there any significance to the name pformat_list? Could I have called it foobar? Why not just pformat? > Some things to note: > > * There's no interface created here. There's no hidden interface > lurking in the background either. You seem to be expecting a particular objection that I don't have. :-) Never in a thousand years would I have thought of interfaces here. What context am I missing? > * It requires cooperation from the original function (pformat -- I'm > using "function" and "method" interchangably). Thereby not helping the poor reader who doesn't understand all of this as well as you and Phillip apparently do. > It does not require any > cooperation from classes like list, similar to adaptation and dissimilar > to magic methods. What do you mean here? Is list used because it appears in the when clause? I'm guessing that you are contrasting it with len(), which could be seen as a special kind of built-in "generic function" if one squints enough, but one that requires the argument to provide the __len__ magic method. But since len() *does* require the magic method, doesn't that disqualify it from competing? > Adaptation also requires cooperation from the caller, > as the adaptation would be applied inside pformat. Huh? Aren't you contradicting yourself here? If the adaptation is done inside pformat (being the callee) what does the caller have to do (except from the obvious "call pformat")? > * The function is mostly self-describing. Perhaps once you've wrapped your head around the when() syntax. To me it's all magic; I feel like I'm back in the situation again where I'm learning a new language and I haven't quite figured out which characters are operators, which are separators, which are part of identifiers, and which have some other magical meaning. IOW it's not describing anything for me, nor (I presume) for most Python users at this point. > If it has certain return > values, then you state what those values are in documentation; there's > no need to be formal about it. In contrast you have to come up with a > whole collection of interfaces to start using adaptation. > > * The function is the hub of all the registration, which seems very > natural, since you are extending the function. Tell us more about the registration machinery. Revealing that (perhaps simplified) could do a lot towards removing the magical feel. > * Like adaptation, you must import a module that defines extra > specializations of the generic function before those are active (just > like you have to import adapter declarations). This strikes me as a > significant problem. I assume ZCML addresses this, but I also assume > that's not a reasonable solution for core Python. You have to import these modules for their side effects (on the registry), not so much because they make some objects or names available that you use directly, right? > * Magic methods do *not* have this import problem, because once you have > an object you have all its methods, including magic methods. Well, of course that works only until you need a new magic method. > RuleDispatch has a bunch more features than just simple type-based > generic functions. But I think that type-based generic functions would > be an easier or more comfortable place to start, and wouldn't preclude a > more featureful implementation later. If RuleDispatch is the thing defining Phillip's full when() syntax, yes, please focus on the simpler rules. > Type-based generic functions and adaptation are more-or-less equivalent. > That is, you can express one in terms of the other, at least > functionally if not syntactically. Could you elaborate this with a concrete example? > If you really wanted adaptation, > then the interface becomes a things-obeying-this-interface factory -- > i.e., a generic function. Generic functions are similer to > multi-adaptaters, where you adapt a tuple of objects, similar to the > tuple of arguments to a function. This is technically like generic > functions, but syntactically rather awkward. > > [Predicate-based dispatching goes considerably further, allowing real > duck typing, e.g., you could implement a pformat method for everything > that has an "__iter__" method and no "next" method (i.e., all iterables, > but not iterators which could lead to unintentionally consuming the > iterator).] > > Anyway, I think generic functions are very compatible with Python syntax > and style, and Python's greater emphasis on what an object or function > can *do*, as opposed to what an object *is*, as well as the use of > functions instead of methods for many operations. People sometimes see > the use of functions instead of methods in Python as a weakness; I think > generic functions turns that into a real strength. Perhaps. The import-for-side-effect requirement sounds like a showstopper though. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.hochberg at ieee.org Tue Apr 4 07:16:16 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon, 03 Apr 2006 22:16:16 -0700 Subject: [Python-3000] Adaptation [was:Re: Iterators for dict keys, values, and items == annoying :)] In-Reply-To: <4431D492.9030409@canterbury.ac.nz> References: <24E21F66-A8A1-48C6-90B6-A2D1D6ABBE83@gmail.com> <442E1FBF.50801@canterbury.ac.nz> <442F212C.9050602@canterbury.ac.nz> <442F5327.1040204@gmail.com> <79990c6b0604020711u141b120em1f63cc46b366145c@mail.gmail.com> <094EECFE-0313-46F2-B79B-3795C7F6A2D7@gmail.com> <4430F138.5090205@gmail.com> <4431D492.9030409@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Nick Coghlan wrote: > > >>In this case, it makes far more sense to me to move the adapter >>registration out to the individual protocols. > > > That would alleviate some of my concerns as well. This > seems more Pythonic: Namespaces are one honking... etc. > > >>In a glorious fit of self-referentiality, one of the first things needed by >>such an approach would be a protocol for the protocol interface that allowed >>other protocol objects to register themselves as implementing it. This would >>then allow the ever popular generic adaptation function to be written as: >> >> def adapt(obj, target_protocol): >> return Protocol.adapt(target_protocol).adapt(obj) > > > Surely the protocol protocol would be well-known enough > that other protocols would simply be duck-typed to it, > rather than having to be adapted to it? > > Also I prefer the idea of the protocol object being > callable, so you just do > > target_protocol(obj) > > to do an adaptation. I've been rewriting copy_reg.pickle to use, or preferably, be replaced by an adaption framework. Copy_reg seems to come up every time adaption does, and this is the registration part of copy_reg. So far I've done generic T->P, generic P->P and this version, let's call it distributed adaption. So far distributed adaption seems to be the best. One caveat -- I've tried to literarly recreate copy_reg.pickle's behaviour, it's possible that something better could be done with a recasting of the whole pickle framework or some such. I'm not going to go there. First, let's look at copy_reg.pickle. I've removed a little bit of it that is marked as vestigal so that we can focus on the remainder: def pickle(ob_type, pickle_function): if type(ob_type) is _ClassType: raise TypeError("copy_reg is not intended for use with classes") if not callable(pickle_function): raise TypeError("reduction functions must be callable") dispatch_table[ob_type] = pickle_function In addition to the basic dispatch table, we need to do two things: check that pickle_function is a callable and test that ob_type is not an instance of _ClassType. The first is probably a good thing for any adaption framework to do, so I went ahead and added that check to the register call in all cases. That leaves us to deal with the ClassType check. 1. "Generic" Type->Protocol Adaption (T->P adaption): ######################################################################## # In copy_reg def pickle(ob_type, pickle_function): if type(ob_type) is _ClassType: raise TypeError("copy_reg ...") register_adapter(pickle_func, ob_type, ipickled) # Usage (same as before) copy_reg.pickle(some_type, some_pickle_function) ######################################################################## In this case, I couldn't find any way to really use the adaption framework to good effect. Sure I managed to shave off two lines of code, but that hardly seems satisfying. 2. "Generic" Protocol-Protocol Adaption (P->P adaption): ######################################################################## # In copy_reg ipickled = Protocol('pickled-object') def pickleable(ob_type): if type(ob_type) is _ClassType: raise TypeError("picklable ...") return get_type_protocol(ob_type) # Usage adaption.register_adapter(some_pickle_function, copy_reg.pickleable(some_type), copy_reg.ipickled) ######################################################################## This seems closer, maybe. At least your actually using the registry that we worked to build. Still you haven't gained much, if anything, and it seems rather kludgy to be almost doing T->P adaption in a P->P framework. If you're wondering, that's what get_type_protocol is doing, it's a helper function to help you fake T->P semantics. The fact that you want it is probably a bad sign. When I implemented these, they both seemed to be complexity magnets. The basics are quite simple, but it seemed like they would need this helper function and that extra feature and so one until the basic simplicity gets obscured. P->P was probably somewhat worse in this regard, but both schemes suffer from this. 3. Distributed Adaption. ######################################################################## # In copy_reg class PickleProtocol(adaption.Protocol): def register(self, adapter, *types): if _ClassType in (type(t) for t in types): raise TypeError("%s is not intended for use with cla adaption.Protocol.register(self, adapter, *types) pickler = PickleProtocol('pickle') # Usage copy_reg.pickler.register(some_pickle_function, some_type) ######################################################################## This seems more like it. You still haven't saved any signifigant code, but this seems clean and extensible. The key feature seems to be that each Protocol can override the register and adapt methods (in this case I'm using __call__ for adapt as Greg suggested). Also, in use this code doesn't seem to want to be complicated to nearly the same extent that the others do. If I find time, I plan to also see how the __index__ protocol would have looked had it been implemented using each of these frameworks. The code that implements T->P and P->P adaption is here: http://members.cox.net/~tim.hochberg/adaption4.py The code that implements distributed adaption is here: http://members.cox.net/~tim.hochberg/adaption5.py Be warned that in the first case the module docstring is, at this point, ridiculously long and meandering and the code has probably accumulated a bit of cruft. Regards, -tim From ianb at colorstudy.com Tue Apr 4 08:03:34 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 01:03:34 -0500 Subject: [Python-3000] Generic functions In-Reply-To: References: <44315A89.9000104@colorstudy.com> Message-ID: <44320C36.7070507@colorstudy.com> Guido van Rossum wrote: > On 4/3/06, Ian Bicking wrote: >> As an alternative to adaptation, I'd like to propose generic functions. >> I think they play much the same role, except they are much simpler to >> use and think about. > > Given that Phillip Eby is another proponent of generic functions I > seriously doubt the latter. Certainly your post causes me to produce > an endless list of questions (some of which I'll interject below). I don't know; I'm not sure if he's on this list, but I'll copy him directly. Phillip wrote PyProtocols before the generic functions; I don't know if he feels as strongly about adaptation after going through that process and seeing the result of each. >> Though RuleDispatch offers considerably more features through predicate >> dispatch, it would probably be best to just consider type based >> dispatch, as that's more equivalent to adaptation. > > I may have missed some context (I have so far skipped all threads > mentioning adaptation), but since you're starting a new thread here, > could you perhaps explain what RuleDispatch refers to? It's the generic function (predicate dispatch) package that Phillip wrote. What I described is a vague hand-wavy interpretation of it. Installation is described on the PEAK front page (http://peak.telecommunity.com/) and an article by David Mertz (http://www-128.ibm.com/developerworks/library/l-cppeak2/) and I'm not actually sure where the main documentation is... >> So, the copy_reg module is one example where adaptation has been >> proposed. The idea is that you adapt an object to a pickleable object, >> or something along those lines. > > That definition doesn't sound right; but since you are waving your > hands and I haven't yet read the thread about adaptation and copy_reg > I'll leave it at that rather than give me own version of how copy_reg > could be seen as adaptation (actually I'm not sure yet). > >> It's a little vague, because while you >> typically adapt an instance coming in, you "adapt" a string to a new >> instance on the way out. > > You only use adaptation for pickling; for unpickling all the > information is in the pickle data. Pickling is a thoroughly asymmetric > API. > >> Or, I dunno, it's not clear to me. In fact, >> though that's the typical example, I'm going to bail on that because the >> pickling protocol is an aside to this and too complex for me to digest >> right now. pprint is a lot easier, and conveniently is much nicer with >> generic functions than adaptation ;) > > Watch it though. it may be a great example to explain generic > functions. But it may be the only example, and its existence may not > be enough of a use case to motivate the introduction of gneric > functions. I can come up with more, to be sure. TurboGears has started using RuleDispatch in several places internally, and I think I saw something about Django using it as well. I've used it some for form generation experiments, which is one of the examples where people have noted adaptation (adapting an abstract field to an HTML input widget, for instance). In TurboGears they are using it for JSON serialization (JSON is a Javascriptish syntax, as an alternative to XML), where it is kind of similar to Pickle (but one way), or more similar to pprint. >> Anyway, pprint could work like: >> >> class PrettyPrinter: >> @generic >> def pformat(self, object): >> """Return the pretty string representation of object""" >> return repr(object) >> # pformat is now "more" than just a function, it's a callable >> # object that does type-based dispatch using an internal registery >> # of implementations, with the implementation above as the fallback. > > Whoa! First of all, my gut reaction is already the same as for > adaptation: having a single global registry somehow feels wrong. (Or > is it not global? "internal" certainly sounds like that's what you > meant; but for methods this seems wrong, one would expect a registry > per class, or something like that.) No, it's pretty local; the registry is stored in the PrettyPrinter.pformat object. This is one of the rather elegant aspects of RuleDispatch, I think -- once you define a function as generic, the function itself contains all the methods you need to extend it. > Second, I'm curious if the fact that the argument name is 'object' > (which is also a built-in type) is an accident, or has significance. Just the existing signature. > Next, I wonder what the purpose of the PrettyPrinter class is. Is it > just there because the real pprint module defines a class by that > name? Or does it have some special significance? It's there because it is matching the pprint module. Also it holds some state which is useful to keep separate from the rest of the arguments, like the current level of indentation. > Are generic functions > really methods? Can they be either? They can be either. I believe that has required special casing methods, because methods and functions act differently. In my experience I also have a hard time writing intelligent decorators that work on both. >> # It also now can be used as a decorator that registers implementations: >> @PrettyPrinter.pformat.when(object=list) >> def pformat_list(self, object): >> s = '[' >> for item in object: >> s += (' '*self.indent) + self.pformat(item) + ',\n' >> return s + (' '*self.indent) + ']' > > Ah, the infamous "when" syntax again, which has an infinite number of > alternative calling conventions, each of which is designed to address > some "but what if...?" objection that might be raised. Yeah, my use here also clashes with what RuleDispatch uses, so don't pay too much attention to that particular. > If pformat is a method of the quite ordinary class PrettyPrinter, why > isn't the pformat_list() method/function declared inside that class? You could, but potentially it could be defined in a different module somewhat unrelated to pprint.PrettyPrinter. > Sooner or later the name conflict between your argument and the > built-in type is going to cause problems, either because you need to > access the built-in type or because the reader is confused. I used "pformat_list" as a name, purely for convenience. I could have used "pformat" or "foo" and it wouldn't matter. The advantage of pformat_list is that it displays better in tracebacks because it has a more unique function name. The function is attached to PrettyPrinter.pformat because I use the decorator attached to that object. > What does when(object=list) mean? Does it do an isinstance() check? Yes; I think RuleDispatch has a form (though I can't remember what the form is -- it isn't .when()). > Is there any significance to the name pformat_list? Could I have > called it foobar? Why not just pformat? Just for tracebacks, and for example to make it greppable. >> Some things to note: >> >> * There's no interface created here. There's no hidden interface >> lurking in the background either. > > You seem to be expecting a particular objection that I don't have. :-) > Never in a thousand years would I have thought of interfaces here. > What context am I missing? To do the same with adaptation you'd have to create an interface. It would be something like: class PrettyPrinter: def pformat(self, object): printable = IPrettyPrintable(object) return printable.pformat(object, self.indent, ...) At least, that's how I'd presume you'd do the same. When talking about copy_reg and other adaptation cases, there's lots of handwaving about adapting something to a pickleable interface or somesuch. In practice you'd actually have to create real interfaces for each of those cases, so I think the interfaces are a hidden cost. >> * It requires cooperation from the original function (pformat -- I'm >> using "function" and "method" interchangably). > > Thereby not helping the poor reader who doesn't understand all of this > as well as you and Phillip apparently do. Hey, you're the one set on functions and methods being interchangeable with an explicit self argument! Mostly everything applies to either case. >> It does not require any >> cooperation from classes like list, similar to adaptation and dissimilar >> to magic methods. > > What do you mean here? Is list used because it appears in the when clause? It means you don't need to add a magic method to the list type, or inject a method into a type to add a magic method. > I'm guessing that you are contrasting it with len(), which could be > seen as a special kind of built-in "generic function" if one squints > enough, but one that requires the argument to provide the __len__ > magic method. But since len() *does* require the magic method, doesn't > that disqualify it from competing? Yes, this is in contrast with len(), which achieves its goal because the people who write the len() function write the entire language, and can put a __len__ on whatever they want ;) For other cases magic-method based systems tend to look like: def pprint(object): if isinstance(object, list): ... elif isinstance(object, tuple): ... ... elif hasattr(object, '__pprint__'): object.pprint() else: print repr(object) That is, all the built in objects get special-cased and other objects define a magic method. >> Adaptation also requires cooperation from the caller, >> as the adaptation would be applied inside pformat. > > Huh? Aren't you contradicting yourself here? If the adaptation is done > inside pformat (being the callee) what does the caller have to do > (except from the obvious "call pformat")? Sorry, too many callers. I don't know what name to give it -- anyway, if pretty printing was made more extensible with adaptation, it would require modifications to PrettyPrinter.pformat, just like generic functions require modifying that same method. With magic methods, all arguments to pformat would have to be extended with a magic method. >> * The function is mostly self-describing. > > Perhaps once you've wrapped your head around the when() syntax. To me > it's all magic; I feel like I'm back in the situation again where I'm > learning a new language and I haven't quite figured out which > characters are operators, which are separators, which are part of > identifiers, and which have some other magical meaning. IOW it's not > describing anything for me, nor (I presume) for most Python users at > this point. The function is described in its docstring, like other functions, as opposed to being partially documented in an interface. The implementation of my simplistic form of generic function isn't too hard. Ignoring keyword arguments, it might work like: class generic(object): def __init__(self, func): self.func = func self.registry = {} def __call__(self, *args): for pattern, implementation in self.registry.items(): for passed, expected in zip(args, pattern): # None is a wildcard here: if (expected is not None and not isinstance(passed, expected)): break else: return implementation(*args) return self.func(*args) def when(self, *args): def decorator(func): self.registry[args] = func return func return decorator def __get__(self, obj, type=None): if obj is None: return self return types.MethodType(self, obj, type) There's lots of details, and handling keyword arguments, dealing intelligently with subclasses, and other things I probably haven't thought of. But anyway, this allows: class PrettyPrinter: def pformat(self, object): ... # Without keyword arguments I have to give a wildcard for the self # argument... @PrettyPrinter.pformat(None, list) def pformat_list(self, object): ... It *is* a new paradigm, just like adaptation is. I think it's a pretty easy paradigm to get used to, and you don't have to introduce auxiliary ideas like interfaces to make it sensible. >> If it has certain return >> values, then you state what those values are in documentation; there's >> no need to be formal about it. In contrast you have to come up with a >> whole collection of interfaces to start using adaptation. >> >> * The function is the hub of all the registration, which seems very >> natural, since you are extending the function. > > Tell us more about the registration machinery. Revealing that (perhaps > simplified) could do a lot towards removing the magical feel. Hopefully the simple implementation I gave makes it clearer. RuleDispatch has something much more clever and faster, and you can give it arbitrary expressions. But in terms of registration it works the same. >> * Like adaptation, you must import a module that defines extra >> specializations of the generic function before those are active (just >> like you have to import adapter declarations). This strikes me as a >> significant problem. I assume ZCML addresses this, but I also assume >> that's not a reasonable solution for core Python. > > You have to import these modules for their side effects (on the > registry), not so much because they make some objects or names > available that you use directly, right? Yes, for the side effects. >> * Magic methods do *not* have this import problem, because once you have >> an object you have all its methods, including magic methods. > > Well, of course that works only until you need a new magic method. Yes, pluses and minuses ;) >> RuleDispatch has a bunch more features than just simple type-based >> generic functions. But I think that type-based generic functions would >> be an easier or more comfortable place to start, and wouldn't preclude a >> more featureful implementation later. > > If RuleDispatch is the thing defining Phillip's full when() syntax, > yes, please focus on the simpler rules. > >> Type-based generic functions and adaptation are more-or-less equivalent. >> That is, you can express one in terms of the other, at least >> functionally if not syntactically. > > Could you elaborate this with a concrete example? Well, in Zope you do something like: class IFoo(Interface): ... And adaptation looks like: fooish_obj = IFoo(obj) There's other proposals, like adapt(obj, IFoo), but it doesn't really change anything. Now, Zope's Interface is a funny class, because it's not really a class, it's an instance you can subclass. So... to ignore that, lets imagine you actually have a class and interface, because I don't want to try to figure out how Zope's Interface class/instance works... class IFoo(Interface): """Calling IFoo()(obj) coerces the object to IFoo""" @generic def __call__(self, obj): raise NotImplementedError Now you have a class, and you want to register an adaptation: class Bar(object): ... class FooBarAdapter(object): def __init__(self, bar): self.bar = bar ... other foo-ish methods ... @IFoo.__call__.when(None, Bar) def coerce_bars(self, obj): return FooBarAdapter(obj) I gave an example of how you'd use adaptation for pretty printing up above. >> If you really wanted adaptation, >> then the interface becomes a things-obeying-this-interface factory -- >> i.e., a generic function. Generic functions are similer to >> multi-adaptaters, where you adapt a tuple of objects, similar to the >> tuple of arguments to a function. This is technically like generic >> functions, but syntactically rather awkward. >> >> [Predicate-based dispatching goes considerably further, allowing real >> duck typing, e.g., you could implement a pformat method for everything >> that has an "__iter__" method and no "next" method (i.e., all iterables, >> but not iterators which could lead to unintentionally consuming the >> iterator).] >> >> Anyway, I think generic functions are very compatible with Python syntax >> and style, and Python's greater emphasis on what an object or function >> can *do*, as opposed to what an object *is*, as well as the use of >> functions instead of methods for many operations. People sometimes see >> the use of functions instead of methods in Python as a weakness; I think >> generic functions turns that into a real strength. > > Perhaps. The import-for-side-effect requirement sounds like a > showstopper though. There's a bunch of places where this is a problem. This is a problem anywhere you want to add functionality to something that doesn't belong to you. The effects of imports is a problem -- but it's a problem we already have, not one that is added just with generic functions or adaptation. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From tjreedy at udel.edu Tue Apr 4 08:55:15 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 4 Apr 2006 02:55:15 -0400 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 References: Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604032032o2e93fee7gde1d29d0c1fdd706 at mail.gmail.com... >I was going to comment about this one "check it in, we'll add to it > later", but then I realized it's not 100% clear whether this is a > feature PEP or a meta-PEP? > In any case let's not let it longer for long. Realistically, I think > it can be a meta-PEP. To me it is definitely a informational meta-PEP, which says 'don't submit a PEP on these topics'. Perhaps 3099. To me, the style is pretty clean and crisp. I have no suggestions for modification and would check it in and hope it gets updated as appropriate. Terry Jan Reedy From tjreedy at udel.edu Tue Apr 4 09:13:58 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 4 Apr 2006 03:13:58 -0400 Subject: [Python-3000] pre-PEP: Process for reviewing/improving stdlibmodules in 3.0 References: Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604032029u475e8e7an2b15764d1ccd008e at mail.gmail.com... >I didn't see any comments on this PEP. For me, it is pretty clear and sensible, hence not much to say. Plus I expected others to says whatever was needed ;-). > I'm not sure that the order in which the > steps are to be carried out is all that important, nor that it's > necessary to do this in the same order for all modules, Such thoughts had occurred to me, but I don't think of anything specific to add. I read the steps as strong guidelines rather than a straightjacket. I imagine refinements will grow out of experience. > but otherwise > the only thing that bugs me is the reference to the great stdlib > renaming (which I'm not sure is such a great idea). This is the main thing that caught my attention since I did not know that you had made a decision, which, obviously now, you haven't. So I would break that sentence into two: There are proposals for a "great stdlib renaming" introducing a hierarchic library namespace. That possibility aside, some module's names are known ... And add PEP reference if and when there is a renaming PEP. > I expect that some of the goals (especially test coverage) are too > ambitious, > but it's worth at least aspiring to great works! The test coverage goal tells me that there is 'room' for me to contribute should I learn to be a good test writer. Terry Jan Reedy From just at letterror.com Tue Apr 4 09:24:35 2006 From: just at letterror.com (Just van Rossum) Date: Tue, 4 Apr 2006 09:24:35 +0200 Subject: [Python-3000] StringIO vs cStringIO, pickle vs cPickle Message-ID: Has there been any discussion yet about StringIO vs cStringIO and pickle vs cPickle? While I appreciate that there are maintained pure Python versions of those modules, as a user it irritates me that I have the choice. The argument so far for not replacing StringIO with cStringIO was that they're subtly incompatible (mostly StringIO's "feature" to support unicode, right?), so Python 3000 seems like a good time to reconsider that option. I don't see any reason to not always want to prefer the faster version. Just From rrr at ronadam.com Tue Apr 4 09:58:59 2006 From: rrr at ronadam.com (Ron Adam) Date: Tue, 04 Apr 2006 02:58:59 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4431CB8D.9040004@canterbury.ac.nz> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Barry Warsaw wrote: > >> I don't much like the $"" prefix > > This was discussed during the last round of formatting > wars, and the conclusion was that having $ both > inside and outside the string would be too visually > confusing. > >> I don't see a good >> way to marry the rich coercion of %-substitution with the simplicity of >> $-substition. > > If I were designing a formatting system from scratch, > I think I'd separate the issue of formatting numbers into > strings from the issue of inserting strings into other > strings. So instead of > > "Answer no. %5d is %8.3f" % (n, x) > > you would say something like > > subst("Answer no. {1} is {2}", format(i, 'd', 5), format(x, 'f', 8, 3)) > And yet another (I'm sure there are many more) ways to look at it... Consider "format" could be used to build strings from data, while "insert" could be used to put data into strings. Then you can build strings in either direction depending on which fits the problem better. Format data using string: (n, x).format("Anser no. {d:5} is {f:8.3}") Insert data into string: "Anser no. {d:5} is {f:8.3}".insert(n, x) If you want to separate the formating from the string further: (n, x).format('d:5', 'f:8.3').format("Answer no. {} is {}.") {n, x).format({1:'d:5', 2:'f:8.3'}) \ .format("Answer no. {1} is {2}.") "Answer no. {} is {}.".insert('d:5', 'f:8.3').insert(n, x) "Answer no. {1} is {2}.".insert({1:'d:5', 2:'f:8.3'}) \ .insert(n, x) And with data structures: string = "Answer no. {1} is {2}." forms = {1:'d:5', 2:'f:8.3'} data = (n, x) string.insert(forms).insert(data) data.format(forms).format(string) I think I'd rather have dot separated format attributes. They would be easy to parse or build with join or split, but considering how short they are it may not matter. {1:d.5} {2:f.8.3} Cheers, Ron From walter at livinglogic.de Tue Apr 4 10:20:10 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue, 04 Apr 2006 10:20:10 +0200 Subject: [Python-3000] Generic functions In-Reply-To: <44320C36.7070507@colorstudy.com> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> Message-ID: <44322C3A.5040509@livinglogic.de> Ian Bicking wrote: > Guido van Rossum wrote: > >[...] >> What does when(object=list) mean? Does it do an isinstance() check? > > Yes; I think RuleDispatch has a form (though I can't remember what the > form is -- it isn't .when()). What happens, if I do the following @PrettyPrinter.pformat.when(object=list) def foo(...): ... @PrettyPrinter.pformat.when(object=object) def foo(...): ... How does it know which isinstance() check to do first? And what happens with performance if I have registered many handler functions? > [...] > > The implementation of my simplistic form of generic function isn't too > hard. Ignoring keyword arguments, it might work like: > > class generic(object): > def __init__(self, func): > self.func = func > self.registry = {} > def __call__(self, *args): > for pattern, implementation in self.registry.items(): > for passed, expected in zip(args, pattern): > # None is a wildcard here: > if (expected is not None and > not isinstance(passed, expected)): > break > else: > return implementation(*args) > return self.func(*args) > def when(self, *args): > def decorator(func): > self.registry[args] = func > return func > return decorator > def __get__(self, obj, type=None): > if obj is None: > return self > return types.MethodType(self, obj, type) > > There's lots of details, and handling keyword arguments, dealing > intelligently with subclasses, and other things I probably haven't > thought of. But anyway, this allows: > > class PrettyPrinter: > def pformat(self, object): ... > > # Without keyword arguments I have to give a wildcard for the self > # argument... > @PrettyPrinter.pformat(None, list) > def pformat_list(self, object): > ... I don't understand! There's no generic in sight here! Bye, Walter D?rwald From ncoghlan at gmail.com Tue Apr 4 11:07:17 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 04 Apr 2006 19:07:17 +1000 Subject: [Python-3000] pre-PEP: Process for reviewing/improving stdlibmodules in 3.0 In-Reply-To: References: Message-ID: <44323745.1080104@gmail.com> Terry Reedy wrote: > "Guido van Rossum" wrote in message > news:ca471dc20604032029u475e8e7an2b15764d1ccd008e at mail.gmail.com... >> I didn't see any comments on this PEP. > > For me, it is pretty clear and sensible, hence not much to say. > Plus I expected others to says whatever was needed ;-). > >> I'm not sure that the order in which the >> steps are to be carried out is all that important, nor that it's >> necessary to do this in the same order for all modules, > > Such thoughts had occurred to me, but I don't think of anything specific to > add. I read the steps as strong guidelines rather than a straightjacket. > I imagine refinements will grow out of experience. That's pretty much what I was thinking. . . (come one, come all, come see a live demonstration of the bystander effect in action!) > This is the main thing that caught my attention since I did not know that > you had made a decision, which, obviously now, you haven't. So I would > break that sentence into two: > > There are proposals for a "great stdlib renaming" introducing a hierarchic > library > namespace. That possibility aside, some module's names are known ... > > And add PEP reference if and when there is a renaming PEP. This wording sounds much better to me, too. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From p.f.moore at gmail.com Tue Apr 4 11:28:43 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 4 Apr 2006 10:28:43 +0100 Subject: [Python-3000] Generic functions In-Reply-To: <44320C36.7070507@colorstudy.com> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> Message-ID: <79990c6b0604040228r768b26fdlcf68e1bece15988f@mail.gmail.com> On 4/4/06, Ian Bicking wrote: > > Perhaps. The import-for-side-effect requirement sounds like a > > showstopper though. > > There's a bunch of places where this is a problem. This is a problem > anywhere you want to add functionality to something that doesn't belong > to you. The effects of imports is a problem -- but it's a problem we > already have, not one that is added just with generic functions or > adaptation. (Generally, I'm going to keep out of this, but I want to comment on this one point). Showstopper is a bit strong. At worst, import-for-side-effect it's bad style, but it's not a requirement of generic functions (or of adaptation, for that matter, where much the same registration issue applies). If you are defining a new class, and want to include pretty-printing support, you'd include the generic function in the module which defines the class. So the import is not "just for side effect", it exports the class itself. If you're an application writer who wants to use an existing class with pprint, but that class doesn't support pretty-printing, you need to write the plumbing (generic function instance) and register it. That's all application code, and how you structure it is up to you. You could have a module that is just imported for effect, but I'd be more likely to write a wrapper module round the module which exported the class - it imports the class, adds the generic functions needed for your application, and re-exports the class. Then you import your ONE module defining the "enhanced" class, rather than importing TWO modules - the original module to get the class, and a "magic" module to add behaviour. But ultimately, it's your choice how you organise your code. All I'm saying is that imports for side effect are *not* required with generic functions or adaptation. Paul. From ncoghlan at gmail.com Tue Apr 4 12:12:35 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 04 Apr 2006 20:12:35 +1000 Subject: [Python-3000] Generic functions In-Reply-To: <44315A89.9000104@colorstudy.com> References: <44315A89.9000104@colorstudy.com> Message-ID: <44324693.9070201@gmail.com> Ian Bicking wrote: > As an alternative to adaptation, I'd like to propose generic functions. > I think they play much the same role, except they are much simpler to > use and think about. Unfortunately the rules for choosing *which* implementation to dispatch to, even if restricting things to concrete types only, are necessarily complex. As Walter pointed out, the deliberately simplistic sample code elsewhere in this thread dispatches to a different method depending on order of registration and hashing idiosyncrasies. To fix that, you either have to stop permitting subclasses of registered argument types, or else you have to define the idea of a "closest" signature match, at which point you've been forced to throw "simple" right out the window. Given this type hierarchy: A B C D E F and a call containing (D(), E(), F()), is the type signature (B, B, C) a closer match than the signature (A, E, F)? In a certain sense, an adapting protocol is just a generic function that only permits dispatch of single argument functions based on type of that argument - as such, adaptation is necessarily simpler than full generic function support. To go back to Ian's pretty printing example, here's an implementation of pretty printing using an adaptation protocol: ------------pprint.py----------------- import adaptation class IPrettyPrintable(adaptation.Protocol): def pformat(self, indent, width, depth): """Return the pretty representation of the object""" # Register known pretty printer functions (e.g. for list and dict) here class PrettyPrinter: # Other normal pretty printer methods go here def pformat(self, obj): printable = IPrettyPrintable(obj) # Permit extension to new types return printable.pformat(self.indent, self.width, self.depth) -------------------------------------- Now all another class would need to do to make itself pretty printable: ------------mymodule.py----------------- from pprint import IPrettyPrintable class MyClass(object): def pformat(self, indent, width, depth): # Do the pretty print operation # based on self return formatted IPrettyPrintable.register_type(MyClass) # No adapter needed -------------------------------------- Suppose, however, I wanted to pretty print someone else's class. That's just as easy: ------------mymodule.py----------------- from pprint import IPrettyPrintable from their_module import TheirClass class MyAdapter(object): def __init__(self, obj): self.obj = obj def pformat(self, indent, width, depth): # Do the pretty print operation # based on self.obj return formatted IPrettyPrintable.register_type(TheirClass, MyAdapter) -------------------------------------- Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From pje at telecommunity.com Tue Apr 4 10:02:19 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 04 Apr 2006 01:02:19 -0700 Subject: [Python-3000] Generic functions In-Reply-To: <44320C36.7070507@colorstudy.com> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> Message-ID: <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> At 11:03 PM 4/3/2006, Ian Bicking wrote: >Guido van Rossum wrote: >>On 4/3/06, Ian Bicking wrote: >>>As an alternative to adaptation, I'd like to propose generic functions. >>> I think they play much the same role, except they are much simpler to >>>use and think about. >>Given that Phillip Eby is another proponent of generic functions I >>seriously doubt the latter. Hm. :) Rather than branch all over the map, let me focus for a moment on a very simple type of generic function - the kind that is essentially equivalent to PEP 246-style adaptation. This will make it easier to see the relationship. In the RuleDispatch package, these simple generic functions are defined using dispatch.on and f.when(), like this: import dispatch @dispatch.on('ob') # name of the argument to dispatch based on def pprint(ob): """This is a pretty-print function""" @pprint.when(object) def pprint(ob): print repr(ob) @pprint.when(list) def pprint(ob): # code for the list case Now, this is exactly equivalent to the much longer code that one would write to define an IPrettyPrintable interface with a pprint() method and adapter classes to define the implementation methods. Yes, it's a convenient example - but it also corresponds to a fairly wide array of problems, that also happen to be a significant number of uses for adaptation. As Ian mentioned, I wrote PyProtocols first, and discovered generic functions afterward. In fact, 'dispatch.on()' and friends are actually implemented internally using PyProtocols adapters. For operations that do not require any state to be saved across method invocations, and for interfaces with a small number of methods (e.g. just one!), generic functions of this type are a more compact and expressive approach. The kicker is this: for this subset of generic functions, it doesn't matter whether you make generic functions primitive, or you make adaptation primitive, because either can be implemented in terms of the other. For example, if you wanted to implement adaptation using generic functions, you would just use a function for each interface or protocol, and the "methods" of the function would be the adapter factories. Anyway, this type of generic function is essentially similar to Python's tables today for things like pretty printing, pickling, and so on. It can be viewed as merely a syntactic convenience coupled with standardization of how such an operation's registry will work, both for registration and lookups. In essence, Python's standard library already has generic functions, they are just implemented differently for each operation that needs such a registry. >>Watch it though. it may be a great example to explain generic >>functions. But it may be the only example, and its existence may not >>be enough of a use case to motivate the introduction of gneric >>functions. It's sufficiently general to encompass just about any "visitor" pattern. I wrote a short article on this a while back: http://peak.telecommunity.com/DevCenter/VisitorRevisited >>Whoa! First of all, my gut reaction is already the same as for >>adaptation: having a single global registry somehow feels wrong. (Or >>is it not global? "internal" certainly sounds like that's what you >>meant; but for methods this seems wrong, one would expect a registry >>per class, or something like that.) This confusion is due to Ian mixing methods and functions in the example. Generic functions are *functions*. If you put them in classes, Python makes them methods. But there's nothing magical about that - it's still a function. The VisitorRevisited article explains this better, and in a way that doesn't delve into the technicalities. Also, its examples are actually working ones based on a fixed spec (i.e., the RuleDispatch implementation), so there's no handwaving to get in the way. >>Next, I wonder what the purpose of the PrettyPrinter class is. Is it >>just there because the real pprint module defines a class by that >>name? Or does it have some special significance? > >It's there because it is matching the pprint module. Also it holds >some state which is useful to keep separate from the rest of the >arguments, like the current level of indentation. Note that this isn't really needed, nor necessarily ideal. If I were really writing a pretty printer, I'd put indentation control in an output stream argument, which would allow me to reuse an IndentedStream class for other purposes. It would then suffice to have a single pprint(ob,stream=IndentedStream(sys.stdout)) generic function, and no need for a PrettyPrinter class. >>Are generic functions >>really methods? Can they be either? > >They can be either. They're really honest-to-goodness Python *functions* (with extra stuff in the function __dict__). That means they behave like ordinary functions when it comes to being methods. They can be instance, class, or static methods, or not methods at all. >>Ah, the infamous "when" syntax again, which has an infinite number of >>alternative calling conventions, each of which is designed to address >>some "but what if...?" objection that might be raised. RuleDispatch actually has a fixed number of when() signatures, and the one Ian gave isn't one of them. Single-dispatch functions' when() takes a type or an interface, or a sequence of types or interfaces. Predicate-dispatch functions take a predicate object, or a string containing a Python expression that will be compiled to create a predicate object. None of RuleDispatch's when() decorators take keyword arguments at the moment, at least not in the way of Ian's example. >>What does when(object=list) mean? Does it do an isinstance() check? > >Yes; I think RuleDispatch has a form (though I can't remember what >the form is -- it isn't .when()). For simple single-dispatch, it's @some_function.when(type). So in this case it'd be @pprint.when(list). >>Is there any significance to the name pformat_list? Could I have >>called it foobar? Why not just pformat? > >Just for tracebacks, and for example to make it greppable. Actually, there's one other reason you might want a separate name, and that's to reuse the code in an explicit upcall. You could then explicitly invoke pformat_list() on something that wasn't an instance of the 'list' type. For example, you could just call "pprint.when(SomeListLikeType)(pformat_list)" to register pformat_list as the method to be called for SomeListLikeType, as well as for 'list' and its subclasses. If it were just called 'pformat', there'd be no name by which you could access the original function. But you *are* allowed to call it whatever you want. If it's called 'pformat' in this case, you just lose access to it; the name 'pformat' remains bound to the overall generic function, rather than to the specific implementation. >>>* It requires cooperation from the original function (pformat -- I'm >>>using "function" and "method" interchangably). >>Thereby not helping the poor reader who doesn't understand all of this >>as well as you and Phillip apparently do. Right. Don't interchange them, they have different meanings. A generic function is just a function. It's not a method unless you make it one. Generic functions can *have* methods, however. Each *implementation* for a generic function, like each reducer in the pickling registry, is called a "method" *of* the generic function. This terminology is lifted straight from Lisp, but I'm not attached to it. If anybody has a better terminology, in fact, I'm all for it! In CLOS, by the way, there are no object methods, only generic function methods. Rather than add methods to classes, you add them to the generic functions. As with adaptation, this is 100% equivalent in computational terms. What's *not* equivalent is the user interface. If you are writing a closed-ended program in any language, it really doesn't matter whether the objects have methods or the functions do. You're just writing a program, so how you organize it is a matter of taste. However, if you are writing an extensible library, then generic functions (especially multiple-dispatch ones) have a *tremendous* advantage, because you aren't forced to pick only one way to expand. And if you don't have generic functions in your language, you'll just reinvent them in your library -- like with pickle and copy_reg in the Python stdlib. The advantage that they offer is twofold: 1) you can add new operations that cut across all types, existing or future. 2) you can add new types that can work with any operation, existing or future. Side note: Ruby dodges this problem by making classes open-ended. Since you can add a new __special__ method to any existing class (even one you didn't write, including built-in types), you don't need to have operation-specific type lookup registries (ala pickle/copy_reg), and for that matter you don't need adaptation! The only problem you might run into is namespace clashes, and I'm not sure how Ruby addresses this. Anyway, there are lots of ways to skin these cats, the main difference is in how the user sees things and expresses their ideas. If you are creating a closed system used by a single developer, you don't need any of this. But if you need an extensible library, you need a mechanism for extension that allows third parties to implement operation A for type B, even if they did not create the libraries containing A and B. Adaptation, generic functions, and Ruby-style open classes are all computationally-equivalent mechanisms for doing this, in that you could translate a given library to use any of the three techniques. >>I'm guessing that you are contrasting it with len(), which could be >>seen as a special kind of built-in "generic function" if one squints >>enough, but one that requires the argument to provide the __len__ >>magic method. But since len() *does* require the magic method, doesn't >>that disqualify it from competing? You could view this as being a generic function with only one method: @len.when(object) def len(ob): return ob.__len__() And which of course is not extensible. :) >Yes, this is in contrast with len(), which achieves its goal because >the people who write the len() function write the entire language, >and can put a __len__ on whatever they want ;) For other cases >magic-method based systems tend to look like: > >def pprint(object): > if isinstance(object, list): ... > elif isinstance(object, tuple): ... > ... > elif hasattr(object, '__pprint__'): > object.pprint() > else: > print repr(object) > >That is, all the built in objects get special-cased and other >objects define a magic method. Yeah, and then there are all the libraries like pydoc that have these huge if-else trees and aren't extensible because they don't even have a magic method escape or any kind of registry you can extend. A uniform way to do this kind of dispatching means uniform extensibility of libraries. >>>* The function is mostly self-describing. >>Perhaps once you've wrapped your head around the when() syntax. To me >>it's all magic; I feel like I'm back in the situation again where I'm >>learning a new language and I haven't quite figured out which >>characters are operators, which are separators, which are part of >>identifiers, and which have some other magical meaning. IOW it's not >>describing anything for me, nor (I presume) for most Python users at >>this point. I'd be interested to know if you still feel that way after reading VisitorRevisited, since it doesn't do any hand-waving around what when() does. >>Tell us more about the registration machinery. Revealing that (perhaps >>simplified) could do a lot towards removing the magical feel. My simple generic function implementation is actually implemented using adaptation - I create a dummy interface for the generic function, and then define adapters that return the implementation functions. IOW, adapt(argument,dummy_interface) returns the function to actually call. The actual generic function object is a wrapper that internally does something like: def wrapper(*args,**kw): return adapt(some_arg, dummy_interface)(*args, **kw) However, if I were implementing a generic function like this "from scratch" today, I'd probably just make the function __dict__ contain a registry dictionary, and the body of the function would be more like: def wrapper(*args,**kw): for cls in type(some_arg).__mro__: if cls in wrapper.registry: return wrapper.registry[cls](*args, **kw) Voila. Simple generic functions. Of course, the actual implementation today is complex because of support for classic classes and even ExtensionClasses, and having C code to speed it up, etc. >>>* Magic methods do *not* have this import problem, because once you have >>>an object you have all its methods, including magic methods. >>Well, of course that works only until you need a new magic method. > >Yes, pluses and minuses ;) Ruby of course solves this by letting you add the magic methods to existing classes, and Python allows this for user-defined types, although we look down on it as "monkey patching". >>>Type-based generic functions and adaptation are more-or-less equivalent. >>> That is, you can express one in terms of the other, at least >>>functionally if not syntactically. >>Could you elaborate this with a concrete example? Here's pprint() written as an interface and adaptation: class IPPrintable(Interface): def pprint(): """pretty print the object"" def pprint(ob): adapt(ob, IPPrintable).pprint() In reality, this would be much more code, because I've left out the adapter classes to adapt each type and give it a pprint() method. >>>Anyway, I think generic functions are very compatible with Python syntax >>>and style, and Python's greater emphasis on what an object or function >>>can *do*, as opposed to what an object *is*, as well as the use of >>>functions instead of methods for many operations. People sometimes see >>>the use of functions instead of methods in Python as a weakness; I think >>>generic functions turns that into a real strength. >>Perhaps. The import-for-side-effect requirement sounds like a >>showstopper though. In practice, it doesn't happen that often, because you're either adding an operation to some type of yours, or adding a type to some operation of yours. And in the case where you're adding operation A to type B and you own neither of the two -- you're still not importing for side-effect except in the sense that the code that defines the A(B) operation is part of your application anyway. The case where there is some *third-party* module that implements A(B) is quite rare in my experience. However, you could have such a scenario *now*, if "operation A" is "pickle", for example, and B is some type that isn't ordinarily picklable. Such a situation requires importing for side effects now, if you're not the one who wrote the pickling operation for it. In fact, it doesn't matter *what* approach you use to provide extensibility; if it allows user C to define operation A on type B, it requires user D to import C's code for side-effects if they want to use it. This equally applies to adaptation and to Ruby's open classes, as it does to generic functions. It's simply the logical consequence of having third-party extensibility! From g.brandl at gmx.net Tue Apr 4 14:00:56 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 04 Apr 2006 14:00:56 +0200 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 In-Reply-To: References: Message-ID: Guido van Rossum wrote: > I was going to comment about this one "check it in, we'll add to it > later", but then I realized it's not 100% clear whether this is a > feature PEP or a meta-PEP? It focuses on features so by that yardstick > it's a feature PEP. But in its list-of-miscellany nature it > approximates a meta-PEP. Hmm, perhaps it ought to be PEP 3999 (which > can be seen as minus one in the numbering scheme for Python-3000 PEPs > :-)? > > In any case let's not let it longer for long. Realistically, I think > it can be a meta-PEP. Since it's a meta-PEP, but closely related to features, I've checked it in with number 3099, that is, "first feature-PEP minus one". Related question: as the current PEP 3000 contains mainly feature proposals, shouldn't it be renamed to 3100? PEP 3000 could then be a quasi-summary of 3xxx PEPs and random important facts that don't fit elsewhere. Georg From g.brandl at gmx.net Tue Apr 4 14:11:57 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 04 Apr 2006 14:11:57 +0200 Subject: [Python-3000] pre-PEP: Process for reviewing/improving stdlibmodules in 3.0 In-Reply-To: References: Message-ID: Terry Reedy wrote: > "Guido van Rossum" wrote in message > news:ca471dc20604032029u475e8e7an2b15764d1ccd008e at mail.gmail.com... >>I didn't see any comments on this PEP. > > For me, it is pretty clear and sensible, hence not much to say. > Plus I expected others to says whatever was needed ;-). > >> I'm not sure that the order in which the >> steps are to be carried out is all that important, nor that it's >> necessary to do this in the same order for all modules, When I numbered the steps, I didn't think of prescribing a specific order of the steps. I'll remove the numbers. > Such thoughts had occurred to me, but I don't think of anything specific to > add. I read the steps as strong guidelines rather than a straightjacket. > I imagine refinements will grow out of experience. > >> but otherwise >> the only thing that bugs me is the reference to the great stdlib >> renaming (which I'm not sure is such a great idea). > > This is the main thing that caught my attention since I did not know that > you had made a decision, which, obviously now, you haven't. So I would > break that sentence into two: > > There are proposals for a "great stdlib renaming" introducing a hierarchic > library > namespace. That possibility aside, some module's names are known ... Yes, that's better. > And add PEP reference if and when there is a renaming PEP. > >> I expect that some of the goals (especially test coverage) are too >> ambitious, >> but it's worth at least aspiring to great works! That's exactly how the proposal is meant. I do not expect all modules to be totally reworked either ;) If there's no other general problem, I'll check the improved version in as PEP 3001. Georg From g.brandl at gmx.net Tue Apr 4 14:16:36 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Tue, 04 Apr 2006 14:16:36 +0200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4431CB8D.9040004@canterbury.ac.nz> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Barry Warsaw wrote: > >> I don't much like the $"" prefix > > This was discussed during the last round of formatting > wars, and the conclusion was that having $ both > inside and outside the string would be too visually > confusing. Another issue is that current string prefixes act at parse time while the $ prefix would act at runtime. Georg From ncoghlan at gmail.com Tue Apr 4 14:40:09 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 04 Apr 2006 22:40:09 +1000 Subject: [Python-3000] Adaptation: T->P vs P->P In-Reply-To: <44315FC3.9030007@cox.net> References: <20060403101814.4vme5g95gogskckw@login.werra.lunarpages.com> <44315FC3.9030007@cox.net> Message-ID: <44326929.3050008@gmail.com> Tim Hochberg wrote: > Michael Chermside wrote: >> Tim's proposal (if I understand it correctly) depends on transitivity -- >> his idea is to have the class authors register as satisfying a small >> number of widely-known protocols (Psrc in his case), >> > Yes. > > >> then use transitivity >> to infer ways to satisfy less well-known protocols. >> > No. > > All I'm saying is that every object is associated with zero or more > protocols. The details of this association are TBD, but look here for > one simple/simplistic approach: > http://members.cox.net/~tim.hochberg/adaption2.py > Adapters are always between two protocols. There is not attempt at > transitivity at all. So adapt looks like this: > > def adapt(obj, Pdest, registry=_global_registry): > protocols = find_protocols(obj) > if Pdest in protocols: # Always return the original object if it > satisfies Pdest > return obj > for Psrc in protocols: > adapter = registry.get((Psrc,Pdest), None) > if adapter is not None: > return adapter(obj) > raise ValueError('adapter not found') > The interesting thing about this T->P1->P2 system is that it is based on two rules: 1. A type can be registered as implementing a protocol 2. A protocol can be registered as adapting to another protocol The annoyance is that there is then no way to register a direct adaptation from MyType to P2. Instead, you have to define a "IMyType" interface that "MyType" implements, and then adapt from IMyType to P2. Not exactly the most convenient API ever :) However, consider the transitive chain involved when adapting an object O of type T, which implements protocol P1, to protocol P2: P2 has an adapter A registered to convert from protocol P1 Type T is registered as implementing protocol P1 Object O is an exact instance of either type T or a subclass of type T Therefore A(O) can be used to adapt object O to protocol P2. The "registered as implementing" transitive step is intriguingly similar to the transitive step in getting from the type T to its instance O. Since class inheritance is assumed to be lossless (interface-wise) unless told otherwise, it doesn't matter whether or not O is an instance of T itself, or an instance of a subclass. This creates the inspiration for a limited form of transitivity which isn't as brain-bending as the full-fledged version in PyProtocols**. Specifically, transitivity would be permitted only when no adaptation is involved in the transitive part of the link. Adaptation would only be permitted on the final non-transitive leg, where the result implements the target protocol. I've got the skeleton of an implementation for that idea (using the distributed adaptation approach and stealing liberally from the concepts in PyProtocols) - I'll try to tidy that up to a postable form. Cheers, Nick. ** For those familiar with adapter chains in PyProtocols, the basic idea is to permit only transitive links with a simplified length of 1, where every step other than the last is a NO_ADAPTER_NEEDED step. The search algorithm in such a simplified system can be a heck of a lot dumber than the one in PyProtocols (since it only needs to keep track of visited nodes in order to avoid loops and redundant searches, and the only error it has to detect is the presence of more than 1 legal adaptation path), while still permitting the registration of a protocol->protocol adapter that will work for any object that implements the source protocol. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From msoulier at digitaltorque.ca Tue Apr 4 14:52:47 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Tue, 4 Apr 2006 08:52:47 -0400 Subject: [Python-3000] A few small py3k wishes In-Reply-To: <200604041322.02900.gmccaughan@synaptics-uk.com> References: <20060402222130.GA12075@tigger.digitaltorque.ca> <200604041322.02900.gmccaughan@synaptics-uk.com> Message-ID: <20060404125247.GC12075@tigger.digitaltorque.ca> On 04/04/06 Gareth McCaughan said: > > It's a minor thing, but it'd be nice to make a method alias for append() on > > mutable sequences called push(), to go with pop(). > > That would confuse anyone used to Lisp, where after > > (defvar foo (list 1 2 3 4)) > (puch 99 foo) > > the value of FOO is (99 1 2 3 4) rather than (1 2 3 4 99). > I take no stand on how much not confusing Lispers matters > to Python 3k. It would ease transition from Perl, and be consistent in the method naming. If one is going to use a method obviously taken from the stack ADT, having a method normally called push() would be less confusing. I know that I found it so, having both used stacks and Perl. I can make a list subclass obviously, but I've always found append() and pop() one of those arrangements that just makes new Python programmers say, huh? Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060404/8180123a/attachment.pgp From gmccaughan at synaptics-uk.com Tue Apr 4 14:22:02 2006 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Tue, 4 Apr 2006 13:22:02 +0100 Subject: [Python-3000] A few small py3k wishes In-Reply-To: <20060402222130.GA12075@tigger.digitaltorque.ca> References: <20060402222130.GA12075@tigger.digitaltorque.ca> Message-ID: <200604041322.02900.gmccaughan@synaptics-uk.com> > It's a minor thing, but it'd be nice to make a method alias for append() on > mutable sequences called push(), to go with pop(). That would confuse anyone used to Lisp, where after (defvar foo (list 1 2 3 4)) (puch 99 foo) the value of FOO is (99 1 2 3 4) rather than (1 2 3 4 99). I take no stand on how much not confusing Lispers matters to Python 3k. -- g From aahz at pythoncraft.com Tue Apr 4 16:17:59 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 4 Apr 2006 07:17:59 -0700 Subject: [Python-3000] StringIO vs cStringIO, pickle vs cPickle In-Reply-To: References: Message-ID: <20060404141759.GB9247@panix.com> On Tue, Apr 04, 2006, Just van Rossum wrote: > > Has there been any discussion yet about StringIO vs cStringIO and pickle > vs cPickle? Now there is! > While I appreciate that there are maintained pure Python versions of > those modules, as a user it irritates me that I have the choice. The > argument so far for not replacing StringIO with cStringIO was that > they're subtly incompatible (mostly StringIO's "feature" to support > unicode, right?), so Python 3000 seems like a good time to reconsider > that option. I don't see any reason to not always want to prefer the > faster version. Suggestion: make string_io (we're renaming the module, right?) and pickle contain a UserStringIO and UserPickle class. Alternatively, they could be moved wholesale into a python_types module, where we collectively maintain Python versions of *all* Python types, including lists and tuples. IOW, I do argue for keeping the Python code somehow, and keeping them officially maintained. Obviously it gets a little absurd with lists and numbers, for example, but I think that showing off good use of __new__ would be a Good Thing. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet From gmccaughan at synaptics-uk.com Tue Apr 4 16:31:25 2006 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Tue, 4 Apr 2006 15:31:25 +0100 Subject: [Python-3000] A few small py3k wishes In-Reply-To: <20060404125247.GC12075@tigger.digitaltorque.ca> References: <200604041322.02900.gmccaughan@synaptics-uk.com> <20060404125247.GC12075@tigger.digitaltorque.ca> Message-ID: <200604041531.26172.gmccaughan@synaptics-uk.com> On Tuesday 2006-04-04 13:52, Michael P. Soulier wrote: > On 04/04/06 Gareth McCaughan said: ... > > That would confuse anyone used to Lisp, where after > > > > (defvar foo (list 1 2 3 4)) > > (puch 99 foo) [of course I meant to type "push". Lisp's operator names are obscure sometimes, but not that obscure] > > the value of FOO is (99 1 2 3 4) rather than (1 2 3 4 99). > > I take no stand on how much not confusing Lispers matters > > to Python 3k. > > It would ease transition from Perl, and be consistent in the method naming. I'd forgotten that Perl has a "push" with that behaviour. And, come to think of it, Lisp has VECTOR-PUSH and VECTOR-PUSH-EXTEND that add onto the end of a vector. Inconsistent naming: Bad Lisp! Bad! No biscuit! Anyway, Python surely has more perlites than lispniks among its newcomers. So: the dubious antiprecedent of Lisp's PUSH is outweighed by the precedent of Perl's and somewhat invalidated by Lisp's internal inconsistency. I retract everything :-). > If one is going to use a method obviously taken from the stack ADT, having a > method normally called push() would be less confusing. I know that I found it > so, having both used stacks and Perl. > > I can make a list subclass obviously, but I've always found append() and pop() > one of those arrangements that just makes new Python programmers say, huh? Oh, yes. The combination is ... peculiar. -- g From tim.hochberg at ieee.org Tue Apr 4 17:04:34 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Tue, 04 Apr 2006 08:04:34 -0700 Subject: [Python-3000] Generic functions In-Reply-To: <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> Message-ID: Phillip J. Eby wrote: > At 11:03 PM 4/3/2006, Ian Bicking wrote: > >>Guido van Rossum wrote: >> >>>On 4/3/06, Ian Bicking wrote: >>> >>>>As an alternative to adaptation, I'd like to propose generic functions. >>>> I think they play much the same role, except they are much simpler to >>>>use and think about. >>> >>>Given that Phillip Eby is another proponent of generic functions I >>>seriously doubt the latter. > > > Hm. :) > > Rather than branch all over the map, let me focus for a moment on a > very simple type of generic function - the kind that is essentially > equivalent to PEP 246-style adaptation. This will make it easier to > see the relationship. > > In the RuleDispatch package, these simple generic functions are > defined using dispatch.on and f.when(), like this: > > import dispatch > > @dispatch.on('ob') # name of the argument to dispatch based on > def pprint(ob): > """This is a pretty-print function""" > > @pprint.when(object) > def pprint(ob): > print repr(ob) > > @pprint.when(list) > def pprint(ob): > # code for the list case > > Now, this is exactly equivalent to the much longer code that one > would write to define an IPrettyPrintable interface with a pprint() > method and adapter classes to define the implementation > methods. Yes, it's a convenient example - but it also corresponds to > a fairly wide array of problems, that also happen to be a significant > number of uses for adaptation. Rewriting this using "distributed" adapters looks like this: >>> pprint = Protocol('pprint') >>> @pprint.when(object) ... def pprint_obj(obj): ... print repr(obj) >>> @pprint.when(list) ... def pprint_list(obj): ... print 'pretty(%r)' % obj Pretty similar! Stealing the when decorator cleans up the use of copy_reg.pickle rewrite a bit as well. I imagine that where these would diverge is when there is more than one argument. Regards, -tim P.S., Here's Protocol. To save a few bytes I've stripped classic class support. class Protocol(object): all_protocols = set() def __init__(self, name): self.name = name self.registry = {} self.all_protocols.add(self) def __repr__(self): return "" % self.name __str__ = __repr__ def __call__(self, obj): #~ mro = _get_mro(obj) mro = type(obj).__mro__ for cls in mro: adapter = self.registry.get(cls, None) if adapter is not None: return adapter(obj) raise ValueError('adapter not found') def register(self, adapter, *types): if not callable(adapter): raise TypeError("adapters must be callable") for t in types: self.registry[t] = adapter def when(self, *types): def decorator(adapter): self.register(adapter, *types) return adapter return decorator From ianb at colorstudy.com Tue Apr 4 17:27:57 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 10:27:57 -0500 Subject: [Python-3000] Generic functions In-Reply-To: <44322C3A.5040509@livinglogic.de> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <44322C3A.5040509@livinglogic.de> Message-ID: <4432907D.8070407@colorstudy.com> Walter D?rwald wrote: > What happens, if I do the following > > @PrettyPrinter.pformat.when(object=list) > def foo(...): > ... > > @PrettyPrinter.pformat.when(object=object) > def foo(...): > ... > > How does it know which isinstance() check to do first? It can see that isinstance(list, object) is true, and orders the checks accordingly. If the condition is entirely ambiguous it raises an error. At least, this is how RuleDispatch works. It has some knowledge of operations like isinstance so that it can tell that the set of objects that are instances of list is a strict subset of the objects that are instances of object. Anyway, for the specific case of type-based dispatch it's fairly straight forward to figure out what condition is most specific. Though, say, if there's a specialized method for (A, object) and a specialized method for (object, B) and you call with (A, B) that's just ambiguous, and without an explicit resolution it should be an error. I guess another take on this, that precedes RuleDispatch, is multiple dispatch, like in http://www-128.ibm.com/developerworks/linux/library/l-pydisp.html -- the article also precedes the decorator syntax I guess (or just doesn't use it); I think the decorator syntax makes this look much nicer. Though single-argument type-based dispatch is probably the most common case. > And what happens with performance if I have registered many handler > functions? Depends on the implementation; I believe RuleDispatch builds a decision tree. >> [...] >> The implementation of my simplistic form of generic function isn't too >> hard. Ignoring keyword arguments, it might work like: >> >> class generic(object): >> def __init__(self, func): >> self.func = func >> self.registry = {} >> def __call__(self, *args): >> for pattern, implementation in self.registry.items(): >> for passed, expected in zip(args, pattern): >> # None is a wildcard here: >> if (expected is not None and >> not isinstance(passed, expected)): >> break >> else: >> return implementation(*args) >> return self.func(*args) >> def when(self, *args): >> def decorator(func): >> self.registry[args] = func >> return func >> return decorator >> def __get__(self, obj, type=None): >> if obj is None: >> return self >> return types.MethodType(self, obj, type) >> >> There's lots of details, and handling keyword arguments, dealing >> intelligently with subclasses, and other things I probably haven't >> thought of. But anyway, this allows: >> >> class PrettyPrinter: >> def pformat(self, object): ... >> >> # Without keyword arguments I have to give a wildcard for the self >> # argument... >> @PrettyPrinter.pformat(None, list) >> def pformat_list(self, object): >> ... > > > I don't understand! There's no generic in sight here! Maybe I should have called this more modest implementation simplistic multiple dispatch. In this case it is dispatching on the second argument (the first is "self" and not as interesting). pformat_list is an implementation for objects of type list, PrettyPrinter.pformat is a fallback implementation. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Tue Apr 4 17:38:16 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 10:38:16 -0500 Subject: [Python-3000] Generic functions In-Reply-To: References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> Message-ID: <443292E8.9000508@colorstudy.com> Tim Hochberg wrote: > Rewriting this using "distributed" adapters looks like this: > > >>> pprint = Protocol('pprint') I suppose this means that Protocol('pprint') implies an interface like: class IPprint(Interface): def __call__(object): """print the pretty representation of the object""" Viewed from this perspective, RuleDispatch-style generic functions use the default implementation as the description of the interface, and only allow for callable interfaces. > >>> @pprint.when(object) > ... def pprint_obj(obj): > ... print repr(obj) > > >>> @pprint.when(list) > ... def pprint_list(obj): > ... print 'pretty(%r)' % obj > > > Pretty similar! Stealing the when decorator cleans up the use of > copy_reg.pickle rewrite a bit as well. I imagine that where these would > diverge is when there is more than one argument. > > > Regards, > > -tim > > > P.S., Here's Protocol. To save a few bytes I've stripped classic class > support. I suppose on the py3k list we don't have to apologize for ignoring classic classes ;) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From msoulier at digitaltorque.ca Tue Apr 4 17:40:16 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Tue, 4 Apr 2006 11:40:16 -0400 Subject: [Python-3000] A few small py3k wishes In-Reply-To: <200604041531.26172.gmccaughan@synaptics-uk.com> References: <200604041322.02900.gmccaughan@synaptics-uk.com> <20060404125247.GC12075@tigger.digitaltorque.ca> <200604041531.26172.gmccaughan@synaptics-uk.com> Message-ID: <20060404154015.GE12075@tigger.digitaltorque.ca> On 04/04/06 Gareth McCaughan said: > [of course I meant to type "push". Lisp's operator names > are obscure sometimes, but not that obscure] Indeed. > I'd forgotten that Perl has a "push" with that behaviour. And, > come to think of it, Lisp has VECTOR-PUSH and VECTOR-PUSH-EXTEND > that add onto the end of a vector. Inconsistent naming: Bad Lisp! > Bad! No biscuit! :) > Anyway, Python surely has more perlites than lispniks among > its newcomers. So: the dubious antiprecedent of Lisp's PUSH > is outweighed by the precedent of Perl's and somewhat invalidated > by Lisp's internal inconsistency. I retract everything :-). Well, I am one disillusioned Perl programmer who came to find Python the answer to my Perl complaints...almost. I'm just trying to fix the almost part now. > Oh, yes. The combination is ... peculiar. Yup. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060404/431be08b/attachment.pgp From ianb at colorstudy.com Tue Apr 4 17:41:39 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 10:41:39 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> Message-ID: <443293B3.1020706@colorstudy.com> Georg Brandl wrote: > Greg Ewing wrote: > >>Barry Warsaw wrote: >> >> >>>I don't much like the $"" prefix >> >>This was discussed during the last round of formatting >>wars, and the conclusion was that having $ both >>inside and outside the string would be too visually >>confusing. > > > Another issue is that current string prefixes act at parse time > while the $ prefix would act at runtime. A $ prefix would act at parse time, i.e., $"answer: $answer" would be equivalent to ("answer: " + str(answer)). like r and u, $ would change the meaning of the literal, it wouldn't be an operator, and so this would be a syntax error: answer = 1 msg = "answer: $answer" print $msg Though I also agree that $"answer: $answer" introduces a confusing number of $'s. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Tue Apr 4 18:38:33 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue, 04 Apr 2006 09:38:33 -0700 Subject: [Python-3000] Generic functions Message-ID: <7.0.1.0.0.20060404091150.0222cc70@after12am.com> Nick Coghlan wrote: >Unfortunately the rules for choosing *which* implementation to dispatch to, >even if restricting things to concrete types only, are necessarily complex. > >As Walter pointed out, the deliberately simplistic sample code elsewhere in >this thread dispatches to a different method depending on order of >registration and hashing idiosyncrasies. The simple code that I posted actually depends only on __mro__. >To fix that, you either have to stop permitting subclasses of registered >argument types, Um, no. :) The single-dispatch case works quite nicely with __mro__ (or the moral equivalent thereof for classic classes). >or else you have to define the idea of a "closest" signature >match, at which point you've been forced to throw "simple" right out >the window. It's no more complex than Python's __mro__ computation is, actually. >Given this type hierarchy: > A > B C >D E F > >and a call containing (D(), E(), F()), is the type signature (B, B, C) a >closer match than the signature (A, E, F)? Neither. It's ambiguous if you're doing symmetric dispatch. The correct algorithm for scanning a set of signatures like this is that you must check *all* signatures for applicability, and then choose the one that is "most specific" by throwing out dominated matches (ones that are a superset of some other match). If you end up with more than one applicable signature, you have a conflict. This is almost exactly the same as Python's metaclass determination algorithm, by the way: it goes through all the candidate metaclasses and throws away dominated metaclasses. If at the end there is more than one candidate remaining, it's a metaclass conflict. Now, I'm not saying that you'd actually want to implement this matching algorithm in such a naive way as a loop over all possibilities. RuleDispatch creates a tree of tests that eliminates entire groups of signatures from consideration while testing conditions in parallel. So in your example, the D() argument has its __mro__ scanned to eliminate any signatures that don't support a 'B' , 'A', or 'D' in the first position. Then the E() is scanned, and finally the F(). In each case, the equivalent of a single isinstance() check is done, and the result used to choose a new sub-branch of the tree. The final node after doing the three quasi-isinstance operations is a node representing the set of applicable signatures. If there is more than one dominant signature, it's an ambiguity error. I'm simplifying a bit here, because RuleDispatch actually allows you to create custom combiners to handle multiple applicable signatures entirely according to your taste. So, you could actually define an alternative resolution that didn't complain about ambiguous methods. For example, you could decide that you would use a left-to-right asymmetric resolution (ala Common Lisp) so that (B,B,C) would be more specific than (A,E,F) because B->A in the first position. Or you could decide that you would break ties using the definition order, which is a popular approach if you're just using the generic function as a kind of "listener" hook, where you plan to call *all* the applicable signatures, even duplicates, because the function is intended as a broadcast hook. (Notice, by the way, that this makes another class of "registration" or "registries" in the stdlib unnecessary; for example, you could ditch the atexit module by just having sys.exitfunc be a generic function that users register methods for!) >In a certain sense, an adapting protocol is just a generic function that only >permits dispatch of single argument functions based on type of that >argument - >as such, adaptation is necessarily simpler than full generic function support. Adaptation is equivalent to a single-dispatch generic function, yes. And multiple dispatch is certainly more complex to implement. However, multiple dispatch is a lot easier to implement if you have single-dispatch generic functions available. A huge amount of RuleDispatch is built on single-dispatch generic functions and adaptation. If I were going to do it over again, I wouldn't use the adaptation bit at all; I'd just start with single-dispatch generics. Or maybe not even those. I've considered that I could actually write the bootstrapping bits of RuleDispatch using a simple "naive" test-all-signatures algorithm, because by the time RuleDispatch was fully imported it would be able to replace those functions with its tree-building algorithms. But that's mostly fantasy at the moment, because RuleDispatch was effectively a sabbatical project for me, and although I've gotten a few nibbles about possible consulting projects built around it, nothing has materialized. So RuleDispatch is unlikely to get any serious development time from me in the near future. From guido at python.org Tue Apr 4 18:57:29 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 09:57:29 -0700 Subject: [Python-3000] hash as attribute/property In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> Message-ID: On 4/3/06, Delaney, Timothy (Tim) wrote: > I've been thinking that `hash` could be an attribute (or property if it > needs to be calculated on-the-fly) rather than the current method call. > > Or it could be an easy thing to add to the "won't change" PEP ... The latter, please. We apparently need more guidance for when it's appropriate to turn something into an attribute. I don't have time to write that up right now; if someone else can give it a shot, I'd appreciate it; or I'll come back to this topic later. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From walter at livinglogic.de Tue Apr 4 20:11:04 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue, 04 Apr 2006 20:11:04 +0200 Subject: [Python-3000] Generic functions In-Reply-To: <4432907D.8070407@colorstudy.com> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <44322C3A.5040509@livinglogic.de> <4432907D.8070407@colorstudy.com> Message-ID: <4432B6B8.7040705@livinglogic.de> Ian Bicking wrote: > Walter D?rwald wrote: >> What happens, if I do the following >> >> @PrettyPrinter.pformat.when(object=list) >> def foo(...): >> ... >> >> @PrettyPrinter.pformat.when(object=object) >> def foo(...): >> ... >> >> How does it know which isinstance() check to do first? > > It can see that isinstance(list, object) is true, and orders the checks > accordingly. If the condition is entirely ambiguous it raises an error. > At least, this is how RuleDispatch works. It has some knowledge of > operations like isinstance so that it can tell that the set of objects > that are instances of list is a strict subset of the objects that are > instances of object. Anyway, for the specific case of type-based > dispatch it's fairly straight forward to figure out what condition is > most specific. Though, say, if there's a specialized method for (A, > object) and a specialized method for (object, B) and you call with (A, > B) that's just ambiguous, and without an explicit resolution it should > be an error. RuleDispatch seems to contain some fairly magical code. But for dispatching on the type of one object, traversing the mro and looking up each type in the registry is sufficient and probably fast enough too. > I guess another take on this, that precedes RuleDispatch, is multiple > dispatch, like in > http://www-128.ibm.com/developerworks/linux/library/l-pydisp.html -- the > article also precedes the decorator syntax I guess (or just doesn't use > it); I think the decorator syntax makes this look much nicer. Though > single-argument type-based dispatch is probably the most common case. > >> And what happens with performance if I have registered many handler >> functions? > > Depends on the implementation; I believe RuleDispatch builds a decision > tree. > >>> [...] >>> The implementation of my simplistic form of generic function isn't >>> too hard. Ignoring keyword arguments, it might work like: >>> >>> class generic(object): >>> def __init__(self, func): >>> self.func = func >>> self.registry = {} >>> def __call__(self, *args): >>> for pattern, implementation in self.registry.items(): >>> for passed, expected in zip(args, pattern): >>> # None is a wildcard here: >>> if (expected is not None and >>> not isinstance(passed, expected)): >>> break >>> else: >>> return implementation(*args) >>> return self.func(*args) >>> def when(self, *args): >>> def decorator(func): >>> self.registry[args] = func >>> return func >>> return decorator >>> def __get__(self, obj, type=None): >>> if obj is None: >>> return self >>> return types.MethodType(self, obj, type) >>> >>> There's lots of details, and handling keyword arguments, dealing >>> intelligently with subclasses, and other things I probably haven't >>> thought of. But anyway, this allows: >>> >>> class PrettyPrinter: >>> def pformat(self, object): ... >>> >>> # Without keyword arguments I have to give a wildcard for the self >>> # argument... >>> @PrettyPrinter.pformat(None, list) >>> def pformat_list(self, object): >>> ... >> >> >> I don't understand! There's no generic in sight here! > > Maybe I should have called this more modest implementation simplistic > multiple dispatch. In this case it is dispatching on the second > argument (the first is "self" and not as interesting). pformat_list is > an implementation for objects of type list, PrettyPrinter.pformat is a > fallback implementation. The way the code is written PrettyPrinter.pformat is a simple unbound method. Was class PrettyPrinter: def pformat(self, object): ... supposed to be class PrettyPrinter: @generic def pformat(self, object): ... or something similar? Bye, Walter D?rwald > From tim.hochberg at cox.net Tue Apr 4 20:17:03 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue, 04 Apr 2006 11:17:03 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <443292E8.9000508@colorstudy.com> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> Message-ID: <4432B81F.1070903@cox.net> Considering generic function in combination adaption led to one more small change in the Protocol implementation that I'm playing with, and all of the sudden I'm left with something that I could actually use. What I realized was that you could factor out the part that potentially has a lot of variation and suddenly you have a relatively simple framework that's extensible in all sorts of ways. By default, everything is the same as in the last iteration, but now it's much easier to change the behaviour by subtyping. Let's return to the case that first bugged me about T->P dispatch. I have some tagged values and I want to assemble two of them to form a complex number. First we need a tagged value type. In reality I'd have this already >>> class TV(object): ... def __init__(self, value, tag): ... self.value = value ... self.tag = tag Then we need to subclass Protocol. I change the behaviour of keysof so that now things are looked up in the registry strictly based on their tags. Note that this can take an arbitrary number of arguments. In the default implementation (see below) only a single argument is allowed, which gives you the basic adapter(obj) -> newobj behaviour. >>> class TVProtocol(Protocol): ... def keysof(self, *args): ... try: ... yield tuple(x.tag for x in args) ... except AttributeError: ... pass >>> ascomplex = TVProtocol('complex_from_tagged') Then I define some converters: >>> import cmath >>> @ascomplex.when(('real', 'imag')) ... def complex_from_real_imag(real, imag): ... return real.value + 1j*imag.value >>> @ascomplex.when(('mag', 'angle')) ... def complex_from_mag_angle(mag, angle): ... return mag.value * cmath.exp(1j * cmath.pi / 180 * angle.value) >>> @ascomplex.when(('db', 'angle')) ... def complex_from_db_angle(db, angle): ... return 10**(db.value/20.0) * cmath.exp(1j * cmath.pi / 180 * angle.value) Here's some values that I can assume came from elsewhere: >>> tv_re, tv_im = TV(1, 'real'), TV(2, 'imag') >>> tv_db, tv_ang, tv_mag = TV(0, 'db'), TV(90, 'angle'), TV(2, 'mag') And here's how I'd use it: >>> ascomplex(tv_re, tv_im) (1+2j) >>> ascomplex(tv_db, tv_ang) (6.1230317691118863e-017+1j) >>> ascomplex(tv_mag, tv_ang) (1.2246063538223773e-016+2j) >>> ascomplex(tv_db, tv_mag) Traceback (most recent call last): ... ValueError: adapter not found All of the sudden this is looking like something I could probably use. I also tried a simple generic function implementation on top of this (no inheritance, keysof just returned a tuple of types). That was also easy. Could full blown generic dispatch be added just by subclassing and adding the correct, and obviously much more complex, version of keysof? It seems likely, but I'm not certain. If so, this is starting to look like a very promising approach. The updated Protocol implementation is below. Regards, -tim class Protocol(object): all_protocols = set() def __init__(self, name, doc=''): self.name = name self.registry = {} self.__doc__ = doc self.all_protocols.add(self) def __repr__(self): return "" % self.name __str__ = __repr__ def __call__(self, *args): for key in self.keysof(*args): adapter = self.registry.get(key, None) if adapter is not None: return adapter(*args) raise ValueError('adapter not found') def keysof(self, *args): if len(args) != 1: raise TypeError("%s expects 1-argument, got %s" (self, len(args))) obj = args[0] mro = type(obj).__mro__ for cls in mro: yield cls def register(self, adapter, *types): if not callable(adapter): raise TypeError("adapters must be callable") for t in types: self.registry[t] = adapter def when(self, *types): def decorator(adapter): self.register(adapter, *types) return adapter return decorator From barry at python.org Tue Apr 4 21:03:13 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 04 Apr 2006 15:03:13 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4431845B.2030601@colorstudy.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> <443165A4.6070807@colorstudy.com> <1144091223.29355.80.camel@resist.wooz.org> <4431845B.2030601@colorstudy.com> Message-ID: <1144177393.17927.31.camel@resist.wooz.org> On Mon, 2006-04-03 at 15:23 -0500, Ian Bicking wrote: > This is all wandering off-topic, except that all these cases make me > think that different kinds of wrapping are very useful. For instance, > if you want to make sure everything is quoted before being inserted: > > class EscapingWrapper: > def __init__(self, d): > self.d = d > def __getitem__(self, item): > return cgi.escape(str(self.d[item]), 1) > > Or if you want expressions: > > class EvalingWrapper: > def __init__(self, d): > self.d = d > def __getitem__(self, item): > return eval(item, d) > > Then you do: > > string.Template(pattern).substitute(EscapingWrapper(EvalingWrapper(locals())) I like this. I'd probably not utilize EvalingWrapper, just because I'd really want to keep translatable strings really really simple. I think most translators can grok simple $-substitutions because they've seen those in many other languages. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060404/3cbbad15/attachment.pgp From barry at python.org Tue Apr 4 21:05:00 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 04 Apr 2006 15:05:00 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4431CB8D.9040004@canterbury.ac.nz> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> Message-ID: <1144177500.17925.34.camel@resist.wooz.org> On Tue, 2006-04-04 at 13:27 +1200, Greg Ewing wrote: > If I were designing a formatting system from scratch, > I think I'd separate the issue of formatting numbers into > strings from the issue of inserting strings into other > strings. This really is a key observation. PEP 292 tried to make inserting strings into other strings really simple to grok, and I think it does that, while still providing a useful level of flexibility for advanced users. I really would hate to mingle that simple functionality with all the complexity of number formatting and the like. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060404/cae7af2d/attachment.pgp From barry at python.org Tue Apr 4 21:10:24 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 04 Apr 2006 15:10:24 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443293B3.1020706@colorstudy.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> <443293B3.1020706@colorstudy.com> Message-ID: <1144177824.17926.36.camel@resist.wooz.org> On Tue, 2006-04-04 at 10:41 -0500, Ian Bicking wrote: > A $ prefix would act at parse time, i.e., $"answer: $answer" would be > equivalent to ("answer: " + str(answer)). Wouldn't it be equivalent to string.Template("answer: $answer") ? In any event... > Though I also agree that $"answer: $answer" introduces a confusing > number of $'s. ...I agree too! :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060404/0a4c8caf/attachment.pgp From barry at python.org Tue Apr 4 21:11:50 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 04 Apr 2006 15:11:50 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144117100.14473.76.camel@localhost.localdomain> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> <1144087267.29376.57.camel@resist.wooz.org> <1144117100.14473.76.camel@localhost.localdomain> Message-ID: <1144177910.17930.39.camel@resist.wooz.org> On Mon, 2006-04-03 at 22:18 -0400, Adam DePrince wrote: > Worse are the maintance problems when you have to change your > expression ... inside every translation of your string. You're right, that's far far worse. One thing about i18n is that it introduces huge inertia against changing any human readable text. When I retire , that's a project I would like to tackle (and have some ideas along those lines). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060404/ea8319a4/attachment.pgp From crutcher at gmail.com Tue Apr 4 21:12:48 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Tue, 4 Apr 2006 12:12:48 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4431CB8D.9040004@canterbury.ac.nz> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> Message-ID: On 4/3/06, Greg Ewing wrote: > Barry Warsaw wrote: > > > I don't much like the $"" prefix > > This was discussed during the last round of formatting > wars, and the conclusion was that having $ both > inside and outside the string would be too visually > confusing. > > > I don't see a good > > way to marry the rich coercion of %-substitution with the simplicity of > > $-substition. > > If I were designing a formatting system from scratch, > I think I'd separate the issue of formatting numbers into > strings from the issue of inserting strings into other > strings. So instead of > > "Answer no. %5d is %8.3f" % (n, x) > > you would say something like > > subst("Answer no. {1} is {2}", format(i, 'd', 5), format(x, 'f', 8, 3)) There are an awful lot of ways to say this, such as: "Answer no. {1} is {2}".subst(int(i).format(5), float(x).format(8, 3)) Or: "Answer no. {1} is {2}".subst(format(i, '5d'), format(x, '8.3f')) Or: "Answer no. {1} is {2}".subst(format('5d', i), format('8.3f', x)) Or: "Answer no. {1} is {2}".subst(format('5d')(i), format('8.3f')(x)) > -- > Greg Ewing, Computer Science Dept, +--------------------------------------+ > University of Canterbury, | Carpe post meridiam! | > Christchurch, New Zealand | (I'm not a morning person.) | > greg.ewing at canterbury.ac.nz +--------------------------------------+ > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/crutcher%40gmail.com > -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From barry at python.org Tue Apr 4 21:14:34 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 04 Apr 2006 15:14:34 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4431D9D7.2050609@canterbury.ac.nz> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> Message-ID: <1144178074.17927.43.camel@resist.wooz.org> On Tue, 2006-04-04 at 14:28 +1200, Greg Ewing wrote: > If we're going to encourage use of named arguments > (which I think we should) I think we also need to > get rid of the need for %(foo)s constructs, which > are extremely error-prone and hard to read. In general, I agree. > I'm -0.7 on having two different formatting > styles (one using % and the other using $) with > partially-overlapping functionality. In Py3k > there should be OOWTDI. Except that there really are two things going on here. One is formatting and the other is insertion. I agree it's awkward to have two styles, but I would really hate to complicate the latter to support the former. Maybe those who are advocating moving away from in-place substitution as the mechanism for formatting have the right idea. > I'm also not all that keen on $, either inside > or outside the string. It seems to me that > something like > > "User {user} has printed {n} pages" > > sets off the parameters from the rest of the > string more readably than > > "User $user has printed $n pages" Here i disagree. $-strings are so familiar to people coming from other languages that they instantly "get it". -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060404/2cbf36c4/attachment.pgp From barry at python.org Tue Apr 4 21:39:53 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 04 Apr 2006 15:39:53 -0400 Subject: [Python-3000] hash as attribute/property In-Reply-To: References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> Message-ID: <1144179593.17928.52.camel@resist.wooz.org> On Tue, 2006-04-04 at 09:57 -0700, Guido van Rossum wrote: > On 4/3/06, Delaney, Timothy (Tim) wrote: > > I've been thinking that `hash` could be an attribute (or property if it > > needs to be calculated on-the-fly) rather than the current method call. > > > > Or it could be an easy thing to add to the "won't change" PEP ... > > The latter, please. > > We apparently need more guidance for when it's appropriate to turn > something into an attribute. I don't have time to write that up right > now; if someone else can give it a shot, I'd appreciate it; or I'll > come back to this topic later. Just a data point. In our API, we've been converting all of our no-arg methods to attributes/properties, each of which may or may not be writable of course. It's simplified our Python code enormously, and made it much more readable. Certainly "foo.get_something()" and "foo.set_something(thing)" are much better spelled "foo.something" and "foo.something = thing". As a general principle, we've been making this conversion for any getters and setters that take no arguments, both for pure-Python types and C-implemented types. While we were at it, we changed everything that returns a newly formed concrete sequence to returning an iterator (which broke code that was doing random access into the sequence, but that's another story). I don't know whether Python 3000 wants to encourage that style, but it works well for us. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060404/d3c29ed8/attachment.pgp From guido at python.org Tue Apr 4 21:44:32 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 12:44:32 -0700 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 In-Reply-To: References: Message-ID: On 4/4/06, Georg Brandl wrote: > Guido van Rossum wrote: > > I was going to comment about this one "check it in, we'll add to it > > later", but then I realized it's not 100% clear whether this is a > > feature PEP or a meta-PEP? It focuses on features so by that yardstick > > it's a feature PEP. But in its list-of-miscellany nature it > > approximates a meta-PEP. Hmm, perhaps it ought to be PEP 3999 (which > > can be seen as minus one in the numbering scheme for Python-3000 PEPs > > :-)? > > > > In any case let's not let it longer for long. Realistically, I think > > it can be a meta-PEP. > > Since it's a meta-PEP, but closely related to features, I've checked it > in with number 3099, that is, "first feature-PEP minus one". Excellent! (Thanks to Terry Reedy for the idea.) > Related question: as the current PEP 3000 contains mainly feature proposals, > shouldn't it be renamed to 3100? PEP 3000 could then be a quasi-summary of > 3xxx PEPs and random important facts that don't fit elsewhere. Good idea. Maybe 3000 should then be the meta-meta PEP with the Process for deciding Python 3000 Processes. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From brett at python.org Tue Apr 4 21:48:43 2006 From: brett at python.org (Brett Cannon) Date: Tue, 4 Apr 2006 12:48:43 -0700 Subject: [Python-3000] hash as attribute/property In-Reply-To: <1144179593.17928.52.camel@resist.wooz.org> References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> <1144179593.17928.52.camel@resist.wooz.org> Message-ID: On 4/4/06, Barry Warsaw wrote: > On Tue, 2006-04-04 at 09:57 -0700, Guido van Rossum wrote: > > On 4/3/06, Delaney, Timothy (Tim) wrote: > > > I've been thinking that `hash` could be an attribute (or property if it > > > needs to be calculated on-the-fly) rather than the current method call. > > > > > > Or it could be an easy thing to add to the "won't change" PEP ... > > > > The latter, please. > > > > We apparently need more guidance for when it's appropriate to turn > > something into an attribute. I don't have time to write that up right > > now; if someone else can give it a shot, I'd appreciate it; or I'll > > come back to this topic later. > > Just a data point. In our API, we've been converting all of our no-arg > methods to attributes/properties, each of which may or may not be > writable of course. It's simplified our Python code enormously, and > made it much more readable. Certainly "foo.get_something()" and > "foo.set_something(thing)" are much better spelled "foo.something" and > "foo.something = thing". > I would amend that to anything that does not cause mutation when calling (this does not include creating new objects, though) or mutate other things when assigning (thanks to properties that is possible). Basically that is my personal rule of thumb for when I code. -Brett > As a general principle, we've been making this conversion for any > getters and setters that take no arguments, both for pure-Python types > and C-implemented types. While we were at it, we changed everything > that returns a newly formed concrete sequence to returning an iterator > (which broke code that was doing random access into the sequence, but > that's another story). > > I don't know whether Python 3000 wants to encourage that style, but it > works well for us. > > -Barry > > > > -----BEGIN PGP SIGNATURE----- > Version: GnuPG v1.4.2.2 (GNU/Linux) > > iQCVAwUARDLLiXEjvBPtnXfVAQJwnwP/fl3UBFDXUMS+DeNMLbQYJwmeJG09v8oJ > ER1yq35Zqs2R3YN9y1ufYiMSXqy/F5szlDs4VnjpQL5t1B7DYmAj+QqW0y6i2CZd > Yd6vzehlEWdMqHc2cmgrXlmPr/AQWodfghrbAemu+kRTjQ4nkrv9jc1Zvfw3YxRg > v2gXHyn66aU= > =MmiI > -----END PGP SIGNATURE----- > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/brett%40python.org > > > From guido at python.org Tue Apr 4 22:04:30 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 13:04:30 -0700 Subject: [Python-3000] hash as attribute/property In-Reply-To: <1144179593.17928.52.camel@resist.wooz.org> References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> <1144179593.17928.52.camel@resist.wooz.org> Message-ID: On 4/4/06, Barry Warsaw wrote: > Just a data point. In our API, we've been converting all of our no-arg > methods to attributes/properties, each of which may or may not be > writable of course. It's simplified our Python code enormously, and > made it much more readable. Certainly "foo.get_something()" and > "foo.set_something(thing)" are much better spelled "foo.something" and > "foo.something = thing". Right. It sounds like your original API was using Java-style getters and setters too much, and then converting to Python-style attributes and properties is an obvious improvement. > As a general principle, we've been making this conversion for any > getters and setters that take no arguments, both for pure-Python types > and C-implemented types. While we were at it, we changed everything > that returns a newly formed concrete sequence to returning an iterator > (which broke code that was doing random access into the sequence, but > that's another story). Let me ask a clarification here. Do you now have properties that return new iterators? That seems a bad idea, for the following reason: I'd expect these two code sequences to have the same effect: a = obj.foo b = obj.foo and a = b = obj.foo If obj.foo were to be a property returning a new iterator on each access, the first example would end up with a and b separate (independent) iterators, while the second example would make a and b aliases for the same iterator. I hope I misunderstood you; something that returns an iterator should definitely not be a property but a real method. > I don't know whether Python 3000 wants to encourage that style, but it > works well for us. I certainly want to continue discouraging setter and getter methods where plain attributes would do. The question is where to draw the line. I don't believe you really meant "all no-args methods". Perhaps you meant "all side-effect free argument-less methods that return the same value each time (or at least until some explicit change to the object is made) and that can be computed reasonably cheap"? The __hash__() method qualifies, which is why it's a tough call (I'll come back on that; I want to come up with a reason why __hash__() should remain a method). The iterator-getting example above doesn't qualify. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Wed Apr 5 00:24:05 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Wed, 5 Apr 2006 08:24:05 +1000 Subject: [Python-3000] hash as attribute/property Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1E647@au3010avexu1.global.avaya.com> Guido van Rossum wrote: > The __hash__() method qualifies, which is why it's a tough call (I'll > come back on that; I want to come up with a reason why __hash__() > should remain a method). The iterator-getting example above doesn't > qualify. Yeah - I felt it was a borderline case, so thought it was a good case for setting a precedent. Pros for making it an attribute/property: 1. It should return the same value each time. 2. Faster access for many types. Cons: 1. Gratuitous breakage with 2.x. 2. If it's a property, subclasses need to override the getter and recreate the property if they want to change the hash of the object (unless the base class uses a lazy property). Tim Delaney From greg.ewing at canterbury.ac.nz Wed Apr 5 01:00:47 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 11:00:47 +1200 Subject: [Python-3000] A few small py3k wishes In-Reply-To: <20060404125247.GC12075@tigger.digitaltorque.ca> References: <20060402222130.GA12075@tigger.digitaltorque.ca> <200604041322.02900.gmccaughan@synaptics-uk.com> <20060404125247.GC12075@tigger.digitaltorque.ca> Message-ID: <4432FA9F.50908@canterbury.ac.nz> Michael P. Soulier wrote: > If > one is going to use a method obviously taken from the stack ADT, having a > method normally called push() would be less confusing. I think the confusion here really starts with using the name pop(), since it's not clear which end of the list it pops from. It might have been better to call it unappend() or something. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 5 01:10:30 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 11:10:30 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443293B3.1020706@colorstudy.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> <443293B3.1020706@colorstudy.com> Message-ID: <4432FCE6.7010002@canterbury.ac.nz> Ian Bicking wrote: > Though I also agree that $"answer: $answer" introduces a confusing > number of $'s. If an interpolation feature like this were considered desirable, maybe the backquote syntax could be repurposed for it in Py3k? -- Greg From ianb at colorstudy.com Wed Apr 5 01:17:39 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 18:17:39 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144177393.17927.31.camel@resist.wooz.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> <443165A4.6070807@colorstudy.com> <1144091223.29355.80.camel@resist.wooz.org> <4431845B.2030601@colorstudy.com> <1144177393.17927.31.camel@resist.wooz.org> Message-ID: <4432FE93.6090405@colorstudy.com> Barry Warsaw wrote: > On Mon, 2006-04-03 at 15:23 -0500, Ian Bicking wrote: > > >>This is all wandering off-topic, except that all these cases make me >>think that different kinds of wrapping are very useful. For instance, >>if you want to make sure everything is quoted before being inserted: >> >>class EscapingWrapper: >> def __init__(self, d): >> self.d = d >> def __getitem__(self, item): >> return cgi.escape(str(self.d[item]), 1) >> >>Or if you want expressions: >> >>class EvalingWrapper: >> def __init__(self, d): >> self.d = d >> def __getitem__(self, item): >> return eval(item, d) >> >>Then you do: >> >>string.Template(pattern).substitute(EscapingWrapper(EvalingWrapper(locals())) > > > I like this. I'd probably not utilize EvalingWrapper, just because I'd > really want to keep translatable strings really really simple. I think > most translators can grok simple $-substitutions because they've seen > those in many other languages. The useful of a wrapper pattern is if some people would use a wrapper, and some would not. One can imagine a formatting wrapper too: class Formatter: def __init__(self, d): self.d = d def __getitem__(self, item): if ':' in item: format, key = item.split(':', 1) else: format, key = '', item value = self.d[item] if format: value = ('%'+format) % value return value Then "${0.2f:price}" would work. Or you could do: class Piper: def __init__(self, d): self.d = d def __getitem__(self, item): parts = item.split('|') value = self.d[parts[0]] for filter in parts[1:] value = self.d[filter](value) return value Then "${price|html}" works (assuming you've added an html() function to your upstream dictionary). And if you are using an evaluating upstream, you got Django-style filters right there. These don't quite work with string.Template, because ${...} has the same content constraints that $... has, so you can't easily put extended expressions in there. I could have sworn I opened an SF bug on this, but it appears not. But that's an aside. Anyway, none of this is very useful if it requires long-winded invocations and imports, which remains a problem. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From greg.ewing at canterbury.ac.nz Wed Apr 5 01:24:13 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 11:24:13 +1200 Subject: [Python-3000] hash as attribute/property In-Reply-To: References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> Message-ID: <4433001D.5040509@canterbury.ac.nz> Guido van Rossum wrote: > We apparently need more guidance for when it's appropriate to turn > something into an attribute. I'd suggest that things which could require substantial computation to obtain should remain methods; hash() seems to fall into that category. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 5 01:44:23 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 11:44:23 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1144178074.17927.43.camel@resist.wooz.org> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> Message-ID: <443304D7.7020307@canterbury.ac.nz> Barry Warsaw wrote: > On Tue, 2006-04-04 at 14:28 +1200, Greg Ewing wrote: > >>I'm also not all that keen on $, either inside >>or outside the string. It seems to me that >>something like >> >> "User {user} has printed {n} pages" >> >>sets off the parameters from the rest of the >>string more readably than >> >> "User $user has printed $n pages" > > > Here i disagree. $-strings are so familiar to people coming from other > languages that they instantly "get it". I'm more interested in long-term readability and maintainability than in pandering to one-off impressions when coming from another language. I don't believe they would have any difficulty "getting" this, anyway. Can you honestly say that if you saw something like subst("User {user} has printed {n} pages", user = user_code, n = page_count) you wouldn't instantly figure out what was going on? There are other problems with the $ syntax too. The lack of an explicit closing marker causes problems when the parameter is immediately followed in the string by an alphanumeric character. You end up having to allow some kind of bracketing as an optional extra anyway, such as subst("I'm a ${what}spotter", what = "train") Seems to me it's simpler as well as clearer to just make the syntax fully bracketed in the first place. -- Greg From ianb at colorstudy.com Wed Apr 5 01:51:48 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 18:51:48 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443304D7.7020307@canterbury.ac.nz> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> Message-ID: <44330694.3060900@colorstudy.com> Greg Ewing wrote: > Barry Warsaw wrote: >> Here i disagree. $-strings are so familiar to people coming from other >> languages that they instantly "get it". > > > I'm more interested in long-term readability > and maintainability than in pandering to one-off > impressions when coming from another language. $-based substitution is very common. Like most languages, Python uses () for calling functions, it uses keywords like "while" and "if", etc. It's okay to use the normal form; tweaking some detail off the norm because it seems slightly more compact or concise seems questionable to me. Using {} instead of $/${} doesn't seem like a big win. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From greg.ewing at canterbury.ac.nz Wed Apr 5 01:59:36 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 11:59:36 +1200 Subject: [Python-3000] hash as attribute/property In-Reply-To: <1144179593.17928.52.camel@resist.wooz.org> References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> <1144179593.17928.52.camel@resist.wooz.org> Message-ID: <44330868.9060409@canterbury.ac.nz> Barry Warsaw wrote: > Certainly "foo.get_something()" and > "foo.set_something(thing)" are much better spelled "foo.something" and > "foo.something = thing". If you already have a get/set pair, that definitely makes sense. If hash() had been spelled get_hash() it might have made sense there too. But it wasn't, and for a reason, I think -- it implies performing a substantial computation rather than just retrieving a pre-existing value from somewhere. Even if it's cached, that's still true, since the computation needs to be done the first time it's called. Another consideration is that if something is a method, you can get a bound method from it. With a property that's much harder, and may be impossible depending on how the property is implemented. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 5 02:04:20 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 12:04:20 +1200 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 In-Reply-To: References: Message-ID: <44330984.7090203@canterbury.ac.nz> Guido van Rossum wrote: > Good idea. Maybe 3000 should then be the meta-meta PEP with the > Process for deciding Python 3000 Processes. And if we get too many more meta-meta-PEPs we'll need a meta-meta-meta-PEP on Procedure for Numbering Meta-Meta-PEPs. Not sure what number it should have, though... -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 5 02:07:59 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 12:07:59 +1200 Subject: [Python-3000] hash as attribute/property In-Reply-To: <1144179593.17928.52.camel@resist.wooz.org> References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> <1144179593.17928.52.camel@resist.wooz.org> Message-ID: <44330A5F.4030501@canterbury.ac.nz> Barry Warsaw wrote: > While we were at it, we changed everything > that returns a newly formed concrete sequence to returning an iterator > (which broke code that was doing random access into the sequence, That sounds like a good use for Py3k-style views! -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 5 02:14:55 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 12:14:55 +1200 Subject: [Python-3000] hash as attribute/property In-Reply-To: References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> <1144179593.17928.52.camel@resist.wooz.org> Message-ID: <44330BFF.8000408@canterbury.ac.nz> Guido van Rossum wrote: > The __hash__() method qualifies, which is why it's a tough call (I'll > come back on that; I want to come up with a reason why __hash__() > should remain a method). Another thought is how frequently the method/attribute is going to be used. I can't even remember when was the last time I had to explicitly hash anything, so the convenience of being able to call it without () seems insignificant. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 5 02:22:30 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 12:22:30 +1200 Subject: [Python-3000] hash as attribute/property In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5FF1E647@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5FF1E647@au3010avexu1.global.avaya.com> Message-ID: <44330DC6.6020007@canterbury.ac.nz> Delaney, Timothy (Tim) wrote: > 2. If it's a property, subclasses need to override the getter and > recreate the property if they want to change the hash of the object > (unless the base class uses a lazy property). In which case the performance advantage becomes negative (instead of just a method call, you have a property lookup plus a method call). -- Greg From guido at python.org Wed Apr 5 02:29:47 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 17:29:47 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions Message-ID: #!/usr/bin/python2.4 """An example of generic functions vs. adaptation. After some face-to-face discussions with Alex I'm presenting here two extremely simplified implementations of second-generation adaptation and generic functions, in order to contrast and compare. As the running example, we use the iterator protocol. Pretend that the iter() built-in function doesn't exist; how would we implement iteration using adaptation or generic functions? """ __metaclass__ = type # Use new-style classes everywhere # 0. The SequenceIter class is shared by all versions. class SequenceIter: def __init__(self, obj): self.obj = obj self.index = 0 def next(self): i = self.index self.index += 1 if i < len(self.obj): return self.obj[i] raise StopIteration def __iter__(self): # This exists so we can use this in a for loop return self # 1. Do it manually. This is ugly but effective... until you need to # iterate over a third party object type whose class you can't modify. def ManualIter(obj): if isinstance(obj, (list, str, unicode)): return SequenceIter(obj) if isinstance(obj, dict): # We can't really do a better job without exposing PyDict_Next() return SequenceIter(obj.keys()) if hasattr(obj, "__iter__"): return obj.__iter__() raise TypeError("Can't iterate over a %s object" % obj.__class__.__name__) # 2. Using adaptation. First I show a simple implementation of # adaptation. In a more realistic situation this would of course be # imported. I call this "second generation adaptation" because the # adaptation from protocol P to type T is invoked as P.adapt(T), and # the registration of an adapter function A for type T is invoked as # P.register(T, A). The only "smart" feature of this implementation # is its support for inheritance (in T, not in P): if T has registered # an adapter A for P, then A is also the default adapter for any # subclass S of T, unless a more specific adapter is registered for S # (or for some base of S that comes before T in S's MRO). class Protocol: def __init__(self, name): self.registry = {} self.name = name def register(self, T, A): self.registry[T] = A def adapt(self, obj): for T in obj.__class__.__mro__: if T in self.registry: return self.registry[T](obj) raise TypeError("Can't adapt %s to %s" % (obj.__class__.__name__, self.name)) def __call__(self, T): # This is invoked when a Protocol instance is used as a decorator. def helper(A): self.register(T, A) return A return helper # Now I show how to define and register the various adapters. In a # more realistic situation these don't all have to be in the same file # of course. AdaptingIterProtocol = Protocol("AdaptingIterProtocol") @AdaptingIterProtocol(list) def _AdaptingSequenceIter(obj): return SequenceIter(obj) AdaptingIterProtocol.register(str, _AdaptingSequenceIter) AdaptingIterProtocol.register(unicode, _AdaptingSequenceIter) @AdaptingIterProtocol(dict) def _AdaptingDictIter(obj): return SequenceIter(obj.keys()) @AdaptingIterProtocol(object) def _AdaptingObjectIter(obj): if hasattr(obj, "__iter__"): return obj.__iter__() raise TypeError("Can't iterate over a %s object" % obj.__class__.__name__) def AdaptingIter(obj): return AdaptingIterProtocol.adapt(obj) # 3. Using generic functions. First I show a simple implementation of # generic functions. In a more realistic situation this would of # course be imported. class GenericFunction: def __init__(self, default_function): self.default_function = default_function self.registry = {} def register(self, *args): def helper(F): self.registry[args] = F return F return helper def __call__(self, *args): types = tuple([obj.__class__ for obj in args]) function = self.registry.get(types, self.default_function) return function(*args) # Now I show how to define a generic function and how to register the # various type-specific implementations. In a more realistic # situation these don't all have to be in the same file of course. @GenericFunction def GenericIter(obj): """This is the docstring for the generic function.""" # The body is the default implementation if hasattr(obj, "__iter__"): return obj.__iter__() raise TypeError("Can't iterate over %s object" % obj.__class__.__name__) @GenericIter.register(list) def _GenericSequenceIter(obj): return SequenceIter(obj) GenericIter.register(str)(_GenericSequenceIter) GenericIter.register(unicode)(_GenericSequenceIter) @GenericIter.register(dict) def _GenericDictIter(obj): return SequenceIter(obj.keys()) # 4. Show that all of these work equivalently. def main(): examples = [ [1, 2, 3, 4, 5], "abcde", u"ABCDE", {"x": 1, "y": 2, "z": 3}, (6, 7, 8, 9, 10), # Not registered, but has __iter__ method 42, # Not registered and has no __iter__ method ] functions = [ManualIter, AdaptingIter, GenericIter] for function in functions: print print "***", function, "***" for example in examples: print ":::", repr(example), ":::" try: iterator = function(example) except Exception, err: print "!!! %s: %s !!!" % (err.__class__.__name__, err) else: for value in function(example): print repr(value), print if __name__ == "__main__": main() # -- # --Guido van Rossum (home page: http://www.python.org/~guido/) From rasky at develer.com Wed Apr 5 02:29:57 2006 From: rasky at develer.com (Giovanni Bajo) Date: Wed, 5 Apr 2006 02:29:57 +0200 Subject: [Python-3000] hash as attribute/property References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com><1144179593.17928.52.camel@resist.wooz.org> Message-ID: <056201c65848$114ee2f0$db4e2a97@bagio> Guido van Rossum wrote: > The question is where to draw the line. I don't believe you really > meant "all no-args methods". Perhaps you meant "all side-effect free > argument-less methods that return the same value each time (or at > least until some explicit change to the object is made) and that can > be computed reasonably cheap"? +1. And "reasonably cheap" also means O(1) (aka "no for loops"), in my own rule of thumb. The worst thing of VB is when you read a value from a property and don't realize how much it's going on behind the scenes. A function call is a function call is a function call. Someone smart said "one thing is to read code which calls functions that you don't know about, another thing is to read code where you do not even know where the function calls are". This is very true, and should be kept in mind when designing code with properties. > The __hash__() method qualifies, which is why it's a tough call (I'll > come back on that; I want to come up with a reason why __hash__() > should remain a method). The iterator-getting example above doesn't > qualify. __hash__() isn't always O(1), so in my world it wouldn't be a property. Giovanni Bajo From guido at python.org Wed Apr 5 02:32:37 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 17:32:37 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: Message-ID: One followup. Alex mentioned that PyPy used to have its own implementation of multiple dispatch (which is really the same as generic functions) but ditched over a year ago. Would Armin or Samuele care to explain why? Do they no longer like multiple dispatch or was the reason for the extraction specific to PyPy? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Wed Apr 5 02:38:24 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Apr 2006 12:38:24 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44330694.3060900@colorstudy.com> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> <44330694.3060900@colorstudy.com> Message-ID: <44331180.9000203@canterbury.ac.nz> Ian Bicking wrote: > Using {} instead of $/${} doesn't seem like a big win. Perhaps this is where we differ. To me it *does* make quite a substantial difference to readability -- easily enough to override whatever small advantage there might be in following what everyone else does. -- Greg From guido at python.org Wed Apr 5 02:40:58 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 17:40:58 -0700 Subject: [Python-3000] hash as attribute/property In-Reply-To: <056201c65848$114ee2f0$db4e2a97@bagio> References: <2773CAC687FD5F4689F526998C7E4E5FF1E63B@au3010avexu1.global.avaya.com> <1144179593.17928.52.camel@resist.wooz.org> <056201c65848$114ee2f0$db4e2a97@bagio> Message-ID: On 4/4/06, Giovanni Bajo wrote: > Guido van Rossum wrote: > > > The question is where to draw the line. I don't believe you really > > meant "all no-args methods". Perhaps you meant "all side-effect free > > argument-less methods that return the same value each time (or at > > least until some explicit change to the object is made) and that can > > be computed reasonably cheap"? > > +1. And "reasonably cheap" also means O(1) (aka "no for loops"), in my own rule > of thumb. The worst thing of VB is when you read a value from a property and > don't realize how much it's going on behind the scenes. > > A function call is a function call is a function call. Someone smart said "one > thing is to read code which calls functions that you don't know about, another > thing is to read code where you do not even know where the function calls are". > This is very true, and should be kept in mind when designing code with > properties. > > > The __hash__() method qualifies, which is why it's a tough call (I'll > > come back on that; I want to come up with a reason why __hash__() > > should remain a method). The iterator-getting example above doesn't > > qualify. > > __hash__() isn't always O(1), so in my world it wouldn't be a property. I like this best of all the arguments presented so far, and consider the case closed. (The weakness of the argument "but hash() can take a long time to compute" is that a good hash function is supposed to be fast. But this doesn't mean it can't have an O(N) component somewhere, e.g. for tuples.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz at pythoncraft.com Wed Apr 5 02:42:29 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 4 Apr 2006 17:42:29 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4432FCE6.7010002@canterbury.ac.nz> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <4431CB8D.9040004@canterbury.ac.nz> <443293B3.1020706@colorstudy.com> <4432FCE6.7010002@canterbury.ac.nz> Message-ID: <20060405004229.GA270@panix.com> On Wed, Apr 05, 2006, Greg Ewing wrote: > Ian Bicking wrote: >> >> Though I also agree that $"answer: $answer" introduces a confusing >> number of $'s. > > If an interpolation feature like this were considered desirable, maybe > the backquote syntax could be repurposed for it in Py3k? Aaagh! ;-) -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet From tdelaney at avaya.com Wed Apr 5 02:47:36 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Wed, 5 Apr 2006 10:47:36 +1000 Subject: [Python-3000] hash as attribute/property Message-ID: <2773CAC687FD5F4689F526998C7E4E5F074368@au3010avexu1.global.avaya.com> Guido van Rossum wrote: >> __hash__() isn't always O(1), so in my world it wouldn't be a >> property. > > I like this best of all the arguments presented so far, and consider > the case closed. > > (The weakness of the argument "but hash() can take a long time to > compute" is that a good hash function is supposed to be fast. But this > doesn't mean it can't have an O(N) component somewhere, e.g. for > tuples.) That seems like a good determinant to me. Perhaps it should be added to PEP err ... 3000? Doesn't quite seem to fit - maybe needs a new section. Tim Delaney From nas at arctrix.com Wed Apr 5 02:59:55 2006 From: nas at arctrix.com (Neil Schemenauer) Date: Tue, 4 Apr 2006 18:59:55 -0600 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44330694.3060900@colorstudy.com> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> <44330694.3060900@colorstudy.com> Message-ID: <20060405005955.GB23489@mems-exchange.org> On Tue, Apr 04, 2006 at 06:51:48PM -0500, Ian Bicking wrote: > Using {} instead of $/${} doesn't seem like a big win. Big enough to be worth considering, IMO. Neil From edcjones at comcast.net Wed Apr 5 03:26:28 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Tue, 04 Apr 2006 21:26:28 -0400 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 In-Reply-To: References: Message-ID: <44331CC4.7000902@comcast.net> Greg Ewing wrote > Subject: Re: [Python-3000] pre-PEP: Things that Will Not Change in > Python 3.0 > To: python-3000 at python.org > Message-ID: <44330984.7090203 at canterbury.ac.nz> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > Guido van Rossum wrote: > >> Good idea. Maybe 3000 should then be the meta-meta PEP with the >> Process for deciding Python 3000 Processes. > > And if we get too many more meta-meta-PEPs we'll need > a meta-meta-meta-PEP on Procedure for Numbering > Meta-Meta-PEPs. Not sure what number it should have, > though... See Kurt Godel, Uber formal unentscheidbare Satze der Principia Mathematica und verwanter Syteme I. From barry at python.org Wed Apr 5 04:13:26 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 4 Apr 2006 22:13:26 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <4432FE93.6090405@colorstudy.com> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> <443165A4.6070807@colorstudy.com> <1144091223.29355.80.camel@resist.wooz.org> <4431845B.2030601@colorstudy.com> <1144177393.17927.31.camel@resist.wooz.org> <4432FE93.6090405@colorstudy.com> Message-ID: <1B2AE36F-4700-4BFF-9360-384CF688BF74@python.org> On Apr 4, 2006, at 7:17 PM, Ian Bicking wrote: > > These don't quite work with string.Template, because ${...} has the > same content constraints that $... has, so you can't easily put > extended expressions in there. I could have sworn I opened an SF > bug on this, but it appears not. But that's an aside. > I may be misunderstanding, but the contents of ${...} is pretty flexible because you control the regular expression that matches the substitution. Heck, you can even use something other than $ delimiters if you want. -Barry From barry at python.org Wed Apr 5 04:16:18 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 4 Apr 2006 22:16:18 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443304D7.7020307@canterbury.ac.nz> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> Message-ID: On Apr 4, 2006, at 7:44 PM, Greg Ewing wrote: > I'm more interested in long-term readability > and maintainability than in pandering to one-off > impressions when coming from another language. > > I don't believe they would have any difficulty > "getting" this, anyway. Can you honestly say > that if you saw something like > > subst("User {user} has printed {n} pages", > user = user_code, n = page_count) > > you wouldn't instantly figure out what was > going on? > > There are other problems with the $ syntax too. > The lack of an explicit closing marker causes > problems when the parameter is immediately > followed in the string by an alphanumeric > character. You end up having to allow some > kind of bracketing as an optional extra > anyway, such as > > subst("I'm a ${what}spotter", what = "train") > > Seems to me it's simpler as well as clearer > to just make the syntax fully bracketed in the > first place. > BTW, with string.Template's defaults, $what and ${what} are exactly equivalent. IOW, totally optional so if your substitution variable is followed by "non-identifier" characters, just use $what, otherwise use ${what}. I haven't tried, but string.Template might even support dropping the delimiter altogether, so that you could use just {what}. Give it a shot and let us know! :) -Barry From barry at python.org Wed Apr 5 04:19:34 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 4 Apr 2006 22:19:34 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44331180.9000203@canterbury.ac.nz> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> <44330694.3060900@colorstudy.com> <44331180.9000203@canterbury.ac.nz> Message-ID: <5F505240-06DC-4AC2-9330-6AC85C9931D3@python.org> On Apr 4, 2006, at 8:38 PM, Greg Ewing wrote: > Ian Bicking wrote: > >> Using {} instead of $/${} doesn't seem like a big win. > > Perhaps this is where we differ. To me it > *does* make quite a substantial difference > to readability -- easily enough to override > whatever small advantage there might be in > following what everyone else does. Yes, I think this is where we differ. $thing in it's most common form is more readable to me than {thing}, and while the former is equivalent to ${thing} with string.Template's defaults, I find it's very rarely used. So I wouldn't be in favor of changing it. Of course, beauty is in the eye of the beholder and all that... -Barry From aleaxit at gmail.com Wed Apr 5 04:59:20 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Tue, 4 Apr 2006 19:59:20 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44331180.9000203@canterbury.ac.nz> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> <44330694.3060900@colorstudy.com> <44331180.9000203@canterbury.ac.nz> Message-ID: On Apr 4, 2006, at 5:38 PM, Greg Ewing wrote: > Ian Bicking wrote: > >> Using {} instead of $/${} doesn't seem like a big win. > > Perhaps this is where we differ. To me it > *does* make quite a substantial difference > to readability -- easily enough to override > whatever small advantage there might be in > following what everyone else does. +1 for {} over the $/${} dichotomy -- it's easier to explain, as well as more readable. Alex From janssen at parc.com Wed Apr 5 05:12:57 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 4 Apr 2006 20:12:57 PDT Subject: [Python-3000] String formating operations in python 3k In-Reply-To: Your message of "Tue, 04 Apr 2006 19:59:20 PDT." Message-ID: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> > +1 for {} over the $/${} dichotomy -- it's easier to explain, as well > as more readable. Perhaps I'm just allergic to bash/perl/java/ruby syntax, but the more I follow this discussion, the more I like the current string formatting system :-). It seems to me that the core of the issue is the current insistence, with %(NAME)FORMAT, on a trailing format type which is frequently forgotten. Why not just remove that insistence, and use %(foo) in py3k? Code will break, sure, but we're going to expect that. If you really need to control the presentation, put that information inside the parens, as in %(foo:.3f). But let it default to "s". There are other issues, such as the need to construct a dictionary, for which reasonable approaches have been offered over the last week. Bill From guido at python.org Wed Apr 5 05:16:55 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 20:16:55 -0700 Subject: [Python-3000] StringIO vs cStringIO, pickle vs cPickle In-Reply-To: <20060404141759.GB9247@panix.com> References: <20060404141759.GB9247@panix.com> Message-ID: On 4/4/06, Aahz wrote: > On Tue, Apr 04, 2006, Just van Rossum wrote: > > Has there been any discussion yet about StringIO vs cStringIO and pickle > > vs cPickle? > > Now there is! > > > While I appreciate that there are maintained pure Python versions of > > those modules, as a user it irritates me that I have the choice. The > > argument so far for not replacing StringIO with cStringIO was that > > they're subtly incompatible (mostly StringIO's "feature" to support > > unicode, right?), so Python 3000 seems like a good time to reconsider > > that option. I don't see any reason to not always want to prefer the > > faster version. > > Suggestion: make string_io (we're renaming the module, right?) Yes, but I don't like module names with underscores. We don't have to decide now though. > and pickle > contain a UserStringIO and UserPickle class. Ah, yuck. UserThis and UserThat classes are a thing from the past, when you couldn't subclass the built-in types. I like what Raymond did with DictMixin though (which should really be called SequenceMixin) -- if you want to create your own mapping type *without* inheriting from dict, you can still get a bunch of standard implementation boilerplate via inheritance. This is what Java does with AbstractList etc. > Alternatively, they could > be moved wholesale into a python_types module, where we collectively > maintain Python versions of *all* Python types, including lists and > tuples. Sorry, another bad idea. The hope of having a comprehensive collection of *all* "standard" types is misguided. Types should go in the appropriate module given their application area. There are tons of types that don't have global names and don't need them (e.g. various iterators). > IOW, I do argue for keeping the Python code somehow, and keeping them > officially maintained. Obviously it gets a little absurd with lists and > numbers, for example, but I think that showing off good use of __new__ > would be a Good Thing. I'm in agreement with the sentiment of keeping the Python code though -- it's incredibly useful for example in Jython and IronPython. Perhaps we should switch to a naming scheme where we have a "pickle" module that you're supposed to import, which dynamically tries to import first "_pickle" and if that fails imports "_pypickle" (or some such naming scheme). If you specifically need to use one or the other you can import it explicitly (assuming it exists). This is a bit similar to how we have os.path and posixpath (although the decision regarding which one os.path refers to is different). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aleaxit at gmail.com Wed Apr 5 05:29:05 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Tue, 4 Apr 2006 20:29:05 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: Message-ID: On Apr 4, 2006, at 5:29 PM, Guido van Rossum wrote: [excellent example, mostly snipped] Just for when this will be published/blogged (as it deserves to be!), tiny nit: > if hasattr(obj, "__iter__"): > return obj.__iter__() > raise TypeError("Can't iterate over a %s object" % > obj.__class__.__name__) I think we should check/get __iter__ from obj.__class__, not from obj itself (new style classes only take special methods from the type, not from the instance). This applies to other occurrences of this pattern later, too. > def register(self, T, A): > self.registry[T] = A Why not accept more than one type (for the same adapter) at one gulp? def register(self, A, *Ts): for T in Ts: self.registry[T] = A > def __call__(self, T): > # This is invoked when a Protocol instance is used as a > decorator. > def helper(A): > self.register(T, A) > return A > return helper And similarly here, take *Ts in the __call__ and pass it on to self.register. > @AdaptingIterProtocol(list) > def _AdaptingSequenceIter(obj): > return SequenceIter(obj) > > AdaptingIterProtocol.register(str, _AdaptingSequenceIter) > AdaptingIterProtocol.register(unicode, _AdaptingSequenceIter) So this could become AdaptingIterProtocol.register(SequenceIter, list, str, unicode) ((no real need for the extra layer represented by _AdaptingSequenceIter, though you could use it, if you insisted, with all the types at one gulp. > def AdaptingIter(obj): > return AdaptingIterProtocol.adapt(obj) Similarly, in the name of cutting down useless layers, AdaptingIter = AdaptingIterProtocol.adapt seems preferable to me. There seems to be an asymmetry in the functionality of adaptation and generics as presented: adaptation as presented supports a search on the mro but for one argument only, generics support N arguments but no mro search. To enable closer comparison, we might have generics support 1 argument and mro search, too, just as to ensure the feature set is identical (they both can be extended, with different degree of difficulty no doubt, but 1 argument each, with or w/o mro search, seems to allow more direct comparison). Alex From ianb at colorstudy.com Wed Apr 5 05:31:44 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 22:31:44 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <1B2AE36F-4700-4BFF-9360-384CF688BF74@python.org> References: <4430A407.5090502@colorstudy.com> <1144065615.11451.53.camel@resist.wooz.org> <44312675.7060702@gmail.com> <1144076154.29376.8.camel@resist.wooz.org> <443149A8.4010401@colorstudy.com> <1144083596.29355.40.camel@resist.wooz.org> <443165A4.6070807@colorstudy.com> <1144091223.29355.80.camel@resist.wooz.org> <4431845B.2030601@colorstudy.com> <1144177393.17927.31.camel@resist.wooz.org> <4432FE93.6090405@colorstudy.com> <1B2AE36F-4700-4BFF-9360-384CF688BF74@python.org> Message-ID: <44333A20.6040208@colorstudy.com> Barry Warsaw wrote: > On Apr 4, 2006, at 7:17 PM, Ian Bicking wrote: >> These don't quite work with string.Template, because ${...} has the >> same content constraints that $... has, so you can't easily put >> extended expressions in there. I could have sworn I opened an SF bug >> on this, but it appears not. But that's an aside. >> > > I may be misunderstanding, but the contents of ${...} is pretty flexible > because you control the regular expression that matches the > substitution. Heck, you can even use something other than $ delimiters > if you want. The problem is that the regex for the ... in $... and ${...} is the same, so if you want to match ${foo:bar}, you also have to match $foo:bar. I'd rather leave $foo constrained to simple variable names, and require ${} for anything more complex. I got around that with LaxTemplate at the bottom of this file: http://svn.pythonpaste.org/Paste/Script/trunk/paste/script/copydir.py -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From aleaxit at gmail.com Wed Apr 5 05:33:36 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Tue, 4 Apr 2006 20:33:36 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> References: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> Message-ID: On Apr 4, 2006, at 8:12 PM, Bill Janssen wrote: >> +1 for {} over the $/${} dichotomy -- it's easier to explain, as well >> as more readable. > > Perhaps I'm just allergic to bash/perl/java/ruby syntax, but the more > I follow this discussion, the more I like the current string > formatting system :-). > > It seems to me that the core of the issue is the current insistence, > with %(NAME)FORMAT, on a trailing format type which is frequently > forgotten. Why not just remove that insistence, and use %(foo) in > py3k? Code will break, sure, but we're going to expect that. If you > really need to control the presentation, put that information inside > the parens, as in %(foo:.3f). But let it default to "s". OK on the substance, but I still prefer Greg's suggestion for simple bracketing (e.g., {}) to any kind of "stropped parentheses" (your % (...) as well as Template's ${...}). As for the % operator, I never liked it -- either a builtin function, or even better a method of string objects, is going to be much more readable (and my preference would be to have it take optional positional arguments, corresponding to {1}, {2}, etc, and optional named arguments, corresponding to {name} &c). Alex From ianb at colorstudy.com Wed Apr 5 05:35:23 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 22:35:23 -0500 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: Message-ID: <44333AFB.2060807@colorstudy.com> Alex Martelli wrote: > On Apr 4, 2006, at 5:29 PM, Guido van Rossum wrote: > [excellent example, mostly snipped] > > Just for when this will be published/blogged (as it deserves to be!), > tiny nit: > >> if hasattr(obj, "__iter__"): >> return obj.__iter__() >> raise TypeError("Can't iterate over a %s object" % >> obj.__class__.__name__) > > I think we should check/get __iter__ from obj.__class__, not from obj > itself (new style classes only take special methods from the type, > not from the instance). This applies to other occurrences of this > pattern later, too. > >> def register(self, T, A): >> self.registry[T] = A > > Why not accept more than one type (for the same adapter) at one gulp? > > def register(self, A, *Ts): > for T in Ts: > self.registry[T] = A Assuming isinstance(obj, tuple_of_types) will still work, the natural analog here would be: def register(self, T, A): if isinstance(T, tuple): for one_T in T: self.registry[one_T] = A else: self.registry[T] = A -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From ianb at colorstudy.com Wed Apr 5 05:37:57 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 22:37:57 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> Message-ID: <44333B95.4010906@colorstudy.com> Alex Martelli wrote: > As for the % operator, I never liked it -- either a builtin function, > or even better a method of string objects, is going to be much more > readable (and my preference would be to have it take optional > positional arguments, corresponding to {1}, {2}, etc, and optional > named arguments, corresponding to {name} &c). Note that if it takes keyword arguments, but doesn't take a single dictionary-like object (like % and string.Template.substitute do), then you lose any ability to use clever or interesting dictionary-like objects for substitution. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From aleaxit at gmail.com Wed Apr 5 05:44:27 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Tue, 4 Apr 2006 20:44:27 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44333B95.4010906@colorstudy.com> References: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> Message-ID: On Apr 4, 2006, at 8:37 PM, Ian Bicking wrote: > Alex Martelli wrote: >> As for the % operator, I never liked it -- either a builtin >> function, or even better a method of string objects, is going to >> be much more readable (and my preference would be to have it take >> optional positional arguments, corresponding to {1}, {2}, etc, >> and optional named arguments, corresponding to {name} &c). > > Note that if it takes keyword arguments, but doesn't take a single > dictionary-like object (like % and string.Template.substitute do), > then you lose any ability to use clever or interesting dictionary- > like objects for substitution. True, that possibility is lost. It remains to be seen if "clever" and "interesting" in this context are to be taken _laudatory_ adjectives, or tantamount to what Sir Humphrey might ``praise'' as "courageous";-). Alex From ianb at colorstudy.com Wed Apr 5 05:51:38 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 04 Apr 2006 22:51:38 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> Message-ID: <44333ECA.3000505@colorstudy.com> Alex Martelli wrote: > On Apr 4, 2006, at 8:37 PM, Ian Bicking wrote: >> Alex Martelli wrote: >>> As for the % operator, I never liked it -- either a builtin >>> function, or even better a method of string objects, is going to be >>> much more readable (and my preference would be to have it take >>> optional positional arguments, corresponding to {1}, {2}, etc, and >>> optional named arguments, corresponding to {name} &c). >> >> Note that if it takes keyword arguments, but doesn't take a single >> dictionary-like object (like % and string.Template.substitute do), >> then you lose any ability to use clever or interesting dictionary-like >> objects for substitution. > > True, that possibility is lost. It remains to be seen if "clever" and > "interesting" in this context are to be taken _laudatory_ adjectives, or > tantamount to what Sir Humphrey might ``praise'' as "courageous";-). Clever and interesting would include my quoting, evaluating, and formatting wrappers. It might also include special gettext expressions, case insensitivity, or other interesting wrappers. Also, a dictionary copy is made everytime you do **, while for string substitution no copy is really needed. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From guido at python.org Wed Apr 5 06:03:58 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 21:03:58 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <4432B81F.1070903@cox.net> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> Message-ID: On 4/4/06, Tim Hochberg wrote: (Could you not change the subject each time? Or if you do, could you assume the reader hasn't necessarily read your previous posts? For gmail users like me, each subject change starts a new thread -- like in newsgroups -- and having so far ignored any thread that's grown to 100 messages or so I eagerly jump into any new thread; only to find that it's hard to follow the first message there since it's assuming I read the tail end of that other thread.) (At least you included your Protocol class at the end, making your post slightly self-contained.) > Considering generic function in combination adaption led to one more > small change in the Protocol implementation that I'm playing with, and > all of the sudden I'm left with something that I could actually use. Cool. Could you comment on the Protocol implementation I posted separately? > What I realized was that you could factor out the part that potentially > has a lot of variation and suddenly you have a relatively simple > framework that's extensible in all sorts of ways. By default, everything > is the same as in the last iteration, but now it's much easier to change > the behaviour by subtyping. Cool. We seem to be converging on a new kind of adaptation implementation (I call it 2nd generation adaptation) where the protocol is an object with register() and adapt() methods. In true duck typing style all we have to do is agree on how to call adapt() and register(), and whether there are other methods (like a convenience to register using a decorator). You & I seem to have chosen slightly different styles, but in essence we're very close. (Except that I can't see I understand the point of your TV example, which seems very artificial.) > Let's return to the case that first bugged me about T->P dispatch. I > have some tagged values and I want to assemble two of them to form a > complex number. > > First we need a tagged value type. In reality I'd have this already > > >>> class TV(object): > ... def __init__(self, value, tag): > ... self.value = value > ... self.tag = tag I suppose this is implementing some theoretical abstraction popular in some literature you've been reading recently? Us non-academicians could use some help in gauging the significance of this class. > Then we need to subclass Protocol. I change the behaviour of keysof so > that now things are looked up in the registry strictly based on their > tags. I can't say I fully fathom the concept of "keysof". Perhaps you can explain it better and then we can come up with a better name? (Actually I think now I do understand it, and I still find the name horrible. It represents all the keys that you want to look up in the registry looking for an adapter. It gets passed the arguments to __call__ -- which is really a shortcut to spell adapt in a cute way; that confused me too -- and the base class assumes __call__ -- or adapt -- is called with a single argument, like a typical adapt() call. But your subclass calls it with tagged values. OK, so what you *really* wanted there was keyword arguments?) > Note that this can take an arbitrary number of arguments. In the > default implementation (see below) only a single argument is allowed, > which gives you the basic adapter(obj) -> newobj behaviour. Which puzzled me at first (I had to read the Protocol class at the end of your post before I could understand the rest). > >>> class TVProtocol(Protocol): > ... def keysof(self, *args): > ... try: > ... yield tuple(x.tag for x in args) > ... except AttributeError: > ... pass > >>> ascomplex = TVProtocol('complex_from_tagged') Shouldn't that be ascomplex = TVProtocol('ascomplex') ? > Then I define some converters: > > >>> import cmath > >>> @ascomplex.when(('real', 'imag')) > ... def complex_from_real_imag(real, imag): > ... return real.value + 1j*imag.value > >>> @ascomplex.when(('mag', 'angle')) > ... def complex_from_mag_angle(mag, angle): > ... return mag.value * cmath.exp(1j * cmath.pi / 180 * angle.value) > >>> @ascomplex.when(('db', 'angle')) > ... def complex_from_db_angle(db, angle): > ... return 10**(db.value/20.0) * cmath.exp(1j * cmath.pi / 180 * angle.value) Can we please get rid of the convention of naming the registration function when()? I don't find it cute at all, and it's not particularly indicative of what it does (registration). > Here's some values that I can assume came from elsewhere: > > >>> tv_re, tv_im = TV(1, 'real'), TV(2, 'imag') > >>> tv_db, tv_ang, tv_mag = TV(0, 'db'), TV(90, 'angle'), TV(2, 'mag') (Aside: I wonder if this would look less awkward if you changed TV.__init__ so that you can write TV(real=1), TV(imag=2) etc.) > And here's how I'd use it: > > >>> ascomplex(tv_re, tv_im) > (1+2j) > >>> ascomplex(tv_db, tv_ang) > (6.1230317691118863e-017+1j) > >>> ascomplex(tv_mag, tv_ang) > (1.2246063538223773e-016+2j) > >>> ascomplex(tv_db, tv_mag) > Traceback (most recent call last): > ... > ValueError: adapter not found Or is anyone of these calling keysof() with multiple args? > All of the sudden this is looking like something I could probably use. I'd like to hear what you think of my version and how you'd refactor it. > I also tried a simple generic function implementation on top of this (no > inheritance, keysof just returned a tuple of types). That was also easy. Post this, please! > Could full blown generic dispatch be added just by subclassing and > adding the correct, and obviously much more complex, version of keysof? > It seems likely, but I'm not certain. If so, this is starting to look > like a very promising approach. It would be useful to compare this to adaptation built on top of generic functions. Alex seems to be convinced that adaptation is more powerful than generic functions; but he hadn't considered the possibility of having a generic function that's a factory (like the iterator-factory in my example). I don't know if that affects his opinion though; he seems to find it important that adaptation can return an object that has multiple methods that belong together. > The updated Protocol implementation is below. > > class Protocol(object): > all_protocols = set() What's the signigificance of all_protocols? You're not using it. The key attraction of this class is that it *doesn't* need to keep track of all protocols. In fact, IMO there's no use for concepts like "all protocols that object X implements". Perhaps there could be a need for this in a narrower concept (I can't rule out that Zope has some use for this) but that could always be done by requiring all *relevant* protocols subclass a certain base implementation that keeps track of this. For general Python protocols I don't think it's necessary. For that matter, after reading Alex's seminal post, I don't see any reason why protocols should have anything to do with interfaces (although it's fine for some framework's interfaces to be protocols). > def __init__(self, name, doc=''): > self.name = name > self.registry = {} > self.__doc__ = doc > self.all_protocols.add(self) > def __repr__(self): > return "" % self.name > __str__ = __repr__ > def __call__(self, *args): > for key in self.keysof(*args): > adapter = self.registry.get(key, None) > if adapter is not None: > return adapter(*args) > raise ValueError('adapter not found') So __call__ is what used to be call adapt. Or, rather, where Alex used to write adapt(x, P) and where I write P.adapt(x), you just write P(). Clever. Perhaps this could become the key to generic functions on top of adaptation? That would be revolutionary! > def keysof(self, *args): > if len(args) != 1: > raise TypeError("%s expects 1-argument, got %s" (self, len(args))) > obj = args[0] > mro = type(obj).__mro__ > for cls in mro: > yield cls > def register(self, adapter, *types): > if not callable(adapter): > raise TypeError("adapters must be callable") > for t in types: > self.registry[t] = adapter > def when(self, *types): > def decorator(adapter): > self.register(adapter, *types) > return adapter > return decorator Sorry for the thoroughly random nature of this post. I kept going back and forth and now I have to go -- but I still want to post it. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From nnorwitz at gmail.com Wed Apr 5 06:05:28 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 4 Apr 2006 20:05:28 -0800 Subject: [Python-3000] StringIO vs cStringIO, pickle vs cPickle In-Reply-To: References: <20060404141759.GB9247@panix.com> Message-ID: On 4/4/06, Guido van Rossum wrote: > > I'm in agreement with the sentiment of keeping the Python code though > -- it's incredibly useful for example in Jython and IronPython. > Perhaps we should switch to a naming scheme where we have a "pickle" > module that you're supposed to import, which dynamically tries to > import first "_pickle" and if that fails imports "_pypickle" (or some > such naming scheme). We could have a subdir, but I'm not sure I like that. We might wind up with a lot of dirs on sys.path if we add a bunch of subdirs. n From guido at python.org Wed Apr 5 06:10:30 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 4 Apr 2006 21:10:30 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: Message-ID: On 4/4/06, Alex Martelli wrote: > > On Apr 4, 2006, at 5:29 PM, Guido van Rossum wrote: > [excellent example, mostly snipped] > > Just for when this will be published/blogged (as it deserves to be!), > tiny nit: > > > if hasattr(obj, "__iter__"): > > return obj.__iter__() > > raise TypeError("Can't iterate over a %s object" % > > obj.__class__.__name__) > > I think we should check/get __iter__ from obj.__class__, not from obj > itself (new style classes only take special methods from the type, > not from the instance). This applies to other occurrences of this > pattern later, too. I prefer to ignore that detail when writing Python code though. > > def register(self, T, A): > > self.registry[T] = A > > Why not accept more than one type (for the same adapter) at one gulp? > > def register(self, A, *Ts): > for T in Ts: > self.registry[T] = A I had that first. :-) Deleted it because it just distracted. > > def __call__(self, T): > > # This is invoked when a Protocol instance is used as a > > decorator. > > def helper(A): > > self.register(T, A) > > return A > > return helper > > And similarly here, take *Ts in the __call__ and pass it on to > self.register. Ditto. > > @AdaptingIterProtocol(list) > > def _AdaptingSequenceIter(obj): > > return SequenceIter(obj) > > > > AdaptingIterProtocol.register(str, _AdaptingSequenceIter) > > AdaptingIterProtocol.register(unicode, _AdaptingSequenceIter) > > So this could become > > AdaptingIterProtocol.register(SequenceIter, list, str, unicode) Which I wrote at some point. :-) > ((no real need for the extra layer represented by > _AdaptingSequenceIter, though you could use it, if you insisted, with > all the types at one gulp. > > > def AdaptingIter(obj): > > return AdaptingIterProtocol.adapt(obj) > > Similarly, in the name of cutting down useless layers, > > AdaptingIter = AdaptingIterProtocol.adapt > > seems preferable to me. Right. In another post, Tim Hochberg uses __call__ instead of adapt; then we could just rename AdaptingIterProtocol to AdaptingIter and cut down one more layer. (Of course this presumes one doesn't have any *other* use in mind for calling a protocol -- which may not work for some frameworks.) > There seems to be an asymmetry in the functionality of adaptation and > generics as presented: adaptation as presented supports a search on > the mro but for one argument only, generics support N arguments but > no mro search. To enable closer comparison, we might have generics > support 1 argument and mro search, too, just as to ensure the feature > set is identical (they both can be extended, with different degree of > difficulty no doubt, but 1 argument each, with or w/o mro search, > seems to allow more direct comparison). Right. But I didn't have the time to do that, and I wanted the examples to be so short that more people could potentially understand them by just reading them top-to-buttom. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Wed Apr 5 06:46:13 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed, 5 Apr 2006 00:46:13 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44333B95.4010906@colorstudy.com> References: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> Message-ID: <200604050046.13884.fdrake@acm.org> On Tuesday 04 April 2006 23:37, Ian Bicking wrote: > Note that if it takes keyword arguments, but doesn't take a single > dictionary-like object (like % and string.Template.substitute do), then > you lose any ability to use clever or interesting dictionary-like > objects for substitution. Indeed; we currently do that so we can have ConfigParser values that refer to ${section:option} to get values from elsewhere in the configuration. Very nice to be able to do that. I've long been in favor of supporting strings that aren't identifiers as substitution names. -Fred -- Fred L. Drake, Jr. From aleaxit at gmail.com Wed Apr 5 07:01:31 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Tue, 4 Apr 2006 22:01:31 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> Message-ID: On Apr 4, 2006, at 9:03 PM, Guido van Rossum wrote: ... > It would be useful to compare this to adaptation built on top of > generic functions. Alex seems to be convinced that adaptation is more > powerful than generic functions; but he hadn't considered the > possibility of having a generic function that's a factory (like the > iterator-factory in my example). I don't know if that affects his > opinion though; he seems to find it important that adaptation can > return an object that has multiple methods that belong together. Hmmm, I guess I wasn't very clear in the last part of our talk today (well, I _was_ sort of rushing towards a meeting, maybe that excuses me;-)...: I _was_ (as in _used to be_) convinced that with generic- functions (which I used in Lisp and Dylan, but those are languages I only played around with -- I never had my income depend on developing code in them, except for a short time at my first job, and that was _before_ Common Lisp and CLOS existed!-) you had to "manually" keep a bunch of related function together, basically like when simulating OOP in a pure functional language (without a facility such as Haskell's typeclasses to group things up). I thought I had made it clear that the "flash of obviousness" about being able to use a generic function that's a factory had converted me on the road to Damascus. It IS now obvious to me that anything I can express as: pFoo = Protocol('foo') class FooFromInt(object): def x(...): def y(...): pFoo.register_adapter(int, FooFromInt) ... def zz(z): z = pFoo.adapt(z) return z.x()+z.y() I could almost-identically express as: gFoo = GenericFunction('foo') class FooFromInt(...as above...) gFoo.specialize(int, FooFromInt) ... def zz(z): z = gFoo(z) return z.x()+z.y() ...or whatever small syntax/semantics variations on either or both of protocols and generic functions one might wish. The two "framings" of the issue -- protocols carrying their specific mappings of types- to-adapter-callables, generic functions specializable by-type on a single argument -- are deeply isomorphic. Framing things in terms of generic functions and their specialization may well have conceptual advantages, such as a more natural, intuitive way to provide a "default" implementation for any other type, than registering an adapter for type 'object' would be with adaptation; or making it more obvious and direct to deal with the most frequent case, where all you ever need to call on objects satisfying protocol x is one single method (in that case, just use the generic function for the purpose, rather than making the genfunc a factory). Also, we avoid seeing "adaptation" spelled "adaption", which I've already seen happen;-). This being the case, I'm perfectly happy to sacrifice PEP 246 in favor of such generic functions, since I am now convinced that I can cover every use case of adaptation with genfuncs. Using genfuncs also kills any issue of "transitivity" since there's obviously no place to fit it, which doesn't faze me one bit;-). How genfuncs could be used as decorators (hopefully to decorate _classes_ too, since they're the easiest and most natural way to make factories;-), whether multimethods should be such (dispatching on the type of _multiple_ args) and how to reconcile that with inheritance ((I like Philip's idea that, if we go to multiple dispatch, it should be an error unless ONE specialization strictly dominates all others)), etc, etc, all are questions that are important and deserve discussion, but maybe these discussions would be more meaningful in the context of some "temporary working Pronouncements" such as "let's talk generics and forget adaptation until further notice", I think. > this. For general Python protocols I don't think it's necessary. For > that matter, after reading Alex's seminal post, I don't see any reason > why protocols should have anything to do with interfaces (although > it's fine for some framework's interfaces to be protocols). And there was much rejoicing in Alexland!-) Alex From martin at v.loewis.de Wed Apr 5 07:36:22 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 05 Apr 2006 07:36:22 +0200 Subject: [Python-3000] StringIO vs cStringIO, pickle vs cPickle In-Reply-To: References: Message-ID: <44335756.6020108@v.loewis.de> Just van Rossum wrote: > While I appreciate that there are maintained pure Python versions of > those modules, as a user it irritates me that I have the choice. The > argument so far for not replacing StringIO with cStringIO was that > they're subtly incompatible (mostly StringIO's "feature" to support > unicode, right?) I believe the rationale for providing them as old-style classes also was to support inheritance. Atleast, that's what the docstring in StringIO.py says: - There's also a much faster implementation in C, called cStringIO, but it's not subclassable. Regards, Martin From tim.hochberg at cox.net Wed Apr 5 07:33:18 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Tue, 04 Apr 2006 22:33:18 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> Message-ID: <4433569E.7030809@cox.net> Guido van Rossum wrote: >On 4/4/06, Tim Hochberg wrote: > >(Could you not change the subject each time? Or if you do, could you >assume the reader hasn't necessarily read your previous posts? For >gmail users like me, each subject change starts a new thread -- like >in newsgroups -- and having so far ignored any thread that's grown to >100 messages or so I eagerly jump into any new thread; only to find >that it's hard to follow the first message there since it's assuming I >read the tail end of that other thread.) > > > Will do. >(At least you included your Protocol class at the end, making your >post slightly self-contained.) > > > >>Considering generic function in combination adaption led to one more >>small change in the Protocol implementation that I'm playing with, and >>all of the sudden I'm left with something that I could actually use. >> >> > >Cool. Could you comment on the Protocol implementation I posted separately? > > I'll have a look at that ASAP. > > >>What I realized was that you could factor out the part that potentially >>has a lot of variation and suddenly you have a relatively simple >>framework that's extensible in all sorts of ways. By default, everything >>is the same as in the last iteration, but now it's much easier to change >>the behaviour by subtyping. >> >> > >Cool. We seem to be converging on a new kind of adaptation >implementation (I call it 2nd generation adaptation) where the >protocol is an object with register() and adapt() methods. In true >duck typing style all we have to do is agree on how to call adapt() >and register(), and whether there are other methods (like a >convenience to register using a decorator). You & I seem to have >chosen slightly different styles, but in essence we're very close. > >(Except that I can't see I understand the point of your TV example, >which seems very artificial.) > > > >>Let's return to the case that first bugged me about T->P dispatch. I >>have some tagged values and I want to assemble two of them to form a >>complex number. >> >>First we need a tagged value type. In reality I'd have this already >> >> >>> class TV(object): >>... def __init__(self, value, tag): >>... self.value = value >>... self.tag = tag >> >> > >I suppose this is implementing some theoretical abstraction popular in >some literature you've been reading recently? Us non-academicians >could use some help in gauging the significance of this class. > > Oh no.It's not the case that I'm guilty of reading any computer literature at all. It's a problem abstracted, probably poorly, from a real issue that I had and solved in a different manner. Essentially I read some large arrays from a file (touchstone format if anyone cares). These files have some minimal metadata including what format the complex numbers are represented in. You might think that I could convert them into complex numbers on the way in and be done with it. However, in my application I care about the "real" phase, not just the phase mod 360, so doing that to mag-angle or db-angle data destroys information that can't be recovered from the resulting real-imag data. Converting it all to mag-angle would work reasonably well except that it's painful to work with . What I actually do is convert it all to log(real+1j*imag). This preserves the phase and is relatively easy to work with in my domain, although it has some other drawbacks, notably for zero values. Had there been a nice framework in place to do adaption of this kind, I might have considered just keeping the data in its original format as long as possible and converting it only when needed. The real problem differs from the example I gave in that there's really only ever one object that contains two values plus the tag (real-imag, mag-angle or db-angle). In the example I was guilty of conflating two things I was interested in: converting a tagged value, and looking up adapters that operate on multiple values -- that probably just led to confusion. >>Then we need to subclass Protocol. I change the behaviour of keysof so >>that now things are looked up in the registry strictly based on their >>tags. >> >> > >I can't say I fully fathom the concept of "keysof". Perhaps you can >explain it better and then we can come up with a better name? > > I'm not fond of the name myself, but I couldn't think of anything better. >(Actually I think now I do understand it, and I still find the name >horrible. It represents all the keys that you want to look up in the >registry looking for an adapter. It gets passed the arguments to >__call__ -- which is really a shortcut to spell adapt in a cute way; >that confused me too -- and the base class assumes __call__ -- or >adapt -- is called with a single argument, like a typical adapt() >call. > Correct. >But your subclass calls it with tagged values. OK, so what you >*really* wanted there was keyword arguments?) > > I'm afraid I don't understand this. (Oh wait, I see you explain further down.) > > >>Note that this can take an arbitrary number of arguments. In the >>default implementation (see below) only a single argument is allowed, >>which gives you the basic adapter(obj) -> newobj behaviour. >> >> > >Which puzzled me at first (I had to read the Protocol class at the end >of your post before I could understand the rest). > > > >> >>> class TVProtocol(Protocol): >>... def keysof(self, *args): >>... try: >>... yield tuple(x.tag for x in args) >>... except AttributeError: >>... pass >> >>> ascomplex = TVProtocol('complex_from_tagged') >> >> > >Shouldn't that be > > ascomplex = TVProtocol('ascomplex') > >? > > Probably. The name is only used for repr and str, so it doesn't necessarily have to be coupled to the name the protocol is initially bound to. Still it was probably silly to include that example (these were all ripped out of a now ridicuously long and meandering docstring that I've been using both to test and to try out various things). >>Then I define some converters: >> >> >>> import cmath >> >>> @ascomplex.when(('real', 'imag')) >>... def complex_from_real_imag(real, imag): >>... return real.value + 1j*imag.value >> >>> @ascomplex.when(('mag', 'angle')) >>... def complex_from_mag_angle(mag, angle): >>... return mag.value * cmath.exp(1j * cmath.pi / 180 * angle.value) >> >>> @ascomplex.when(('db', 'angle')) >>... def complex_from_db_angle(db, angle): >>... return 10**(db.value/20.0) * cmath.exp(1j * cmath.pi / 180 * angle.value) >> >> > >Can we please get rid of the convention of naming the registration >function when()? I don't find it cute at all, and it's not >particularly indicative of what it does (registration). > > OK, no more when. I'm not entirely happy with register for a decorator name because it sounds like the argument (type/tag/whatever) is what's getting registered, not the following function. I'll try it out though and see how it looks and perhaps something better will come along. >>Here's some values that I can assume came from elsewhere: >> >> >>> tv_re, tv_im = TV(1, 'real'), TV(2, 'imag') >> >>> tv_db, tv_ang, tv_mag = TV(0, 'db'), TV(90, 'angle'), TV(2, 'mag') >> >> > >(Aside: I wonder if this would look less awkward if you changed >TV.__init__ so that you can write TV(real=1), TV(imag=2) etc.) > > Ah, here's where the keyword args come in. Yes that probably would be better. The example's artificial in any event. It's really all read from a file. > > >>And here's how I'd use it: >> >> >>> ascomplex(tv_re, tv_im) >>(1+2j) >> >>> ascomplex(tv_db, tv_ang) >>(6.1230317691118863e-017+1j) >> >>> ascomplex(tv_mag, tv_ang) >>(1.2246063538223773e-016+2j) >> >>> ascomplex(tv_db, tv_mag) >>Traceback (most recent call last): >> ... >>ValueError: adapter not found >> >> > >Or is anyone of these calling keysof() with multiple args? > > They all are. In each of the cases above, keysof is getting passed two objects and in turn returning a 2-tuple of tags. >>All of the sudden this is looking like something I could probably use. >> >> > >I'd like to hear what you think of my version and how you'd refactor it. > > I'll definately look that over. >>I also tried a simple generic function implementation on top of this (no >>inheritance, keysof just returned a tuple of types). That was also easy. >> >> > >Post this, please! > > class SimplePolyargProtocol(Protocol): """Build a simple multiple dispatch mechanism on top of the basic Protocol >>> addnum = SimplePolyargProtocol('addnum') >>> @addnum.when((int, int), (int, float), (float, int), (float, float)) ... def add(a, b): return repr(a + b) >>> @addnum.when((str, int), (str, str), (str, float), (int, str), (float, str)) ... def add(a, b): return '%s+%s' % (a, b) >>> addnum(5, 7) '12' >>> addnum(5, '33') '5+33' """ def keysof(self, *args): if not len(args): raise TypeError("%s expects at least one argument" (self, len(args))) yield tuple(type(x) for x in args) Yes, the example in the docstring is sortof bizzaro. No time to clean it up right now. Actually looking at this now, I suppose the len(args) check is not really needed. If someone wants to register a function to work on no arguments, who am I sto stop them? > > >>Could full blown generic dispatch be added just by subclassing and >>adding the correct, and obviously much more complex, version of keysof? >>It seems likely, but I'm not certain. If so, this is starting to look >>like a very promising approach. >> >> > >It would be useful to compare this to adaptation built on top of >generic functions. Alex seems to be convinced that adaptation is more >powerful than generic functions; but he hadn't considered the >possibility of having a generic function that's a factory (like the >iterator-factory in my example). I don't know if that affects his >opinion though; he seems to find it important that adaptation can >return an object that has multiple methods that belong together. > > > >>The updated Protocol implementation is below. >> >>class Protocol(object): >> all_protocols = set() >> >> > >What's the signigificance of all_protocols? You're not using it. > Right. It's vestigial from an earlier version that did 1st-generation adaption. In that it really did look like you wanted ways to search the protocols. When I started writing this version I still had that on the brain, so all_protocols got included. It got ripped out shortly after I posted my previous message. > The >key attraction of this class is that it *doesn't* need to keep track >of all protocols. In fact, IMO there's no use for concepts like "all >protocols that object X implements". Perhaps there could be a need for >this in a narrower concept (I can't rule out that Zope has some use >for this) but that could always be done by requiring all *relevant* >protocols subclass a certain base implementation that keeps track of >this. For general Python protocols I don't think it's necessary. For >that matter, after reading Alex's seminal post, I don't see any reason >why protocols should have anything to do with interfaces (although >it's fine for some framework's interfaces to be protocols). > > > >> def __init__(self, name, doc=''): >> self.name = name >> self.registry = {} >> self.__doc__ = doc >> self.all_protocols.add(self) >> def __repr__(self): >> return "" % self.name >> __str__ = __repr__ >> def __call__(self, *args): >> for key in self.keysof(*args): >> adapter = self.registry.get(key, None) >> if adapter is not None: >> return adapter(*args) >> raise ValueError('adapter not found') >> >> > >So __call__ is what used to be call adapt. Or, rather, where Alex used >to write adapt(x, P) and where I write P.adapt(x), you just write P(). >Clever. > >Perhaps this could become the key to generic functions on top of >adaptation? That would be revolutionary! > > > >> def keysof(self, *args): >> if len(args) != 1: >> raise TypeError("%s expects 1-argument, got %s" (self, len(args))) >> obj = args[0] >> mro = type(obj).__mro__ >> for cls in mro: >> yield cls >> def register(self, adapter, *types): >> if not callable(adapter): >> raise TypeError("adapters must be callable") >> for t in types: >> self.registry[t] = adapter >> def when(self, *types): >> def decorator(adapter): >> self.register(adapter, *types) >> return adapter >> return decorator >> >> > >Sorry for the thoroughly random nature of this post. I kept going back >and forth and now I have to go -- but I still want to post it. > > Not a problem, regards, -tim From rrr at ronadam.com Wed Apr 5 07:38:04 2006 From: rrr at ronadam.com (Ron Adam) Date: Wed, 05 Apr 2006 00:38:04 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44331180.9000203@canterbury.ac.nz> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> <44330694.3060900@colorstudy.com> <44331180.9000203@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Ian Bicking wrote: > >> Using {} instead of $/${} doesn't seem like a big win. > > Perhaps this is where we differ. To me it > *does* make quite a substantial difference > to readability -- easily enough to override > whatever small advantage there might be in > following what everyone else does. > > -- > Greg I agree with you Greg about {}. +1 There is also a factor of underlying consistency in grouping that I like. In most every place else in python a subgroup is delimited by like or opposing symbols. So this just seems to be an extension of that. Most of the examples have been reasonably easy to read because they contain white space at the right places. To better compare use harder to read examples without white spaces. "abc%sefg" "abc$(s)efg" "abc{s}efg" <-- I think this is an improvement. Cheers, Ron From janssen at parc.com Wed Apr 5 08:02:36 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 4 Apr 2006 23:02:36 PDT Subject: [Python-3000] String formating operations in python 3k In-Reply-To: Your message of "Tue, 04 Apr 2006 20:33:36 PDT." Message-ID: <06Apr4.230244pdt."58633"@synergy1.parc.xerox.com> > As for the % operator, I never liked it -- either a builtin function, > or even better a method of string objects, is going to be much more > readable (and my preference would be to have it take optional > positional arguments, corresponding to {1}, {2}, etc, and optional > named arguments, corresponding to {name} &c). Sure, I prefer the builtin function myself to the % operator. But I still like the printf-like syntax of the formatting commands, and think that %(foo) is a reasonable extension to that syntax. Bill From tjreedy at udel.edu Wed Apr 5 08:30:58 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 5 Apr 2006 02:30:58 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions References: Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604041729s227336bdp41c122700886380c at mail.gmail.com... > #!/usr/bin/python2.4 > > """An example of generic functions vs. adaptation. Nice. For me, the idea of 'generic function' goes into my brain much more smoothly than the idea of 'protocol adaptation' even though the effect seems to be about the same. That is because I think of Python as a language for writing generic algorithms. Untyped polymorphic expressions, coupled with special methods that plug user types into the syntax of such expressions, are the main means to do so. But when these are inadequate (because different types require different expressions, or because some types benefit from specialized expressions) or difficult (due to builtin-ness or inavailability of code), a more generic means might be a nice addition. One test of whether a specific implementation of the idea should be added is whether it would be used to replace some of the ad hoc special-case methods, as with pickle reduce. Terry Jan Reedy From tim.hochberg at ieee.org Wed Apr 5 09:09:00 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 05 Apr 2006 00:09:00 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: Message-ID: <44336D0C.1010409@ieee.org> Guido van Rossum wrote: > #!/usr/bin/python2.4 > > """An example of generic functions vs. adaptation. > > After some face-to-face discussions with Alex I'm presenting here two > extremely simplified implementations of second-generation adaptation > and generic functions, in order to contrast and compare. > > As the running example, we use the iterator protocol. Pretend that > the iter() built-in function doesn't exist; how would we implement > iteration using adaptation or generic functions? > > """ > > > __metaclass__ = type # Use new-style classes everywhere > > > # 0. The SequenceIter class is shared by all versions. > > class SequenceIter: > def __init__(self, obj): > self.obj = obj > self.index = 0 > def next(self): > i = self.index > self.index += 1 > if i < len(self.obj): > return self.obj[i] > raise StopIteration > def __iter__(self): > # This exists so we can use this in a for loop > return self > > > # 1. Do it manually. This is ugly but effective... until you need to > # iterate over a third party object type whose class you can't modify. > > def ManualIter(obj): > if isinstance(obj, (list, str, unicode)): > return SequenceIter(obj) > if isinstance(obj, dict): > # We can't really do a better job without exposing PyDict_Next() > return SequenceIter(obj.keys()) > if hasattr(obj, "__iter__"): > return obj.__iter__() > raise TypeError("Can't iterate over a %s object" % obj.__class__.__name__) > > > # 2. Using adaptation. First I show a simple implementation of > # adaptation. In a more realistic situation this would of course be > # imported. I call this "second generation adaptation" because the > # adaptation from protocol P to type T is invoked as P.adapt(T), and > # the registration of an adapter function A for type T is invoked as > # P.register(T, A). The only "smart" feature of this implementation > # is its support for inheritance (in T, not in P): if T has registered > # an adapter A for P, then A is also the default adapter for any > # subclass S of T, unless a more specific adapter is registered for S > # (or for some base of S that comes before T in S's MRO). > > class Protocol: > def __init__(self, name): > self.registry = {} > self.name = name > def register(self, T, A): > self.registry[T] = A > def adapt(self, obj): > for T in obj.__class__.__mro__: > if T in self.registry: > return self.registry[T](obj) > raise TypeError("Can't adapt %s to %s" % > (obj.__class__.__name__, self.name)) > def __call__(self, T): > # This is invoked when a Protocol instance is used as a decorator. > def helper(A): > self.register(T, A) > return A > return helper [Guido in another thread where I had posted a Protocol class] > I'd like to hear what you think of my version and how you'd refactor it. The above ended up being almost identical to the Protocol class I came up with. It hopefully speaks well of the general concept that different people came up with essentially the same thing. There are two minor spelling differences, how to spell adapt (I spelled it __call__) and how to spell decorate (I spelled it 'when', here it's spelled __call__). So far, I like how code looks when adapt is spelled __call__, still using __call__ to spell decorate sidesteps the need to come up with a good name ('when' is right out). There are two other differences. The first is that I allowed multiple type arguments (using *args) in three locations. The first two are in register and call (decorate); this is done to allow multiple types to be registered at once. This requires that the order of the A and T arguments be switched in register so that the signature becomes register(A, *types). The other location is in adapt. This is done to allow subclasses to support multiple arguments. Thus the signature to adapt is adapt(self,*args). The second difference is that I factored out how to get keys for the registry. With this change adapt becomes: def adapt(self, *args): for T in self.keysof(obj): if T in self.registry: return self.registry[T](obj) raise TypeError("Can't adapt %s to %s" % (obj.__class__.__name__, self.name)) def keysof(self, *args): if len(args) != 1: raise TypeError('expected 1 argument') return args[0].__mro__ (and no, no ones happy with the name keysof -- suggestions welcome). At first this seems a step backwards; Things got longer and marginally more complicated. However, factoring out keysof gives you another level of flexibility. For example, you can now build a simple multiple dispatch mechanism on top of protocol by simple overriding keysof. class MultipleDispatchProtocol(Protocol): def keysof(self, *args): return [tuple(type(x) for x in args)] This only dispatches on types, it doesn't do any mro traversals or anything. Still, I found it pretty impressive that it was so simple. It's possible that this factoring out of keysof was a mistake; adapt is so simple that perhaps overriding it completely is the right approach. On the other hand, if one were to try to implement full blown multimethods on top of this, I think all of the complexity would live in keysof, leaving adapt simple. On the third hand, perhaps only that class would need to factor out the keysof logic. I think I'll try unfactoring it and see how my examples end up looking. So after all of that, I think my conclusion is that I wouldn't refactor this at all, at least not yet. I'd add support for multiple registration and possibly spell adapt as __call__, otherwise I'd leave it alone. My opinion may change after I try ripping out keysof and see how it looks. Regards, -tim > > # Now I show how to define and register the various adapters. In a > # more realistic situation these don't all have to be in the same file > # of course. > > AdaptingIterProtocol = Protocol("AdaptingIterProtocol") > > @AdaptingIterProtocol(list) > def _AdaptingSequenceIter(obj): > return SequenceIter(obj) > > AdaptingIterProtocol.register(str, _AdaptingSequenceIter) > AdaptingIterProtocol.register(unicode, _AdaptingSequenceIter) > > @AdaptingIterProtocol(dict) > def _AdaptingDictIter(obj): > return SequenceIter(obj.keys()) > > @AdaptingIterProtocol(object) > def _AdaptingObjectIter(obj): > if hasattr(obj, "__iter__"): > return obj.__iter__() > raise TypeError("Can't iterate over a %s object" % obj.__class__.__name__) > > def AdaptingIter(obj): > return AdaptingIterProtocol.adapt(obj) > > > # 3. Using generic functions. First I show a simple implementation of > # generic functions. In a more realistic situation this would of > # course be imported. > > class GenericFunction: > def __init__(self, default_function): > self.default_function = default_function > self.registry = {} > def register(self, *args): > def helper(F): > self.registry[args] = F > return F > return helper > def __call__(self, *args): > types = tuple([obj.__class__ for obj in args]) > function = self.registry.get(types, self.default_function) > return function(*args) > > # Now I show how to define a generic function and how to register the > # various type-specific implementations. In a more realistic > # situation these don't all have to be in the same file of course. > > @GenericFunction > def GenericIter(obj): > """This is the docstring for the generic function.""" > # The body is the default implementation > if hasattr(obj, "__iter__"): > return obj.__iter__() > raise TypeError("Can't iterate over %s object" % obj.__class__.__name__) > > @GenericIter.register(list) > def _GenericSequenceIter(obj): > return SequenceIter(obj) > > GenericIter.register(str)(_GenericSequenceIter) > GenericIter.register(unicode)(_GenericSequenceIter) > > @GenericIter.register(dict) > def _GenericDictIter(obj): > return SequenceIter(obj.keys()) > > > # 4. Show that all of these work equivalently. > > def main(): > examples = [ > [1, 2, 3, 4, 5], > "abcde", > u"ABCDE", > {"x": 1, "y": 2, "z": 3}, > (6, 7, 8, 9, 10), # Not registered, but has __iter__ method > 42, # Not registered and has no __iter__ method > ] > > functions = [ManualIter, AdaptingIter, GenericIter] > > for function in functions: > print > print "***", function, "***" > for example in examples: > print ":::", repr(example), ":::" > try: > iterator = function(example) > except Exception, err: > print "!!! %s: %s !!!" % (err.__class__.__name__, err) > else: > for value in function(example): > print repr(value), > print > > if __name__ == "__main__": > main() > > # -- > # --Guido van Rossum (home page: http://www.python.org/~guido/) From g.brandl at gmx.net Wed Apr 5 09:18:21 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Wed, 05 Apr 2006 09:18:21 +0200 Subject: [Python-3000] pre-PEP: Things that Will Not Change in Python 3.0 In-Reply-To: References: Message-ID: Guido van Rossum wrote: > On 4/4/06, Georg Brandl wrote: >> Guido van Rossum wrote: >> > I was going to comment about this one "check it in, we'll add to it >> > later", but then I realized it's not 100% clear whether this is a >> > feature PEP or a meta-PEP? It focuses on features so by that yardstick >> > it's a feature PEP. But in its list-of-miscellany nature it >> > approximates a meta-PEP. Hmm, perhaps it ought to be PEP 3999 (which >> > can be seen as minus one in the numbering scheme for Python-3000 PEPs >> > :-)? >> > >> > In any case let's not let it longer for long. Realistically, I think >> > it can be a meta-PEP. >> >> Since it's a meta-PEP, but closely related to features, I've checked it >> in with number 3099, that is, "first feature-PEP minus one". > > Excellent! (Thanks to Terry Reedy for the idea.) > >> Related question: as the current PEP 3000 contains mainly feature proposals, >> shouldn't it be renamed to 3100? PEP 3000 could then be a quasi-summary of >> 3xxx PEPs and random important facts that don't fit elsewhere. > > Good idea. Maybe 3000 should then be the meta-meta PEP with the > Process for deciding Python 3000 Processes. I've now moved PEP 3000 to PEP 3100. However, I don't consider myself metapythonical enough to write the meta-meta-PEP 3000 ;) Georg From walter at livinglogic.de Wed Apr 5 09:43:04 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 05 Apr 2006 09:43:04 +0200 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> Message-ID: <44337508.2030505@livinglogic.de> Guido van Rossum wrote: > On 4/4/06, Tim Hochberg wrote: > [...] >> def __init__(self, name, doc=''): >> self.name = name >> self.registry = {} >> self.__doc__ = doc >> self.all_protocols.add(self) >> def __repr__(self): >> return "" % self.name >> __str__ = __repr__ >> def __call__(self, *args): >> for key in self.keysof(*args): >> adapter = self.registry.get(key, None) >> if adapter is not None: >> return adapter(*args) >> raise ValueError('adapter not found') > > So __call__ is what used to be call adapt. Or, rather, where Alex used > to write adapt(x, P) and where I write P.adapt(x), you just write P(). > Clever. I've used that in my example too (see http://mail.python.org/pipermail/python-3000/2006-April/000308.html) But might closes off one route we could take if we're adapting to something that must provide more than one function/method (i.e. an interface) and we'd like to have the interface and the adaption registry be identical. Suppose we're designing a new interface for sequence. A sequence must provide getitem() and len() methods. If the adapt(at)ion API only used register and adapt methods we could do something like this: class Interface(object): class __metaclass__(type): def __new__(mcl, name, bases, dict): # Give each class it's own registry dict["registry"] = {} return type.__new__(mcl, name, bases, dict) @classmethod def register(cls, adapter, T): cls.registry[T] = adapter @classmethod def adapt(cls, obj): for base in type(obj).__mro__: try: return cls.registry[base](obj) except KeyError: pass raise TypeError("can't adapt %r to %r" % (obj, cls)) class Sequence(Interface): def getitem(self, index): pass def len(self): pass class PythonSeqAsSequence(Sequence): def __init__(self, obj): self.obj = obj def getitem(self, index): return self.obj[i] def len(self): return len(self.obj) Sequence.register(PythonSeqAsSequence, list) print Sequence.adapt([1,2,3]).len() But if adapting is done via __call__() we have a problem: Sequence already provides a __call__, the constructor. Of course if this worked print Sequence([1,2,3]).len() would look much better than print Sequence.adapt([1,2,3]).len() but I'm not sure if it's possible to work around the constructor/adapt clash. Bye, Walter D?rwald From pedronis at strakt.com Wed Apr 5 10:19:15 2006 From: pedronis at strakt.com (Samuele Pedroni) Date: Wed, 05 Apr 2006 10:19:15 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: Message-ID: <44337D83.7050802@strakt.com> Guido van Rossum wrote: > One followup. Alex mentioned that PyPy used to have its own > implementation of multiple dispatch (which is really the same as > generic functions) but ditched over a year ago. Would Armin or Samuele > care to explain why? Do they no longer like multiple dispatch or was > the reason for the extraction specific to PyPy? > No, PyPy is still using multi-methods, both in the translation chain and in the Python implementation itself. They are both implementation details, we are not offering at the moment user-level multimethods. In the translation chain itself we use a simple 2 argument multiple dispatching based on a pairtype metaclass: http://codespeak.net/svn/pypy/dist/pypy/annotation/pairtype.py both for annotation (our type inference): (example) see class __extend__(pairtype(SomeObject, SomeObject)): ... etc. in http://codespeak.net/svn/pypy/dist/pypy/annotation/binaryop.py and the part of the translation that converts high-level operations to low-level operations: (example) see class __extend__(pairtype(StringRepr, SliceRepr)): ... class __extend__(pairtype(StringRepr, StringRepr)): ... etc. in http://codespeak.net/svn/pypy/dist/pypy/rpython/rstr.py For this kind of code multi-dispatching is very natural. For the interpreter itself we use a form of general asymmetric multi-dispatching, implemented at the moment as chained double-dispatching although other strategies are possible, for the special methods implementations (what in CPython would go in the type table slots). I think the difference Alex is referring to is that at some point we were exposing the multimethods as __xxx__ directly, but this didn't match descriptor and Python semantics properly, so we moved to expose as __xxx__ restrictions of the multimethods on the relevant part of the hierarchy. We use multimethods for implementing Python types behavior because it makes easy to have multiple inter-operating internal implementations appearing as the same user level type (this should make it simple for example to have integers implemented as both short tagged integers and normal boxed ones for larger ones, or to even include long integers under the "int" type). About multi-dispatching, when I suggested switching to C3 for the mro, it was also because it's a good basis for multi-dispatching (it was developed in such a context). In Python is relatively easy to roll your own multi-dispatching when you need it. It Is also true that languages offering multi-dispatching usually offer rather sophisticated versions of it, because it is natural to need to tweak dispatching in practice, or go for very general forms like predicate dispatching. As a side note I think, while multi-dispatching makes sense as a language design level addition, adaptation feels more like a software engineering/library level tool. regards. From ncoghlan at gmail.com Wed Apr 5 11:35:03 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 05 Apr 2006 19:35:03 +1000 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <44337508.2030505@livinglogic.de> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> Message-ID: <44338F47.3050905@gmail.com> Walter D?rwald wrote: > But if adapting is done via __call__() we have a problem: Sequence > already provides a __call__, the constructor. Of course if this worked > > print Sequence([1,2,3]).len() > > would look much better than > > print Sequence.adapt([1,2,3]).len() > > but I'm not sure if it's possible to work around the constructor/adapt > clash. You can redefine __call__ in the metaclass to fix it: Py> class ClassCall: ... class __metaclass__(type): ... def __call__(cls): ... print "Calling", cls ... Py> ClassCall() Calling So your example would become: class MetaInterface(type): def __new__(mcl, name, bases, dict): # Give each class it's own registry cls = type.__new__(mcl, name, bases, dict) cls.registry = {} return cls def register(cls, adapter, T): cls.registry[T] = adapter def register_for(cls, T): # Registration decorator def register_func(adapter): cls.register(adapter, T) return register_func def __call__(cls, obj): # Cannot construct interface instances # Instead, calling the class triggers adaptation for base in type(obj).__mro__: try: adapter = cls.registry[base] except KeyError: pass if adapter is None: return obj return adapter(obj) raise TypeError("can't adapt %r to %r" % (obj, cls)) class Interface(object): __metaclass__ = MetaInterface class Sequence(Interface): def getitem(self, index): pass def len(self): pass class PythonSeqAsSequence(object): # Adapter is a normal class! def __init__(self, obj): self.obj = obj def getitem(self, index): return self.obj[i] def len(self): return len(self.obj) Sequence.register(PythonSeqAsSequence, list) print Sequence([1,2,3]).len() # Or using the function decorator version @Sequence.register_for(tuple) def tuple_as_seq(obj): class AdaptedTuple(object): def getitem(self, index): return obj[i] def len(self): return len(obj) return AdaptedTuple() -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Wed Apr 5 13:43:30 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 05 Apr 2006 21:43:30 +1000 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44336D0C.1010409@ieee.org> References: <44336D0C.1010409@ieee.org> Message-ID: <4433AD62.80406@gmail.com> Tim Hochberg wrote: > So after all of that, I think my conclusion is that I wouldn't refactor > this at all, at least not yet. I'd add support for multiple registration > and possibly spell adapt as __call__, otherwise I'd leave it alone. My > opinion may change after I try ripping out keysof and see how it looks. I was curious to see how the adaptation version actually looked with your and Guido's versions mixed. While writing it, I also noticed two interesting cases worth simplifying: 1. the "no adapter needed case" (for registering that a type implements a protocol directly) 2. the "missing adapter case" (for providing a default adaptation, as in the generic function case) Here's what the whole thing ended up looking like: def null_adapter(*args): """Adapter used when adaptation isn't actually needed""" if len(args) > 1: return args else: return args[0] class Protocol(object): """Declare a named protocol""" def __init__(self, name): self.registry = {} self.name = name def register(self, adapter, *keys): """Register an adapter from given registry keys to the protocol""" if adapter is None: adapter = null_adapter for key in keys: self.registry[key] = adapter def register_for(self, *keys): """Function decorator to register as an adapter for given keys""" def helper(adapter): self.register(adapter, *keys) return adapter return helper def candidate_keys(self, call_args): """Find candidate registry keys for given call arguments""" # Default behaviour dispatches on the type of the first argument return type(call_args[0]).__mro__ def default_adapter(self, *args): """Call result when no adapter was found""" raise TypeError("Can't adapt %s to %s" % (args[0].__class__.__name__, self.name)) def __call__(self, *args): """Adapt supplied arguments to this protocol""" for key in self.candidate_keys(args): try: adapter = self.registry[key] except KeyError: pass else: return adapter(*args) return self.default_adapter(*args) # The adapting iteration example class AdaptingIterProtocol(Protocol): def __init__(self): Protocol.__init__(self, "AdaptingIter") def default_adapter(self, obj): if hasattr(obj, "__iter__"): return obj.__iter__() raise TypeError("Can't iterate over a %s object" % obj.__class__.__name__) AdaptingIter = AdaptingIterProtocol() AdaptingIter.register(SequenceIter, list, str, unicode) @AdaptingIter.register_for(dict) def _AdaptingDictIter(obj): return SequenceIter(obj.keys()) # Building a generic function on top of that Protocol class GenericFunction(Protocol): def __init__(self, default): Protocol.__init__(self, default.__name__) self.__doc__ = default.__doc__ self.default_adapter = default def candidate_keys(self, call_args): """Find candidate registry keys for given call arguments""" arg_types = tuple(type(x) for x in call_args) if len(call_args) == 1: yield arg_types[0] # Allow bare type for single args yield arg_types # Always try full argument tuple # The generic iteration example @GenericFunction def GenericIter(obj): """This is the docstring for the generic function.""" # The body is the default implementation if hasattr(obj, "__iter__"): return obj.__iter__() raise TypeError("Can't iterate over %s object" % obj.__class__.__name__) @GenericIter.register(list) def _GenericSequenceIter(obj): return SequenceIter(obj) GenericIter.register(str)(_GenericSequenceIter) GenericIter.register(unicode)(_GenericSequenceIter) @GenericIter.register(dict) def _GenericDictIter(obj): return SequenceIter(obj.keys()) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From walter at livinglogic.de Wed Apr 5 13:52:47 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 05 Apr 2006 13:52:47 +0200 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <44338F47.3050905@gmail.com> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <44338F47.3050905@gmail.com> Message-ID: <4433AF8F.50602@livinglogic.de> Nick Coghlan wrote: > [...] > So your example would become: > > class MetaInterface(type): > def __new__(mcl, name, bases, dict): > # Give each class it's own registry > cls = type.__new__(mcl, name, bases, dict) > cls.registry = {} > return cls > > def register(cls, adapter, T): > cls.registry[T] = adapter > > def register_for(cls, T): > # Registration decorator > def register_func(adapter): > cls.register(adapter, T) > return register_func > > def __call__(cls, obj): > # Cannot construct interface instances > # Instead, calling the class triggers adaptation > for base in type(obj).__mro__: > try: > adapter = cls.registry[base] > except KeyError: > pass > if adapter is None: > return obj > return adapter(obj) > raise TypeError("can't adapt %r to %r" % (obj, cls)) > > class Interface(object): > __metaclass__ = MetaInterface > > class Sequence(Interface): > def getitem(self, index): pass > def len(self): pass > > > class PythonSeqAsSequence(object): # Adapter is a normal class! > def __init__(self, obj): > self.obj = obj > > def getitem(self, index): > return self.obj[i] > > def len(self): > return len(self.obj) > > Sequence.register(PythonSeqAsSequence, list) > > print Sequence([1,2,3]).len() > > # Or using the function decorator version > @Sequence.register_for(tuple) > def tuple_as_seq(obj): > class AdaptedTuple(object): > def getitem(self, index): > return obj[i] > def len(self): > return len(obj) > return AdaptedTuple() Of course it would be better, if we had class decorators: @Sequence.register_for(tuple) class AdaptedTuple(object): def __init__(self, obj): self.obj = obj def getitem(self, index): return self.obj[i] def len(self): return len(self.obj) But this doesn't look too bad. Not being able to instantiate an interface actually makes sense. What looks debatable to me is that it adapter doesn't subclass the interface (it would then inherit some default implementations of methods). Servus, Walter From tim.hochberg at ieee.org Wed Apr 5 15:55:26 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 05 Apr 2006 06:55:26 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <4433AD62.80406@gmail.com> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> Message-ID: <4433CC4E.7090901@ieee.org> Nick Coghlan wrote: > Tim Hochberg wrote: > >>So after all of that, I think my conclusion is that I wouldn't refactor >>this at all, at least not yet. I'd add support for multiple registration >>and possibly spell adapt as __call__, otherwise I'd leave it alone. My >>opinion may change after I try ripping out keysof and see how it looks. > > > I was curious to see how the adaptation version actually looked with your and > Guido's versions mixed. While writing it, I also noticed two interesting cases > worth simplifying: > 1. the "no adapter needed case" (for registering that a type implements a > protocol directly) > 2. the "missing adapter case" (for providing a default adaptation, as in > the generic function case) > > Here's what the whole thing ended up looking like: > > def null_adapter(*args): > """Adapter used when adaptation isn't actually needed""" > if len(args) > 1: > return args > else: > return args[0] > > class Protocol(object): > """Declare a named protocol""" > def __init__(self, name): > self.registry = {} > self.name = name > > def register(self, adapter, *keys): > """Register an adapter from given registry keys to the protocol""" > if adapter is None: > adapter = null_adapter > for key in keys: > self.registry[key] = adapter > > def register_for(self, *keys): > """Function decorator to register as an adapter for given keys""" > def helper(adapter): > self.register(adapter, *keys) > return adapter > return helper > > def candidate_keys(self, call_args): > """Find candidate registry keys for given call arguments""" > # Default behaviour dispatches on the type of the first argument > return type(call_args[0]).__mro__ > > def default_adapter(self, *args): > """Call result when no adapter was found""" > raise TypeError("Can't adapt %s to %s" % > (args[0].__class__.__name__, self.name)) > > def __call__(self, *args): > """Adapt supplied arguments to this protocol""" > for key in self.candidate_keys(args): > try: > adapter = self.registry[key] > except KeyError: > pass > else: > return adapter(*args) > return self.default_adapter(*args) I like this version. The naming seems an improvement and the default_adapter seems like a worthy addition. I do keep wondering if there's some reason for a user of Protocol to call candidate_keys directly or it's only an implementation detail. If the there is such a reason, and we could figure it out, it would probably immediately obvious what to call it. If there isn't, perhaps it should be prefixed with '_' to indicate that it's not part of the public interface. > # The adapting iteration example > class AdaptingIterProtocol(Protocol): > def __init__(self): > Protocol.__init__(self, "AdaptingIter") > > def default_adapter(self, obj): > if hasattr(obj, "__iter__"): > return obj.__iter__() > raise TypeError("Can't iterate over a %s object" % > obj.__class__.__name__) > > AdaptingIter = AdaptingIterProtocol() > > AdaptingIter.register(SequenceIter, list, str, unicode) > > @AdaptingIter.register_for(dict) > def _AdaptingDictIter(obj): > return SequenceIter(obj.keys()) > > > # Building a generic function on top of that Protocol > class GenericFunction(Protocol): > def __init__(self, default): > Protocol.__init__(self, default.__name__) > self.__doc__ = default.__doc__ > self.default_adapter = default > > def candidate_keys(self, call_args): > """Find candidate registry keys for given call arguments""" > arg_types = tuple(type(x) for x in call_args) > if len(call_args) == 1: > yield arg_types[0] # Allow bare type for single args > yield arg_types # Always try full argument tuple > > # The generic iteration example > @GenericFunction > def GenericIter(obj): > """This is the docstring for the generic function.""" > # The body is the default implementation > if hasattr(obj, "__iter__"): > return obj.__iter__() > raise TypeError("Can't iterate over %s object" % obj.__class__.__name__) > > @GenericIter.register(list) > def _GenericSequenceIter(obj): > return SequenceIter(obj) > > GenericIter.register(str)(_GenericSequenceIter) > GenericIter.register(unicode)(_GenericSequenceIter) > > @GenericIter.register(dict) > def _GenericDictIter(obj): > return SequenceIter(obj.keys()) > These should all be "GenericIter.register_for", right? Regards, -tim From ncoghlan at gmail.com Wed Apr 5 16:16:34 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 06 Apr 2006 00:16:34 +1000 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <4433CC4E.7090901@ieee.org> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> Message-ID: <4433D142.7050203@gmail.com> Tim Hochberg wrote: > I like this version. The naming seems an improvement and the > default_adapter seems like a worthy addition. I do keep wondering if > there's some reason for a user of Protocol to call candidate_keys > directly or it's only an implementation detail. If the there is such a > reason, and we could figure it out, it would probably immediately > obvious what to call it. If there isn't, perhaps it should be prefixed > with '_' to indicate that it's not part of the public interface. It was a toss-up, but even if it was private, it would still need to be documented so subclasses knew how to override it. And if it's public, then it might be a useful introspection tool on protocols. For example: def can_adapt(protocol, *args): for key in protocol.candidate_keys(args): if key in protocol.registry: return True return False Although, now that I think about it, mere presence in the registry doesn't necessarily signify much - it's trivial to register an "anti-adapter" that raises TypeError. So having this method be public may make it an attractive nuisance, rather than something useful. . . >> # The generic iteration example >> @GenericFunction >> def GenericIter(obj): >> """This is the docstring for the generic function.""" >> # The body is the default implementation >> if hasattr(obj, "__iter__"): >> return obj.__iter__() >> raise TypeError("Can't iterate over %s object" % >> obj.__class__.__name__) >> >> @GenericIter.register(list) >> def _GenericSequenceIter(obj): >> return SequenceIter(obj) >> >> GenericIter.register(str)(_GenericSequenceIter) >> GenericIter.register(unicode)(_GenericSequenceIter) >> >> @GenericIter.register(dict) >> def _GenericDictIter(obj): >> return SequenceIter(obj.keys()) >> > > These should all be "GenericIter.register_for", right? I don't what happened there - I must have hit Ctrl-C instead of Ctrl-V or something. . . That part should have looked like: @GenericFunction def GenericIter(obj): """This is the docstring for the generic function.""" # The body is the default implementation if hasattr(obj, "__iter__"): return obj.__iter__() raise TypeError("Can't iterate over %s object" % obj.__class__.__name__) GenericIter.register(SequenceIter, list, str, unicode) @GenericIter.register_for(dict) def _GenericDictIter(obj): return SequenceIter(obj.keys()) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From walter at livinglogic.de Wed Apr 5 16:30:52 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 05 Apr 2006 16:30:52 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <4433AD62.80406@gmail.com> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> Message-ID: <4433D49C.70900@livinglogic.de> Nick Coghlan wrote: > [...] > def default_adapter(self, *args): > """Call result when no adapter was found""" > raise TypeError("Can't adapt %s to %s" % > (args[0].__class__.__name__, self.name)) +1 This makes it easy to add fallbacks: class XReprProtocol(Protocol): def default_adapter(self, *args): return repr(*args) xrepr = XReprProtocol("xrepr") @XReprProtocol.register_for(list) def xrepr_list(obj): return "[%s]" % ", ".join(xrepr(x) for x in obj) @XReprProtocol.register_for(tuple) def xrepr_tuple(obj): return "(%s)" % ", ".join(xrepr(x) for x in obj) @XReprProtocol.register_for(int, long) def xrepr_number(obj): return hex(obj) For all other objects it falls back to the normal repr(). > def __call__(self, *args): > """Adapt supplied arguments to this protocol""" > for key in self.candidate_keys(args): > try: > adapter = self.registry[key] > except KeyError: > pass > else: > return adapter(*args) > return self.default_adapter(*args) It would be great if __call__() would pass on keywords arguments to the adapter too. (This would probably mean that candidate_keys would need *args, **kwargs arguments). But then calling the adapter is totally generic. Servus, Walter From tim.hochberg at ieee.org Wed Apr 5 19:20:30 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 05 Apr 2006 10:20:30 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <4433D142.7050203@gmail.com> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> Message-ID: <4433FC5E.6070103@ieee.org> Nick Coghlan wrote: > Tim Hochberg wrote: > >>I like this version. The naming seems an improvement and the >>default_adapter seems like a worthy addition. I do keep wondering if >>there's some reason for a user of Protocol to call candidate_keys >>directly or it's only an implementation detail. If the there is such a >>reason, and we could figure it out, it would probably immediately >>obvious what to call it. If there isn't, perhaps it should be prefixed >>with '_' to indicate that it's not part of the public interface. > > > It was a toss-up, but even if it was private, it would still need to be > documented so subclasses knew how to override it. And if it's public, then it > might be a useful introspection tool on protocols. For example: > > def can_adapt(protocol, *args): > for key in protocol.candidate_keys(args): > if key in protocol.registry: > return True > return False > > Although, now that I think about it, mere presence in the registry doesn't > necessarily signify much - it's trivial to register an "anti-adapter" that > raises TypeError. So having this method be public may make it an attractive > nuisance, rather than something useful. . . Perhaps one could avoid this by providing a predefined AntiAdapter class and recomending that instances of that class (and possibly its sublasses) be used to disable adaption. Then something like can_adapt could return False rather than True if the first adapter it came across is an instance of AntiAdapter. (I think a better name than AntiAdapter might be in order though). Let me float another name for candidate_keys: 'signatures'. Let me try to explain protocols in terms of signatures and see how it goes. A protocol adapts an object by examining each of the object's signatures in turn and checking whether an adapter has been registered from that signature. If so, the result of found_adapter(the_object) is returned. If no adapters are found for any of the signatures, the protocol returns self.default_adapter(the_object). An object's signatures can vary by protocol, but are type(obj).__mro__ in the default case. [I've neglected the multiargument case for now to avoid twisting myself into knots writing "object or objects" all over the place] That doesn't seem too bad. Certainly better than talking about an object's keys, which really doesn't make any sense. However, it's possible that this use of signature will be confusing because it conflicts with the terminology "function signature". The resulting change in __call__ results in a slightly clearer code IMO: def __call__(self, *args): """Adapt supplied arguments to this protocol""" for key in self.signatures(args): try: adapter = self.registry[key] except KeyError: pass else: return adapter(*args) return self.default_adapter(*args) Regards, -tim From jjl at pobox.com Wed Apr 5 21:12:31 2006 From: jjl at pobox.com (John J Lee) Date: Wed, 5 Apr 2006 20:12:31 +0100 (GMT Standard Time) Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44330694.3060900@colorstudy.com> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> <44330694.3060900@colorstudy.com> Message-ID: [Greg Ewing] > I'm more interested in long-term readability > and maintainability than in pandering to one-off > impressions when coming from another language. [Ian Bicking] > $-based substitution is very common. Like most languages, Python uses > () for calling functions, it uses keywords like "while" and "if", etc. > It's okay to use the normal form; tweaking some detail off the norm > because it seems slightly more compact or concise seems questionable to > me. Using {} instead of $/${} doesn't seem like a big win. Well, if you're interested in precedents, I guess it depends on the audience you're most concerned with: MS Word, when you use its Mail Merge feature, uses European-style quotes (which look a bit like a scaled-down version of <>) -- at least, that's what it does on my PC. I guess the *potential* audience for whom that's the most familiar interpolation syntax is much larger than the number of people coming from other programming languages. Presumably if, like Guido, you have an interest in making programming languages accessible to a wider audience, this might by your own (Ian's) argument favour Greg's proposed syntax. FWIW, ReportLab's "preppy" templating system uses {{expression}}, which ISTR was inspired by MS's choice (and if it wasn't, it should have been ;-). John From jjl at pobox.com Wed Apr 5 21:13:08 2006 From: jjl at pobox.com (John J Lee) Date: Wed, 5 Apr 2006 20:13:08 +0100 (GMT Standard Time) Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443304D7.7020307@canterbury.ac.nz> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> Message-ID: On Wed, 5 Apr 2006, Greg Ewing wrote: [...] > There are other problems with the $ syntax too. > The lack of an explicit closing marker causes > problems when the parameter is immediately > followed in the string by an alphanumeric > character. You end up having to allow some > kind of bracketing as an optional extra > anyway, such as > > subst("I'm a ${what}spotter", what = "train") > > Seems to me it's simpler as well as clearer > to just make the syntax fully bracketed in the > first place. One argument against the plain {} syntax would be that, if you believe there should be one syntax that handles both simple substitution and formatting-with-format-characters nicely, you need somewhere to put the format characters. Putting them at the end is problematic, as we know. But how about putting them at the start, after the $? That way, the format character can't get mixed up with the text following it. For example, $r{foo} would mean the same as the current %(foo)r. John From jjl at pobox.com Wed Apr 5 21:15:57 2006 From: jjl at pobox.com (John J Lee) Date: Wed, 5 Apr 2006 20:15:57 +0100 (GMT Standard Time) Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <5F505240-06DC-4AC2-9330-6AC85C9931D3@python.org> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> <44330694.3060900@colorstudy.com> <44331180.9000203@canterbury.ac.nz> <5F505240-06DC-4AC2-9330-6AC85C9931D3@python.org> Message-ID: On Tue, 4 Apr 2006, Barry Warsaw wrote: [...] > Yes, I think this is where we differ. $thing in it's most common > form is more readable to me than {thing}, and while the former is > equivalent to ${thing} with string.Template's defaults, I find it's > very rarely used. Regardless of readability, I think that very rarity, and the presence of two rules in place of Greg's single rule, is likely a problem for new users. The ${} case becomes an odd little corner case you forget about, then bump into later. If you're new to all this, you may not even have understood the need for it in the first place. It's a relatively small problem, but certainly not a non-existant one. Exactly that sort of problem, piled high, is what makes Perl so hard to learn compared with Python. Piled less high, it's still a real barrier. > So I wouldn't be in favor of changing it. Of > course, beauty is in the eye of the beholder and all that... Clarity and difficulty of learning, OTOH, may not be. Those are questions that could perfectly well be objectively tested (though I doubt anybody has ever done so, or is likely to). John From talin at acm.org Wed Apr 5 21:17:37 2006 From: talin at acm.org (Talin) Date: Wed, 5 Apr 2006 19:17:37 +0000 (UTC) Subject: [Python-3000] String formating operations in python 3k References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > I'm also not all that keen on $, either inside > or outside the string. It seems to me that > something like > > "User {user} has printed {n} pages" FWIW, this is very similar to the syntax used in the .Net function String.Format (I think Java uses a similar syntax as well). I don't think it supports named args, though. In fact, a lot of the proposals in the followup postings are eerily similar to the .Net syntax for formatting options.) So for those who are concerned with the ease of migrating from other languages, well, it really depends on which set of users you are interested in :) Here's a quick summary: "{0}" -- positional arg "{{some text}}" -- escaped braces "{0:x8}" -- format as 8-digit hexadecimal String.Format(?{0:(###) ###-####}?, 8005551212) ==> "(800) 555-1212" Here's a link to a reference page for the various formatting options: http://blog.stevex.net/index.php/string-formatting-in-csharp/ In general, I would be very much in favor of a convention that allows mixed positional and named arguments. As much as I like the % operator, I find that the "all or nothing" leads to some awkwardness in practice. -- Talin From tim.hochberg at ieee.org Wed Apr 5 21:26:46 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 05 Apr 2006 12:26:46 -0700 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44333B95.4010906@colorstudy.com> References: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> Message-ID: <443419F6.70103@ieee.org> Ian Bicking wrote: > Alex Martelli wrote: > >>As for the % operator, I never liked it -- either a builtin function, >>or even better a method of string objects, is going to be much more >>readable (and my preference would be to have it take optional >>positional arguments, corresponding to {1}, {2}, etc, and optional >>named arguments, corresponding to {name} &c). > > > Note that if it takes keyword arguments, but doesn't take a single > dictionary-like object (like % and string.Template.substitute do), then > you lose any ability to use clever or interesting dictionary-like > objects for substitution. > Why doesn't the ** take care of this? To take your example, why doesn't this work? string.Template(pattern).substitute(**EscapingWrapper(**EvalingWrapper(**locals())) Is it that you feel that adding ** is too much of a syntactic burden, or am I just missing something? Regards, -tim From ianb at colorstudy.com Wed Apr 5 21:34:28 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 05 Apr 2006 14:34:28 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443419F6.70103@ieee.org> References: <06Apr4.201259pdt."58633"@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> <443419F6.70103@ieee.org> Message-ID: <44341BC4.6090601@colorstudy.com> Tim Hochberg wrote: > Ian Bicking wrote: >> Alex Martelli wrote: >>> As for the % operator, I never liked it -- either a builtin >>> function, or even better a method of string objects, is going to be >>> much more readable (and my preference would be to have it take >>> optional positional arguments, corresponding to {1}, {2}, etc, and >>> optional named arguments, corresponding to {name} &c). >> >> >> >> Note that if it takes keyword arguments, but doesn't take a single >> dictionary-like object (like % and string.Template.substitute do), >> then you lose any ability to use clever or interesting dictionary-like >> objects for substitution. >> > > Why doesn't the ** take care of this? To take your example, why doesn't > this work? > > string.Template(pattern).substitute(**EscapingWrapper(**EvalingWrapper(**locals())) > > > Is it that you feel that adding ** is too much of a syntactic burden, or > am I just missing something? func(**kw) actually passes a dictionary to func. EvalingWrapper isn't a dictionary, it's just an object with a __getitem__. ** can't be changed in this way, either -- it really has to enumerate all the keys in the thing passed to it, and there's no way to enumerate all the keys in a wrapper like EvalingWrapper. Any string substitution only needs __getitem__, so these constraints when calling a function with ** do not apply to the actual string substition. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From guido at python.org Wed Apr 5 21:39:17 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Apr 2006 12:39:17 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <4433FC5E.6070103@ieee.org> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> Message-ID: Fascinating ideas in this thread! I spent some time blogging about this on artima: http://www.artima.com/weblogs/viewpost.jsp?thread=155123 I have to write my slides for a talk about Py3K later today, but I'll be back. In the mean time I've rejected PEPs 245 and 246 in expectation of something better that's imminent! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From walter at livinglogic.de Wed Apr 5 22:16:32 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 05 Apr 2006 22:16:32 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> Message-ID: <443425A0.3050101@livinglogic.de> Guido van Rossum wrote: > Fascinating ideas in this thread! > > I spent some time blogging about this on artima: > http://www.artima.com/weblogs/viewpost.jsp?thread=155123 > > I have to write my slides for a talk about Py3K later today, but I'll > be back. In the mean time I've rejected PEPs 245 and 246 in > expectation of something better that's imminent! What's still missing IMHO is a way for an adapter to defer to the next adapter in the chain, i.e. something like: class Protocol(object): def register(self, ...): ... def __call__(self, ...): ... def default(self, *args, **kwargs): raise TypeError def candidates(self, *args, **kwargs): return type(args[0]).__mro__ def super(self, *args, **kwargs): search = iter(self.candidates(*args, **kwargs)) for key in search: if key in self.registry: break try: return search.next()(*args, **kwargs) except StopIteration: return self.default(*args, **kwargs) With this an adapter could use the super() method to call the next adapter in the chain: p = Protocol() @p.register(Foo) def p_foo(foo): p.super(foo) Servus, Walter From ianb at colorstudy.com Wed Apr 5 22:23:11 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 05 Apr 2006 15:23:11 -0500 Subject: [Python-3000] Interfaces (was: Re: Adaptation vs. Generic Functions) In-Reply-To: References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> Message-ID: <4434272F.2050201@colorstudy.com> Guido van Rossum wrote: > Fascinating ideas in this thread! It's certainly been interesting to see convergence coming from both directions independently. > I spent some time blogging about this on artima: > http://www.artima.com/weblogs/viewpost.jsp?thread=155123 > > I have to write my slides for a talk about Py3K later today, but I'll > be back. In the mean time I've rejected PEPs 245 and 246 in > expectation of something better that's imminent! Is PEP 245 ("Python Interface Syntax") that related to adaptation or generic functions? Certainly they've gone hand-in-hand for some time, but adaptation as an idea isn't really going away (just rephrased), and the motivation for the formalism of an interface isn't going away either. So I don't think this invalidates the interface PEP. Maybe a more minimal PEP is called for. Or one revised with consideration of Zope interfaces; though that implementation mixes interfaces and adaptation in a way that the original PEP does not. Potentially the whole thing could just be a module in the standard library; the only real language support called for at all is an "interface IFoo:" special form. The PEP also gives a form "class Foo implements IBar:", but that doesn't seem needed or in line with the way decorators work; class decorators would resolve this issue (+1 on "@class decorator" in the class body). For documentation alone I would find interfaces very useful, and I write ad hoc (non-Zope) interfaces from time to time for that reason. But there's no tool support then. OTOH, I'd be non-plused if interfaces and interface-related assertions popped up all over the place; I really only like them when using objects between library boundaries. Anyway, it seems like interfaces deserve their own discussion. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From guido at python.org Wed Apr 5 22:37:33 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Apr 2006 13:37:33 -0700 Subject: [Python-3000] Interfaces (was: Re: Adaptation vs. Generic Functions) In-Reply-To: <4434272F.2050201@colorstudy.com> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <4434272F.2050201@colorstudy.com> Message-ID: On 4/5/06, Ian Bicking wrote: > Guido van Rossum wrote: > > I have to write my slides for a talk about Py3K later today, but I'll > > be back. In the mean time I've rejected PEPs 245 and 246 in > > expectation of something better that's imminent! > > Is PEP 245 ("Python Interface Syntax") that related to adaptation or > generic functions? Certainly they've gone hand-in-hand for some time, > but adaptation as an idea isn't really going away (just rephrased), and > the motivation for the formalism of an interface isn't going away > either. So I don't think this invalidates the interface PEP. Correct; it's more that that PEP is also 5+ years old without getting traction. I think we need a new PEP for interfaces, less focused on syntax and more on use cases and requirements. Rejection of the PEP doesn't render it inaccessible; I'm just acknowledging that PEP 245 isn't ever going to be implemented in that form. (And yes, there are other PEPs of which the same could be said; I'll weed those out later.) If you feel like writing a new Interfaces PEP draft, please go ahead! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 5 22:37:33 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Apr 2006 13:37:33 -0700 Subject: [Python-3000] Interfaces (was: Re: Adaptation vs. Generic Functions) In-Reply-To: <4434272F.2050201@colorstudy.com> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <4434272F.2050201@colorstudy.com> Message-ID: On 4/5/06, Ian Bicking wrote: > Guido van Rossum wrote: > > I have to write my slides for a talk about Py3K later today, but I'll > > be back. In the mean time I've rejected PEPs 245 and 246 in > > expectation of something better that's imminent! > > Is PEP 245 ("Python Interface Syntax") that related to adaptation or > generic functions? Certainly they've gone hand-in-hand for some time, > but adaptation as an idea isn't really going away (just rephrased), and > the motivation for the formalism of an interface isn't going away > either. So I don't think this invalidates the interface PEP. Correct; it's more that that PEP is also 5+ years old without getting traction. I think we need a new PEP for interfaces, less focused on syntax and more on use cases and requirements. Rejection of the PEP doesn't render it inaccessible; I'm just acknowledging that PEP 245 isn't ever going to be implemented in that form. (And yes, there are other PEPs of which the same could be said; I'll weed those out later.) If you feel like writing a new Interfaces PEP draft, please go ahead! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.hochberg at cox.net Wed Apr 5 22:44:22 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed, 05 Apr 2006 13:44:22 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <443425A0.3050101@livinglogic.de> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> Message-ID: <44342C26.70009@cox.net> Walter D?rwald wrote: > Guido van Rossum wrote: > >> Fascinating ideas in this thread! >> >> I spent some time blogging about this on artima: >> http://www.artima.com/weblogs/viewpost.jsp?thread=155123 >> >> I have to write my slides for a talk about Py3K later today, but I'll >> be back. In the mean time I've rejected PEPs 245 and 246 in >> expectation of something better that's imminent! > > > What's still missing IMHO is a way for an adapter to defer to the next > adapter in the chain, i.e. something like: > > class Protocol(object): > def register(self, ...): > ... > > def __call__(self, ...): > ... > > def default(self, *args, **kwargs): > raise TypeError > > def candidates(self, *args, **kwargs): > return type(args[0]).__mro__ > > def super(self, *args, **kwargs): > search = iter(self.candidates(*args, **kwargs)) > for key in search: > if key in self.registry: > break > try: > return search.next()(*args, **kwargs) > except StopIteration: > return self.default(*args, **kwargs) > > > With this an adapter could use the super() method to call the next > adapter in the chain: > > p = Protocol() > > @p.register(Foo) > def p_foo(foo): > p.super(foo) The concept seems good, but I find the implementation baffling, OK, I finally figured it out. It only seems to work once though -- you can't chain super calls. Why not simply raise a DeferToNextAdapter exception? That seems much simpler and it chains correctly. class DeferToNextAdapter(Exception): pass class Protocol(object): #.... def __call__(self, *args): """Adapt supplied arguments to this protocol""" for key in self.signatures(args): try: try: adapter = self.registry[key] except pass else: return adapter(*args) except DeferToNextAdapter: pass return self.default_adapter(*args) p = Protocol() @p.register(Foo) def p_foo(foo): raise DeferToNextAdapter ========================================= FWIW, I'm also not very enthusiastic about the inner try/except block that showed up at some point. The normal case is to fail here, so it seem like this results in unnecessary overhead. That's a minor issue though. Regards, -tim From guido at python.org Wed Apr 5 22:47:56 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Apr 2006 13:47:56 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44342C26.70009@cox.net> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> Message-ID: On 4/5/06, Tim Hochberg wrote: > Walter D?rwald wrote: > > What's still missing IMHO is a way for an adapter to defer to the next > > adapter in the chain, i.e. something like: [...] > The concept seems good, but I find the implementation baffling, OK, I > finally figured it out. It only seems to work once though -- you can't > chain super calls. Why not simply raise a DeferToNextAdapter exception? I haven't used adapters in years, and I don't recall seeing this in Alex's recent posts; can either of you explain slowly what the use case is and how it's typically used? The code fragments shown so far aren't helping. :-( -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Wed Apr 5 23:07:31 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 05 Apr 2006 16:07:31 -0500 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <443425A0.3050101@livinglogic.de> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> Message-ID: <44343193.4030201@colorstudy.com> Walter D?rwald wrote: > Guido van Rossum wrote: > > >>Fascinating ideas in this thread! >> >>I spent some time blogging about this on artima: >>http://www.artima.com/weblogs/viewpost.jsp?thread=155123 >> >>I have to write my slides for a talk about Py3K later today, but I'll >>be back. In the mean time I've rejected PEPs 245 and 246 in >>expectation of something better that's imminent! > > > What's still missing IMHO is a way for an adapter to defer to the next > adapter in the chain, i.e. something like: Sounds like next_method in generic functions, which is an area of generic functions we haven't touched upon (but which seem important). I think in RuleDispatch it goes like: @foo.when(...) def incremented_foo(next_method, arg1, arg2): return next_method(arg1, arg2) + 1 By using a first argument named "next_method" you tell the generic function machinery to pass in this special object, which acts like the original generic function ("foo" in this case) but ignores the current method. I assume there won't be very positive reaction here to behavior based on an argument name. But ignoring that aspect, it's a somewhat important feature of typical generic function implementations, and very similar to super. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From tim.hochberg at cox.net Wed Apr 5 23:39:35 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed, 05 Apr 2006 14:39:35 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> Message-ID: <44343917.4060503@cox.net> Guido van Rossum wrote: >On 4/5/06, Tim Hochberg wrote: > > >>Walter D?rwald wrote: >> >> >>>What's still missing IMHO is a way for an adapter to defer to the next >>>adapter in the chain, i.e. something like: >>> >>> >[...] > > >>The concept seems good, but I find the implementation baffling, OK, I >>finally figured it out. It only seems to work once though -- you can't >>chain super calls. Why not simply raise a DeferToNextAdapter exception? >> >> > >I haven't used adapters in years, and I don't recall seeing this in >Alex's recent posts; can either of you explain slowly what the use >case is and how it's typically used? The code fragments shown so far >aren't helping. :-( > > I'm hoping that Walter can give some more realistic examples since I don't have real-world experience here. The basic idea is simply to let an adapter give up and let the protocol try the next adapter. This could happen in a generic function, for instance, if you wanted to try some fast algorithm for some specific subtypes, but the algorithm might be inappropriate depending on the values of the subtypes. You don't find that out till your in the adapter itself. Consider: @GenericFunction def power(a, b): """return a**b""" return a**b @power.register_for((int, int), (float, int), (complex, int)) def intpower(a, b): "Compute a**b quickly for some integral cases. Punt otherwise." if b == 0: return 1 elif b == 1: return a elif b == 2: return a*a else: raise DeferToNextAdapter This uses Nick Coghlan's GenericFunction implementation from a few posts ago with the DeferToNextAdapter stuff added. The example is contrived since I would really just call a**b directly in the else clause. However, I can see how in a case with more complicated / powerful type dispatching you might not know in advance what the next adapter was going to do, as you do here, so you'd want to hand responsibility back off to the GenericFunction instance. I'll give this some more thought and see if I can come up with non-contrived examples. Regards, -tim From guido at python.org Wed Apr 5 23:53:29 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Apr 2006 14:53:29 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44343917.4060503@cox.net> References: <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> Message-ID: On 4/5/06, Tim Hochberg wrote: > I'm hoping that Walter can give some more realistic examples since I > don't have real-world experience here. The basic idea is simply to let > an adapter give up and let the protocol try the next adapter. This could > happen in a generic function, for instance, if you wanted to try some > fast algorithm for some specific subtypes, but the algorithm might be > inappropriate depending on the values of the subtypes. You don't find > that out till your in the adapter itself. Hm... The alternative, of course, instead of some magic feature, is to explicitly call the fallback algorithm. That seems totally reasonable for this example, and is similar to Python's single-inheritance-style explicit super-call e.g. return BaseClass.methodName(self, arguments...). Does anybody know if Phillip Eby's generic functions support this? I would think that the definition of "next candidate" can be problematic there, in case you've broken a tie between two ambiguous possibilities by registering an explicit tie-breaker. Remove the tie-breaker and neither alternative dominates the other, so the next candidate is undefined. This is different from Python's MRO calculations, which always create a perfect linearization of the base classes. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Thu Apr 6 00:00:49 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 5 Apr 2006 18:00:49 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44342C26.70009@cox.net> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> Message-ID: > Walter D?rwald wrote: > > What's still missing IMHO is a way for an adapter > > to defer to the next adapter in the chain Tim Hochberg wrote: > Why not simply raise a DeferToNextAdapter exception? It sounds good at first, but it may be overengineered. If that means "NotImplemented", then why bother to register the adapter? If you're just speeding lookup, then go ahead and register the "Next" adapter explicitly. So I'm assuming that you really meant to support cooperative adapters. If the adapters themselves inherit properly, that isn't a problem; just use super. If you really mean "whichever adapter is next, even one I never heard about", then there is a problem with diamond inheritance. As Guido's blog points out, a decision between (Exact, Base) and (Base, Exact) isn't clearcut, but can be resolved by requiring an explicit registration for disambiguation. nextmethod then has exactly the same problem... Even calling all potential next methods expcitly isn't really an answer, if some of them end up redelegating to (Base, Base). Or are you suggesting that each adapter function should itself be treated as a class, with a lookup actually returning a newly created derived class that includes each relevant adapter in its MRO? A part of me likes that idea, but I can't help wondering if it wouldn't make the adaptation protocols too heavyweight. Tim Hochberg wrote: > FWIW, I'm also not very enthusiastic about the > inner try/except block that showed up at some point. Are you worried about the implementation or the explanation? Do you think it would read better as for key in candidates: adapter = self.registry.get(key) if adapter is not None: return adapter(*args, **kwargs) From walter at livinglogic.de Thu Apr 6 00:01:08 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 06 Apr 2006 00:01:08 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> Message-ID: <44343E24.9010906@livinglogic.de> Guido van Rossum wrote: > On 4/5/06, Tim Hochberg wrote: >> I'm hoping that Walter can give some more realistic examples since I >> don't have real-world experience here. The basic idea is simply to let >> an adapter give up and let the protocol try the next adapter. This could >> happen in a generic function, for instance, if you wanted to try some >> fast algorithm for some specific subtypes, but the algorithm might be >> inappropriate depending on the values of the subtypes. You don't find >> that out till your in the adapter itself. > > Hm... The alternative, of course, instead of some magic feature, is to > explicitly call the fallback algorithm. That seems totally reasonable > for this example, and is similar to Python's single-inheritance-style > explicit super-call e.g. return BaseClass.methodName(self, > arguments...). For methods you know what your bases classes are, but for adapters that's not necessarily so. Somewhat could have registered another adapter for the base class. > Does anybody know if Phillip Eby's generic functions support this? If I understood Ian's posting (http://mail.python.org/pipermail/python-3000/2006-April/000481.html) correctly, they do. > I > would think that the definition of "next candidate" can be problematic > there, in case you've broken a tie between two ambiguous possibilities > by registering an explicit tie-breaker. Remove the tie-breaker and > neither alternative dominates the other, so the next candidate is > undefined. This is different from Python's MRO calculations, which > always create a perfect linearization of the base classes. BTW, another useful feature might be the ability to copy the protocol. Then I could register a few other adapters for the copy and use the modified copy for my own purpose. Servus, Walter From walter at livinglogic.de Thu Apr 6 00:06:19 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 06 Apr 2006 00:06:19 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44343917.4060503@cox.net> References: <44336D0C.1010409@ieee.org> <4433AD62.80406@gmail.com> <4433CC4E.7090901@ieee.org> <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> Message-ID: <44343F5B.5040908@livinglogic.de> Tim Hochberg wrote: > Guido van Rossum wrote: > >> On 4/5/06, Tim Hochberg wrote: >> >> >>> Walter D?rwald wrote: >>> >>> >>>> What's still missing IMHO is a way for an adapter to defer to the next >>>> adapter in the chain, i.e. something like: >>>> >>>> >> [...] >> >> >>> The concept seems good, but I find the implementation baffling, OK, I >>> finally figured it out. It only seems to work once though -- you can't >>> chain super calls. Why not simply raise a DeferToNextAdapter exception? >>> >>> >> I haven't used adapters in years, and I don't recall seeing this in >> Alex's recent posts; can either of you explain slowly what the use >> case is and how it's typically used? The code fragments shown so far >> aren't helping. :-( >> >> > I'm hoping that Walter can give some more realistic examples since I > don't have real-world experience here. The basic idea is simply to let > an adapter give up and let the protocol try the next adapter. This could > happen in a generic function, for instance, if you wanted to try some > fast algorithm for some specific subtypes, but the algorithm might be > inappropriate depending on the values of the subtypes. You don't find > that out till your in the adapter itself. What I had in mind was something like this for an extensible repr(): xrepr = Protocol("xrepr") @xrepr.register(list) def xrepr_list(obj): return "[%s]" % ", ".join(xrepr(x) for x in obj) class MyList(list): ... @xrepr.register(MyList) def xrepr_MyList(obj): return "MyList(%s)" % xrepr.super(obj) Servus, Walter > This uses Nick Coghlan's GenericFunction implementation from a few posts > ago with the DeferToNextAdapter stuff added. But raising a exception ends the execute of the adapter there. I can't use the result of the super adapter. Servus, Walter From guido at python.org Thu Apr 6 00:07:04 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Apr 2006 15:07:04 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44343E24.9010906@livinglogic.de> References: <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> Message-ID: On 4/5/06, Walter D?rwald wrote: > Guido van Rossum wrote: > > > On 4/5/06, Tim Hochberg wrote: > >> I'm hoping that Walter can give some more realistic examples since I > >> don't have real-world experience here. The basic idea is simply to let > >> an adapter give up and let the protocol try the next adapter. This could > >> happen in a generic function, for instance, if you wanted to try some > >> fast algorithm for some specific subtypes, but the algorithm might be > >> inappropriate depending on the values of the subtypes. You don't find > >> that out till your in the adapter itself. > > > > Hm... The alternative, of course, instead of some magic feature, is to > > explicitly call the fallback algorithm. That seems totally reasonable > > for this example, and is similar to Python's single-inheritance-style > > explicit super-call e.g. return BaseClass.methodName(self, > > arguments...). > > For methods you know what your bases classes are, Not necessarily in the case of multiple inheritance; that's why Python 2.2 added super(). > but for adapters > that's not necessarily so. Somewhat could have registered another > adapter for the base class. > > Does anybody know if Phillip Eby's generic functions support this? > > If I understood Ian's posting > (http://mail.python.org/pipermail/python-3000/2006-April/000481.html) > correctly, they do. > > > I > > would think that the definition of "next candidate" can be problematic > > there, in case you've broken a tie between two ambiguous possibilities > > by registering an explicit tie-breaker. Remove the tie-breaker and > > neither alternative dominates the other, so the next candidate is > > undefined. This is different from Python's MRO calculations, which > > always create a perfect linearization of the base classes. > > BTW, another useful feature might be the ability to copy the protocol. > Then I could register a few other adapters for the copy and use the > modified copy for my own purpose. Let's not go wild with features; let's first collect use cases. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From walter at livinglogic.de Thu Apr 6 00:26:58 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 06 Apr 2006 00:26:58 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> Message-ID: <44344432.5030704@livinglogic.de> Guido van Rossum wrote: > On 4/5/06, Walter D?rwald wrote: >> Guido van Rossum wrote: >> >>> On 4/5/06, Tim Hochberg wrote: >>>> I'm hoping that Walter can give some more realistic examples since I >>>> don't have real-world experience here. The basic idea is simply to let >>>> an adapter give up and let the protocol try the next adapter. This could >>>> happen in a generic function, for instance, if you wanted to try some >>>> fast algorithm for some specific subtypes, but the algorithm might be >>>> inappropriate depending on the values of the subtypes. You don't find >>>> that out till your in the adapter itself. >>> Hm... The alternative, of course, instead of some magic feature, is to >>> explicitly call the fallback algorithm. That seems totally reasonable >>> for this example, and is similar to Python's single-inheritance-style >>> explicit super-call e.g. return BaseClass.methodName(self, >>> arguments...). >> For methods you know what your bases classes are, > > Not necessarily in the case of multiple inheritance; that's why Python > 2.2 added super(). True. >> but for adapters >> that's not necessarily so. Somewhat could have registered another >> adapter for the base class. > >>> Does anybody know if Phillip Eby's generic functions support this? >> If I understood Ian's posting >> (http://mail.python.org/pipermail/python-3000/2006-April/000481.html) >> correctly, they do. >> >>> I >>> would think that the definition of "next candidate" can be problematic >>> there, in case you've broken a tie between two ambiguous possibilities >>> by registering an explicit tie-breaker. Remove the tie-breaker and >>> neither alternative dominates the other, so the next candidate is >>> undefined. This is different from Python's MRO calculations, which >>> always create a perfect linearization of the base classes. >> BTW, another useful feature might be the ability to copy the protocol. >> Then I could register a few other adapters for the copy and use the >> modified copy for my own purpose. > > Let's not go wild with features; OK! (Adding a copy method is trivial anyway). > let's first collect use cases. For a potential use case see http://bugs.python.org/1351692 Someone wanted to add hex output for ints and long to pprint.pprint(). If pprint used a protocol, he could have copied the pprint protocol object, replaced the adapter for int and long and used that. But maybe a better option would be to "subclass" it, so that if the adapter isn't found in the subclassed protocol it is searched in the base protocol. This has the advantage that the copied protocol picks up changes in the base protocol. Servus, Walter From jimjjewett at gmail.com Thu Apr 6 00:50:16 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 5 Apr 2006 18:50:16 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44333B95.4010906@colorstudy.com> References: <44333B95.4010906@colorstudy.com> Message-ID: On 4/4/06, Ian Bicking wrote: > Alex Martelli wrote: > > As for the % operator, I never liked it -- either a builtin function, > > or even better a method of string objects, is going to be much more > > readable (and my preference would be to have it take optional > > positional arguments, corresponding to {1}, {2}, etc, and optional > > named arguments, corresponding to {name} &c). > > Note that if it takes keyword arguments, but doesn't take a single > dictionary-like object (like % and string.Template.substitute do), then > you lose any ability to use clever or interesting dictionary-like > objects for substitution. If "%" is replaced by a method, there is no reason not to have multiple methods. ";lasdf;lks{1} asdf {0} asfd".subst_seq(["a", "b", "C"]) "asdfs{a} asdf {b} asfd".subst_map(locals()) "asdlfkjaslfj{1}{a}{b}".format(16, a=18, b=19) -jJ From pje at telecommunity.com Thu Apr 6 01:39:34 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 05 Apr 2006 16:39:34 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions Message-ID: <7.0.1.0.0.20060405161502.02127ad0@after12am.com> Guido wrote: >Does anybody know if Phillip Eby's generic functions support this? Yes, and yes, they do. :) At least, the multiple-dispatch ones do. You can add a 'next_method' argument to the front of a method's arguments, and it's automatically passed the next applicable callable. (Which can also define next_method, and so on.) See: http://peak.telecommunity.com/DevCenter/CombiningResults#using-next-method > I >would think that the definition of "next candidate" can be problematic >there, in case you've broken a tie between two ambiguous possibilities >by registering an explicit tie-breaker. Remove the tie-breaker and >neither alternative dominates the other, so the next candidate is >undefined. Yes, but in that case you would want to use explicit upcalls anyway - the moral equivalent of doing BaseClass.foo(self,...) instead of super(Subclass,self).foo(...). There are also various ways to customize the way that methods are chained, too, which are described in the docs at: http://peak.telecommunity.com/DevCenter/CombiningResults >This is different from Python's MRO calculations, which >always create a perfect linearization of the base classes. Indeed. RuleDispatch's multiple-dispatch generic functions can't actually use the standard MRO for this reason. The single-dispatch API uses the MRO, but multiple dispatch uses a simpler algorithm that ignores the order of __bases__ for purposes of determining signature dominance. In other words, signature dominance is a graph or "partial ordering", rather than a linearization. > > BTW, another useful feature might be the ability to copy the protocol. > > Then I could register a few other adapters for the copy and use the > > modified copy for my own purpose. > >Let's not go wild with features; let's first collect use cases. Indeed. RuleDispatch's single-dispatch generic functions support cloning, as described in the VisitorRevisited article. However, I haven't used that feature as often as I thought I would, because it's really just a poor substitute for multiple dispatch. That is, the use cases that lead one to want to copy a protocol or single-dispatch generic function have usually turned out to be better served by rethinking in terms of a multiple-dispatch generic function. Also, you can achieve the same effect by chaining even single-dispatch generic functions, for example: @generic def g1(foo): pass @generic def g2(foo): return g1(foo) And you have now "copied" g1 to g2, simply by delegating the default case of g2 to g1. So, the motivation for the use case is sound, but in my experience, copying isn't really the best way to achieve it. Copying is IMO an artifact of "thinking in adaptation", or at least that's how I ended up thinking it was needed originally. From tim.hochberg at cox.net Thu Apr 6 01:43:08 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Wed, 05 Apr 2006 16:43:08 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44344432.5030704@livinglogic.de> References: <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> Message-ID: <4434560C.8030203@cox.net> Walter D?rwald wrote: > Guido van Rossum wrote: > >> On 4/5/06, Walter D?rwald wrote: >> >>> Guido van Rossum wrote: >>> >>>> On 4/5/06, Tim Hochberg wrote: >>>> >>>>> I'm hoping that Walter can give some more realistic examples since I >>>>> don't have real-world experience here. The basic idea is simply to >>>>> let >>>>> an adapter give up and let the protocol try the next adapter. This >>>>> could >>>>> happen in a generic function, for instance, if you wanted to try some >>>>> fast algorithm for some specific subtypes, but the algorithm might be >>>>> inappropriate depending on the values of the subtypes. You don't find >>>>> that out till your in the adapter itself. >>>> >>>> Hm... The alternative, of course, instead of some magic feature, is to >>>> explicitly call the fallback algorithm. That seems totally reasonable >>>> for this example, and is similar to Python's single-inheritance-style >>>> explicit super-call e.g. return BaseClass.methodName(self, >>>> arguments...). >>> >>> For methods you know what your bases classes are, >> >> >> Not necessarily in the case of multiple inheritance; that's why Python >> 2.2 added super(). > > > True. > >>> but for adapters >>> that's not necessarily so. Somewhat could have registered another >>> adapter for the base class. >> >> >>>> Does anybody know if Phillip Eby's generic functions support this? >>> >>> If I understood Ian's posting >>> (http://mail.python.org/pipermail/python-3000/2006-April/000481.html) >>> correctly, they do. >>> >>>> I >>>> would think that the definition of "next candidate" can be problematic >>>> there, in case you've broken a tie between two ambiguous possibilities >>>> by registering an explicit tie-breaker. Remove the tie-breaker and >>>> neither alternative dominates the other, so the next candidate is >>>> undefined. This is different from Python's MRO calculations, which >>>> always create a perfect linearization of the base classes. >>> >>> BTW, another useful feature might be the ability to copy the protocol. >>> Then I could register a few other adapters for the copy and use the >>> modified copy for my own purpose. >> >> >> Let's not go wild with features; > > > OK! (Adding a copy method is trivial anyway). > >> let's first collect use cases. > > > For a potential use case see http://bugs.python.org/1351692 > > Someone wanted to add hex output for ints and long to pprint.pprint(). > If pprint used a protocol, he could have copied the pprint protocol > object, replaced the adapter for int and long and used that. > > But maybe a better option would be to "subclass" it, so that if the > adapter isn't found in the subclassed protocol it is searched in the > base protocol. This has the advantage that the copied protocol picks > up changes in the base protocol. It seems like you can chain protocols like that simply by setting up an appropriate default_adapter method. For example, if we already had a pprinter protocol, we could define one that subclassed it and customized it: class CustomPPrintProtocol(Protocol): def default_adapter(self, *args): return pprinter(*args) hex_pprinter = CustomPPrintProtocol("hex_pprinter") @hex_pprinter.register_for(int, long): def hex_pprint(obj): return hex(obj) While on the subject of pretty print, I just hacked pretty print to allow site wide customizations using Protocols. In addition to adding the protocol class wholesale (it's not on my python path right now), it took only three lines worth of changes. I added: simple_formatter = Protocol('simple_formatter') simple_formatter.register(repr, object) Near the top and changed _safe_repr to use simple_formatter: def _safe_repr(object, context, maxlevels, level): # .... rep = simple_formatter(object) # was repr(object) return rep, (rep and not rep.startswith('<')), False With that in place, I can easily do sitewise changes to stuff like integer display: import pprint data = ( # From fbots librarybook "this is a string", [1, 2, 3, 4], ("more tuples", 1.0, 2.3, 4.5), "this is yet another string" ) pprint.pprint(data) print pprint.simple_formatter.register(hex, int, long) pprint.pprint(data) ==> ('this is a string', [1, 2, 3, 4], ('more tuples', 1.0, 2.2999999999999998, 4.5), 'this is yet another string') ('this is a string', [0x1, 0x2, 0x3, 0x4], ('more tuples', 1.0, 2.2999999999999998, 4.5), 'this is yet another string') Nice! I expect that with another dozen lines of code I could make custom pretty printers possible, although the way the customization happens might be a little kludgy. I'll look into it. -tim From walter at livinglogic.de Thu Apr 6 01:59:37 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 06 Apr 2006 01:59:37 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <4434560C.8030203@cox.net> References: <4433D142.7050203@gmail.com> <4433FC5E.6070103@ieee.org> <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> Message-ID: <443459E9.50908@livinglogic.de> Tim Hochberg wrote: > Walter D?rwald wrote: > >> Guido van Rossum wrote: >> >>> On 4/5/06, Walter D?rwald wrote: >>> >>>> Guido van Rossum wrote: >>>> >>>>> On 4/5/06, Tim Hochberg wrote: >>>>> >>>>>> I'm hoping that Walter can give some more realistic examples since I >>>>>> don't have real-world experience here. The basic idea is simply to >>>>>> let >>>>>> an adapter give up and let the protocol try the next adapter. This >>>>>> could >>>>>> happen in a generic function, for instance, if you wanted to try some >>>>>> fast algorithm for some specific subtypes, but the algorithm might be >>>>>> inappropriate depending on the values of the subtypes. You don't find >>>>>> that out till your in the adapter itself. >>>>> >>>>> Hm... The alternative, of course, instead of some magic feature, is to >>>>> explicitly call the fallback algorithm. That seems totally reasonable >>>>> for this example, and is similar to Python's single-inheritance-style >>>>> explicit super-call e.g. return BaseClass.methodName(self, >>>>> arguments...). >>>> >>>> For methods you know what your bases classes are, >>> >>> >>> Not necessarily in the case of multiple inheritance; that's why Python >>> 2.2 added super(). >> >> >> True. >> >>>> but for adapters >>>> that's not necessarily so. Somewhat could have registered another >>>> adapter for the base class. >>> >>> >>>>> Does anybody know if Phillip Eby's generic functions support this? >>>> >>>> If I understood Ian's posting >>>> (http://mail.python.org/pipermail/python-3000/2006-April/000481.html) >>>> correctly, they do. >>>> >>>>> I >>>>> would think that the definition of "next candidate" can be problematic >>>>> there, in case you've broken a tie between two ambiguous possibilities >>>>> by registering an explicit tie-breaker. Remove the tie-breaker and >>>>> neither alternative dominates the other, so the next candidate is >>>>> undefined. This is different from Python's MRO calculations, which >>>>> always create a perfect linearization of the base classes. >>>> >>>> BTW, another useful feature might be the ability to copy the protocol. >>>> Then I could register a few other adapters for the copy and use the >>>> modified copy for my own purpose. >>> >>> >>> Let's not go wild with features; >> >> >> OK! (Adding a copy method is trivial anyway). >> >>> let's first collect use cases. >> >> >> For a potential use case see http://bugs.python.org/1351692 >> >> Someone wanted to add hex output for ints and long to pprint.pprint(). >> If pprint used a protocol, he could have copied the pprint protocol >> object, replaced the adapter for int and long and used that. >> >> But maybe a better option would be to "subclass" it, so that if the >> adapter isn't found in the subclassed protocol it is searched in the >> base protocol. This has the advantage that the copied protocol picks >> up changes in the base protocol. > > It seems like you can chain protocols like that simply by setting up an > appropriate default_adapter method. For example, if we already had a > pprinter protocol, we could define one that subclassed it and customized > it: > > class CustomPPrintProtocol(Protocol): > def default_adapter(self, *args): > return pprinter(*args) > hex_pprinter = CustomPPrintProtocol("hex_pprinter") > > @hex_pprinter.register_for(int, long): > def hex_pprint(obj): > return hex(obj) The problem with this is that it looks in the base protocol only if the the class can't be found in the registry of the subprotocol. This mean that an adapter from the subprotocol might be used, although the base protocol has a "better" adapter (i.e. one whose class that is nearer to real class of the object. Instead for each class in the mro chain the subprotocol must traverse the list of its base protocols to check if this class is in the registry of the base protocols. This guarantees that the best match is found. > While on the subject of pretty print, I just hacked pretty print to > allow site wide customizations using Protocols. In addition to adding > the protocol class wholesale (it's not on my python path right now), it > took only three lines worth of changes. I added: > > simple_formatter = Protocol('simple_formatter') > simple_formatter.register(repr, object) > > > Near the top and changed _safe_repr to use simple_formatter: > > def _safe_repr(object, context, maxlevels, level): > # .... > rep = simple_formatter(object) # was repr(object) > return rep, (rep and not rep.startswith('<')), False > > > With that in place, I can easily do sitewise changes to stuff like > integer display: > > import pprint > data = ( # From fbots librarybook > "this is a string", [1, 2, 3, 4], ("more tuples", > 1.0, 2.3, 4.5), "this is yet another string" > ) > pprint.pprint(data) > print > pprint.simple_formatter.register(hex, int, long) > pprint.pprint(data) > > ==> > > ('this is a string', > [1, 2, 3, 4], > ('more tuples', 1.0, 2.2999999999999998, 4.5), > 'this is yet another string') > > ('this is a string', > [0x1, 0x2, 0x3, 0x4], > ('more tuples', 1.0, 2.2999999999999998, 4.5), > 'this is yet another string') > > Nice! > > I expect that with another dozen lines of code I could make custom > pretty printers possible, although the way the customization happens > might be a little kludgy. I'll look into it. I'm looking forward to the results! Servus, Walter From greg.ewing at canterbury.ac.nz Thu Apr 6 01:57:55 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Apr 2006 11:57:55 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <06Apr4.201259pdt.58633@synergy1.parc.xerox.com> Message-ID: <44345983.2010006@canterbury.ac.nz> Alex Martelli wrote: > > and my preference would be to have it take optional > positional arguments, corresponding to {1}, {2}, etc, and optional > named arguments, corresponding to {name} &c). Yes, certainly. -- Greg From greg.ewing at canterbury.ac.nz Thu Apr 6 02:03:37 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Apr 2006 12:03:37 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44333B95.4010906@colorstudy.com> References: <06Apr4.201259pdt.58633@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> Message-ID: <44345AD9.9030207@canterbury.ac.nz> Ian Bicking wrote: > Note that if it takes keyword arguments, but doesn't take a single > dictionary-like object (like % and string.Template.substitute do), then > you lose any ability to use clever or interesting dictionary-like > objects for substitution. If you subclass your clever object from dict you can use it with **. And expanding the range of objects acceptable to ** would be a good idea for Py3k too. -- Greg From guido at python.org Thu Apr 6 02:05:51 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Apr 2006 17:05:51 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <4434560C.8030203@cox.net> References: <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> Message-ID: On 4/5/06, Tim Hochberg wrote: > It seems like you can chain protocols like that simply by setting up an > appropriate default_adapter method. For example, if we already had a > pprinter protocol, we could define one that subclassed it and customized it: > > class CustomPPrintProtocol(Protocol): > def default_adapter(self, *args): > return pprinter(*args) > hex_pprinter = CustomPPrintProtocol("hex_pprinter") > > @hex_pprinter.register_for(int, long): > def hex_pprint(obj): > return hex(obj) > > > While on the subject of pretty print, I just hacked pretty print to > allow site wide customizations using Protocols. In addition to adding > the protocol class wholesale (it's not on my python path right now), it > took only three lines worth of changes. I added: > > simple_formatter = Protocol('simple_formatter') > simple_formatter.register(repr, object) > > > Near the top and changed _safe_repr to use simple_formatter: > > def _safe_repr(object, context, maxlevels, level): > # .... > rep = simple_formatter(object) # was repr(object) > return rep, (rep and not rep.startswith('<')), False > > > With that in place, I can easily do sitewise changes to stuff like > integer display: > > import pprint > data = ( # From fbots librarybook > "this is a string", [1, 2, 3, 4], ("more tuples", > 1.0, 2.3, 4.5), "this is yet another string" > ) > > pprint.pprint(data) > print > pprint.simple_formatter.register(hex, int, long) > pprint.pprint(data) > > ==> > > ('this is a string', > [1, 2, 3, 4], > ('more tuples', 1.0, 2.2999999999999998, 4.5), > 'this is yet another string') > > ('this is a string', > [0x1, 0x2, 0x3, 0x4], > ('more tuples', 1.0, 2.2999999999999998, 4.5), > 'this is yet another string') > > Nice! > > I expect that with another dozen lines of code I could make custom > pretty printers possible, although the way the customization happens > might be a little kludgy. I'll look into it. Neat indeed. What I'd like to see is how this changes if we use a generic functions (multiple dispatch) implementation instead of adaptation. Adaptation is pretty much isomorphic with a generic function of one argument (see my blog). But the register_for() method couldn't (shouldn't) interpret multiple arguments as multiple registrations; it should probably take this to match the call signature. Then multiple registrations would slightly more awkward; fortunately they don't have to be as ugly as they were in my earlier post since you can stack them: @hex_pprinter.register_for(int, long) def ... could become @hex_pprinter.register_for(int) @hex_pprinter.register_for(long) def ... I also wonder if the explicit (non-decorator) registration API is rare enough that we could use a single register() method that favors use as a decorator but can be used for explicit registration in a pinch. Then the decorator could look like this: @hex_pprinter.register(int) def ... and explicit registration could be done like this: pprint.simple_formatter.register(int)(hex) pprint.simple_formatter.register(long)(hex) For multiple-argument signatures the call would become e.g. foo.register(int, str, C)(myfunc) which isn't so bad compared to foo.register(myfunc, (int, str, C)) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Apr 6 02:07:10 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Apr 2006 12:07:10 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44333ECA.3000505@colorstudy.com> References: <06Apr4.201259pdt.58633@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> <44333ECA.3000505@colorstudy.com> Message-ID: <44345BAE.50301@canterbury.ac.nz> Ian Bicking wrote: > Clever and interesting would include my quoting, evaluating, and > formatting wrappers. It might also include special gettext expressions, > case insensitivity, or other interesting wrappers. Hmmm, *that* degree of cleverness would be better served by a different interface to the formatting system. But there should still be a simple keyword arg interface for the vast majority of cases when you don't want to be clever. -- Greg From guido at python.org Thu Apr 6 02:08:22 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 5 Apr 2006 17:08:22 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <443459E9.50908@livinglogic.de> References: <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443459E9.50908@livinglogic.de> Message-ID: On 4/5/06, Walter D?rwald wrote: > The problem with this is that it looks in the base protocol only if the > the class can't be found in the registry of the subprotocol. This mean > that an adapter from the subprotocol might be used, although the base > protocol has a "better" adapter (i.e. one whose class that is nearer to > real class of the object. It also doesn't look like it will work right for recursive invocations of the adapter by some implementation (as happens to be common for the pprint example). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Apr 6 02:53:56 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Apr 2006 12:53:56 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44341BC4.6090601@colorstudy.com> References: <06Apr4.201259pdt.58633@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> <443419F6.70103@ieee.org> <44341BC4.6090601@colorstudy.com> Message-ID: <443466A4.8060600@canterbury.ac.nz> Ian Bicking wrote: > ** can't be changed > in this way, either -- it really has to enumerate all the keys in the > thing passed to it, In Py3k this could perhaps be different in the case where you're calling a function defined like def f(*args, **kwds): ... In that case there are no named arguments to look up in the dict, so when it's called using f(**obj) the obj could be passed straight through to the kwds argument. -- Greg From ianb at colorstudy.com Thu Apr 6 03:16:24 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 05 Apr 2006 20:16:24 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44345BAE.50301@canterbury.ac.nz> References: <06Apr4.201259pdt.58633@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> <44333ECA.3000505@colorstudy.com> <44345BAE.50301@canterbury.ac.nz> Message-ID: <44346BE8.3070106@colorstudy.com> Greg Ewing wrote: > Ian Bicking wrote: > >> Clever and interesting would include my quoting, evaluating, and >> formatting wrappers. It might also include special gettext >> expressions, case insensitivity, or other interesting wrappers. > > Hmmm, *that* degree of cleverness would be better > served by a different interface to the formatting > system. But there should still be a simple keyword > arg interface for the vast majority of cases when > you don't want to be clever. Why? The wrappers looked pretty darn simple to me. I suppose they'd be slightly simpler if you replaced __getitem__ with __call__. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From ianb at colorstudy.com Thu Apr 6 03:20:01 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 05 Apr 2006 20:20:01 -0500 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <443466A4.8060600@canterbury.ac.nz> References: <06Apr4.201259pdt.58633@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> <443419F6.70103@ieee.org> <44341BC4.6090601@colorstudy.com> <443466A4.8060600@canterbury.ac.nz> Message-ID: <44346CC1.6010200@colorstudy.com> Greg Ewing wrote: > Ian Bicking wrote: > >> ** can't be changed in this way, either -- it really has to enumerate >> all the keys in the thing passed to it, > > In Py3k this could perhaps be different in the case where > you're calling a function defined like > > def f(*args, **kwds): > ... > > In that case there are no named arguments to look up > in the dict, so when it's called using > > f(**obj) > > the obj could be passed straight through to the kwds > argument. I thought of that, but that seemed much too clever. For instance, if you then change the signature to: f(a, **obj) Then it doesn't work anymore, because you have to test if a keyword of 'a' was passed in. ** in the signature and ** in the call are really not very symmetric. With current substitution this isn't that big a deal, because you have either positional *or* named markers (or for string.Template positional are not allowed), and if you see any named marker you assume you were passed a dictionary-like object. Some of the proposals that mix {1} with {name} in the same substitution are harder. Also, they have to look at the name of the marker and see if it is an integer, and thus a dictionary with a key like '1' cannot be used. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From tim.hochberg at ieee.org Thu Apr 6 04:50:22 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 05 Apr 2006 19:50:22 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> Message-ID: <443481EE.6060303@ieee.org> Guido van Rossum wrote: [TAHs simple pprint customization example snipped] >> >>Nice! >> >>I expect that with another dozen lines of code I could make custom >>pretty printers possible, although the way the customization happens >>might be a little kludgy. I'll look into it. > > > Neat indeed. > > What I'd like to see is how this changes if we use a generic functions > (multiple dispatch) implementation instead of adaptation. Adaptation > is pretty much isomorphic with a generic function of one argument (see > my blog). But the register_for() method couldn't (shouldn't) interpret > multiple arguments as multiple registrations; it should probably take > this to match the call signature. > > Then multiple registrations would slightly more awkward; fortunately > they don't have to be as ugly as they were in my earlier post since > you can stack them: > > @hex_pprinter.register_for(int, long) > def ... > > could become > > @hex_pprinter.register_for(int) > @hex_pprinter.register_for(long) > def ... > > I also wonder if the explicit (non-decorator) registration API is rare > enough that we could use a single register() method that favors use as > a decorator but can be used for explicit registration in a pinch. Then > the decorator could look like this: > > @hex_pprinter.register(int) > def ... > > and explicit registration could be done like this: > > pprint.simple_formatter.register(int)(hex) > pprint.simple_formatter.register(long)(hex) > > For multiple-argument signatures the call would become e.g. > > foo.register(int, str, C)(myfunc) > > which isn't so bad compared to > > foo.register(myfunc, (int, str, C)) FWIW, I consistently type protocol.register(type, adapter) instead of the other way around. The change you propose switches the order around to what seems natural and cleans up the API as well, so +1 from me. I was sucessful on the pprint front. I'll clean up the example tomorrow and post either a URL or a diff (any preferences?) but here's how it's used: ############## import pprint # Any instance of pprint.PPrint is a separately cusomizable # pretty printer. pprint.pprint is just another instance. mypprint = pprint.PPrint() # Some data I stole from the fbots librarybook for testing data = ( "this is a string", [1, 2, 3, 4], ("more tuples", 1.0, 2.3, 4.5), "this is yet another string" ) # This just pprints as usual pprint.pprint(data) # So does this. mypprint(data) # Now we can register an adapter to mypprint and # ints and longs will be printed in hex. Note # that mypprint is not really an adapter, it just # forwards register calls to an underlying # Protocol. (Actually a SubProtocol that should handle # "inheritance" correctly, although it's not well # tested yet). mypprint.register(hex, int, long) mypprint(data) # The output of this is still unchanged however. pprint.pprint(data) ############## This is pretty cool, but it's really only halfway there. Right now, only the 'basic' types that were previously handled by repr are handled by the custom formatter. So, you can't customize tuples and dicts and such that are hardcoded into _safe_repr and _format. The full solution tosses those into the formatter as well and gets rid of _safe_repr and _format altogether. I have a feeling that once you start pulling on this thread the whole module will unravel and all that will be left is a couple of protocols and a bunch of helper functions. That would be cool to see, but it might take a while. Regards, -tim From greg.ewing at canterbury.ac.nz Thu Apr 6 05:32:28 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 06 Apr 2006 15:32:28 +1200 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <44346BE8.3070106@colorstudy.com> References: <06Apr4.201259pdt.58633@synergy1.parc.xerox.com> <44333B95.4010906@colorstudy.com> <44333ECA.3000505@colorstudy.com> <44345BAE.50301@canterbury.ac.nz> <44346BE8.3070106@colorstudy.com> Message-ID: <44348BCC.9070807@canterbury.ac.nz> Ian Bicking wrote: > Why? The wrappers looked pretty darn simple to me. I suppose they'd be > slightly simpler if you replaced __getitem__ with __call__. What I meant was that another method or function could be provided that takes a mapping object as a direct argument, rather than trying to stuff it through **. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From ian.bollinger at gmail.com Thu Apr 6 06:02:39 2006 From: ian.bollinger at gmail.com (Ian D. Bollinger) Date: Thu, 6 Apr 2006 00:02:39 -0400 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> Message-ID: <86a6a97b0604052102n4508b70ey34b2442106ec6913@mail.gmail.com> On 4/5/06, John J Lee wrote: > > But how about putting them at the start, after the $? That way, the > format character can't get mixed up with the text following it. For > example, $r{foo} would mean the same as the current %(foo)r. > This is somewhat tangental, but hopefully an $r formatting operation won't be necessary in Py3k, assuming repr() and str() can somehow be combined. -- - Ian D. Bollinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060406/65324201/attachment.htm From walter at livinglogic.de Thu Apr 6 07:46:14 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 06 Apr 2006 07:46:14 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443459E9.50908@livinglogic.de> Message-ID: <4434AB26.1040909@livinglogic.de> Guido van Rossum wrote: > On 4/5/06, Walter D?rwald wrote: >> The problem with this is that it looks in the base protocol only if the >> the class can't be found in the registry of the subprotocol. This mean >> that an adapter from the subprotocol might be used, although the base >> protocol has a "better" adapter (i.e. one whose class that is nearer to >> real class of the object. > > It also doesn't look like it will work right for recursive invocations > of the adapter by some implementation (as happens to be common for the > pprint example). If the protocol was a class instead of an instance we could use the mro of the protocol: class MetaInterface(type): def __new__(mcl, name, bases, dict): # Give each class it's own registry cls = type.__new__(mcl, name, bases, dict) cls.registry = {} return cls def register(cls, *types): def decorator(adapter): for type in types: cls.registry[type] = adapter return decorator def default(cls, *args, **kwargs): raise TypeError("can adapt %r to %r" % (args[0], cls)) def __call__(cls, *args, **kwargs): # Cannot construct protocol instances # Instead, calling the class triggers adaptation for basetype in type(args[0]).__mro__: for baseprotocol in cls.__mro__: registry = getattr(baseprotocol, "registry", None) if registry is not None: adapter = registry.get(basetype) if adapter is not None: # Must pass the protocol to the adapter # so that it can dispath to the right protocol return adapter(cls, *args, **kwargs) return cls.default(*args, **kwargs) class Protocol(object): __metaclass__ = MetaInterface # Extensible repr protocol class xrepr(Protocol): @classmethod def default(cls, *args, **kwargs): return repr(*args, **kwargs) @xrepr.register(list) def xrepr_list(protocol, obj): return "[%s]" % ", ".join(protocol(x) for x in obj) # Subprotocol that overwrites int/long adaption class hexrepr(xrepr): pass @hexrepr.register(int, long) def hexrepr_number(protocol, obj): return hex(obj) print hexrepr(range(4)) This prints [0x0, 0x1, 0x2, 0x3] Servus, Walter From ncoghlan at gmail.com Thu Apr 6 11:09:30 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 06 Apr 2006 19:09:30 +1000 Subject: [Python-3000] String formating operations in python 3k In-Reply-To: <86a6a97b0604052102n4508b70ey34b2442106ec6913@mail.gmail.com> References: <443062D8.50701@livinglogic.de> <44316EBA.3020302@colorstudy.com> <4431D9D7.2050609@canterbury.ac.nz> <1144178074.17927.43.camel@resist.wooz.org> <443304D7.7020307@canterbury.ac.nz> <86a6a97b0604052102n4508b70ey34b2442106ec6913@mail.gmail.com> Message-ID: <4434DACA.2050308@gmail.com> Ian D. Bollinger wrote: > On 4/5/06, *John J Lee* > wrote: > > But how about putting them at the start, after the $? That way, the > format character can't get mixed up with the text following it. For > example, $r{foo} would mean the same as the current %(foo)r. > > > This is somewhat tangental, but hopefully an $r formatting operation > won't be necessary in Py3k, assuming repr() and str() can somehow be > combined. repr() (machine readable) and str() (human readable) do different things, so why would combining them even be considered? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From mcherm at mcherm.com Thu Apr 6 13:57:49 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 06 Apr 2006 04:57:49 -0700 Subject: [Python-3000] Removing repr Message-ID: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> Ian D. Bollinger writes: > This is somewhat tangental, but hopefully an $r formatting operation > won't be necessary in Py3k, assuming repr() and str() can somehow be > combined. Nick Coghlan responds: > repr() (machine readable) and str() (human readable) do different things, > so why would combining them even be considered? First, the history as I understand it: repr() and str() are both concerned with turning an object into a string that vaguely describes it. (Turning it into a string that completely describes it is called serialization, and neither of these functions performs that reliably.) In the special (but rather important) case of strings (including unicode) there is a basic ambiguity of whether you want to display the string itself (after all, it's already a string) or provide quoting and escaping. This ambiguity was solved by creating two different methods, str() and repr(). A general rule was promulgated: str() should produce human readable strings, and repr() should produce something which could be fed to eval() to recreate the original *if that's easy to do*, and something readable in other cases. In practice, a huge number of classes implement the two identically. The argument in favor of combining the two (the result would be named "str", hence the subject line): There are NOT two different fundamental ideas here, so we don't *really* need two different functions. Perhaps we need some way (unspecified... lots of clever ways could be suggested) to control whether strings are escaped-and-quoted or not, but that's it. Making repr() produce something which can be used with eval() to recreate the original object was always a dream: it fails in an enormous number of real cases and there are better ways to do serialization anyhow. Furthermore, the existing situation creates confusion. It is difficult to explain to newbies, and even experts like Nick wind up eroniously beliving that repr() generates a "machine readable" form. (I'm going out on a limb here, claiming that I'm right and Nick is wrong, but I'm going to stand by the claim anyhow. Anything which is machine readable "sometimes" is, by my definition, NOT machine readable!) The argument AGAINST combining the two is as follows: There would be a huge amount of gratuitous code breakage, for the sake of eliminating a little confusion (which isn't even a major source of bugs). It isn't worth the effort of re-writing code (even though python 3000 may break code, most not-too-clever code from python 2.x ought to work fine with perhaps a few modifications). In my opinion, the argument against wins. I move we list this in PEP 3099. -- Michael Chermside From msoulier at digitaltorque.ca Thu Apr 6 15:13:33 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Thu, 6 Apr 2006 09:13:33 -0400 Subject: [Python-3000] Removing repr In-Reply-To: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> Message-ID: <20060406131333.GM12075@tigger.digitaltorque.ca> On 06/04/06 Michael Chermside said: > repr() and str() are both concerned with turning an object into a > string that vaguely describes it. (Turning it into a string that I thought that the distinction was as follows. str() should return a human-readable representation of something. repr() should return a machine-readable representation of something, suitable for an exec command. That still has value, no? Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060406/14b302d5/attachment.pgp From gustavo at niemeyer.net Thu Apr 6 15:08:01 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 6 Apr 2006 10:08:01 -0300 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <443425A0.3050101@livinglogic.de> <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> Message-ID: <20060406130801.GA9078@localhost.localdomain> > What I'd like to see is how this changes if we use a generic functions > (multiple dispatch) implementation instead of adaptation. Adaptation > is pretty much isomorphic with a generic function of one argument (see > my blog). But the register_for() method couldn't (shouldn't) interpret > multiple arguments as multiple registrations; it should probably take > this to match the call signature. Zope interfaces support multi-adapters, and they're very useful in certain cases. I belive this would match pretty well with a generic function with more than a single argument. If somebody is curious about the logic used to lookup adapters, the interesting part is in zope.interface.adapter.AdapterLookup.lookup(). -- Gustavo Niemeyer http://niemeyer.net From jjl at pobox.com Thu Apr 6 16:02:36 2006 From: jjl at pobox.com (John J Lee) Date: Thu, 6 Apr 2006 15:02:36 +0100 (GMT Standard Time) Subject: [Python-3000] Removing repr In-Reply-To: <20060406131333.GM12075@tigger.digitaltorque.ca> References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> <20060406131333.GM12075@tigger.digitaltorque.ca> Message-ID: On Thu, 6 Apr 2006, Michael P. Soulier wrote: [...] > I thought that the distinction was as follows. > > str() should return a human-readable representation of something. > > repr() should return a machine-readable representation of something, suitable > for an exec command. [...] Is this old chestnut really still around? repr() is for debugging; it does not have to return an eval()-able string. The __repr__ docs explain that well. Perhaps the documentation for repr() is still partly to blame for this confusion. It says: """ Return a string containing a printable representation of an object. This is the same value yielded by conversions (reverse quotes). It is sometimes useful to be able to access this operation as an ordinary function. For many types, this function makes an attempt to return a string that would yield an object with the same value when passed to eval(). """ I suggest (for Py3k, anyway): """ Return a string containing a printable representation of an object. This is useful for debugging. Often, this function returns a string of the form "<...some useful description...>". For some types, this function returns a string that would yield an object with the same value when passed to eval(). """ > That still has value, no? I find the 'debug string' / 'human-readable string' distinction valuable. John From ncoghlan at iinet.net.au Thu Apr 6 16:30:53 2006 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri, 07 Apr 2006 00:30:53 +1000 Subject: [Python-3000] Sane transitive adaptation Message-ID: <4435261D.1000602@iinet.net.au> One issue with generic functions and adaptation as currently being discussed (and something Tim mentioned a while back), it that it is very focused on dispatching based solely on obj.__class__.__mro__. That's all well and good, but (to use an example of Tim's), suppose we have a couple of frameworks we're using, where those frameworks have each defined their own version of a "wibbly-wobbly" object: frmwrk_A.IWibble frmwrk_B.IWobble The methods and signatures included in these protocols may be the same, IWibble may be a subset of IWobble, or vice versa. Both frameworks have defined their interfaces (input and output) in terms of these protocols, rather than in terms of concrete types. As Tim pointed out, pure type-based dispatch would require that every type be registered with both protocols, even if the two protocols are identical. Doing that manually would be a serious pain if the frameworks were non-trivial. The first thought may be to allow adaptation between protocols - but the issue with that is that at adaptation time, we only have the object to work with, and won't know what protocols it claims to implement (this is where either __conform__ or a global registry comes in with PEP 246). Even if we did know, the cost of doing a search on all equivalent protocols on each call would add up. We don't really want to be finding ways to make functions calls even slower than usual ;) There is, however, an alternative, which would be to include a mechanism for telling a protocol about other equivalent protocols, and updating the registration mechanism to distribute any registration updates to the equivalent protocols. For example (not tested code!): def _update(self, updates, updated=None): # Update ourselves with a change # This update does NOT overwrite existing mappings changes = {} for signature, adapter in updates.iteritems(): if signature not in self.registry: changes[signature] = adapter self.registry.update(changes) self._distribute_update(changes, updated) def _distribute_update(self, updates, updated=None): # We don't want to get this update back if updated is None: updated = set() else: if self in updated: return updated.add(self) # Update any equivalent protocols # that have not yet been updated for protocol in self.subprotocols: if protocol not in updated: protocol._update(updates, updated) def _register_callable(self, signature, callable) # Explicit registration overrides existing mapping self.registry[signature] = adapter self._distribute_update({signature:callable}) def register_subprotocol(self, protocol): # Specified protocol is a subset of this one # so it can safely use this protocol's adapters protocol._update(self.registry) self.subprotocols.add(protocol) def register(self, *key) def helper(adapter): self.register(adapter, *key) return adapter return helper Using such a mechanism, the interface mismatch above could be addressed in one of the following ways: Suppose framework A specifies fewer methods than framework B, but those methods match. Then you can write: frmwrk_B.IWobble.register_subprotocol(frmwrk_A.IWibble) # Now registration for framework B also registers you for the narrower # interface in framework A You can turn that around, if A is the one that is more prescriptive: frmwrk_A.IWibble.register_subprotocol(frmwrk_B.IWobble) # Now it is registration for framework A that registers you for the narrower # interface in framework B And finally, if the two interfaces are identical: frmwrk_A.IWibble.register_subprotocol(frmwrk_B.IWobble) frmwrk_B.IWobble.register_subprotocol(frmwrk_A.IWibble) # Now registration for either framework registers you for both Now, suppose, however, that mapping from A to B required a slight tweak to A's interface - one of the method signatures didn't line up right (e.g. one of A's methods has the wrong name). This can be handled with an explicit protocol adapter, which would provide a modified update method like so: class ProtocolAdapter(object): def __init__(self, src, target): self.target = target src.register_subprotocol(self) def _update(self, updates, updated=None): if self.target in updated: return wrapped_updates = {} for signature, adapter in updates.iteritems(): def wrapped_adapter(*args, **kwds): return self.adapt(adapter(*args, **kwds)) wrapped_updates[signature] = wrapped_adapter self.target._update(wrapped_updates, updated) def __call__(self, adapter): self.adapt = adapter class AdaptWibbletoWobble(object): def __init__(self, obj): self.obj = obj def __iter__(x, y, z): return self.obj.method2(x, y, z) ProtocolAdapter(frmwork_A.IWibble, frmwrk_B.IWobble)(AdaptWibbleToWobble) The equivalent of the above for generic functions is the case where "generic function A" does a very similar thing to "generic function B", and you want to be able to do a wholesale delegation from A to B of everything that B handles more specifically than A. Aside from the fact that I think any such transitivity mechanism should be observer based, and that the updates should happen at registration time rather than lookup time, I'm not really wedded to any of the implementation details above. But I figured it was something worth throwing out there :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From aahz at pythoncraft.com Thu Apr 6 16:45:43 2006 From: aahz at pythoncraft.com (Aahz) Date: Thu, 6 Apr 2006 07:45:43 -0700 Subject: [Python-3000] Kill "generic functions"! Message-ID: <20060406144543.GC23101@panix.com> I cringe every time I see the phrase "generic function". It's way too generic -- it doesn't contain any hint of its meaning, and I find it confusing because Python already has what I think of as generic functions: def adder(a, b): return a+b That's a perfectly good generic function that depends on Python's duck-typing. "Multimethods" works, but Python is in many ways function-oriented rather than method-oriented. Why not call them "multifunctions"? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet From ianb at colorstudy.com Thu Apr 6 17:50:46 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 06 Apr 2006 10:50:46 -0500 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <4435261D.1000602@iinet.net.au> References: <4435261D.1000602@iinet.net.au> Message-ID: <443538D6.5020704@colorstudy.com> Nick Coghlan wrote: > One issue with generic functions and adaptation as currently being discussed > (and something Tim mentioned a while back), it that it is very focused on > dispatching based solely on obj.__class__.__mro__. RuleDispatch allows any function to be used in selecting an implementation, so implements() works as well as isinstance(). Well, truly making it work as well is some extra work, because RuleDispatch can determine the dominant implementation with isinstance(), but can't do so for an arbitrary function. Since this aspect of RuleDispatch hasn't come up, it goes like: @foo.when("isinstance(a, Widget) and implements(b, IWibble)") def foo(a, b): ... If general predicate dispatch doesn't go into py3k, that seems fine to me since it doesn't keep anyone from using RuleDispatch. And having the concept in the core language can only benefit more sophisticated implementations. But... interfaces. Without predicate dispatch, how might it work? I think you basically need two operations to do simple dispatch: a boolean, and something to detect when one condition is a subset of another. For instance, if you have an implementation for A, and one for B, and B subclasses A, then you know with issubclass(B, A) that the B condition is a subset of A. I guess you can get more complex about it too, with all the __mro__ discussion... but I haven't really kept track of those details. Then you need a condition, isinstance(obj, B). Since we don't know what interfaces look like, we don't know what these functions might be named. But lets say issubinterface(IList, ISequence) will be true, and implements([], IList) is true. So you just need to get your hands on these two functions... umm... @generic def is_predicate_subset(pred1, pred2): """Is pred1 a subset of pred2? If this is true, then an implementation specialized for pred1 is dominant over an implementation specialized for pred2, because anything selecting for pred1 is statically identifiable as being more specific. """ return False @is_predicate_subset.when(type, type) def predicate_classes(pred1, pred2): return issubclass(pred1, pred2) @is_predicate_subset.when(interface, interface) def predicate_interfaces(pred1, pred2): return issubinterface(pred1, pred2) ## though I guess you could also do: # is_predicate_subset.when(type, type)(issubclass) @generic def matches_predicate(obj, pred): """Does obj satisfy the given predicate? """ raise NotImplementedError @matches_predicate.when(object, type): def matches_class(obj, pred): return isinstance(obj, pred) @matches_predicate.when(object, interface) def matches_interface(obj, pred): return implements(obj, pred) For performance reasons, it seems like you would want to resolve matches_predicate before the actual generic function is called. So it might be: @generic def predicate_matcher(pred): """Return a function that tests if an object satisfies the predicate """ raise NotImplementedError @predicate_matcher.when(type) def match_type(pred): return lambda obj: isinstance(obj, pred) @predicate_matcher.when(interface) def match_interface(pred): return lambda obj: implements(obj, pred) So... in theory you could register strings, and thus implement RuleDispatches behavior (where those strings represent expressions). Or, ideally, py3k will have something better than strings to represent unevaluated expressions. Though there's lots of optimizations in RuleDispatch that would be lost when abstracted in this way. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From tim.hochberg at ieee.org Thu Apr 6 18:15:54 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 06 Apr 2006 09:15:54 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <4434AB26.1040909@livinglogic.de> References: <44342C26.70009@cox.net> <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443459E9.50908@livinglogic.de> <4434AB26.1040909@livinglogic.de> Message-ID: Walter D?rwald wrote: > Guido van Rossum wrote: > > >>On 4/5/06, Walter D?rwald wrote: >> >>>The problem with this is that it looks in the base protocol only if the >>>the class can't be found in the registry of the subprotocol. This mean >>>that an adapter from the subprotocol might be used, although the base >>>protocol has a "better" adapter (i.e. one whose class that is nearer to >>>real class of the object. >> >>It also doesn't look like it will work right for recursive invocations >>of the adapter by some implementation (as happens to be common for the >>pprint example). > > > If the protocol was a class instead of an instance we could use the mro > of the protocol: > > class MetaInterface(type): > def __new__(mcl, name, bases, dict): [...] > > class Protocol(object): > __metaclass__ = MetaInterface > > # Extensible repr protocol > class xrepr(Protocol): > @classmethod > def default(cls, *args, **kwargs): > return repr(*args, **kwargs) > > @xrepr.register(list) > def xrepr_list(protocol, obj): > return "[%s]" % ", ".join(protocol(x) for x in obj) > [...] Here's another approach to this that doesn't use metaclasses (they make my head hurt). First we define a ChainedDict class that forwards requests for items it doesn't have onto it's parent: _missing = object() class ChainedDict(dict): """A dict like object that forwards item requests to its parent if neeeded.""" def __init__(self, parent): dict.__init__(self) self.parent = parent def __contains__(self, key): return dict.__contains__(self, key) or (key in self.parent) def __getitem__(self, key): x = self.get(key, _missing) if x is _missing: raise KeyError('not found') return x def get(self, key, default=None): x = dict.get(self, key, _missing) if x is _missing: return self.parent.get(key, default) return x [I believe that this satisfies the properties that Walter wants, but I've already been wrong on this once already so we'll see.] With that in place, we make a minor change to the constructor of Protocol: class Protocol(object): """Declare a named protocol""" def __init__(self, parent=None): if parent is None: self.registry = {} else: self.registry = ChainedDict(parent.registry) #.... Now "subclassing" a protocol is acomplished by passing it a parent. subprotocol = Protocol(parentprotocol) This is what I'm using in pprint and so far it seems to be working well. I think that this may be relevant to the stuff that Nick just posted about sane transitive adaption. I haven't fully digested that yet, but it seems that it would be simple enough to extend ChainedDict and Protocol to have multiple parents. Then combining two Protocols could be achieved by: comboprotocol = Protocol(protocol_1, protocol_2) I'll think about this more and post in the other thread if anything comes of it. You'll note that I've ditched the name parameter in the Protocol constructor. It seemed nice to be able to have some minimal information about the protocol available by introspection, but in practice it seemed more and more like cruft. I think the correct spelling of a protocol with a specific name may be: class MyProtocol(Protocol): pass myprotocol = MyProtocol() print myprotocol => regards, -tim From pje at telecommunity.com Thu Apr 6 18:32:12 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Apr 2006 09:32:12 -0700 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: References: Message-ID: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> At 08:51 AM 4/6/2006, Aahz wrote: >I cringe every time I see the phrase "generic function". It's way too >generic -- it doesn't contain any hint of its meaning, And worse - "generic" in US English often implies: * commodity * ordinary * cheap knock-off * low-quality etc. Not the sort of ideas to associate with Python. :) >and I find it >confusing because Python already has what I think of as generic >functions: > >def adder(a, b): > return a+b > >That's a perfectly good generic function that depends on Python's >duck-typing. "Multimethods" works, but Python is in many ways >function-oriented rather than method-oriented. Why not call them >"multifunctions"? As long as we're inventing our own terminology, I vote for "extensible functions", as this emphasizes the benefit rather than the technology. Without extensibility, there's nothing really special about them. We wouldn't even be talking about them if you had to define all of their "methods" in one place. We also need a term for their methods, as this gets confusing when talking about a gener^H^H^H^H^Hextensible function that's being used as an object method. I don't have as unequivocal a suggestion for that, but lean towards "extensions". So, you would "extend" an extensible function by adding an extension: @extensible def some_function(...): """blah""" @some_function.extend(...) def other_thing(...): """bah""" Although, depending on how extensibility is implemented, I'd suggest that in Py3K there should be no need for @extensible; functions could simply be extensible by nature, in which case, this would be sufficient: def some_function(...): """blah""" @extends(some_function, ...) def other_thing(...): """bah""" Note that @extends() itself could be a function that is extensible, which could open the door to alternative means of doing the extending, so that people could have add-on libraries providing more advanced forms of extension that what the language itself might provide. (e.g. RuleDispatch's level of customizability) I understand that Ruby 2 is probably going to add before/after/around decorators that basically provide a form of @extends that allows you to monkeypatch an existing method. If that's all that was built in to Python 3000 (i.e., a way to monkeypatch a function without messing with func_code), then all the more sophisticated forms of generic functions could be implemented on top of that. And for really simple situations, like where you want to just add *one* more extension to an almost perfect function, that monkeypatching ability alone would work great, especially if it could be done in such a way that invoking the previous definition of the function didn't create function call overhead. As soon as you add a few levels of function call, RuleDispatch's dispatch tree starts looking speedy by comparison. Anyway, syntactically, that would probably look like: @extends(some_function) def some_extension(...): if some_condition: ... else: next_method(...) And then other people could make fancier versions of @extends that put the conditional information into the decorator arguments instead of the function body. From pje at telecommunity.com Thu Apr 6 18:44:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Apr 2006 09:44:38 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: Message-ID: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> At 08:51 AM 4/6/2006, Nick Coghlan wrote: >There is, however, an alternative, which would be to include a mechanism for >telling a protocol about other equivalent protocols, and updating the >registration mechanism to distribute any registration updates to the >equivalent protocols. I don't want to discourage people from working out their own ideas, but a lot of the stuff that's being discussed here about protocol adaptation is already implemented in PyProtocols. For example, what you just described is called "implication" in PyProtocols: one protocol can "imply" another by way of an adapter, and in the equivalence case, the adapter is a special adapter called NO_ADAPTER_NEEDED. So, your concept of equivalence is a subset of PyProtocols' concept of "implication". Indeed, many other proposals that have been made here regarding protocol objects as registries, callable protools that adapt to themselves, etc. are all already implemented in PyProtocols, often in ways not too different from the sample code that people have posted with their proposals, except that PyProtocols handles a lot of edge cases and quirks (e.g. classic class issues, weak references, metaclass level issues, etc.) that you'd only know about from actually trying to use them. Sadly, I believe one of the main reasons PyProtocols didn't get much uptake is simply that I wrote the documentation with *this* audience in mind: that is, people who want to develop a protocol/adaptation system. Thus, I documented the metasystem and tradeoffs in the design of such a system, rather than putting much emphasis on the uses of adaptation. So the PyProtocols docs suck -- unless you're trying to design or develop an adaptation system, that is. :) From guido at python.org Thu Apr 6 18:51:36 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Apr 2006 09:51:36 -0700 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <20060406144543.GC23101@panix.com> References: <20060406144543.GC23101@panix.com> Message-ID: On 4/6/06, Aahz wrote: > I cringe every time I see the phrase "generic function". It's way too > generic -- it doesn't contain any hint of its meaning, and I find it > confusing because Python already has what I think of as generic > functions: > > def adder(a, b): > return a+b > > That's a perfectly good generic function that depends on Python's > duck-typing. No, the term for this is polymorphism. I don't like "generic" either though, because e.g. in Java (and C++?) the term "generics" is associated with parameterized types a la Collection. > "Multimethods" works, but Python is in many ways > function-oriented rather than method-oriented. It's also confusing because these things don't directly relate to the methods we already have. (There's a way to view it differently, of course, but it's not a simple transformation from a method to a multimethod.) > Why not call them "multifunctions"? What about overloaded functions, which is the term used in C++ and Java for a similar mechanism (that happens at compile-time -- but it's standard Python practice to move things to run-time). We could then also talk about overloaded methods when we're using the mechanism inside a class. Phillip Eby proposes "extensible"; but that's almost as meaningless as "generic"; it has so many meanings already. "Function overloading" is a pretty well-defined technical term with a narrow meaning that applies well to this concept. We already use the term overloading for operator overloading; I don't think it's going to cause confusion though; C++ has the same double use of the terminology. (As does Java, in the sense that it's commonly know that "Java does not have operator overloading." :-) I'll try to go with this for a while and see how it feels. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Thu Apr 6 18:59:57 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Apr 2006 09:59:57 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: Message-ID: <7.0.1.0.0.20060406094516.0214a760@telecommunity.com> At 08:51 AM 4/6/2006, Ian Bicking wrote: >So... in theory you could register strings, and thus implement >RuleDispatches behavior (where those strings represent expressions). >Or, ideally, py3k will have something better than strings to represent >unevaluated expressions. Though there's lots of optimizations in >RuleDispatch that would be lost when abstracted in this way. I wouldn't worry about that. Give me an AST and the ability to edit and compile it, and I'll move the world. :) Really, if functions had a writable 'func_ast' attribute, RuleDispatch could be a *lot* simpler than it currently is. Even the syntax would be better, since when() could take a lambda function, and I thus wouldn't have to do string parsing or have the mini-interpreter. RuleDispatch could just manipulate the ASTs. My plans for future versions of RuleDispatch had called for using bytecode generation in order to produce more specialized dispatch patterns based on the configuration and use of a particular function; it would be *so* nice not to have to do it with bytecode. In fact, the only other dream feature I'd ask for to make RuleDispatch complete would be an AST abstraction for a hash table lookup; i.e., an optimized form of: if x==int: ... elif x==str: ... That is, I'd like to be able to put a structure like the above in a while loop that's walking something's mro, and have it be as efficient as doing a dictionary lookup that just branched within the function instead of incurring a function call overhead to do a table-based dispatch. With that ability (which could also just be that setting func_ast to such a dispatch tree automatically optimized the if-elifs to a "computed goto"), RuleDispatch wouldn't have to create a special "dispatch tree" structure except during compilation -- the AST would be the runtime dispatch tree. And, Psyco-style, it could leave some branches of the AST unexpressed except for a stub that requests that branch be built out at runtime. In other words, delayed specialization for unneeded branches. From ianb at colorstudy.com Thu Apr 6 19:07:32 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 06 Apr 2006 12:07:32 -0500 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> Message-ID: <44354AD4.90801@colorstudy.com> Phillip J. Eby wrote: > As long as we're inventing our own terminology, I vote for > "extensible functions", as this emphasizes the benefit rather than > the technology. Without extensibility, there's nothing really > special about them. We wouldn't even be talking about them if you > had to define all of their "methods" in one place. Extensible is kind of vague. Dispatching functions? The idea is that you have a kind of central function, but there are several actual implementations, one of which is used. There's also the thing-that-gets-dispatched-to. Calling these "methods" is probably a bad idea, because we already have dispatching methods in addition to dispatching functions. Some ideas: * Specific (specified?) functions * Dispatching implementations * Just "implementation" * Concrete functions or concrete implementations I think "extensible" or other extend-ish terms imply a purpose, not a mechanism. "Multimethod" or "multifunction" refers to dispatching on multiple arguments based on type (where current class dispatch is done by dispatching only on the type of the first argument). Generic functions can do more than type dispatching, and anyway few people think of methods as single-argument type dispatching, so the referring to multi-argument type dispatching as simply "multi" is pretty vague. Guido suggests "overloaded functions", which also seems pretty reasonable to me. "Overloaded" implies that there's more than one implementation for this thing -- in part it implies this because of the C++ terminology, but also creates images of a function as a container, which is a useful and accurate way of thinking of this. What to call the implementations is still an open issue then. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Apr 6 19:16:31 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 06 Apr 2006 12:16:31 -0500 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <7.0.1.0.0.20060406094516.0214a760@telecommunity.com> References: <7.0.1.0.0.20060406094516.0214a760@telecommunity.com> Message-ID: <44354CEF.2010205@colorstudy.com> Phillip J. Eby wrote: > At 08:51 AM 4/6/2006, Ian Bicking wrote: > >> So... in theory you could register strings, and thus implement >> RuleDispatches behavior (where those strings represent expressions). >> Or, ideally, py3k will have something better than strings to represent >> unevaluated expressions. Though there's lots of optimizations in >> RuleDispatch that would be lost when abstracted in this way. > > > I wouldn't worry about that. Give me an AST and the ability to edit and > compile it, and I'll move the world. :) I was thinking about how RuleDispatch might extend built-in type-based generic functions, and I don't think you'd have the opportunity to do the same optimizations. That is, instead of RuleDispatch providing an alternate decorator, it would simply provide an alternate kind of predicate. My understanding of RuleDispatch is that given two predictaes "foo(a) > 10" and "foo(a) < 20", you calculate "foo(a)" once and apply both comparisons on that value. And that you also combine predicates to create a decision tree. However, if all RuleDispatch can do is give opaque boolean-producing functions for its predicates, the generic function machinery would have to scan through all predicates to determine what matched and if there was any ambiguity (except for the occasional condition you wouldn't have to test because of a dominance relationship). > Really, if functions had a writable 'func_ast' attribute, RuleDispatch > could be a *lot* simpler than it currently is. Even the syntax would be > better, since when() could take a lambda function, and I thus wouldn't > have to do string parsing or have the mini-interpreter. RuleDispatch > could just manipulate the ASTs. I personally would very much like a general syntax and associated support for unevaluated (and maybe unevaluatable) expressions, beyond just lambda; RuleDispatch is one of several projects already consuming such expressions. But that's a separate topic I was hoping to bring up once the generic function/adapter discussion died down some. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From rrr at ronadam.com Thu Apr 6 19:26:56 2006 From: rrr at ronadam.com (Ron Adam) Date: Thu, 06 Apr 2006 12:26:56 -0500 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <4435261D.1000602@iinet.net.au> References: <4435261D.1000602@iinet.net.au> Message-ID: > But I figured it was something worth throwing out there :) > > Cheers, > Nick. I'm finding most of this discussion very interesting although the fine details haven't clicked for me just yet. This reminds me of Guido's multi-method blog entry a while back. In both the cases being discussed, here as well as in Guido's multi-method blog, I find that the concept of logical equivalence seems to be very important. Would implementing a logical equivalence function or operator simplify the problem any? (not simple in it self) A <=> B -> True if A->B; B->C; and A==C Or would determining logical equivalence be a use case for an adapter registry or general-function look-up method? It seems so, because there must first be a path from B to A and A to B. But that doesn't necessarily mean they are logically equivalent. So I would think adapters are a subset of converters, where adapted objects are logically equivalent to each other, while converted objects may not be. Just a few thoughts of which I'm not sure how relevant they may be. I'll go back to reading and learning now. ;-) Cheers, Ron From pje at telecommunity.com Thu Apr 6 19:34:13 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Apr 2006 10:34:13 -0700 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <44354AD4.90801@colorstudy.com> References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> Message-ID: <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> At 10:07 AM 4/6/2006, Ian Bicking wrote: >I think "extensible" or other extend-ish terms imply a purpose, not >a mechanism. Which is why it's a good thing, IMO. >"Multimethod" or "multifunction" refers to dispatching on multiple >arguments based on type (where current class dispatch is done by >dispatching only on the type of the first argument). Actually, IIRC, "multi function" is actually a distributed programming term and means something else entirely. Also, it's a really lousy word for Googling. :) >Guido suggests "overloaded functions", which also seems pretty >reasonable to me. "Overloaded" implies that there's more than one >implementation for this thing -- in part it implies this because of >the C++ terminology, but also creates images of a function as a >container, which is a useful and accurate way of thinking of >this. What to call the implementations is still an open issue then. I have two problems with "overload": 1) You have to already be a programmer to make any sense of the term... 2) ...and in any place you've seen it before, the overloading is *static* and completely unlike this kind. In Java and C++, any overloading that takes place has to be known statically to the compiler. Not only does it have to know all the *definitions* involved in the overloading, it also statically determines what will be called at the *call sites*. That is, for an overloaded function F, the Java or C++ compiler knows for every call of F(), what overloaded version of F will be invoked. Runtime types have *zero* to do with it. So overloading and extensible functions are different in two ways: 1. New overloads are added at runtime 2. Implementation selection occurs at runtime It's possible I'm overreacting here, of course. Maybe a typical Java or C++ programmer, upon being exposed to this kind of overloading, will shout "hurrah! That's what I really wanted anyway". In fact, I'm rather sure they will. What I'm really more concerned about is that they won't *know* that it does that because of "verbal overshadowing" - the term being so strongly associated in their minds with another concept that it will be hard to get them to notice. From an educational POV, it seems to me better to use a term that is either completely new (which you can then say is a kind of "dynamic overloading"), or else borrow a term from a language that actually has the same thing. For example, many functional languages have "pattern matching", which is similar to this in that implementation selection occurs at runtime. The part that's different is that pattern-matching languages generally require you to define all the patterns in one place, with no extensibility. And so "extensible pattern matching" could work, except it's too bloody long, especially as an adjective for "function"! So, when it comes right down to it, the only existing terminology used in other languages that precisely matches this thing is "generic function". And in languages where they're called generic functions, the things in them are called methods. I think that Python has a reasonable chance to invent a new term for these things and make it stick, but I don't think that overloading "overloading" with even more meanings than it already has is a good idea. :) From guido at python.org Thu Apr 6 19:56:53 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Apr 2006 10:56:53 -0700 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> Message-ID: On 4/6/06, Phillip J. Eby wrote: > I have two problems with "overload": > > 1) You have to already be a programmer to make any sense of the term... You gotta be kidding. This wouldn't be true for other terms like class or method? I really don't care about non-programmers "getting" this the first time they hear about it; especially not such a relatively advanced feature (I don't expect this to show up in the CS1 textbook about programming by John Zelle, for example, until perhaps the last chapter whose main purpose it is to whet your appetite for CS2...). > 2) ...and in any place you've seen it before, the overloading is > *static* and completely unlike this kind. And as I said, Python moves stuff to run-time all the time. If you don't have a better argument against "overloaded", then "overloaded" it is. > So overloading and extensible functions are different in two ways: > > 1. New overloads are added at runtime > 2. Implementation selection occurs at runtime > > It's possible I'm overreacting here, of course. Maybe a typical Java > or C++ programmer, upon being exposed to this kind of overloading, > will shout "hurrah! That's what I really wanted anyway". In fact, > I'm rather sure they will. What I'm really more concerned about is > that they won't *know* that it does that because of "verbal > overshadowing" - the term being so strongly associated in their minds > with another concept that it will be hard to get them to > notice. Well, the same is true for concepts like "class", "import", and "variable". Tese all have dynamic run-time meanings while they are compile-time concepts in Java/C++. > From an educational POV, it seems to me better to use a term > that is either completely new (which you can then say is a kind of > "dynamic overloading"), or else borrow a term from a language that > actually has the same thing. I disagree on principle -- if you were to invent a mechanism to do function overloading purely at run-time, then surely you would come up with something akin to what we're planning to do here. > For example, many functional languages have "pattern matching", which > is similar to this in that implementation selection occurs at > runtime. The part that's different is that pattern-matching > languages generally require you to define all the patterns in one > place, with no extensibility. And so "extensible pattern matching" > could work, except it's too bloody long, especially as an adjective > for "function"! > > So, when it comes right down to it, the only existing terminology > used in other languages that precisely matches this thing is "generic > function". And in languages where they're called generic functions, > the things in them are called methods. > > I think that Python has a reasonable chance to invent a new term for > these things and make it stick, but I don't think that overloading > "overloading" with even more meanings than it already has is a good idea. :) And I respectfully disagree. Python's predecessor, ABC, introduced a boatload of new terms because they were afraid the traditional terms would be misunderstood by newbies. I intentionally didn't borrow any of those terms (except list, which I gave a totally different meaning :-), going back to traditional terms. I don't want Python to add to the language except for the word "Pythonic". -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 6 20:09:29 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Apr 2006 11:09:29 -0700 Subject: [Python-3000] Removing repr In-Reply-To: References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> <20060406131333.GM12075@tigger.digitaltorque.ca> Message-ID: You are all correct. :-) The way I think of it is that str() is used by print; repr() is used by the interactive prompt. Clearly you want print 'abc' to print abc without quotes; but in the interactive prompt you want different output for the number 1 and the string '1', and that's what repr() is for. FWIW, for types that implement them the same, I'd like to see them return something from which the type is obvious, i.e. repr()-like. Just yesterday I helped someone debug an AttributeError where he was convinced that an object was a string because its repr() looked like a string; but it was a dbapi DateTime object (don't recall which implementation). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Thu Apr 6 21:16:17 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 06 Apr 2006 14:16:17 -0500 Subject: [Python-3000] Removing repr In-Reply-To: References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> <20060406131333.GM12075@tigger.digitaltorque.ca> Message-ID: <44356901.3000300@colorstudy.com> Guido van Rossum wrote: > FWIW, for types that implement them the same, I'd like to see them > return something from which the type is obvious, i.e. repr()-like. > Just yesterday I helped someone debug an AttributeError where he was > convinced that an object was a string because its repr() looked like a > string; but it was a dbapi DateTime object (don't recall which > implementation). Maybe this should go in PEP 8. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Thu Apr 6 21:30:38 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Apr 2006 12:30:38 -0700 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> Message-ID: <7.0.1.0.0.20060406121637.0218a898@telecommunity.com> At 10:56 AM 4/6/2006, Guido van Rossum wrote: >I disagree on principle -- if you were to invent a mechanism to do >function overloading purely at run-time, then surely you would come up >with something akin to what we're planning to do here. The issue is more that Java programmers think they need to use staticmethod because that sounds to them like a Java "static method", instead of doing something more appropriate like a classmethod or a module-level function. I worry that "overloading" will make them think it should be used the way it's used in Java, where it often substitutes for Java's lack of default arguments. Not only that, but overloading in Java means you have to write different implementations for different types, even in cases where duck typing would be more appropriate. It's especially confusing given Java's mix of object-based and non-object numeric types. ISTM that C++ has even worse connotations for this. So, when I say that "overload" has the wrong connotations, I'm not talking about some abstract understanding of what it is or does, I'm talking about the wrong connotations for their practical intuition of how the thing should be *used*. The use cases for overloading in the languages that have it are way different, and if you move those concepts over to generic functions without a critical reinterpretation of the concept, you will write atrocious Python that will be hard to read and perform badly. >And I respectfully disagree. Python's predecessor, ABC, introduced a >boatload of new terms because they were afraid the traditional terms >would be misunderstood by newbies. I intentionally didn't borrow any >of those terms (except list, which I gave a totally different meaning >:-), going back to traditional terms. I don't want Python to add to >the language except for the word "Pythonic". I agree 100% that it doesn't matter what we call it for newbies. My point is that it would be much better to *not* have the term's familiarity get in the way for the *experienced* people. I want a term that we can say is "like overloading, but dynamic", rather than just saying it's overloading. Because otherwise it's like that psychology experiment where they print the names of colors in colors other than the name, and ask people to read them fast. If you see the word "brown" and it's actually printed in blue, it's going to take you longer to figure out what it says. :) And this happens now with things like 'staticmethod', when people blindly apply their previous intuitions from other languages. Anyway, that's my best and only argument, so if you don't agree, I'll support "overload". I'll just try to stick in "dynamic" wherever possible to help reforming C++ and Java programmers out a bit. :) From tim.hochberg at ieee.org Thu Apr 6 21:36:04 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 06 Apr 2006 12:36:04 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <4435261D.1000602@iinet.net.au> References: <4435261D.1000602@iinet.net.au> Message-ID: <44356DA4.5050503@ieee.org> Nick Coghlan wrote: > One issue with generic functions and adaptation as currently being discussed > (and something Tim mentioned a while back), it that it is very focused on > dispatching based solely on obj.__class__.__mro__. > > That's all well and good, but (to use an example of Tim's), suppose we have a > couple of frameworks we're using, where those frameworks have each defined > their own version of a "wibbly-wobbly" object: > > frmwrk_A.IWibble > frmwrk_B.IWobble > > The methods and signatures included in these protocols may be the same, > IWibble may be a subset of IWobble, or vice versa. > > Both frameworks have defined their interfaces (input and output) in terms of > these protocols, rather than in terms of concrete types. > > As Tim pointed out, pure type-based dispatch would require that every type be > registered with both protocols, even if the two protocols are identical. Doing > that manually would be a serious pain if the frameworks were non-trivial. > > The first thought may be to allow adaptation between protocols - but the issue > with that is that at adaptation time, we only have the object to work with, > and won't know what protocols it claims to implement (this is where either > __conform__ or a global registry comes in with PEP 246). > > Even if we did know, the cost of doing a search on all equivalent protocols on > each call would add up. We don't really want to be finding ways to make > functions calls even slower than usual ;) > > There is, however, an alternative, which would be to include a mechanism for > telling a protocol about other equivalent protocols, and updating the > registration mechanism to distribute any registration updates to the > equivalent protocols. > [Implementation snipped] In the "Adaptation vs. Generic Functions" thread I proposed that the registry be implemented using a ChainedDict that searched it's parent for any keys that it didn't have itself. This allows extending existing protocols using: extended_protocol = Protocol(base_protocol) [Note that the name parameter has gone away. I'll include the Protocol and ChainedDict implementation at the end of this post.] It's a relatively short step from here to allowing the extending of multiple protocols. And a relatively short step from there to an alternative implementation of register_subprotocol. > > Using such a mechanism, the interface mismatch above could be addressed in one > of the following ways: > > Suppose framework A specifies fewer methods than framework B, but those > methods match. Then you can write: > > frmwrk_B.IWobble.register_subprotocol(frmwrk_A.IWibble) > # Now registration for framework B also registers you for the narrower > # interface in framework A Given the above, if framework A is created with knowledge of framework B, we can use: # in frmwrk_A IWibble = Protocol(frmwrk_B.I_Wobble) If instead we want to make IWibble a subprotocol of IWobble after the fact, we can instead do: frmwrk_B.IWobble.registry.parents.append(frmwrk_A.IWibble.registry) This may need some sugar to make it more palatable. For instance, one could have still have register_subprotocol as a method, or perhaps a helper function, that implemented the above. I'll try add_parent and see how that looks: class Protocol: #... def add_parent(self, protocol): self.registry.parents.append(protocol.registr) Then we can dynamically make IWibble a subprotocol of IWobble using: frmwrk_B.IWobble.add_parent(frmwrk_A.IWibble) > > You can turn that around, if A is the one that is more prescriptive: > > frmwrk_A.IWibble.register_subprotocol(frmwrk_B.IWobble) > # Now it is registration for framework A that registers you for the narrower > # interface in framework B > > And finally, if the two interfaces are identical: > > frmwrk_A.IWibble.register_subprotocol(frmwrk_B.IWobble) > frmwrk_B.IWobble.register_subprotocol(frmwrk_A.IWibble) > # Now registration for either framework registers you for both ChainedDict is careful not to loop, so this case can be spelled as: frmwrk_A.IWibble.registry.parents.append(frmwrk_A.IWobble.registry) frmwrk_B.IWobble.registry.parents.append(frmwrk_B.IWibble.registry) or perhaps more readably as: frmwrk_B.IWobble.add_parent(frmwrk_A.IWibble) frmwrk_A.IWibble.add_parent(frmwrk_B.IWobble) > Now, suppose, however, that mapping from A to B required a slight tweak to A's > interface - one of the method signatures didn't line up right (e.g. one of A's > methods has the wrong name). This can be handled with an explicit protocol > adapter, which would provide a modified update method like so: > > class ProtocolAdapter(object): > def __init__(self, src, target): > self.target = target > src.register_subprotocol(self) > > def _update(self, updates, updated=None): > if self.target in updated: > return > wrapped_updates = {} > for signature, adapter in updates.iteritems(): > def wrapped_adapter(*args, **kwds): > return self.adapt(adapter(*args, **kwds)) > wrapped_updates[signature] = wrapped_adapter > self.target._update(wrapped_updates, updated) > > def __call__(self, adapter): > self.adapt = adapter > > class AdaptWibbletoWobble(object): > def __init__(self, obj): > self.obj = obj > def __iter__(x, y, z): > return self.obj.method2(x, y, z) > > ProtocolAdapter(frmwork_A.IWibble, frmwrk_B.IWobble)(AdaptWibbleToWobble) Clever. The naming makes that last statement pretty opaque until you squint at the code for a while though. I don't think this would be a problem to layer on top of chained dict if needed. I think something like the following would work: WrappedDict(object): def __init__(self, obj, wrapper): self._obj = obj self._wrapper = wrapper def get(self, key, default=None): try: return self[key] except KeyError: return default def __getitem___(self, key): return self._wrapper(self._obj[key]) def __contains__(self, key): return key in self._obj class Protocol: #.... def add_parent(self, protocol, wrapper=None): registry = protocol.registry if wrapper is not None: registry = WrappedDict(registry) self.registry.parents.append(protocol.registry) I'm not sure this stuff needs to be methods -- it might be better relegated to helper functions to keep the core Protocol object simple. > The equivalent of the above for generic functions is the case where "generic > function A" does a very similar thing to "generic function B", and you want to > be able to do a wholesale delegation from A to B of everything that B handles > more specifically than A. > > Aside from the fact that I think any such transitivity mechanism should be > observer based, and that the updates should happen at registration time rather > than lookup time, I'm not really wedded to any of the implementation details > above. But I figured it was something worth throwing out there :) I can see why you would want to this for performance reasons. However, the implementation crosses the threshold into too complex in my mind. In a sense, I've just pushed the complexity off into ChainedDict, but it's easier for me to digest in smaller chunks. I can also see some caching strategies that would make ChainedDict as fast as the observer strategy as long as we don't need to worry too much about people removing keys from the registries. Regards, -tim ====================================================== _missing = object() class ChainedDict(dict): """A dict like object that forwards item requests to its parent if neeeded.""" def __init__(self, *parents): dict.__init__(self) self.parents = list(parents) def __contains__(self, key): return self.get(key, _missing) is not _missing def __getitem__(self, key): x = self.get(key, _missing) if x is _missing: raise KeyError('not found') return x def _get(self, key, visited): # Get a value from self or parents. Return _missing on failure # visited contains the ids of objects searched so far. myid = id(self) if myid in visited: return _missing visited.add(myid) x = dict.get(self, key, _missing) if x is not _missing: return x for p in self.parents: if isinstance(p, ChainedDict): x = p._get(key, visited) else: x = p.get(key, _missing) if x is not _missing: return x return _missing def get(self, key, default=None): x = self._get(key, set()) if x is _missing: return default return x def null_adapter(*args): """Adapter used when adaptation isn't actually needed""" if len(args) > 1: return args else: return args[0] class Protocol(object): """Declare a protocol object that subclasses parents if provided""" def __init__(self, *parents): self.registry = ChainedDict(*(x.registry for x in parents)) def register(self, *args): """Function decorator to register as an adapter for given keys""" if len(args) == 1: args = args[0] def helper(adapter): if adapter is None: adapter = null_adapter self.registry[args] = adapter return adapter return helper def signatures(self, *args): """Find signatures for given call arguments""" # Default behaviour dispatches on the type of the first argument if len(args) != 1: raise TypeError("%s expected 1 argument, got %s" % (self, len(args))) return type(args[0]).__mro__ def default_adapter(self, *args): """Call result when no adapter was found""" raise TypeError("Can't adapt <%s> to %s" % (', '.join(x.__class__.__name__ for x in args), self.__class__.__name__)) def __call__(self, *args): """Adapt supplied arguments to this protocol""" for key in self.signatures(*args): adapter = self.registry.get(key, _missing) if adapter is not _missing: return adapter(*args) return self.default_adapter(*args) From robey at lag.net Thu Apr 6 21:15:15 2006 From: robey at lag.net (Robey Pointer) Date: Thu, 6 Apr 2006 12:15:15 -0700 Subject: [Python-3000] Removing repr In-Reply-To: References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> <20060406131333.GM12075@tigger.digitaltorque.ca> Message-ID: On 6 Apr 2006, at 7:02, John J Lee wrote: > On Thu, 6 Apr 2006, Michael P. Soulier wrote: > [...] >> I thought that the distinction was as follows. >> >> str() should return a human-readable representation of something. >> >> repr() should return a machine-readable representation of >> something, suitable >> for an exec command. > [...] > > Is this old chestnut really still around? repr() is for debugging; it > does not have to return an eval()-able string. The __repr__ docs > explain > that well. Yes, I find this to be one of the advantages python has over java: In python, coercion to string (str) is separate from providing a description for debugging (repr). Until your email, I was starting to think I was unusual in this interpretation. robey From guido at python.org Thu Apr 6 22:33:46 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Apr 2006 13:33:46 -0700 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <7.0.1.0.0.20060406121637.0218a898@telecommunity.com> References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> <7.0.1.0.0.20060406121637.0218a898@telecommunity.com> Message-ID: On 4/6/06, Phillip J. Eby wrote: > The issue is more that Java programmers think they need to use > staticmethod because that sounds to them like a Java "static method", > instead of doing something more appropriate like a classmethod or a > module-level function. I worry that "overloading" will make them > think it should be used the way it's used in Java, where it often > substitutes for Java's lack of default arguments. Not only that, but > overloading in Java means you have to write different implementations > for different types, even in cases where duck typing would be more > appropriate. It's especially confusing given Java's mix of > object-based and non-object numeric types. ISTM that C++ has even > worse connotations for this. But at least C++ has default argument values. I don't believe that Java programmers using Python will attempt to write multiple overloaded versions of a method for different types that have the same API (duck typing). If they do, they have more serious problems. > So, when I say that "overload" has the wrong connotations, I'm not > talking about some abstract understanding of what it is or does, I'm > talking about the wrong connotations for their practical intuition of > how the thing should be *used*. The use cases for overloading in the > languages that have it are way different, and if you move those > concepts over to generic functions without a critical > reinterpretation of the concept, you will write atrocious Python that > will be hard to read and perform badly. I think that the solution to this would be a book or course "Python for Java programmers". Not using the most natural name for a feature because Java programmers might misunderstand it sound like the wrong motivation. Plus, the syntax for using overloaded functions/methods is sufficiently verbose that the elegance of using default argument values is hard to miss. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 6 23:15:10 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Apr 2006 14:15:10 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> Message-ID: On 4/6/06, Phillip J. Eby wrote: > I don't want to discourage people from working out their own ideas, > but a lot of the stuff that's being discussed here about protocol > adaptation is already implemented in PyProtocols. That's great. I believe that if we derive things from first principles here and arrive at the same choices as PyProtocols, we haven't wasted anybody's time; rather, we've (a) educated everyone who participated a whole lot on how to build a good dynamic function overloading package; (b) validated the choices you made in PyProtocols; (c) explained a lot of the warts in PyProtocols. That to me seems an invaluable contribution, and I expect it's easier to do it this way than to try and understand PyProtocols by attempting to understand its code and docs. Lots of people learn by doing rather than by seeing. I'm one of them. For example, I still owe you an update to the examples in PEP 342. One reason I haven't done this is that after starting from first principles, after a few days I arrived at something very similar to your trampoline example. For me, the *only* way to understand the ready-made trampoline code that you aput in the PEP was to try to create a similar thing from scratch, and evolve it until it had the same functionality. I still have the 3 or 4 evolutionary steps I made in a directory on an old laptop. Without the first few steps I wasn't able to understand the rationale behind the design of the completed example. I expect that many others have the same problem with that example. (Hands please; who here understands example 3 in PEP 342?) At the moment I'm doing the same for overloadable functions. However, instead of doing it in the privacy of a hotel room in a temp directory on a disconnected laptop, I'll blog about it. There'll be a lot of thinking aloud, a couple of dead alleys and red herrings, and every once in a while I'll have to buy a clue from PyProtocols. But I really see no other way to do it; when I look at the completed PyProtocols my eyes just start glazing over. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Thu Apr 6 23:56:22 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 6 Apr 2006 17:56:22 -0400 Subject: [Python-3000] Removing repr In-Reply-To: References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> <20060406131333.GM12075@tigger.digitaltorque.ca> Message-ID: On 4/6/06, John J Lee wrote: [suggests changing the docs for repr for Py3k] The current help(repr) may be even more misleading. I would suggest making the change as early as 2.4 (2.5 for the docstring. Changes: (1) It doesn't mention backquotes. They are already deprecated, so it makes sense to mention the equivalency in the backquote docs, rather than the repr docs. (2) Saying "for debugging" instead of "canonical" makes it less of an attractive nuisance. (canonical can mean "the one right way") (3) Giving the < > example along with -- or even before -- the eval makes it clear that you shouldn't rely on the eval loop in the general case. These are good changes, even from a 2.x perspective. As to the exact wording -- what are the current guarantees on the type of repr's return. Can it be unicode? If we don't want to explicitly ban unicode, I would modify your suggestion (even today) to """repr(object) -> string Return a printable representation of an object, useful for debugging. For simple objects, repr may the inverse of eval, so that eval(repr(obj)) == obj. More complex objects typically use '<...useful info ...>' such as '<__main__.C instance at 0x00B818C8>'. """ -jJ From solipsis at pitrou.net Fri Apr 7 00:52:44 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 07 Apr 2006 00:52:44 +0200 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> <7.0.1.0.0.20060406121637.0218a898@telecommunity.com> Message-ID: <1144363964.5672.71.camel@fsol> Le jeudi 06 avril 2006 ? 13:33 -0700, Guido van Rossum a ?crit : > I think that the solution to this would be a book or course "Python > for Java programmers". Not using the most natural name for a feature > because Java programmers might misunderstand it sound like the wrong > motivation. > > Plus, the syntax for using overloaded functions/methods is > sufficiently verbose that the elegance of using default argument > values is hard to miss. But overloaded methods as they exist now have exactly the same expression as non-overloaded methods, all the machinery (mro walking etc.) being hidden from the programmer. On the contrary, this new "overloaded function" mechanism requires the explicit use of a special decorator to specify that the function can be overloaded, and another special decorator to specify that we are currently overloading a function. Thus you would be referring to two different programming schemes with the same word. It's confusing IMO. From pje at telecommunity.com Fri Apr 7 01:31:10 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Apr 2006 16:31:10 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> Message-ID: <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> At 02:15 PM 4/6/2006, Guido van Rossum wrote: >On 4/6/06, Phillip J. Eby wrote: > > I don't want to discourage people from working out their own ideas, > > but a lot of the stuff that's being discussed here about protocol > > adaptation is already implemented in PyProtocols. > >That's great. I believe that if we derive things from first principles >here and arrive at the same choices as PyProtocols, we haven't wasted >anybody's time; I agree; that's why I said "I don't want to discourage people from working out ... ideas". I just wanted to point out that for those who prefer to learn from worked-out examples, there are some available. One reason I like to steal designs from *built* systems is that making something real forces lots of tradeoffs into the open, that don't come up when you're just brainstorming. That having been said, please consider this a "spoiler warning": the rest of this message reveals where I think you're all going to eventually end up. If it'll spoil anyone's fun and/or learning to know this ahead of time, don't read the rest of this message. :) Hint: premature optimization is the root of all evil. I was actually trying to hint in my previous message that it would be valuable to learn from PyProtocols' example by *not* repeating it! I was not suggesting that PyProtocols is a model to which you should all aspire, but rather consider why I wish I hadn't bothered to do it in the first place! >At the moment I'm doing the same for overloadable functions. However, >instead of doing it in the privacy of a hotel room in a temp directory >on a disconnected laptop, I'll blog about it. There'll be a lot of >thinking aloud, a couple of dead alleys and red herrings, and every >once in a while I'll have to buy a clue from PyProtocols. But I really >see no other way to do it; when I look at the completed PyProtocols my >eyes just start glazing over. No criticism of your methods was intended or implied; it's just that I personally hate to struggle to reinvent something that's already been done. More specifically, I wish somebody could've pointed *me* to PyProtocols' docs before I wrote them, or better yet, sat me down and showed me why generic, uh, overloadable functions were an even better idea and sold me on why I shouldn't bother writing PyProtocols, but jump straight to the good stuff. :) This is particularly so in the area of multiple-argument dispatch, as some people have been bringing up implementation ideas that I see as painful dead-ends that took me months to chew my way out of. So I've actually been doing an *excellent* job of sitting on my hands so far and not stealing anybody's learning opportunities. :) However, continuing in the vein of "things I wished somebody told me", I would say that the biggest hurdle to implementing a good multiple-argument dispatch system, IMO, is realizing that there is only one basic algorithm that works, and that there are no shortcuts. You can find lots of ways to make the basic algorithm efficient (indexing, dispatch trees, partial evaluation, etc.), but there are really no shortcuts. The good news, however, is that once you realize there is only one basic algorithm (eliminate non-matching cases and prioritize what's left), you find that there are a lot of implementation shortcuts, especially if you have single-dispatch functions available to build the multiple-dispatch implementation. You just can't directly shoehorn multiple-dispatch into single-dispatch, although you can waste a *lot* of time trying. I literally could've had RuleDispatch done maybe 2 years sooner if I took all the time I spent thinking about how to figure out something "easier" to implement than the Chambers&Chen optimization of the One True Algorithm, and just worked on implementing it instead. Anyway, the point of my comment was not to interfere with anybody's learning, it was just to mention stuff that I would have wished to have, when I was trying to work these things out. I'll go back to sitting on my hands now, except for occasional drooling at the idea of a writable func_ast attribute. :) Actually, I like the func_ast idea so much that it tempts me to write something that would emulate it, like a pair of set_ast and get_ast functions for Python 2.x. (I could use PyPy's abstract interpretation approach to turn bytecode back into a pseudo-AST, perhaps.) It would be useful as a proof-of-concept to demonstrate how simple it would be to implement trivial overloading and then full multiple/predicate dispatch using AST manipulation, because all this mucking about with dictionaries is way too forest-for-the-trees if your goal is to understand the *applications* for overloading. Overloadable functions are easiest to understand first (IMO) in terms of syntactic transforms. After you've got a grasp at that level, you can come up with other optimizations. Better still, with the new AST branch and PyPy, we now have a better chance of just optimizing the ASTs (or compiling them to C) directly, rather than painfully working out a bunch of low-level MRO and registry issues. Those are costly premature optimizations, and I wish I had not been sucked into that trap. If I had it to do over, I would proceed directly to AST manipulation, and then work on ways to make it faster (like the Chambers & Chen-inspired dispatch indexes, and by moving the AST manipulations to C after they were tested). So, I would gently suggest that, now that we have overloaded functions on the table, that it might be better to focus on "what should construct X translate to in naive Python code", than worrying about implementation feasibility of optimized code. One of the interesting findings of Chambers and Chen was that a significant number of overloadable functions "in the wild" have a very small number of methods, and then there are a few that have gobs and gobs of methods. My focus on premature optimization thus blinded me to the fact that the most efficient way to deal with those common cases is just to translate it to a simple set of of if-then statements. The cases where you need big guns like registries are much fewer and further between. Indeed, if you examine PEAK, Zope, or Twisted, you will also find that most interfaces have only a handful of adapter implementations (or direct method implementations in classes), and then there are some that have zillions. And -- here's the key bit -- if you have a tool that lets you handle those common cases (of only a few distinct methods), you can then *use* them as a stepping stone to implement the more complex approaches. RuleDispatch makes extensive use of simple single-dispatch functions to bootstrap the multiple-dispatch implementation. If I had a multiple dispatch implementation whose only drawback was that it was slow if you had more than a handful of overloads for the function, then creating an *efficient* multi-dispatch system on top of that is a piece of cake, especially if you make the "overloading function" itself overloadable. That is, once the system is bootstrapped, you just overload "overload()" to be able to use the more efficient implementation approaches whenever the number of overloads reaches a certain threshold. (Analogy: 'type' is its own type in today's Python, but it's built on a simpler C-level notion of types as a data structure with function pointers that it needs in order to be bootstrapped. In the same way, you can bootstrap overload() using an extremely simple form of overloading, not far removed from monkeypatching.) The reason I haven't done much on documenting RuleDispatch's internals to date is because I basically got to a point where the implementation I have, although relatively efficient for large overload sets, is not as efficient as the naive approach (unoptimized if-then trees) for the common cases, and isn't as customizable or optimized as I'd like for the "zillions of methods" cases. Its internals are hard-wired for things in a way that makes it hard to add more special-case optimizations. Thus, RuleDispatch is effectively optimized for an excluded middle where you have enough rules that the naive approach is too slow, but not so many rules that you need advanced customization features or speed that matches what can be done by hand-tuning. In other words: for use cases that don't actually exist! (This same problem exists to a lesser degree for PyProtocols.) Thus, I've pretty much decided that I need to go back and reimplement the RuleDispatch internals so that they are themselves based on generic functions, and my plan was to use a naive approach to code generation for those functions, and then "overload the overload() function". In truth, my suggestion for Py3K is that simply providing a writable func_ast and a simple overloading implementation that uses it, and is itself overloadable, seems like the simplest thing that would possibly work, and it allows unlimited implementation-level optimizations. For example, let's say that naive if-then overloading is too slow for pickle() or pprint(). So you overload "overload()" to allow overloads to pickle to modify a registry instead of modifying code. Voila! So this is why I think that worrying about protocol objects and MROs and registries is losing the forest for the trees. It's easier to think about those things when you're grounded in specific use cases. And, an important feature of dynamic overloading is that you get to change your mind about how things work - just create another overload! This is something that was totally not obvious to me before I got to a certain point of working on RuleDispatch. It took a while for my thinking to shift to a world where overloading is *cheap*. In today's Python overloading requires interfaces or registries or adaptation; it's not as simple of a thing, so you don't use it unless you know you have to. But if you can overload functions as easily as breathing, then overloadability can be the *default* case, and then you realize that any *single* implementation of a function is necessarily a premature optimization! And just as important, trying to only ever design *one* implementation of an algorithm is necessarily a wasteful compromise! You see, overloading allows you to write correct algorithms, and then overload the abstract operations for a specific implementation scenario. Almost like PyPy generating a custom VM -- you can create a new "back-end" for any given special case. So special cases aren't special enough to break the rules -- but with overloading, you can bend the rules to accomodate them. Or as Uncle Timmy put it, "1) Everything is a trivial special case of something else, and 2) death is a bunch of blue spheres." :) From guido at python.org Fri Apr 7 02:10:13 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Apr 2006 17:10:13 -0700 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <1144363964.5672.71.camel@fsol> References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> <7.0.1.0.0.20060406121637.0218a898@telecommunity.com> <1144363964.5672.71.camel@fsol> Message-ID: On 4/6/06, Antoine Pitrou wrote: > But overloaded methods as they exist now have exactly the same > expression as non-overloaded methods, all the machinery (mro walking > etc.) being hidden from the programmer. What type of overloading are you referring to? The way I think about it, Python doesn't *have* overloading. Perhaps you're thinking of overriding? Totally different beast! > On the contrary, this new "overloaded function" mechanism requires the > explicit use of a special decorator to specify that the function can be > overloaded, and another special decorator to specify that we are > currently overloading a function. Thus you would be referring to two > different programming schemes with the same word. It's confusing IMO. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Fri Apr 7 02:23:42 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Apr 2006 12:23:42 +1200 Subject: [Python-3000] Removing repr In-Reply-To: <20060406131333.GM12075@tigger.digitaltorque.ca> References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> <20060406131333.GM12075@tigger.digitaltorque.ca> Message-ID: <4435B10E.3030005@canterbury.ac.nz> Michael P. Soulier wrote: > I thought that the distinction was as follows. > > str() should return a human-readable representation of something. > > repr() should return a machine-readable representation of something, suitable > for an exec command. As Michael pointed out, this is totally impractical in the majority of cases. The guideline I use is quite different: str() is for normal program output, repr() is for debugging output. To expand a bit more: repr() should return something that unambiguously indicates the *type* of the object. If something about the value can be indicated as well without being too verbose, that's a bonus, but it's not the most important thing. I suggest that the passage in the docs about repr() output being eval-able be deleted, since it misleads more than it helps, and replaced with something to the above effect. BTW, these days I normally don't bother giving my classes a __repr__ at all, and just make do with the default . If I want to know anything more, I insert prints at the appropriate place to probe the object further. Anyway, -1 on combining them. In my view they *do* serve two very clearly distinct purposes -- just not the ones the docs say! -- Greg From greg.ewing at canterbury.ac.nz Fri Apr 7 02:30:26 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Apr 2006 12:30:26 +1200 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <20060406144543.GC23101@panix.com> References: <20060406144543.GC23101@panix.com> Message-ID: <4435B2A2.4090106@canterbury.ac.nz> Aahz wrote: > I cringe every time I see the phrase "generic function". It's way too > generic "Multiple dispatch" seems to capture the important features. -- Greg From greg.ewing at canterbury.ac.nz Fri Apr 7 02:42:39 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Apr 2006 12:42:39 +1200 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> Message-ID: <4435B57F.4040904@canterbury.ac.nz> Phillip J. Eby wrote: > So overloading and extensible functions are different in two ways: > > 1. New overloads are added at runtime > 2. Implementation selection occurs at runtime Maybe "dynamically overloaded"? -- Greg From edloper at gradient.cis.upenn.edu Fri Apr 7 02:56:04 2006 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Thu, 06 Apr 2006 20:56:04 -0400 Subject: [Python-3000] Removing repr In-Reply-To: <4435B10E.3030005@canterbury.ac.nz> References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> <20060406131333.GM12075@tigger.digitaltorque.ca> <4435B10E.3030005@canterbury.ac.nz> Message-ID: <4435B8A4.4030707@gradient.cis.upenn.edu> Greg Ewing wrote: > The guideline I use is quite different: str() is for > normal program output, repr() is for debugging > output. I think that another important difference between str() and repr() is that repr() is often used as an embedded string in other objects' representations; but str() usually is not. Thus, the string returned by repr() should not span multiple lines, and it should be easy to match up its beginning with its end. str() doesn't have these restrictions. So actually, it can sometimes be convenient to use both for debugging purposes, with repr() for a short string, suitable for embedding, and str() for a more verbose, multiline, formatted string. (Another -1 for combining them.) -Edward From guido at python.org Fri Apr 7 06:23:56 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 6 Apr 2006 21:23:56 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443459E9.50908@livinglogic.de> <4434AB26.1040909@livinglogic.de> Message-ID: #!/usr/bin/env python2.5 # overloading.py """Here's another executable email. This is an implementation of (dynamically, or run-time) overloaded functions, formerly known as generic functions or multi-methods. I actually blogged on Artima about multi-methods about a year ago; but at the time I hadn't figured out the trick of explicitly declaring the MM before registering the implementations (Phillip helpfully pointed that out in a comment); also, I was using a global registry then. This version is an improvement over my earlier attempts (both last year and at the start of this thread) because it supports subclasses in call signatures. If an implementation is registered for a signature (T1, T2), then a call with a signature (S1, S2) is acceptable, assuming that S1 is a subclass of T1, S2 a subclass of T2, and there is no ambiguity in the match (see below). I came up with an algorithm for doing this that may or may not resemble the one in PEAK's RuleDispatch. I kind of doubt that it's all that similar because RuleDispatch supports arbitrary predicates. In contrast, I'm just using the argument types for dispatch, similar to (compile-time) overloaded functions in C++ and methods in Java. I do use a concept that I overheard Phillip mention: if there are multiple matches and one of those doesn't *dominate* the others, the match is deemed ambiguous and an exception is raised. I added one refinement of my own: if, after removing the dominated matches, there are still multiple matches left, but they all map to the same function, then the match is not deemed ambiguous and that function is used. Read the method find_func() below for details. The example is a bit lame; it's not a very good pretty-printer and it only dispatches on a single argument; but it does exercise the weeding out of dominant matches. I'll try to post a complete unit test suite later. Python 2.5 is required due to the use of predicates any() and all(). """ # Make the environment more like Python 3.0 __metaclass__ = type from itertools import izip as zip class overloaded: # An implementation of overloaded functions. def __init__(self, default_func): # Decorator to declare new overloaded function. self.registry = {} self.cache = {} self.default_func = default_func def register(self, *types): # Decorator to register an implementation for a specific set of types. # .register(t1, t2)(f) is equivalent to .register_func((t1, t2), f). def helper(func): self.register_func(types, func) return func return helper def register_func(self, types, func): # Helper to register an implementation. self.registry[tuple(types)] = func self.cache = {} # Clear the cache (later we can optimize this). def __call__(self, *args): # Call the overloaded function. func = self.find_func(args) return func(*args) def find_func(self, args): # Find the appropriate overloaded function; don't call it. # NB. This won't work for old-style classes or classes without __mro__. types = tuple(type(a) for a in args) if types in self.cache: # First easy case -- direct hit in cache. return self.cache[types] if types in self.registry: # Second easy case -- direct hit in registry, update cache. self.cache[types] = func = self.registry[types] return func # I can't help myself -- this is going to be intense functional code. # Find all possible candidate signatures. mros = tuple(t.__mro__ for t in types) n = len(mros) candidates = [sig for sig in self.registry if len(sig) == n and all(t in mro for t, mro in zip(sig, mros))] if not candidates: # No match at all -- use the default function. self.cache[types] = func = self.default_func return func if len(candidates) == 1: # Unique match -- that's an easy case. self.cache[types] = func = self.registry[candidates[0]] return func # More than one match -- weed out the subordinate ones. def dominates(dom, sub, orders=tuple(dict((t, i) for i, t in enumerate(mro)) for mro in mros)): # Predicate to decide whether dom strictly dominates sub. # Strict domination is defined as domination without equality. # The arguments dom and sub are type tuples of equal length. # The orders argument is a precomputed auxiliary data structure # giving dicts of ordering information corresponding to the # positions in the type tuples. # A type d dominates a type s iff order[d] <= order[s]. # A type tuple (d1, d2, ...) dominates a type tuple of equal length # (s1, s2, ...) iff d1 dominates s1, d2 dominates s2, etc. if dom is sub: return False return all(order[d] <= order[s] for d, s, order in zip(dom, sub, orders)) # I suppose I could inline dominates() but it wouldn't get any clearer. candidates = [cand for cand in candidates if not any(dominates(dom, cand) for dom in candidates)] if len(candidates) == 1: # There's exactly one candidate left. self.cache[types] = func = self.registry[candidates[0]] return func # Perhaps these multiple candidates all have the same implementation? funcs = set(self.registry[cand] for cand in candidates) if len(funcs) == 1: self.cache[types] = func = funcs.pop() return func # No, the situation is irreducibly ambiguous. raise TypeError("ambigous call; types=%r; candidates=%r" % (types, candidates)) # Example showing how to create an overloaded function. class List(list): pass class SubList(List): pass @overloaded def pprint(obj): return repr(obj) @pprint.register(List) @pprint.register(list) def pprint_list(obj): if not obj: return "[]" s = "[" for item in obj: s += pprint(item).replace("\n", "\n ") + ",\n " return s[:-3] + "]" @pprint.register(tuple) def pprint_tuple(obj): if not obj: return "()" s = "(" for item in obj: s += pprint(item).replace("\n", "\n ") + ",\n " if len(obj) == 1: return s[:-2] + ")" return s[:-3] + ")" @pprint.register(dict) def pprint_dict(obj): if not obj: return "{}" s = "{" for key, value in obj.iteritems(): s += (pprint(key).replace("\n", "\n ") + ": " + pprint(value).replace("\n", "\n ") + ",\n ") return s[:-3] + "}" @pprint.register(set) def pprint_set(obj): if not obj: return "{/}" s = "{" for item in obj: s += pprint(item).replace("\n", "\n ") + ",\n " return s[:-3] + "}" # Example showing how to use the above overloaded function. data = ( "this is a string", [1, 2, 3, 4], ("more tuples", 1.0, 2.3, 4.5), "this is yet another string", (99,) ) print pprint(data) print pprint(List(data)) print pprint(SubList(data)) #--Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Fri Apr 7 08:31:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 06 Apr 2006 23:31:15 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: Message-ID: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> Guido wrote: > mros = tuple(t.__mro__ for t in types) > n = len(mros) > candidates = [sig for sig in self.registry > if len(sig) == n and > all(t in mro for t, mro in zip(sig, mros))] I believe this can be simplified to: n = len(types) candidates = [sig for sig in self.registry if len(sig) == n and all(issubclass(s,t) for s,t in zip(sig, types))] Similarly, I believe the dominates function can be moved out to a top-level function in similar fashion: def dominates(dom, sub): return all(d is not s and issubclass(d,s) for d,s in zip(dom, sub)) Using issubclass() is more accurate in this case than using __mro__, because __mro__ considers the order of the __bases__ to be significant, which can lead to unexpected results due to false precision. (i.e., because it'll be guessing about cases that are actually ambiguous) Unfortunately, this entire approach is subject to similar scale issues as PyProtocols/RuleDispatch. It is much slower than a few hand-written if-then's for the common cases, and it's even slower and too inflexible for the "zillions of methods" cases like Zope's "view" registry (which dispatches based on some types and a string). You could get better performance in the common cases by just creating a dominance hierarchy among all the methods, and then writing out Python code that executes a series of if-then tests with the dominant cases being tested first (and some extra branches to check for ambiguity). This code would then run at virtually the same speed as an identical hand-written function for the most common use cases (which tend towards having only a handful of overloads at most), and it would also not require any extra memory allocations for temporary lists and tuples on each invocation of the function (because all the state is effectively tracked in the program counter as it threads its way among the if-then tests). To make the same approach also work for the zillions-of-methods cases, all that would be needed is replacing some of the if-thens with dictionary lookups to allow N-way branches instead of just true/false tests, and some optimization to avoid re-testing conditions or recomputing values that have already been computed. Most of RuleDispatch's complexity stems from the fact that I effectively implemented a mini-interpreter for executing such N-way dispatch trees, instead of just generating bytecode. I made that choice because there's unfortunately no "computed goto" equivalent in today's bytecode. (Bytecode is also rather annoyingly low-level for this kind of work.) From solipsis at pitrou.net Fri Apr 7 11:14:18 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 07 Apr 2006 11:14:18 +0200 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> <7.0.1.0.0.20060406121637.0218a898@telecommunity.com> <1144363964.5672.71.camel@fsol> Message-ID: <1144401259.5722.20.camel@fsol> Le jeudi 06 avril 2006 ? 17:10 -0700, Guido van Rossum a ?crit : > On 4/6/06, Antoine Pitrou wrote: > > But overloaded methods as they exist now have exactly the same > > expression as non-overloaded methods, all the machinery (mro walking > > etc.) being hidden from the programmer. > > What type of overloading are you referring to? The way I think about > it, Python doesn't *have* overloading. Perhaps you're thinking of > overriding? Totally different beast! Hmm, perhaps... ;) Let me explain my confusion then: 1. "Overloading" and "overriding" look very much the same for most people. In French the same word "surcharge" is used for both. 2. Python does have "operrator overloading". From ncoghlan at gmail.com Fri Apr 7 11:55:27 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 07 Apr 2006 19:55:27 +1000 Subject: [Python-3000] Removing repr In-Reply-To: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> References: <20060406045749.q415ubkbecowo80k@login.werra.lunarpages.com> Message-ID: <4436370F.7070502@gmail.com> Michael Chermside wrote: > (I'm going out on a limb here, claiming that I'm right and Nick > is wrong, but I'm going to stand by the claim anyhow. Anything > which is machine readable "sometimes" is, by my definition, NOT > machine readable!) I was just being sloppy with my terminology by including "standardised format for representations that won't round trip through eval()" under the heading of 'machine readable' (the machine can't do anything very useful with it, but it can certainly recognise it for what it is). Other than that, I agree entirely with what you wrote, and am +1 for including this (leaving repr() alone) in PEP 3099 :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at iinet.net.au Fri Apr 7 12:52:26 2006 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri, 07 Apr 2006 20:52:26 +1000 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> Message-ID: <4436446A.5050804@iinet.net.au> Phillip J. Eby wrote: > At 02:15 PM 4/6/2006, Guido van Rossum wrote: >> On 4/6/06, Phillip J. Eby wrote: >> > I don't want to discourage people from working out their own ideas, >> > but a lot of the stuff that's being discussed here about protocol >> > adaptation is already implemented in PyProtocols. >> >> That's great. I believe that if we derive things from first principles >> here and arrive at the same choices as PyProtocols, we haven't wasted >> anybody's time; > > I agree; that's why I said "I don't want to discourage people from > working out ... ideas". I just wanted to point out that for those who > prefer to learn from worked-out examples, there are some available. One > reason I like to steal designs from *built* systems is that making > something real forces lots of tradeoffs into the open, that don't come > up when you're just brainstorming. > > That having been said, please consider this a "spoiler warning": the > rest of this message reveals where I think you're all going to > eventually end up. If it'll spoil anyone's fun and/or learning to know > this ahead of time, don't read the rest of this message. :) > > Hint: premature optimization is the root of all evil. I was actually > trying to hint in my previous message that it would be valuable to learn > from PyProtocols' example by *not* repeating it! I was not suggesting > that PyProtocols is a model to which you should all aspire, but rather > consider why I wish I hadn't bothered to do it in the first place! I've been stealing liberally from PyProtocols all along, usually trying to grasp ways to take *concepts* from it that I liked, and turn them into something that seemed *usable* (with aesthetics playing a large part in that). That's why one of the first things I hit on in rereading PEP 246 was "hang on, why are we use a pair of ad hoc protocols to build an adaptation system"? And lo and behold, PyProtocols had a much easier to use system based on concrete protocol objects, where you manipulated the target protocol directly. And my last message about adaptation, which was a (massively) simplified version of some of the ideas in PyProtocols, let me understand how transitivity would apply in a generic function context: you have function_A with a certain signature, and existing operations and function_B that can be implemented in terms of function_A. All there then needs to be is a way to tell function_B about it: @function_A.when_specialized def notify_function_B(signature, specialization): # Give function B a new specialization # based on the specialization in A # If the signatures are the same, then # we can just pass the info along function_B.specialize(signature)(specialization) I have no idea whether providing a mechanism for that is a good idea or not, but the important point is that it covers the last thing that I understood how to do with protocols, but not with generic functions. And given that *normal* functions are already in the language as a fundamental building block, then it makes a lot of sense to explore how to make them generic. I like being able to use 'signature' and 'specialization' as the terms, too, as they describe exactly what's going on. Further, if all builtin function objects (including C function descriptors and method wrappers) were potentially generic, then it would be possible to write: @specialize(str).for_signature(SomeClass) def str_SomeClass(obj): return obj.stringify() The use of the "specialize" function (rather than invoking the method directly) then means that arbitrary callables can support specialization by defining it as something like: def specialize(obj): return obj.__call__ @specialize.for_signature(type(specialize)) def specialize_function(func): return func Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From mcherm at mcherm.com Fri Apr 7 13:53:05 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Fri, 07 Apr 2006 04:53:05 -0700 Subject: [Python-3000] Kill "generic functions"! Message-ID: <20060407045305.twzae7mu83y804s8@login.werra.lunarpages.com> Greg Ewing writes: > "Multiple dispatch" +1 How is the noun spelled? "Multiple Dispatch Function"? Too wordy perhaps? I like it anyhow. -- Michael Chermside From ncoghlan at gmail.com Fri Apr 7 14:42:03 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 07 Apr 2006 22:42:03 +1000 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> <7.0.1.0.0.20060406121637.0218a898@telecommunity.com> Message-ID: <44365E1B.3000309@gmail.com> Guido van Rossum wrote: > On 4/6/06, Phillip J. Eby wrote: >> So, when I say that "overload" has the wrong connotations, I'm not >> talking about some abstract understanding of what it is or does, I'm >> talking about the wrong connotations for their practical intuition of >> how the thing should be *used*. The use cases for overloading in the >> languages that have it are way different, and if you move those >> concepts over to generic functions without a critical >> reinterpretation of the concept, you will write atrocious Python that >> will be hard to read and perform badly. > > I think that the solution to this would be a book or course "Python > for Java programmers". Not using the most natural name for a feature > because Java programmers might misunderstand it sound like the wrong > motivation. The problems I have with "overload" are that the adjective form is lousy, the noun forms are identical to some of the verb forms, and the *English* connotations are bad. My own suggestion of "specialize" (mentioned somewhere in the generic functions thread) is just too damn long. "extend" doesn't have any of these problems. === Noun and Verb forms === A quick side-by-side word form comparison: Verb forms: extend overload extending overloading extended overloaded extends overloads Adjective forms: extensible overloadable Noun forms: extension overload extensions overloads And some comparative sentences using those terms: Py3k functions will be extensible Py3k functions will be overloadable Function extensions are used to extend extensible functions Function overloads are used to overload overloadable functions An extended function may delegate a call to one of its extensions An overloaded function may delegate a call to one of its overloads Here's an example of registering a function extension: @extend(str).for_signature(SomeClass) def stringify(obj): return obj.stringify() No problem here - this function extends the "str" function when that function is called with an object of type "SomeClass". Now here's the exact same example as a function overload: @overload(str).for_signature(SomeClass) def stringify(obj): return obj.stringify() Uh-oh - now the verb/noun duality of the word "overload" comes back to bite us. Are we registering that "str" is a known overload of *this* function? Or that this function overloads "str"? The latter is definitely the more likely interpretation, but there's still room for a bit of doubt. === English connotations === On the topic of "overload" as an English word, I've always suspected whoever chose it for the concept of signature-based function dispatch was deliberately trying to discourage people from using overloaded functions. Overloading something or someone is normally considered a very *bad* thing, as it means asking that thing or person to do more than it or they can handle. When a word is described as being "overloaded with meanings" it generally means that the word has *so many* meanings that it can no longer be used safely if you desire clear communications. So I consider "overload" to be far from a natural term for a mechanism that's perfectly safe, and believe there are some very sound reasons to prefer the name "extensible functions". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From pje at telecommunity.com Fri Apr 7 18:40:16 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 07 Apr 2006 09:40:16 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> Message-ID: <7.0.1.0.0.20060407093812.02148630@telecommunity.com> At 07:37 AM 4/7/2006, Jim Jewett wrote: >On 4/7/06, Phillip J. Eby wrote: > > Guido wrote: > > > mros = tuple(t.__mro__ for t in types) > > > n = len(mros) > > > candidates = [sig for sig in self.registry > > > if len(sig) == n and > > > all(t in mro for t, mro in zip(sig, mros))] > > > > I believe this can be simplified to: > > n = len(types) > > candidates = [sig for sig in self.registry > > if len(sig) == n and > > all(issubclass(s,t) for s,t in zip(sig, types))] > >Am I going nuts, or should that issubclass test be reversed? No, you're right, I goofed. To me, the "signature" was what the function was *called* with, so it seemed to read correctly to me. i.e., I read it as "the calling signature item is a subclass of the target method's type"... which is the right idea, but the wrong variables. :) From guido at python.org Fri Apr 7 19:21:05 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 10:21:05 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <4436446A.5050804@iinet.net.au> References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> Message-ID: On 4/7/06, Nick Coghlan wrote: > I've been stealing liberally from PyProtocols all along, usually trying to > grasp ways to take *concepts* from it that I liked, and turn them into > something that seemed *usable* (with aesthetics playing a large part in that). Perhaps you could give us a hint on where the parts of PyProtocols that you like are documented on the web? Scanning the peak.telecommunity.com site I always seem to get lost. For example, I have so far had no luck finding either the source code or the docs for RuleDispatch there. I got tantalizingly close when I found a reference to a module named 'dispatch' in core.py in the peak svn tree, but there was nothing named dispatch or dispatch.py nearby... :-( The "API docs" similarly apear as a forest of dead leaves to me. > That's why one of the first things I hit on in rereading PEP 246 was "hang on, > why are we use a pair of ad hoc protocols to build an adaptation system"? And > lo and behold, PyProtocols had a much easier to use system based on concrete > protocol objects, where you manipulated the target protocol directly. > > And my last message about adaptation, which was a (massively) simplified > version of some of the ideas in PyProtocols, let me understand how > transitivity would apply in a generic function context: you have function_A > with a certain signature, and existing operations and function_B that can be > implemented in terms of function_A. This is clear as mud to me. > All there then needs to be is a way to > tell function_B about it: > > @function_A.when_specialized > def notify_function_B(signature, specialization): > # Give function B a new specialization > # based on the specialization in A > # If the signatures are the same, then > # we can just pass the info along > function_B.specialize(signature)(specialization) This just looks like a rearrangement of names to me, none of which have syntax. It's no more meaningful to me than if it had said @A.wq def nB(p,q): B.qq(p)(q) The problem I have with examples like this (some examples given for Peak suffer from this too) is that unless yuo already know what some parts mean, it's impossible to guess at the meaning. > I have no idea whether providing a mechanism for that is a good idea or not, > but the important point is that it covers the last thing that I understood how > to do with protocols, but not with generic functions. And given that *normal* > functions are already in the language as a fundamental building block, then it > makes a lot of sense to explore how to make them generic. Maybe it would help me understand what you are saying if you gave an actual example with real code. Explaining what the various magic methods/decorators do would help too. > I like being able to use 'signature' and 'specialization' as the terms, too, > as they describe exactly what's going on. Not to me. I think of "signature" as a tuple of types, and maybe that's what you mean it to describe. But what's a specialization? Is it a callable? Is it a string? What properties does a specialization have? (This reminds me of the kind of conversation I sometimes have with Orlijn. Sometimes he'll say "I saw a frizzlefrazzle today!" I'll say "what's a frizzlefrazzle?" He'll say something that sounds to me like "Not frizzlefrazzle, frizzlefrazzle!" After a few tries I give up, and I have to change my line of questioning to get him to describe a frizzlefrazzle. The most effective questions are usually "what does a frizzlefrazzle do?" or "what does a frizzlefrazzle have?" That's what I'm asking you here. :-) > Further, if all builtin function > objects (including C function descriptors and method wrappers) were > potentially generic, This seems a rather extreme anticipation. I don't think anybody has assumed this is where we might be going. > then it would be possible to write: > > @specialize(str).for_signature(SomeClass) > def str_SomeClass(obj): > return obj.stringify() OK, now at least I have an understanding of what you're trying to do. You're saying that str(obj) should call obj.stringify() whenever obj is a SomeClass. Forgetting that str is really a type; assuming str were an @overloaded function (I'm sticking to my terminology :-) then I could have written that as @str.register(SomeClass) def str_SomeClass(obj): return obj.stringify() I'm not sure though what @specialize(str) adds to this. Assuming we can't change str into an @overloaded function but we do want to allow this style of overriding, I guess we could have some external registry object which str consults before looking for a __str__ method or using hardcoded knowledge. > The use of the "specialize" function (rather than invoking the method > directly) then means that arbitrary callables can support specialization by > defining it as something like: > > def specialize(obj): > return obj.__call__ > > @specialize.for_signature(type(specialize)) > def specialize_function(func): > return func Ouch. My head just exploded again. :-( -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 7 19:36:17 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 10:36:17 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <44337508.2030505@livinglogic.de> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> Message-ID: On 4/5/06, Walter D?rwald wrote: > But might closes off one route we could take if we're adapting to > something that must provide more than one function/method (i.e. an > interface) and we'd like to have the interface and the adaption registry > be identical. Suppose we're designing a new interface for sequence. A > sequence must provide getitem() and len() methods. If the adapt(at)ion > API only used register and adapt methods we could do something like this: > > > class Interface(object): > class __metaclass__(type): > def __new__(mcl, name, bases, dict): > # Give each class it's own registry > dict["registry"] = {} > return type.__new__(mcl, name, bases, dict) > > @classmethod > def register(cls, adapter, T): > cls.registry[T] = adapter > > @classmethod > def adapt(cls, obj): > for base in type(obj).__mro__: > try: > return cls.registry[base](obj) > except KeyError: > pass > raise TypeError("can't adapt %r to %r" % (obj, cls)) > > > class Sequence(Interface): > def getitem(self, index): pass > def len(self): pass > > > class PythonSeqAsSequence(Sequence): > def __init__(self, obj): > self.obj = obj > > def getitem(self, index): > return self.obj[i] > > def len(self): > return len(self.obj) > > Sequence.register(PythonSeqAsSequence, list) > > > print Sequence.adapt([1,2,3]).len() > > > But if adapting is done via __call__() we have a problem: Sequence > already provides a __call__, the constructor. Of course if this worked > > print Sequence([1,2,3]).len() > > would look much better than > > print Sequence.adapt([1,2,3]).len() > > but I'm not sure if it's possible to work around the constructor/adapt > clash. Using @overloaded functions I would create an explicit class variable which is the @overloaded adapter rather than trying to make the interface also *be* the adapter. I would define the Interface class like this: class InterfaceMetaclass(type): # I hate anonymous metaclasses def __new__(mcl, name, bases, dict): # Give each class it's own registry dict["adapter"] = overloaded(None) # set default_function to None return type.__new__(mcl, name, bases, dict) class Interface: __metaclass__ = InterfaceMetaclass # No need for adapt and register methods here The registration would then look like this: Sequence.adapter.register(list)(PythonSeqAsSequence) and the invocation would look like this: print Sequence.adapter([1,2,3]).len() -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 7 19:57:24 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 10:57:24 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> References: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> Message-ID: On 4/6/06, Phillip J. Eby wrote: > Guido wrote: > > mros = tuple(t.__mro__ for t in types) > > n = len(mros) > > candidates = [sig for sig in self.registry > > if len(sig) == n and > > all(t in mro for t, mro in zip(sig, mros))] > > I believe this can be simplified to: > n = len(types) > candidates = [sig for sig in self.registry > if len(sig) == n and > all(issubclass(s,t) for s,t in zip(sig, types))] [but with issubclass(t,s) as Jim Jewett pointed out] Aha. This would solve the problem that it currently only works with "proper" types (that inherit from type and hence define __mro__). > Similarly, I believe the dominates function can be moved out to a > top-level function in similar fashion: > > def dominates(dom, sub): > return all(d is not s and issubclass(d,s) for d,s in zip(dom, sub)) > > Using issubclass() is more accurate in this case than using __mro__, > because __mro__ considers the order of the __bases__ to be > significant, which can lead to unexpected results due to false > precision. (i.e., because it'll be guessing about cases that are > actually ambiguous) Here I'm not so sure. Are you sure those results are unexpected? It seems you are saying that if I have a class C deriving from A and B (in that order) where both A and B implement a method foo(), and C doesn't, the call C().foo() would be ambiguous. But in fact the whole point of having a defined MRO is to foster the expectation that it will call A's foo(), because that one comes first in the MRO. So at least for the single-argument variant I would think that my algorithm gives results consistent with the definition of inheritance in new-style classes. > Unfortunately, this entire approach is subject to similar scale > issues as PyProtocols/RuleDispatch. Even with the cache I put in? The hairy algorithm doesn't get invoked more than once per actual signature (type tuple). > It is much slower than a few > hand-written if-then's for the common cases, and it's even slower and > too inflexible for the "zillions of methods" cases like Zope's "view" > registry (which dispatches based on some types and a string). I don't recall the details of Zope view dispatch (and they changed since I left) but isn't the string a red herring that can be solved by having a separate dispatcher for each string? > You > could get better performance in the common cases by just creating a > dominance hierarchy among all the methods, and then writing out > Python code that executes a series of if-then tests with the dominant > cases being tested first (and some extra branches to check for > ambiguity). I like the idea of analyzing the methods for dominance relationships ahead of time. But I'm not sure why we should then resort to generating code. Shouldn't it be just as easy to create a data structure that can be walked by fixed code? (Just like a recursive-descent parser can be written as a fixed "machine" that interprets a data structure representing the parse information.) > This code would then run at virtually the same speed as > an identical hand-written function for the most common use cases > (which tend towards having only a handful of overloads at most), and > it would also not require any extra memory allocations for temporary > lists and tuples on each invocation of the function (because all the > state is effectively tracked in the program counter as it threads its > way among the if-then tests). If the cache is doing its work, the only allocation on each call is the expression types = tuple(type(a) for a in args) Tuple allocation is pretty fast so I'm surprised you're worried about this. Have you ever measured this kind of thing? > To make the same approach also work for the zillions-of-methods > cases, all that would be needed is replacing some of the if-thens > with dictionary lookups to allow N-way branches instead of just > true/false tests, and some optimization to avoid re-testing > conditions or recomputing values that have already been > computed. > Most of RuleDispatch's complexity stems from the fact that > I effectively implemented a mini-interpreter for executing such N-way > dispatch trees, instead of just generating bytecode. I made that > choice because there's unfortunately no "computed goto" equivalent in > today's bytecode. (Bytecode is also rather annoyingly low-level for > this kind of work.) I would not recommend anyone even looking into bytecode generation. If you *have* to generate code, at least generate Python code, so Jython and IronPython aren't left behind. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Fri Apr 7 20:09:42 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 7 Apr 2006 14:09:42 -0400 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> Message-ID: On 4/5/06, Walter D?rwald wrote: > class Sequence(Interface): > def getitem(self, index): pass > def len(self): pass > class PythonSeqAsSequence(Sequence): > def __init__(self, obj): > self.obj = obj > def getitem(self, index): > return self.obj[i] > def len(self): > return len(self.obj) > Sequence.register(PythonSeqAsSequence, list) > But if adapting is done via __call__() we have a problem: Sequence > already provides a __call__, the constructor. So? I can't see any reason why you would want to construct an (abstract) Sequence. What would it mean? What you might want to do is to construct a specific Sequence, such as Sequence([1,2,3]), or the empty Sequence() . Just as the list() constructor takes arguments and returns something that looks like a list, the Sequence() constructor should take an argument and return something that looks like a sequence. The difference from a "regular" class constructor is that you are less likely to create a new object if it can be done more efficiently, and there is no expectation that Sequence will appear in the object's mro. Actually calling the sole method of an interface is admittedly more debatable. -jJ From walter at livinglogic.de Fri Apr 7 20:16:42 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 07 Apr 2006 20:16:42 +0200 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> Message-ID: <4436AC8A.50802@livinglogic.de> Guido van Rossum wrote: > On 4/5/06, Walter D?rwald wrote: >> But might closes off one route we could take if we're adapting to >> something that must provide more than one function/method (i.e. an >> interface) and we'd like to have the interface and the adaption registry >> be identical. Suppose we're designing a new interface for sequence. A >> sequence must provide getitem() and len() methods. If the adapt(at)ion >> API only used register and adapt methods we could do something like this: >> >> >> class Interface(object): >> class __metaclass__(type): >> def __new__(mcl, name, bases, dict): >> # Give each class it's own registry >> dict["registry"] = {} >> return type.__new__(mcl, name, bases, dict) >> >> @classmethod >> def register(cls, adapter, T): >> cls.registry[T] = adapter >> >> @classmethod >> def adapt(cls, obj): >> for base in type(obj).__mro__: >> try: >> return cls.registry[base](obj) >> except KeyError: >> pass >> raise TypeError("can't adapt %r to %r" % (obj, cls)) >> >> >> class Sequence(Interface): >> def getitem(self, index): pass >> def len(self): pass >> >> >> class PythonSeqAsSequence(Sequence): >> def __init__(self, obj): >> self.obj = obj >> >> def getitem(self, index): >> return self.obj[i] >> >> def len(self): >> return len(self.obj) >> >> Sequence.register(PythonSeqAsSequence, list) >> >> >> print Sequence.adapt([1,2,3]).len() >> >> >> But if adapting is done via __call__() we have a problem: Sequence >> already provides a __call__, the constructor. Of course if this worked >> >> print Sequence([1,2,3]).len() >> >> would look much better than >> >> print Sequence.adapt([1,2,3]).len() >> >> but I'm not sure if it's possible to work around the constructor/adapt >> clash. > > Using @overloaded functions I would create an explicit class variable > which is the @overloaded adapter rather than trying to make the > interface also *be* the adapter. I would define the Interface class > like this: > > class InterfaceMetaclass(type): > # I hate anonymous metaclasses > def __new__(mcl, name, bases, dict): > # Give each class it's own registry > dict["adapter"] = overloaded(None) # set default_function to None > return type.__new__(mcl, name, bases, dict) > > class Interface: > __metaclass__ = InterfaceMetaclass > # No need for adapt and register methods here > > The registration would then look like this: > > Sequence.adapter.register(list)(PythonSeqAsSequence) > > and the invocation would look like this: > > print Sequence.adapter([1,2,3]).len() This looks reasonable enough, as the adapter comes for free and it avoids the problem of a __call__() in the Interface or its metaclass. But if I'm implementing an adapter for a certain type, I still can't make the returned object an instance of Interface (or I could, but it would inherit an adapter, which is useless), so I would have: class Interface: def getitem(self, index): ... def len(self, index): ... Interface.adapter() which I have to call to do the adaption and class PythonSeqAsSequence: def getitem(self, index): ... def len(self, index): ... which implements the Sequence protocol, but shouldn't subclass Sequence. Somehow this feels awkward to me. Servus, Walter From guido at python.org Fri Apr 7 20:33:25 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 11:33:25 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <4436AC8A.50802@livinglogic.de> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> Message-ID: On 4/7/06, Walter D?rwald wrote: > > Using @overloaded functions I would create an explicit class variable > > which is the @overloaded adapter rather than trying to make the > > interface also *be* the adapter. I would define the Interface class > > like this: > > > > class InterfaceMetaclass(type): > > # I hate anonymous metaclasses > > def __new__(mcl, name, bases, dict): > > # Give each class it's own registry > > dict["adapter"] = overloaded(None) # set default_function to None > > return type.__new__(mcl, name, bases, dict) > > > > class Interface: > > __metaclass__ = InterfaceMetaclass > > # No need for adapt and register methods here > > > > The registration would then look like this: > > > > Sequence.adapter.register(list)(PythonSeqAsSequence) > > > > and the invocation would look like this: > > > > print Sequence.adapter([1,2,3]).len() > > This looks reasonable enough, as the adapter comes for free and it > avoids the problem of a __call__() in the Interface or its metaclass. > > But if I'm implementing an adapter for a certain type, I still can't > make the returned object an instance of Interface (or I could, but it > would inherit an adapter, which is useless), I'm not following. Why would an adapter ever return an instance of Interface (which is just a token object, a reification of the concept of an interface)? > so I would have: > > class Interface: > def getitem(self, index): ... > def len(self, index): ... > > Interface.adapter() which I have to call to do the adaption and > > class PythonSeqAsSequence: > def getitem(self, index): ... > def len(self, index): ... > > which implements the Sequence protocol, but shouldn't subclass Sequence. > > Somehow this feels awkward to me. Why would the wrapper class need to subclass the interface class? OTOH if you really like to express implementing an interface as inheriting from that interface, what's the problem with inheriting the adapter? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Fri Apr 7 20:40:20 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 7 Apr 2006 14:40:20 -0400 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <4436AC8A.50802@livinglogic.de> Message-ID: <200604071440.20992.fdrake@acm.org> On Friday 07 April 2006 14:33, Guido van Rossum wrote: > OTOH if you really like to express implementing an interface as > inheriting from that interface, what's the problem with inheriting the > adapter? Ugh! Inheritance should only be done to get implementation. If there's value in default implementation of methods, a base class can be provided independently of the interface. -Fred -- Fred L. Drake, Jr. From tim.hochberg at cox.net Fri Apr 7 20:43:48 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Fri, 07 Apr 2006 11:43:48 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <4436AC8A.50802@livinglogic.de> References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> Message-ID: <4436B2E4.1090900@cox.net> Walter D?rwald wrote: >Guido van Rossum wrote: > > > >>On 4/5/06, Walter D?rwald wrote: >> >> >>>But might closes off one route we could take if we're adapting to >>>something that must provide more than one function/method (i.e. an >>>interface) and we'd like to have the interface and the adaption registry >>>be identical. Suppose we're designing a new interface for sequence. A >>>sequence must provide getitem() and len() methods. If the adapt(at)ion >>>API only used register and adapt methods we could do something like this: >>> >>> >>>class Interface(object): >>> class __metaclass__(type): >>> def __new__(mcl, name, bases, dict): >>> # Give each class it's own registry >>> dict["registry"] = {} >>> return type.__new__(mcl, name, bases, dict) >>> >>> @classmethod >>> def register(cls, adapter, T): >>> cls.registry[T] = adapter >>> >>> @classmethod >>> def adapt(cls, obj): >>> for base in type(obj).__mro__: >>> try: >>> return cls.registry[base](obj) >>> except KeyError: >>> pass >>> raise TypeError("can't adapt %r to %r" % (obj, cls)) >>> >>> >>>class Sequence(Interface): >>> def getitem(self, index): pass >>> def len(self): pass >>> >>> >>>class PythonSeqAsSequence(Sequence): >>> def __init__(self, obj): >>> self.obj = obj >>> >>> def getitem(self, index): >>> return self.obj[i] >>> >>> def len(self): >>> return len(self.obj) >>> >>>Sequence.register(PythonSeqAsSequence, list) >>> >>> >>>print Sequence.adapt([1,2,3]).len() >>> >>> >>>But if adapting is done via __call__() we have a problem: Sequence >>>already provides a __call__, the constructor. Of course if this worked >>> >>>print Sequence([1,2,3]).len() >>> >>>would look much better than >>> >>>print Sequence.adapt([1,2,3]).len() >>> >>>but I'm not sure if it's possible to work around the constructor/adapt >>>clash. >>> >>> >>Using @overloaded functions I would create an explicit class variable >>which is the @overloaded adapter rather than trying to make the >>interface also *be* the adapter. I would define the Interface class >>like this: >> >>class InterfaceMetaclass(type): >> # I hate anonymous metaclasses >> def __new__(mcl, name, bases, dict): >> # Give each class it's own registry >> dict["adapter"] = overloaded(None) # set default_function to None >> return type.__new__(mcl, name, bases, dict) >> >>class Interface: >> __metaclass__ = InterfaceMetaclass >> # No need for adapt and register methods here >> >>The registration would then look like this: >> >>Sequence.adapter.register(list)(PythonSeqAsSequence) >> >>and the invocation would look like this: >> >>print Sequence.adapter([1,2,3]).len() >> >> > >This looks reasonable enough, as the adapter comes for free and it >avoids the problem of a __call__() in the Interface or its metaclass. > >But if I'm implementing an adapter for a certain type, I still can't >make the returned object an instance of Interface (or I could, but it >would inherit an adapter, which is useless), so I would have: > >class Interface: > def getitem(self, index): ... > def len(self, index): ... > >Interface.adapter() which I have to call to do the adaption and > >class PythonSeqAsSequence: > def getitem(self, index): ... > def len(self, index): ... > >which implements the Sequence protocol, but shouldn't subclass Sequence. > >Somehow this feels awkward to me. > > Perhaps this is a hint that protocol isn't the right term here. The Protocol class in its current implementation doesn't describe an object, it's really just a factory function for creating objects that satisfy some other interface. Trying to do both in one object looks like it's just going to cause trouble. Suppose that instead of calling these protocols, we just call them factories. Then it's clear that these objects don't describe anything, they just make stuff out of other stuff. As it stands, Walter's question seems inevitable, but if these were called something else the idea would probably seem nonsensical. The answer to "but what if I I'd like to have the interface and the factory function be identical?", assuming it were ever asked, could almost certainly be "don't do that!". Regards, -tim From walter at livinglogic.de Fri Apr 7 20:43:48 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 07 Apr 2006 20:43:48 +0200 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> Message-ID: <4436B2E4.8010607@livinglogic.de> Jim Jewett wrote: > On 4/5/06, Walter D?rwald wrote: > >> class Sequence(Interface): >> def getitem(self, index): pass >> def len(self): pass > >> class PythonSeqAsSequence(Sequence): >> def __init__(self, obj): >> self.obj = obj > >> def getitem(self, index): >> return self.obj[i] > >> def len(self): >> return len(self.obj) > >> Sequence.register(PythonSeqAsSequence, list) > > >> But if adapting is done via __call__() we have a problem: Sequence >> already provides a __call__, the constructor. > > So? I can't see any reason why you would want to construct an > (abstract) Sequence. What would it mean? The official adaption/overloading API might specify that an adapter must be a callable object that has a register method, which can be used as a decorator. If adaption is done to a single method protocol, calling the protocol should return the result of calling the adapter, but if adaption is done to a protocol with several methods, calling the protocol returns an object that has those methods. Now if we want the Interface declaration to be the protocol, this means that we call the Interface class to get an object that provides this interface for the object passed in. > What you might want to do is to construct a specific Sequence, such as > Sequence([1,2,3]), or the empty Sequence() . > > Just as the list() constructor takes arguments and returns something > that looks like a list, the Sequence() constructor should take an > argument and return something that looks like a sequence. You use the int() protocol to get an int from a string: int("42"), the same way as you use the Sequence() protocol to get a Sequence. (And if you're passing a long string, (i.e. int(100*"42")) you don't even get an int. > The difference from a "regular" class constructor is that you are less > likely to create a new object if it can be done more efficiently, and > there is no expectation that Sequence will appear in the object's mro. > > Actually calling the sole method of an interface is admittedly more debatable. Servus, Walter From guido at python.org Fri Apr 7 20:45:43 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 11:45:43 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <200604071440.20992.fdrake@acm.org> References: <44315A89.9000104@colorstudy.com> <4436AC8A.50802@livinglogic.de> <200604071440.20992.fdrake@acm.org> Message-ID: On 4/7/06, Fred L. Drake, Jr. wrote: > On Friday 07 April 2006 14:33, Guido van Rossum wrote: > > OTOH if you really like to express implementing an interface as > > inheriting from that interface, what's the problem with inheriting the > > adapter? > > Ugh! Inheritance should only be done to get implementation. If there's value > in default implementation of methods, a base class can be provided > independently of the interface. That's what I was thinking -- I was only trying to guess what Walter was after. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From walter at livinglogic.de Fri Apr 7 20:49:51 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 07 Apr 2006 20:49:51 +0200 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <44320C36.7070507@colorstudy.com> <7.0.1.0.0.20060403235420.021f3d40@telecommunity.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> Message-ID: <4436B44F.5090407@livinglogic.de> Guido van Rossum wrote: > On 4/7/06, Walter D?rwald wrote: >>> Using @overloaded functions I would create an explicit class variable >>> which is the @overloaded adapter rather than trying to make the >>> interface also *be* the adapter. I would define the Interface class >>> like this: >>> >>> class InterfaceMetaclass(type): >>> # I hate anonymous metaclasses >>> def __new__(mcl, name, bases, dict): >>> # Give each class it's own registry >>> dict["adapter"] = overloaded(None) # set default_function to None >>> return type.__new__(mcl, name, bases, dict) >>> >>> class Interface: >>> __metaclass__ = InterfaceMetaclass >>> # No need for adapt and register methods here >>> >>> The registration would then look like this: >>> >>> Sequence.adapter.register(list)(PythonSeqAsSequence) >>> >>> and the invocation would look like this: >>> >>> print Sequence.adapter([1,2,3]).len() >> This looks reasonable enough, as the adapter comes for free and it >> avoids the problem of a __call__() in the Interface or its metaclass. >> >> But if I'm implementing an adapter for a certain type, I still can't >> make the returned object an instance of Interface (or I could, but it >> would inherit an adapter, which is useless), > > I'm not following. Why would an adapter ever return an instance of > Interface (which is just a token object, a reification of the concept > of an interface)? An Interface is an abstract class that you subclass to make it a concrete implementation. If you have an adapter that adapts to the FooInterface why shouldn't it return a FooInterface object? >> so I would have: >> >> class Interface: >> def getitem(self, index): ... >> def len(self, index): ... >> >> Interface.adapter() which I have to call to do the adaption and >> >> class PythonSeqAsSequence: >> def getitem(self, index): ... >> def len(self, index): ... >> >> which implements the Sequence protocol, but shouldn't subclass Sequence. >> >> Somehow this feels awkward to me. > > Why would the wrapper class need to subclass the interface class? > > OTOH if you really like to express implementing an interface as > inheriting from that interface, what's the problem with inheriting the > adapter? The adapter isn't a class, it's either a fancy function or an object with a __call__(). Servus, Walter From guido at python.org Fri Apr 7 21:19:17 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 12:19:17 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> Message-ID: On 4/7/06, Guido van Rossum wrote: > Even with the cache I put in? The hairy algorithm doesn't get invoked > more than once per actual signature (type tuple). OK, I timed this, and the @overloaded function (when it hits the cache) is indeed 4-7 times slower than a handwritten decision tree with about 5 cases (depending on which case gets hit). But I can make the @overloaded function twice as fast by in-lining the first few lines of find_func() and using a different expression for calculating the types tuple: tuple(map(type, args)) (despite allocating a throw-away list first) is more than twice as fast as tuple(type(a) for a in args) and that's where most of the time went. A bit of speedup also came from avoiding to call find_func() if there's a cache hit. So I think that with some additional effort and/or a tiny bit of C code, the speed issue *when the cache works* can be solved. Is the performance of the full-fledged algorithm really that much of an issue? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 7 21:22:15 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 12:22:15 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <4436B44F.5090407@livinglogic.de> References: <44315A89.9000104@colorstudy.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> <4436B44F.5090407@livinglogic.de> Message-ID: On 4/7/06, Walter D?rwald wrote: > An Interface is an abstract class that you subclass to make it a > concrete implementation. That's not the view of most people who use the word interface these days. Not in Zope (see Fred Drake's post), not in Java. > If you have an adapter that adapts to the > FooInterface why shouldn't it return a FooInterface object? Because implementing an interface and extending a class are separate concepts. Did you forget duck typing? Something can be a sequence without subclassing a common base class. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake at acm.org Fri Apr 7 21:39:42 2006 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri, 7 Apr 2006 15:39:42 -0400 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <4436B2E4.8010607@livinglogic.de> References: <44315A89.9000104@colorstudy.com> <4436B2E4.8010607@livinglogic.de> Message-ID: <200604071539.42676.fdrake@acm.org> On Friday 07 April 2006 14:43, Walter D?rwald wrote: > You use the int() protocol to get an int from a string: > int("42"), the same way as you use the Sequence() protocol to get a > Sequence. (And if you're passing a long string, (i.e. int(100*"42")) you > don't even get an int. Not all single-arg calls that return something based on the input are adaptations. While this particular case *might* be reasonable to describe as adaptation (it's unclear to me), the fact that this call happens to be to the target type is not necessarily significant. -Fred -- Fred L. Drake, Jr. From guido at python.org Fri Apr 7 21:52:26 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 12:52:26 -0700 Subject: [Python-3000] A few small py3k wishes In-Reply-To: <200604041322.02900.gmccaughan@synaptics-uk.com> References: <20060402222130.GA12075@tigger.digitaltorque.ca> <200604041322.02900.gmccaughan@synaptics-uk.com> Message-ID: On 4/4/06, Gareth McCaughan wrote: > I take no stand on how much not confusing Lispers matters > to Python 3k. But I do: none whatsoever. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tjreedy at udel.edu Sat Apr 8 00:31:09 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 7 Apr 2006 18:31:09 -0400 Subject: [Python-3000] Kill "generic functions"! References: <20060407045305.twzae7mu83y804s8@login.werra.lunarpages.com> Message-ID: "Michael Chermside" wrote in message news:20060407045305.twzae7mu83y804s8 at login.werra.lunarpages.com... > Greg Ewing writes: >> "Multiple dispatch" > > +1 > > How is the noun spelled? "Multiple Dispatch Function"? Too wordy perhaps? > I > like it anyhow. -1 To me, multiple dispatch mean dispatching on multiple arguments, whereas the simple and most common of what we are talking about is, I believe, sinple arg dispatch to multiple implementations. tjr From ianb at colorstudy.com Sat Apr 8 00:55:04 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 07 Apr 2006 17:55:04 -0500 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: References: <20060407045305.twzae7mu83y804s8@login.werra.lunarpages.com> Message-ID: <4436EDC8.8050407@colorstudy.com> Terry Reedy wrote: > "Michael Chermside" wrote in message > news:20060407045305.twzae7mu83y804s8 at login.werra.lunarpages.com... > >>Greg Ewing writes: >> >>>"Multiple dispatch" >> >>+1 >> >>How is the noun spelled? "Multiple Dispatch Function"? Too wordy perhaps? >>I >>like it anyhow. > > > -1 > > To me, multiple dispatch mean dispatching on multiple arguments, whereas > the > simple and most common of what we are talking about is, I believe, > sinple arg dispatch to multiple implementations. Drop the "multiple" part? There's no other function that dispatches. Methods can be thought of as dispatching on their first argument... but even that's a stretch, and you'd only think about it that way coming from a static typed background, and maybe then only if you came to statically typed OO programming after doing non-OO programming for a while. In Python methods are selected based on an object (and that happens on the callers side), and that object is also passed in as the first argument (which happens in the class machinery), and none of that is done by the function/method object itself. So I think calling current method invocation a "dispatch" is not accurate, nor natural. Though calling any current behavior overloading is not accurate or natural either. But I don't know, if we're concerned about how people coming from Java or C++ might misunderstand this... well, I at least am not one of those people ;) Overloaded doesn't bother me, though I still prefer dispatching functions and methods. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From crutcher at gmail.com Sat Apr 8 03:21:19 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Fri, 7 Apr 2006 18:21:19 -0700 Subject: [Python-3000] List-Comp style for loops? Message-ID: Basic idea: [for ... in ...]+ if ...: body A) it should be syntatically simple to parse. B) it establishes a nice symetry with list comprehensions. -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From guido at python.org Sat Apr 8 05:44:58 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 20:44:58 -0700 Subject: [Python-3000] List-Comp style for loops? In-Reply-To: References: Message-ID: On 4/7/06, Crutcher Dunnavant wrote: > Basic idea: > > [for ... in ...]+ if ...: > body > > A) it should be syntatically simple to parse. > B) it establishes a nice symetry with list comprehensions. Are you in cahoots with the person who posted the first response to my Artima blog on today? :-) (http://www.artima.com/forums/flat.jsp?forum=106&thread=155514) I can't make heads or tails of this. What is it supposed to do? What is it supposed to replace? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From crutcher at gmail.com Sat Apr 8 05:55:55 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Fri, 7 Apr 2006 20:55:55 -0700 Subject: [Python-3000] List-Comp style for loops? In-Reply-To: References: Message-ID: I'm not really sure what that guy is talking about. Sorry, On 4/7/06, Guido van Rossum wrote: > On 4/7/06, Crutcher Dunnavant wrote: > > Basic idea: > > > > [for ... in ...]+ if ...: > > body > > > > A) it should be syntatically simple to parse. > > B) it establishes a nice symetry with list comprehensions. > > Are you in cahoots with the person who posted the first response to my > Artima blog on today? :-) > (http://www.artima.com/forums/flat.jsp?forum=106&thread=155514) > > I can't make heads or tails of this. What is it supposed to do? What > is it supposed to replace? I'm not really sure what that guy is talking about. When list comps are explained, they are often described as being similar to: L = [] for i in ...: for j in ...: for k in ...: if ...: L.append(somefunof(i,j,k)) Well, if it is nice to have these constructs when we want the list, it will be nice to have them when we _don't_ care about the list. I find myself occasionally using listcomps to build looping constructs just cause they're nice, without saving the list, but of course, this is inefficient. So what I'm asking for is that this (without anything getting the comp value, and in hackish BNF): [ EXPR (for VARSEXPR in SEQ)+ [if TEST]] be equivalent to: (for VARSEXPR in SEQ)+ [if TEST]: EXPR So, for example: [ x(i) for i in range(n) if y(i) ] Could be represented as: for i in range(n) if y(i): x(i) -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From guido at python.org Sat Apr 8 05:59:37 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 7 Apr 2006 20:59:37 -0700 Subject: [Python-3000] List-Comp style for loops? In-Reply-To: References: Message-ID: On 4/7/06, Crutcher Dunnavant wrote: > So, for example: > > [ x(i) for i in range(n) if y(i) ] > > Could be represented as: > > for i in range(n) if y(i): > x(i) I don't think many Python users will consider that an improvement. It's really just saving an indent level and a colon. I really need to write up my list of guidelines for Python 3000 proposals. One of the guidelines will be "no wild ideas"; another "no gratuitous changes". I declare this proposal to be excluded by at least one of those... :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From vinj at alumni.rice.edu Sat Apr 8 06:13:26 2006 From: vinj at alumni.rice.edu (Vineet Jain) Date: Sat, 08 Apr 2006 00:13:26 -0400 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? Message-ID: <44373866.9070305@alumni.rice.edu> I'm working on an application that allows user to write models (currently python scripts) that get run on my grid of servers. My application is written in python, pyrex and C and I would have liked to use python for the user scripts as well. However, given that there is no way to run restricted python scripts, I'm in the process of currently moving from python to Lua for the user scripts. Is it wishful thinking to have a python restricted exec mode which allows for: 1. Limit the memory consumed by the script 2. Limit access to file system and other system resources 3. Limit cpu time that the script will take 4. Be able to specify which modules are available for import VJ From ncoghlan at gmail.com Sat Apr 8 06:21:26 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 08 Apr 2006 14:21:26 +1000 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> Message-ID: <44373A46.6060900@gmail.com> Guido van Rossum wrote: > On 4/7/06, Nick Coghlan wrote: >> I've been stealing liberally from PyProtocols all along, usually trying to >> grasp ways to take *concepts* from it that I liked, and turn them into >> something that seemed *usable* (with aesthetics playing a large part in that). > > Perhaps you could give us a hint on where the parts of PyProtocols > that you like are documented on the web? Scanning the > peak.telecommunity.com site I always seem to get lost. For example, I > have so far had no luck finding either the source code or the docs for > RuleDispatch there. I got tantalizingly close when I found a reference > to a module named 'dispatch' in core.py in the peak svn tree, but > there was nothing named dispatch or dispatch.py nearby... :-( The "API > docs" similarly apear as a forest of dead leaves to me. Mostly just the API docs: http://peak.telecommunity.com/protocol_ref/module-protocols.html This was the specific page about using a concrete API on the protocols themselves: http://peak.telecommunity.com/protocol_ref/protocols-calling.html > >> That's why one of the first things I hit on in rereading PEP 246 was "hang on, >> why are we use a pair of ad hoc protocols to build an adaptation system"? And >> lo and behold, PyProtocols had a much easier to use system based on concrete >> protocol objects, where you manipulated the target protocol directly. >> >> And my last message about adaptation, which was a (massively) simplified >> version of some of the ideas in PyProtocols, let me understand how >> transitivity would apply in a generic function context: you have function_A >> with a certain signature, and existing operations and function_B that can be >> implemented in terms of function_A. > > This is clear as mud to me. It's not really that important, except in convincing me personally that overloadable functions really are a superset of protocol adaptation. Another way of looking at it: 1. Suppose function A and function B do the roughly the same thing 2. Suppose function A can be overloaded/extended 3. It would then be straightforward to add additional extensions to function A that delegated certain call signatures to function B (possibly with a wrapper to deal with slight differences in the order of arguments or the type of the return value. Turning that into *transitivity* is a matter of asking the question, "well , what if B can be overloaded as well?". The idea then is that you would pass B a callback to be invoked whenever a new signature was registered with B. The callback would take the new signature, rearrange it if necessary, and register the new version with function A. @function_B.call_for_new_registrations def extending_B_extends_A_too(signature, extension): # The decorator means this function would be called whenever a new # extension was registered with function B # This particular callback implements the rule "any extension of B # is automatically an extension of A as well" function_A.register(signature)(extension) (using PJE's terminology here, because I'm not sure of the right word for the extending functions when the concept is called "overloading") >> I like being able to use 'signature' and 'specialization' as the terms, too, >> as they describe exactly what's going on. > > Not to me. I think of "signature" as a tuple of types, and maybe > that's what you mean it to describe. But what's a specialization? Is > it a callable? Is it a string? What properties does a specialization > have? > > (This reminds me of the kind of conversation I sometimes have with > Orlijn. Sometimes he'll say "I saw a frizzlefrazzle today!" I'll say > "what's a frizzlefrazzle?" He'll say something that sounds to me like > "Not frizzlefrazzle, frizzlefrazzle!" After a few tries I give up, and > I have to change my line of questioning to get him to describe a > frizzlefrazzle. The most effective questions are usually "what does a > frizzlefrazzle do?" or "what does a frizzlefrazzle have?" That's what > I'm asking you here. :-) A specialization would be a tailored version of the function registered to deal with a particular signature. An adapter in protocol terms, an extension in PJE's terms, and, as noted above, I'm not sure what to call it when the concept is referred to as function overloading :) >> Further, if all builtin function >> objects (including C function descriptors and method wrappers) were >> potentially generic, > > This seems a rather extreme anticipation. I don't think anybody has > assumed this is where we might be going. There's certain attractive properties to doing it though - if we can get it down to the point where the runtime hit is minimal for non-extended functions, then it avoids the scenario "if only framework X had made function Y extensible, the fixing this would be easy!". OTOH, if there's an unavoidable call time cost, then a trivial @extensible or @overloadable decorator would do the trick. > Forgetting that str is really a type; assuming str were an @overloaded > function (I'm sticking to my terminology :-) then I could have written > that as > > @str.register(SomeClass) > def str_SomeClass(obj): > return obj.stringify() > > I'm not sure though what @specialize(str) adds to this. > > Assuming we can't change str into an @overloaded function but we do > want to allow this style of overriding, I guess we could have some > external registry object which str consults before looking for a > __str__ method or using hardcoded knowledge. And that's exactly where the benefits of having an "overload" or "extend" function comes in. Using PJE's terms, we might write: _cls_extensions = {} class ClassRegistry(object): """This would expose all the methods of an extensible function""" def __init__(self, cls): self.cls = cls self.registry = {} def for_signature(self, *signature): # This is the same as register() # but reads better with the preceding function call def helper(f): self.registry[signature] = f return f return helper @extend(extend).for_signature(type) def get_registry(cls): # This function retrieves the extension registry for a class try: registry = _cls_extensions[cls] except KeyError: registry = _cls_extensions[cls] = ClassRegistry(cls) return registry With a change in type.__call__() to check the class extension registry before calling cls.__new__(), the above would put us well on the way to making all types extensible. For example, we could make a class of our own behave like a dictionary, returning its keys when converted to a list, and its key-value pairs when converted to a dictionary: @extend(list).for_signature(MyMapping): def get_keys(obj) return obj.keys() @extend(dict).for_signature(MyMapping): def get_items(obj) return obj.items() Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sat Apr 8 06:45:12 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 08 Apr 2006 14:45:12 +1000 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <44373A46.6060900@gmail.com> References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> Message-ID: <44373FD8.3080504@gmail.com> Nick Coghlan wrote: > With a change in type.__call__() to check the class extension registry > before calling cls.__new__(), the above would put us well on the way to making > all types extensible. For example, we could make a class of our own behave > like a dictionary, returning its keys when converted to a list, and its > key-value pairs when converted to a dictionary: > > @extend(list).for_signature(MyMapping): > def get_keys(obj) > return obj.keys() > > @extend(dict).for_signature(MyMapping): > def get_items(obj) > return obj.items() Oops, better make that: @extend(list).for_signature(MyMapping): def get_keys(obj) return list(obj.keys()) @extend(dict).for_signature(MyMapping): def get_items(obj) return dict(obj.items()) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From pje at telecommunity.com Sat Apr 8 09:37:01 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 03:37:01 -0400 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <44373A46.6060900@gmail.com> References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> Message-ID: <5.1.1.6.0.20060408033158.03968f50@mail.telecommunity.com> At 02:21 PM 4/8/2006 +1000, Nick Coghlan wrote: >>>Further, if all builtin function >>>objects (including C function descriptors and method wrappers) were >>>potentially generic, >>This seems a rather extreme anticipation. I don't think anybody has >>assumed this is where we might be going. > >There's certain attractive properties to doing it though - if we can get >it down to the point where the runtime hit is minimal for non-extended >functions, then it avoids the scenario "if only framework X had made >function Y extensible, the fixing this would be easy!". Right, this is why I'm so obssessed about the performance - to ensure that the mechanism *can* be ubiquitous, and therefore will get used in preference to explicit type checks and just plain lack of extensibility. In today's CPython, you can manage this "delayed overloading" by swapping out func_code, but a uniform, higher-level mechanism would be more desirable, especially one that includes alternative Python implementations. From pje at telecommunity.com Sat Apr 8 10:28:24 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 04:28:24 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> Message-ID: <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> At 10:57 AM 4/7/2006 -0700, Guido van Rossum wrote: >Here I'm not so sure. Are you sure those results are unexpected? It >seems you are saying that if I have a class C deriving from A and B >(in that order) where both A and B implement a method foo(), and C >doesn't, the call C().foo() would be ambiguous. But in fact the whole >point of having a defined MRO is to foster the expectation that it >will call A's foo(), because that one comes first in the MRO. > >So at least for the single-argument variant I would think that my >algorithm gives results consistent with the definition of inheritance >in new-style classes. The ambiguity is only in context, or rather, the requirement for context. If you use the argument's MRO, then you can't statically determine the dominance hierarchy, because whether A or B dominates depends on the argument you're actually testing at runtime! This makes it harder for you to tell ahead of time whether you've defined the right methods. In particular, note that if you have cases for A and for B, you can easily be misled into thinking that one (let's say A) dominates the other because you currently only have class C(A,B). And you can then conclude that your list of cases is comprehensive and unambiguous -- until somebody creates class D(B,A) and suddenly your rules are ambiguous again. By the way, my rewrite of dominates() actually had a bug in it also, as did your original. Yours used an 'is' to compare the tuple signatures where it should've used ==, and mine checked individual types with 'is' instead of comparing the entire tuple with ==. Instead of: all(d is not s and issubclass(d,s) for d,s in zip(dom, sub)) I should've written: dom != sub and all(issubclass(d,s) for d,s in zip(dom, sub)) > > Unfortunately, this entire approach is subject to similar scale > > issues as PyProtocols/RuleDispatch. > >Even with the cache I put in? The hairy algorithm doesn't get invoked >more than once per actual signature (type tuple). You've measured it now, but yes, my own measurements of similar techniques in RuleDispatch showed the same effect, and as with your measurement, it is the Python code to build the tuple, rather than the tuple allocation itself, that produces the delays. The current version of RuleDispatch goes to some lengths to avoid creating any temporary data structures (if possible) for this reason. >I like the idea of analyzing the methods for dominance relationships >ahead of time. But I'm not sure why we should then resort to >generating code. Shouldn't it be just as easy to create a data >structure that can be walked by fixed code? (Just like a >recursive-descent parser can be written as a fixed "machine" that >interprets a data structure representing the parse information.) That's what RuleDispatch does currently, but it's again considerably slower than an AST-manipulation approach would be, and is also much slower when it has to handle expressions other than "plain old arguments". Calling it "generating code", however, is a misleading way of thinking about it, in that it implies something much more complex than is necessary. The data structure created to do dispatching is not much different from an AST, and in fact RuleDispatch uses its own simple AST for processing expressions anyway. The parts of RuleDispatch that process expressions are effectively recapitulating the Python interpreter, so I would much rather in a future version just take an AST right off of a function, like this: @some_function.when(lambda: x<27 and isinstance(y,int)) def twenty_seven_and_int(x,y): ... to replace my current use of string parsing for these conditions. At that point RuleDispatch would just basically be performing relatively simple AST pruning and grafting, to take the conditional expression(s) and function bodies and graft them into a larger dispatch tree. Since we already have the mechanisms to manage such ASTs and generate bytecode, and since alternative implementations are going to have to provide the same AST APIs at some point, ISTM that this is the natural way to go for the long term, and provides the most opportunities for customized dispatching techniques, such as RuleDispatch's custom method combinations. (Which allow you to control such things as whether there are "next methods" or not, among other things.) Note that such AST prune-and-graft operations could also potentially implement things like a 'next_method()' call, by substituting the function's previous AST. This would also allow you to do a simple monkeypatching style of overloading (i.e, just replacing the existing function body and chaining) for the very simplest cases with only a handful of methods. And, setting a function's AST needn't immediately cause compilation; it could be delayed until the function is actually called, thus nullifying any cost besides the tree grafting itself, for functions that aren't used in a particular program run. Granted, this definition-time cost might still be higher than the defintion-time cost of a more limited dispatch facility, such as the "tuple of types" approach you've been prototyping. In truth, my thoughts on AST munging are still essentially speculation; they reflect my design for the next version of RuleDispatch, rather than an implemented and tested design at this point. I should also clarify that I think of dispatch from a slightly different perspective, since RuleDispatch allows tests on arbitrary conditions and expressions. As a result, there are constraints on ordering of tests, because of things like a condition for "x>0 and y/x<27", where the second expression would produce an error if it was evaluated out of order. RuleDispatch already has the necessary machinery to manage these issues and build a single dispatch tree for interpretation, however, and it's actually not that complex: RuleDispatch in total is only about 4000 lines of Python, not including tests, but including all those blank lines I put between screenfuls of code. :) It does not reuse any of the stdlib compiler library, either, as I found it too slow and too big for the relatively simple task of processing logical expressions -- the expression processor is ~700 lines, or ~1100 if you include the concrete syntax->abstract syntax converter. Some of this machinery wouldn't be needed if there were a nice fast AST API and I could pull the ASTs directly from functions instead of parsing them. That leaves about 2900 lines of "meat": basically the index classes for handling class lookups and other comparison operators, and the code to build and interpret dispatch trees. The interpretation part would of course go away under an AST regime, while the rest would stay about the same. Naturally, if your goals are more modest, dealing only with actual arguments, no expressions, no operators other than 'isinstance()', no ability to apply boolean logic to the conditions (e.g. "isinstance(arg1,A) and isinstance(arg1,B)", or perhaps "not isinstance(arg1,C)!"), then the logic for an interpreter is indeed much simpler. In that case, you can just use a series of dictionaries that each argument is checked against, one at a time, and you can build those dictionaries in a straightforward way at registration time. In that approach, the lookup loop could look something like: reg = self.root_registry for arg in args: for cls in type(arg).__mro__: # not exactly, but sort of if cls in reg: reg = reg[cls] break else: raise NoMatch # (or fall back to default) # reg is now a function or a placeholder indicating ambiguity You wouldn't actually use __mro__, in order to avoid the unstable dominance hierarchy problem, but you get the gist. If you use RuleDispatch and restrict what operations you use, you'll actually get almost exactly the above; it's just that RuleDispatch may not evaluate the args in strictly left-to-right order, depending on how "good" an argument is for dispatching. (RuleDispatch evaluates the "goodness" of expressions (such as arguments) by how well they fan out and flatten the dispatch tree, so it may decide that the last argument is in fact a better thing to check first. This approach of course has more hash table lookups than your approach, so these are all tradeoffs to be considered, and more importantly, measured. But in general I tend to see this space through the lens of "arbitrary boolean expressions" and thus may occasionally intuit wrong answers for the more limited scope you're contemplating here. Actually, I keep feeling like you're "cheating" because you can get away with so little code -- and I keep having to remind myself that it's because there are so many things that the cached type tuple approach cannot. (The approach which you are using, and which other folks wrote papers about, prior to the Chambers & Chen paper on using dispatch trees to handle arbitrary predicates.) Anyway, I just got home from a 6+ hour flight, and I'm probably not being very coherent right now, so I'm going to go get some sleep. :) From martin at v.loewis.de Sat Apr 8 12:01:32 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 08 Apr 2006 12:01:32 +0200 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <44373866.9070305@alumni.rice.edu> References: <44373866.9070305@alumni.rice.edu> Message-ID: <443789FC.3000802@v.loewis.de> Vineet Jain wrote: > Is it wishful thinking > to have a python restricted exec mode which allows for: I would say so, yes: this needs some dedicated maintainer who comes up with an architecture, provides an initial implementation, and then monitors ongoing changes to make sure nobody can bypass the restrictions. I don't think any of the regular contributors is interested in that kind of project. Regards, Martin From ncoghlan at gmail.com Sat Apr 8 12:28:50 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 08 Apr 2006 20:28:50 +1000 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443789FC.3000802@v.loewis.de> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> Message-ID: <44379062.1070004@gmail.com> Martin v. L?wis wrote: > Vineet Jain wrote: >> Is it wishful thinking >> to have a python restricted exec mode which allows for: > > I would say so, yes: this needs some dedicated maintainer who > comes up with an architecture, provides an initial implementation, > and then monitors ongoing changes to make sure nobody can bypass > the restrictions. > > I don't think any of the regular contributors is interested in > that kind of project. I'm interested, but I'm also aware of how much work it would be. I'm disinclined to trust any mechanism which allows the untrusted code to run in the same process, as the implications of being able to do: self.__class__.__mro__[-1].__subtypes__() are somewhat staggering, and designing an in-process sandbox to cope with that is a big ask (and demonstrating that the sandbox actually *achieves* that goal is even tougher). And once you move to an out-of-process sandbox, then the only Python specific issue remaining is improving the support for inter-process communications. The security issues are then more in the domain of the OS: - controlling file access permissions on a per-process basis - controlling network access permissions on a per-process basis - limiting the memory usage of a process - limiting the CPU usage of a process Studying rexec/Bastion and their flaws, investigating OS-based solutions like chroot jails, looking at distributed processing frameworks like Kamaelia, and then figuring out how to combine the various concepts into a secure cross-platform Python restricted execution mechanism is something I would love to do. But it would only be feasible if some angel came along and offered me (or somebody else) a full-time job doing it, because there's no way I could do it in my spare time and make it work in a usable fashion. Something like this would also need the support of an organisation behind it in order to be able to jump on security problems quickly. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From walter at livinglogic.de Sat Apr 8 16:23:46 2006 From: walter at livinglogic.de (=?iso-8859-1?Q?Walter_D=F6rwald?=) Date: Sat, 8 Apr 2006 16:23:46 +0200 (CEST) Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> <4436B44F.5090407@livinglogic.de> Message-ID: <61479.89.54.32.6.1144506226.squirrel@isar.livinglogic.de> Guido van Rossum sagte: > On 4/7/06, Walter D?rwald wrote: >> An Interface is an abstract class that you subclass to make it a concrete implementation. > > That's not the view of most people who use the word interface these days. Not in Zope (see Fred Drake's post), not in Java. > >> If you have an adapter that adapts to the >> FooInterface why shouldn't it return a FooInterface object? > > Because implementing an interface and extending a class are separate concepts. > > Did you forget duck typing? Something can be a sequence without > subclassing a common base class. True, with adaption it's only relevant whether there's a dispatch condition in the registry that is true. The return type of the adaption call is irrelevant. But somehow this changes duck typing: It's no longer relevant which operation you want to perform (by doing EAFP with the method you want to call), but whether someone has registered a adapter with a dispatch condition that matches your object (which itself might use duck typing): @overload def sequence(obj): raise TypeError("can't convert to sequence") @sequence.when("hasattr(obj, '__getitem__') and hasattr(obj, '__len__')") def pythonseqassequence(obj): class Sequence(object): def __init__(self, obj): self.obj = obj def getitem(self, index): return self.obj[index] def len(self): return len(self.obj) return pythonseqassequence(obj) Servus, Walter From vinj at alumni.rice.edu Sat Apr 8 16:25:38 2006 From: vinj at alumni.rice.edu (Vineet Jain) Date: Sat, 08 Apr 2006 10:25:38 -0400 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <44379062.1070004@gmail.com> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> Message-ID: <4437C7E2.40402@alumni.rice.edu> Nick Coghlan wrote: > are somewhat staggering, and designing an in-process sandbox to cope > with that is a big ask (and demonstrating that the sandbox actually > *achieves* that goal is even tougher). I was thinking along the lines of: 1. Start a "light" python interpreter, which by default will not allow you to import anything including any of the standard python libraries. 2. This light python interpreter will have hooks around memory allocation: Limit maximum memory used by the user. So users should not be able to do: [1]*100000000000 bytecode execution: Limit time of execution of each python line. Users should not be able to: while True: i = i + 1 3. Have the ability to transfer objects between the main and the light python interpreter. user_interpreters = {} for user_name, user_module in all_user_modules: pl = python_light(memory_limit_per_instruction = "0.1M", execution_limit = "2S", memory_limit_interpreter="2M", allow_modules=None) pl.load_module(user_script) user_interpreters[user_name] = pl for user_name in user_interpreters: pl = user_interpreters[user_name] update_globa_env(pl) #application specific function which will setup data to be used by the python interpreter try: user_function_value = pl.user_function() except Exceeded_Memory:_Per_Instruction: except Exceeded_Execution_Time_Limit: except Exceeded_Total_Memory_Limit: handle_exception() It is up the the application builder to ensure that any and all modules/object that are added to the "light" python interpreter are secure. Vineet From guido at python.org Sat Apr 8 16:58:06 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Apr 2006 07:58:06 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> References: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> Message-ID: On 4/8/06, Phillip J. Eby wrote: > By the way, my rewrite of dominates() actually had a bug in it also, as did > your original. Yours used an 'is' to compare the tuple signatures where it > should've used ==, and mine checked individual types with 'is' instead of > comparing the entire tuple with ==. Instead of: > > all(d is not s and issubclass(d,s) for d,s in zip(dom, sub)) > > I should've written: > > dom != sub and all(issubclass(d,s) for d,s in zip(dom, sub)) But since d and s are types, and AFAIK == for types is defined as 'is', it doesn't matter? (I'm being distracted so I'll respond to more later.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 8 17:01:41 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Apr 2006 08:01:41 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> References: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> Message-ID: On 4/8/06, Phillip J. Eby wrote: > >Even with the cache I put in? The hairy algorithm doesn't get invoked > >more than once per actual signature (type tuple). > > You've measured it now, but yes, my own measurements of similar techniques > in RuleDispatch showed the same effect, and as with your measurement, it is > the Python code to build the tuple, rather than the tuple allocation > itself, that produces the delays. The current version of RuleDispatch goes > to some lengths to avoid creating any temporary data structures (if > possible) for this reason. Check out the accelerated version in time_overloading.py in the svn sandbox/overloading/. It's mostly faster than the manual version! Anyway, you seem to be confirming that it's the speed with which we can do a cache lookup. I'm not worried about that now. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 8 17:29:30 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Apr 2006 08:29:30 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> References: <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> Message-ID: On 4/8/06, Phillip J. Eby wrote: > Naturally, if your goals are more modest, dealing only with actual > arguments, no expressions, no operators other than 'isinstance()', no > ability to apply boolean logic to the conditions (e.g. "isinstance(arg1,A) > and isinstance(arg1,B)", or perhaps "not isinstance(arg1,C)!"), then the > logic for an interpreter is indeed much simpler. In that case, you can > just use a series of dictionaries that each argument is checked against, > one at a time, and you can build those dictionaries in a straightforward > way at registration time. Yes, I'd like to explore this approach first: (a) Java and C++ seem to get quite a bit of mileage out of it; (b) I'm not sure that *all* if/else tests at the top of a function should be frowned upon; the ones that do type tests are usually the most suspect. Of course, there are applications for more general predicates (like a dungeons&dragons game dispatcher) but to me they all seem a bit specialized to me. > In that approach, the lookup loop could look something like: > > reg = self.root_registry > for arg in args: > for cls in type(arg).__mro__: # not exactly, but sort of > if cls in reg: > reg = reg[cls] > break > else: > raise NoMatch # (or fall back to default) > > # reg is now a function or a placeholder indicating ambiguity I was thinking of something like this (nested dicts) as a strategy to deal with hundreds of registered methods. But I'm not sure if it matters in practice give that this code is only executed once to fill up the cache. I see more timing tests in my future... :-) > You wouldn't actually use __mro__, in order to avoid the unstable dominance > hierarchy problem, but you get the gist. If you use RuleDispatch and > restrict what operations you use, you'll actually get almost exactly the > above; it's just that RuleDispatch may not evaluate the args in strictly > left-to-right order, depending on how "good" an argument is for > dispatching. (RuleDispatch evaluates the "goodness" of expressions (such > as arguments) by how well they fan out and flatten the dispatch tree, so it > may decide that the last argument is in fact a better thing to check first. > > This approach of course has more hash table lookups than your approach, so > these are all tradeoffs to be considered, and more importantly, > measured. But in general I tend to see this space through the lens of > "arbitrary boolean expressions" and thus may occasionally intuit wrong > answers for the more limited scope you're contemplating here. Actually, I > keep feeling like you're "cheating" because you can get away with so little > code -- and I keep having to remind myself that it's because there are so > many things that the cached type tuple approach cannot. (The approach > which you are using, and which other folks wrote papers about, prior to the > Chambers & Chen paper on using dispatch trees to handle arbitrary predicates.) I'm not sure if it's worth the complexity. Also, responding to stuff I snipped that you wrote about access to ASTs is incredibly far in the future; it's too close to macros or extensible syntax, which I am explicitly ruling out from Python 3000 (I'm saying this so we don't have to engage in an endless discussion). > Anyway, I just got home from a 6+ hour flight, and I'm probably not being > very coherent right now, so I'm going to go get some sleep. :) And I have a 4-year-old next to me playing Elmo so I'm going to give my family some time! :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Sat Apr 8 19:44:55 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 09 Apr 2006 03:44:55 +1000 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <4437C398.9040600@alumni.rice.edu> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> Message-ID: <4437F697.2010606@gmail.com> Vineet Jain wrote: > Nick Coghlan wrote: >> are somewhat staggering, and designing an in-process sandbox to cope >> with that is a big ask (and demonstrating that the sandbox actually >> *achieves* that goal is even tougher). > I was thinking along the lines of: > > 1. Start a "light" python interpreter, which by default will not allow > you to import anything including any of the standard python libraries. But will it allow you to use numbers or strings? If yes, then you can get to object(), and hence to pretty much whatever C builtins you want. So its not enough to try to hide dangerous builtins like file(), you want to remove them from the light version entirely (routing all file system and network access requests through the main application). But if the file objects are gone, what happens to the Python machinery that relies on them (like import)? Python's powerful introspection is a severe drawback from a security POV - it is *really* hard to make a user stay in a box you put them in without crippling some part of the language as a side effect. So while I agree with your approach in principle, there's a big chunk of work hiding behind that word "light". What inconveniences can be tolerated in the restricted code when the payoff is that a user can trust that the code is unable to do anything "bad" to the system? What are the kinds of cases where rexec and Bastion broke down, and is it possible to avoid them? Can rexec/Bastion be fixed, or is it necessary to start from scratch? Would it make sense to modify the parser to disallow the use of getattr/setattr/delattr, and make any identifier using double-preceding and double-trailing underscores a SyntaxError? The latter disables Python-level access to most magic attributes, while the former would eliminate functions that otherwise provide easy workarounds for the latter restriction. This disables most introspection, permitting other techniques that might otherwise be easily worked around to be somewhat effective. And more fundamental technology questions: If the restricted interpreter runs in a separate process, should it be a CPython compile time option, or a full-blown fork? Or would something based on PyPy be a better idea? Does Python's introspection make it a better idea to go with an Access Control List based system rather than a Capability based system? IOW, I think there's a lot of work to be done just to figure out the real scope of the problem to be solved, as well as researching previous efforts (and why they failed) and comparable efforts (like the Java sandbox, or the .NET security model, and Lua's lightweight core), before proceeding on to try to create something new. And that's even leaving out the part about trying to persuade people that what you've built actually *is* secure, and being prepared to back that up by promptly fixing any discovered holes. Hmm, anyone looking for a thesis topic? ;) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From sdeibel at wingware.com Sat Apr 8 20:00:16 2006 From: sdeibel at wingware.com (sdeibel) Date: Sat, 8 Apr 2006 14:00:16 -0400 (EDT) Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: References: Message-ID: From: Nick Coghlan : > But it would only be feasible if some angel came along and offered me (or > somebody else) a full-time job doing it, because there's no way I could do it > in my spare time and make it work in a usable fashion. Something like this > would also need the support of an organisation behind it in order to be able > to jump on security problems quickly. Well, I remember this was one of the key goals Mitch Kapor outlined for Python in his keynote at PyCon. He blogged about it too: http://blogs.osafoundation.org/mitch/000559.html So maybe this is a possibility in the context of the OSAF? It's probably a bit ahead of what the PSF can commit to, but a good example of what it might fund in the future as an ongoing effort if it can't be sustained any other way. If you (or somebody) is interested enough, I could try to ask around at OSAF and mention it to the PSF board too. Just a thought... Stephan Deibel Chairman of the Board Python Software Foundation http://python.org/psf From ronaldoussoren at mac.com Sat Apr 8 21:00:41 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 8 Apr 2006 21:00:41 +0200 Subject: [Python-3000] List-Comp style for loops? In-Reply-To: References: Message-ID: <81E05B62-EBE8-4FDC-8CF2-54C89B8D7F82@mac.com> On 8-apr-2006, at 5:44, Guido van Rossum wrote: > On 4/7/06, Crutcher Dunnavant wrote: >> Basic idea: >> >> [for ... in ...]+ if ...: >> body >> >> A) it should be syntatically simple to parse. >> B) it establishes a nice symetry with list comprehensions. > > Are you in cahoots with the person who posted the first response to my > Artima blog on today? :-) > (http://www.artima.com/forums/flat.jsp?forum=106&thread=155514) > > I can't make heads or tails of this. What is it supposed to do? What > is it supposed to replace? My guess is that he doesn't get generic/overloaded/extensible functions and proposes a way to extend function bodies through some very odd syntax. foo = if x: print "x!" foo += elif y: print "y!" would probably be get you a function like this: def foo(): if x: print "x!" elif y: print "y!" I'd say this wouldn't be an improvement to python ;-) Ronald > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/ > ronaldoussoren%40mac.com From crutcher at gmail.com Sat Apr 8 21:31:23 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Sat, 8 Apr 2006 12:31:23 -0700 Subject: [Python-3000] List-Comp style for loops? In-Reply-To: <81E05B62-EBE8-4FDC-8CF2-54C89B8D7F82@mac.com> References: <81E05B62-EBE8-4FDC-8CF2-54C89B8D7F82@mac.com> Message-ID: On 4/8/06, Ronald Oussoren wrote: > > On 8-apr-2006, at 5:44, Guido van Rossum wrote: > > > On 4/7/06, Crutcher Dunnavant wrote: > >> Basic idea: > >> > >> [for ... in ...]+ if ...: > >> body > >> > >> A) it should be syntatically simple to parse. > >> B) it establishes a nice symetry with list comprehensions. > > > > Are you in cahoots with the person who posted the first response to my > > Artima blog on today? :-) > > (http://www.artima.com/forums/flat.jsp?forum=106&thread=155514) > > > > I can't make heads or tails of this. What is it supposed to do? What > > is it supposed to replace? > > My guess is that he doesn't get generic/overloaded/extensible functions > and proposes a way to extend function bodies through some very odd > syntax. > > foo = if x: > print "x!" > > foo += elif y: > print "y!" > > would probably be get you a function like this: > > def foo(): > if x: > print "x!" > elif y: > print "y!" > > I'd say this wouldn't be an improvement to python ;-) Perhaps Guido's poster is suggesting this, but I'm not. I don't want assignment from loops, I just want the same semantics we get in list comps in the for ... if ... section. > > > > -- > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > _______________________________________________ > > Python-3000 mailing list > > Python-3000 at python.org > > http://mail.python.org/mailman/listinfo/python-3000 > > Unsubscribe: http://mail.python.org/mailman/options/python-3000/ > > ronaldoussoren%40mac.com > > -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From pje at telecommunity.com Sat Apr 8 22:29:21 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 16:29:21 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060408162440.01e2b148@mail.telecommunity.com> At 07:58 AM 4/8/2006 -0700, Guido van Rossum wrote: >On 4/8/06, Phillip J. Eby wrote: > > By the way, my rewrite of dominates() actually had a bug in it also, as did > > your original. Yours used an 'is' to compare the tuple signatures where it > > should've used ==, and mine checked individual types with 'is' instead of > > comparing the entire tuple with ==. Instead of: > > > > all(d is not s and issubclass(d,s) for d,s in zip(dom, sub)) > > > > I should've written: > > > > dom != sub and all(issubclass(d,s) for d,s in zip(dom, sub)) > >But since d and s are types, and AFAIK == for types is defined as >'is', it doesn't matter? It matters that the comparison is moved out of the loop, because before it was basically was saying that a pair of signatures were ambiguous if any of the types were the same! That was the bug in my version. The bug in your version was that there was no way for dom to be 'is' sub, because they were tuples. Oh wait -- your version was okay, because the only case where dom==sub in your implementation is the case where you compared a signature against itself. My intuition failed me here because in RuleDispatch it *is* possible to register multiple methods with the same signature, so an 'is' comparison would be inadequate, and a dominance comparison would require an equality test. From pje at telecommunity.com Sat Apr 8 22:32:46 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 16:32:46 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060408163048.01e17198@mail.telecommunity.com> At 08:01 AM 4/8/2006 -0700, Guido van Rossum wrote: >Check out the accelerated version in time_overloading.py in the svn >sandbox/overloading/. It's mostly faster than the manual version! > >Anyway, you seem to be confirming that it's the speed with which we >can do a cache lookup. I'm not worried about that now. Btw, the cache should probably hold tuples of weakrefs to types, so as not to leak classes in programs that create and discard classes dynamically. (E.g., they create temporary classes in closures that get called a lot.) From guido at python.org Sat Apr 8 22:39:14 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Apr 2006 13:39:14 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <44373A46.6060900@gmail.com> References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> Message-ID: On 4/7/06, Nick Coghlan wrote: > Mostly just the API docs: > http://peak.telecommunity.com/protocol_ref/module-protocols.html I just finally found this too. PJE's website is really a confusing maze; the only *source* for PyProtocols I've found so far is a CVS repository that seems to be telling me not to use it because it's been converted to SVN. But the SVN repository doesn't seem to have PyProtocols, only PEAK. :-( Anyway, the quote at the top of that page also explains why I'm still grappling for use cases: I don't think of my own use of Python as that of "an integrator, someone who is connecting components from various vendors." Or at least, that's not what it feels like. I write plenty of code that uses multiple 3rd party libraries; for example my current Google project blends Django templates, wsgiref, a home-grown Perforce wrapper, a Google-internal database-ish API, and some other minor Google-specific APIs. But it doesn't *feel* to me like there's a need to adapt one package's API to that of another; instead, my application is in control and makes calls into each package as needed. I wonder if it would feel different to you, Phillip and Alex, and I'm just not seeing the opportunities for adaptation, or if it really does depend on the typical application. > This was the specific page about using a concrete API on the protocols themselves: > http://peak.telecommunity.com/protocol_ref/protocols-calling.html Aha! That's probably the earliest description of "adaptation by calling the protocol/interface object" and exactly the lightbulb idea in my blog. [...] > Another way of looking at it: > > 1. Suppose function A and function B do the roughly the same thing That's not a very clear way of saying it, but I get it. > 2. Suppose function A can be overloaded/extended > > 3. It would then be straightforward to add additional extensions to > function A that delegated certain call signatures to function B (possibly > with a wrapper to deal with slight differences in the order of arguments or > the type of the return value. > > Turning that into *transitivity* is a matter of asking the question, "well , > what if B can be overloaded as well?". The idea then is that you would pass B > a callback to be invoked whenever a new signature was registered with B. The > callback would take the new signature, rearrange it if necessary, and register > the new version with function A. Hm, couldn't you do this is A's default method? > @function_B.call_for_new_registrations > def extending_B_extends_A_too(signature, extension): > # The decorator means this function would be called whenever a new > # extension was registered with function B > # This particular callback implements the rule "any extension of B > # is automatically an extension of A as well" > function_A.register(signature)(extension) > > (using PJE's terminology here, because I'm not sure of the right word for the > extending functions when the concept is called "overloading") I've been use "method" instead of "extension" or "specialization". Most people already know that a method is a callable, and I've heard a few others call it that too. OTOH it's quite different from methods elsewhere in Python so I'm not too keen on it either. [...] > There's certain attractive properties to doing it though - if we can get it > down to the point where the runtime hit is minimal for non-extended functions, > then it avoids the scenario "if only framework X had made function Y > extensible, the fixing this would be easy!". That's not the feeling that occurred to me when I was writing my Google project (see above). I can think of a few things that were inflexible. For example, the version of Django I'm using doesn't let you override how it searches for templates, and I had to monkey-patch the implementation to allow requesting the template source from an abstract API that gets it from a zip file, rather than going to the filesystem directly. But I don't see how adaptation or extensible functions would have helped. I suppose you could *reframe* it to benefit from adaptation, by proposing that there should be a "template source" object, and then all you have to do is (a) replace the template source with a zip file, and (b) write an adapter from zip files to the required "template source" protocol. But that's a much more elaborate refactoring than the simple load-template hook that would have solved my problem. You could even say that the load-template hook existed -- my monkey-patching required replacing an existing two-line function with one I wrote (and arguably the hook *did* exist, in the form of this single function that acted as a choke point for the entire template loading system). > OTOH, if there's an unavoidable call time cost, then a trivial @extensible or > @overloadable decorator would do the trick. Before we worry about the implementation overhead, we need to make sure that the resulting paradigm shift isn't going to destroy Python as we know it. We already have well-established and well-understood machinery for method lookup in a class hierarchy. Are we sure that replacing all that with overloadable/extensible functions is really a good idea? While classes are technically mutable, in practice few classes are modified at run-time. The overloadable functions approach seems to encourage tinkering with globally defined (and used) functions. For example, what if one library overloads bool(str) so that bool("False") == False, while another library assumes the traditional meaning (non-zero string => True)? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Sat Apr 8 22:53:39 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 16:53:39 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> <7.0.1.0.0.20060406215753.0218f800@telecommunity.com> <5.1.1.6.0.20060408033723.03962670@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060408163350.037303a8@mail.telecommunity.com> At 08:29 AM 4/8/2006 -0700, Guido van Rossum wrote: >Of course, there are applications for more general predicates (like a >dungeons&dragons game dispatcher) but to me they all seem a bit >specialized to me. Sure - but it would certainly be nice if RuleDispatch could tie into the Python-supplied mechanism and extend it to offer the predicates. They're also quite useful for data-driven security rules: http://peak.telecommunity.com/DevCenter/SecurityRules and other pricing/business applications: http://peak.telecommunity.com/DevCenter/CombiningResults So it's not just games that benefit. :) >I'm not sure if it's worth the complexity. Fair enough - check out the use cases illustrated by the pages above. Again, I just want to be able to make RuleDispatch look like an extension of the base mchanism; I'm not trying to convince you to implement all of RuleDispatch's features as part of Py3K. :) (Although that would certainly be cool. ;) ) > Also, responding to stuff I >snipped that you wrote about access to ASTs is incredibly far in the >future; it's too close to macros or extensible syntax, How is it extensible syntax if you have to use a *Python* AST? I also don't think macros in the general sense can be practically implemented using mere AST access of the type I described, since *each* function using such "macros" would have to be decorated individually. IMO it would be a PITA at best, especially since you couldn't change Python's syntax in any way. To me an AST access feature would only be useful for changing the "VM" that Python code runs on. For example, there is an object-relational mapper out there (I think it's Robert Brewer's DejaVu system) that takes Python generator expressions and converts them into SQL. Right now it does it by bytecode hacking, because the only alternative is to parse a string. From pje at telecommunity.com Sat Apr 8 22:59:26 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 16:59:26 -0400 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: <44373A46.6060900@gmail.com> <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> Message-ID: <5.1.1.6.0.20060408165520.03734780@mail.telecommunity.com> At 01:39 PM 4/8/2006 -0700, Guido van Rossum wrote: >For example, what if one library overloads bool(str) so >that bool("False") == False, while another library assumes the >traditional meaning (non-zero string => True)? Presumably, the author of that code would be taken out and shot. :) I suspect, however, we should look to Ruby for an answer to that question, since it's possible for such things to happen there as well. i.e., "consenting adults" is therefore much more of a requirement there, especially the "adult" part. :) From pje at telecommunity.com Sat Apr 8 23:22:35 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 17:22:35 -0400 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: <44373A46.6060900@gmail.com> <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> Message-ID: <5.1.1.6.0.20060408170200.0374bf18@mail.telecommunity.com> At 01:39 PM 4/8/2006 -0700, Guido van Rossum wrote: >On 4/7/06, Nick Coghlan wrote: > > Mostly just the API docs: > > http://peak.telecommunity.com/protocol_ref/module-protocols.html > >I just finally found this too. PJE's website is really a confusing >maze; the only *source* for PyProtocols I've found so far is a CVS >repository that seems to be telling me not to use it because it's been >converted to SVN. But the SVN repository doesn't seem to have >PyProtocols, only PEAK. :-( > >Anyway, the quote at the top of that page also explains why I'm still >grappling for use cases: Don't bother - PyProtocols has few use cases that aren't better served by overloadable functions, even of the single-dispatch kind. The use cases for overloadable functions, in turn, tend to break down into single dispatch, double dispatch, and predicate dispatch. I haven't actually encountered a use case for dispatching on 3 or more arguments that didn't also require some other kind of predicate besides 'isinstance()'. > I don't think of my own use of Python as that >of "an integrator, someone who is connecting components from various >vendors." Or at least, that's not what it feels like. I write plenty >of code that uses multiple 3rd party libraries; for example my current >Google project blends Django templates, wsgiref, a home-grown Perforce >wrapper, a Google-internal database-ish API, and some other minor >Google-specific APIs. But it doesn't *feel* to me like there's a need >to adapt one package's API to that of another; instead, my application >is in control and makes calls into each package as needed. Those are mostly libraries, not frameworks, and for the most part you're not *integrating* them. You're not trying to make Perforce store information in your database, or to put a database API over Perforce, for example. > I wonder if >it would feel different to you, Phillip and Alex, and I'm just not >seeing the opportunities for adaptation, or if it really does depend >on the typical application. It depends mainly on the library. A lot of PEAK is there because existing Python libraries or frameworks simply weren't extensible or configurable enough for the tasks my team was doing at Verio. It was very, very frustrating, because there were a lot of libraries that were almost perfect - except for some dependency on an internal type that couldn't be mixed with the types from some other library. Source patching or monkey patching just didn't seem like a viable solution for a lot of these problems. My later work on setuptools was inspired by realizing that the real answer wasn't reinventing competing wheels, but rather to build more roads so that more people would actually use the wheels and complain to the wheels' makers if they weren't round enough. :) >That's not the feeling that occurred to me when I was writing my >Google project (see above). I can think of a few things that were >inflexible. For example, the version of Django I'm using doesn't let >you override how it searches for templates, and I had to monkey-patch >the implementation to allow requesting the template source from an >abstract API that gets it from a zip file, rather than going to the >filesystem directly. But I don't see how adaptation or extensible >functions would have helped. Well, it would've given you a less obscure way of monkeypatching, since it would be part of the language. And, the authors would be more likely to think about "what functions might people want to overload?" and therefore include choke points of this kind. If everything's overloadable, then offering users choices is cheap. Here's an example of a problem I ran into: pydoc and epydoc hate things with custom metaclasses, callable objects not of FunctionType or MethodType, and often any descriptor that wasn't a built-in type (like classmethod, property, etc.). They both have big if-then trees dispatching on type. So, I simply couldn't use them to generate documentation, even though everything *else* about them was great, without extensive patching (because the trees were spread out everywhere in their code bases). If overloading was in the language, however, the One Obvious Way to implement those libraries would've been overloading -- which means they would have been extensible as a *natural side effect*. So this is the way in which overloading will be *good* for the language, because in this case, the right thing for the library user will become the right thing for the library author as well. From guido at python.org Sun Apr 9 00:26:21 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Apr 2006 15:26:21 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <5.1.1.6.0.20060408170200.0374bf18@mail.telecommunity.com> References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> <5.1.1.6.0.20060408170200.0374bf18@mail.telecommunity.com> Message-ID: On 4/8/06, Phillip J. Eby wrote: > Those are mostly libraries, not frameworks, and for the most part you're > not *integrating* them. You're not trying to make Perforce store > information in your database, or to put a database API over Perforce, for > example. OK -- then I feel vindicated that the characterization of "most Python programmers" as integrators doesn't apply to me. I guess I'm not a typical Python programmer anyway. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Sun Apr 9 02:05:17 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat, 08 Apr 2006 20:05:17 -0400 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: <5.1.1.6.0.20060408170200.0374bf18@mail.telecommunity.com> <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> <5.1.1.6.0.20060408170200.0374bf18@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060408195301.01e08128@mail.telecommunity.com> At 03:26 PM 4/8/2006 -0700, Guido van Rossum wrote: >On 4/8/06, Phillip J. Eby wrote: > > Those are mostly libraries, not frameworks, and for the most part you're > > not *integrating* them. You're not trying to make Perforce store > > information in your database, or to put a database API over Perforce, for > > example. > >OK -- then I feel vindicated that the characterization of "most Python >programmers" as integrators doesn't apply to me. I guess I'm not a >typical Python programmer anyway. :-) I suspect that the perspective of people like Alex, Jim, and myself is distorted by our being in consulting organizations (like Strakt, my group at Verio, and Zope corp.) where the focus is on having a toolkit that can be used to produce *multiple* applications, or configurable applications. Frameworks, in other words. Verio does a lot of "private label" hosting, where each client has significantly different business rules. Being able to have a core framework whose behavior can be significantly changed by just importing a different ruleset is a very useful concept. Indeed, the one nibble I got about possible consulting involving RuleDispatch was from another firm doing private label web services of a different sort; the same kind of needs apply. I suspect that Zope and Strakt have had similar influences shaping their perception of what a "typical Python programmer" is or does, due to the nature of consulting work. I doubt, however, that we are really "typical", in the sense that most Python programmers do not work for consulting organizations that maintain an organizational "stash" of tools used to create the same kinds of applications over and over again, with variations. They thus don't obtain any economic leverage from having tools that allow extreme customization while retaining modularity. So, approaches that may seem reasonable to you or other typical Python programmers will sometimes be objected to by folks like Alex and Jim and I, in that we will tend to think of a whole series of related applications being written with the approach, rather than just one. It probably also explains why we are usually so intently interested in "meta" features and often pioneer them; again we're thinking about writing entire families of programs. Ironically, this no longer applies to me from a career perspective; I'm not in that business any more. It frequently surprises me when I realize how easy it is to use what I think of as cheap hacks -- but which are quite adequate approaches if you intend to write only one program of the kind you are working on. :) Anyway, as to Alex's "most Python programmers are integrators" comment, I used to think it was reasonable. And it probably sounded reasonable in his organization, and in Zope's. After all, "most Python programmers" in such organizations would be a reasonable assessment. It's just not reasonable in the overall Python ecosphere. From greg.ewing at canterbury.ac.nz Sun Apr 9 02:36:13 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 09 Apr 2006 12:36:13 +1200 Subject: [Python-3000] Kill "generic functions"! In-Reply-To: <1144401259.5722.20.camel@fsol> References: <7.0.1.0.0.20060406090915.0202c180@telecommunity.com> <44354AD4.90801@colorstudy.com> <7.0.1.0.0.20060406101627.020cbd80@telecommunity.com> <7.0.1.0.0.20060406121637.0218a898@telecommunity.com> <1144363964.5672.71.camel@fsol> <1144401259.5722.20.camel@fsol> Message-ID: <443856FD.90602@canterbury.ac.nz> Antoine Pitrou wrote: > 1. "Overloading" and "overriding" look very much the same for most > people. In French the same word "surcharge" is used for both. In statically typed languages, the distinction is that overloading is resolved at compile time, whereas overriding is resolved (at least partly) at run time. In Python, everything is resolved at run time, so it's all overriding. > 2. Python does have "operrator overloading". Although by the above definition, it should really be called "operator overriding". -- Greg From nnorwitz at gmail.com Sun Apr 9 03:18:13 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sat, 8 Apr 2006 18:18:13 -0700 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <44373866.9070305@alumni.rice.edu> References: <44373866.9070305@alumni.rice.edu> Message-ID: On 4/7/06, Vineet Jain wrote: > to have a python restricted exec mode which allows for: > > 1. Limit the memory consumed by the script > 2. Limit access to file system and other system resources > 3. Limit cpu time that the script will take > 4. Be able to specify which modules are available for import I don't think any of these issues are necessarily 3k only. I will probably implement #1 (or perhaps find someone to do it). I think it is useful to limit memory and should be pretty easy to do. I started thinking about this a few weeks ago. I probably won't get around to it before 2.5 goes final. Without thinking much, I expect this would be a special build (ie, require a compile time flag to activate). #3 is easy to do a simple, naive implementation. I don't know what your needs are. If you just want to say "exit this script if it ran more than N seconds" you can just modify the eval loop (*). But I suspect there are many more features that would be required and it's not so simple. As other people have discussed in this thread, the others are very difficult in total. No one has volunteered to help lead such a task. It would be great if you found a bunch of interested people and were able to complete all of your ideas. n -- (*) here's the naive impl. You could use cpu time instead of wall time if you wanted. Making this a real patch is left as an exercise to the reader. Index: Python/ceval.c =================================================================== --- Python/ceval.c (revision 43738) +++ Python/ceval.c (working copy) @@ -16,6 +16,8 @@ #include +static time_t stop_time = 0; + #ifndef WITH_TSC #define READ_TIMESTAMP(var) @@ -744,6 +746,9 @@ assert(stack_pointer != NULL); f->f_stacktop = NULL; /* remains NULL unless yield suspends frame */ + /* XXX: This is just here to initialize. It should be elsewhere. */ + if (!stop_time) + stop_time = time(NULL) + 5; #ifdef LLTRACE lltrace = PyDict_GetItemString(f->f_globals, "__lltrace__") != NULL; #endif @@ -842,6 +847,11 @@ } fast_next_opcode: + if (stop_time < time(NULL)) { + fprintf(stderr, "Exceeded maximum time allowed.\n"); + exit(1); + } + f->f_lasti = INSTR_OFFSET(); /* line-by-line tracing support */ From greg.ewing at canterbury.ac.nz Sun Apr 9 03:22:06 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 09 Apr 2006 13:22:06 +1200 Subject: [Python-3000] List-Comp style for loops? In-Reply-To: References: Message-ID: <443861BE.5090508@canterbury.ac.nz> Crutcher Dunnavant wrote: > Basic idea: > > [for ... in ...]+ if ...: > body -1, not enough advantage over writing out the nested loops. -- Greg From aahz at pythoncraft.com Sun Apr 9 03:28:08 2006 From: aahz at pythoncraft.com (Aahz) Date: Sat, 8 Apr 2006 18:28:08 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: <5.1.1.6.0.20060408195301.01e08128@mail.telecommunity.com> References: <5.1.1.6.0.20060408170200.0374bf18@mail.telecommunity.com> <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> <5.1.1.6.0.20060408170200.0374bf18@mail.telecommunity.com> <5.1.1.6.0.20060408195301.01e08128@mail.telecommunity.com> Message-ID: <20060409012808.GA13622@panix.com> On Sat, Apr 08, 2006, Phillip J. Eby wrote: > > I suspect that the perspective of people like Alex, Jim, and myself is > distorted by our being in consulting organizations (like Strakt, my > group at Verio, and Zope corp.) where the focus is on having a toolkit > that can be used to produce *multiple* applications, or configurable > applications. Frameworks, in other words. Well, that's my current job, too. And we do a fair number of things where adaptation or something like it might help. However, my perspective as a "peon programmer" is that such toys make it more difficult to figure out what the system is doing. (We have a fairly elaborate plugin-and-hooks mechanism.) I don't think adaptation by itself would help much; at most it would change the framework structure a bit. I'm not arguing against adaptation, mind, I just wonder which people are going to benefit. Keep in mind that we're still stuck with Python 2.2 as our limiting version and that our application was started with 1.4, so we haven't really even thought of trying out new-style classes. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Look, it's your affair if you want to play with five people, but don't go calling it doubles." --John Cleese anticipates Usenet From nnorwitz at gmail.com Sun Apr 9 03:42:26 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sat, 8 Apr 2006 18:42:26 -0700 Subject: [Python-3000] drop vars() builtin? Message-ID: Is there a benefit to keeping vars()? "vars([object]) -> dictionary\n\ \n\ Without arguments, equivalent to locals().\n\ With an argument, equivalent to object.__dict__."); n From greg.ewing at canterbury.ac.nz Sun Apr 9 04:06:58 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 09 Apr 2006 14:06:58 +1200 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <4437F697.2010606@gmail.com> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> Message-ID: <44386C42.90803@canterbury.ac.nz> Nick Coghlan wrote: > If yes, then you can get to object(), and hence to pretty much whatever C > builtins you want. Perhaps instead of attributes like __class__ and __subtypes__ there should be functions like classof() and subtypes() that could be excluded from a restricted environment. > But if > the file objects are gone, what happens to the Python machinery that relies on > them (like import)? Regardless of security issues, I think it would be a good idea to make the core machinery independent of the existence of concrete things like file(). This would make it easier to build Python interpreters for unusual environments which aren't based on a traditional OS and file system. -- Greg From guido at python.org Sun Apr 9 04:30:06 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 8 Apr 2006 19:30:06 -0700 Subject: [Python-3000] drop vars() builtin? In-Reply-To: References: Message-ID: On 4/8/06, Neal Norwitz wrote: > Is there a benefit to keeping vars()? > > "vars([object]) -> dictionary\n\ > \n\ > Without arguments, equivalent to locals().\n\ > With an argument, equivalent to object.__dict__."); I dunno; but let's not rush it. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From talin at acm.org Sun Apr 9 07:27:46 2006 From: talin at acm.org (Talin) Date: Sun, 9 Apr 2006 05:27:46 +0000 (UTC) Subject: [Python-3000] Sane transitive adaptation References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> Message-ID: Guido van Rossum python.org> writes: > Before we worry about the implementation overhead, we need to make > sure that the resulting paradigm shift isn't going to destroy Python > as we know it. We already have well-established and well-understood > machinery for method lookup in a class hierarchy. Are we sure that > replacing all that with overloadable/extensible functions is really a > good idea? While classes are technically mutable, in practice few > classes are modified at run-time. The overloadable functions approach > seems to encourage tinkering with globally defined (and used) > functions. For example, what if one library overloads bool(str) so > that bool("False") == False, while another library assumes the > traditional meaning (non-zero string => True)? You know, on the one hand, I've been reading this thread, and I'm excited about the possibility of generic functions (dynamic dispatch, duckaplex-typing, whatever you want to call it), but at the same time I kind of wonder about the eventual effect on the language. For example, it seems to me that most of the various magic methods, such as __cmp__, __add__, etc. would be better done with generics, especially since it gives you much finer control over the type of the other argument. So instead of having to write: class X: def __cmp__( self, other ): if isinstance( other, X ): return cmp( self.something, other.something ) return NotImplemented # or raise TypeError or whatever Instead you can just write: @generic( X, X ) cmp( a, b ): return cmp( a.something, b.something ) Similarly, if you add an integer to a string, the overloading machinery will report the error, instead of having to check it manually in the __add__ method. The addition of a funamentally new method of control flow - especially one that is a superset of an existing control flow method - is something that could potentially ripple through all of the standard libraries, transforming them beyond recognition. Given the obvious advantages, it would be hard to justify sticking with the older methods of doing things. While that's cool and I would look forward to working with them, I recognize that there are some downsides. For one thing, if generic functions are used in a way that truly leverages their potential, the structure of the code is going to be so different from 2.x that there won't be any possibility of automated migration - unless you decide up front that the use of generics is going to be restrained and controlled, at least initially. Personally I'm fine with breaking from the past. But I wanted to point out what some of the potential consequences would be. -- Talin From talin at acm.org Sun Apr 9 08:13:21 2006 From: talin at acm.org (Talin) Date: Sun, 9 Apr 2006 06:13:21 +0000 (UTC) Subject: [Python-3000] string.format Message-ID: I was browsing through the C# documentation for the String.format function, because I'd noticed that a number of the suggestions made in the string formatting thread were syntactically similar to what is used in .Net. A quick review of .Net string formatting: Substitution fields are specified using a number in braces: {0}, where the number indicates the argument number to substitute (named arguments are not supported.) Formatting options are indicated by putting a colon after the number, followed by a formatting string: {0:8x}, for example, to indicate an 8-digit hex number. Note that the formatting options are *options*, they are not type specifiers. There is no equivalent to %s or %d, and in a dynamically typed language, there is little need for such. One interesting thing is that the formatting options are interpreted differently for different types. Thus, if printing a date, you can specify something like {0:dd:mm:yy}, whereas this same formatting option would be meaningless in the case of an integer. (Note that only the first colon is interpreted as a separator, the rest are passed on to the type-specific function uninterpreted.) There are two ways in which you can customize the interpretation of the formatting string. If you are writing a subclass, you can overload the ToString() function, which is like Python's __str__ function, except that it takes an optional format string (which is simply the literal text of the format options following the colon.) The other method is to supply a 'custom formatter' which is passed in as an argument to String.Format, and which can override the various object's decisions as to how to present themselves. (I wonder if generic dispatch would be useful here - being able to override the interpretation of the format string on a per-type basis might be a cleaner way to do it. Problem is, however, you might want to have different overrides at different times.) In general, I kind of like the brace syntax, assuming of course that named arguments would be supported. I agree that $name, while popular, can be troublesome, and while I'm used to using ${name} with Kid templates, I wonder if one really needs both the prefix and the braces, especially when you consider that they are only meaningful when calling format - it's not like Perl where $name works on every string. Other details: In .Net literal braces are escaped with {{ and }}, however \{ and \} seem a bit more consistent to me. I am definately not in favor of being able to put arbitrary expressions embedded in strings (its easy enough to move the expression outside, and its a potential security hole), however it seems to me that you might want to add a few convenience features for cases where you are just passing in locals() as a dict. Perhaps it would be enough to support just dot (.) and brackets: "{name.name} and {name[index]}" Of course, there's no guarantee that the __getattr__ and __getitem__ functions aren't going to do something potentially bad when invoked by a maliciously-written format string, but when it comes to that, there's no guarantee that __str__ has been written sanely either! In general, however, its a good bet that for a given random object, __getattr__ and __getitem__ have fewer side effects than __call__. All right I'm done. Back to my other Python programming... :) -- Talin From listsub at wickedgrey.com Fri Apr 7 21:58:46 2006 From: listsub at wickedgrey.com (Eli Stevens (WG.c)) Date: Fri, 07 Apr 2006 12:58:46 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: References: <44315A89.9000104@colorstudy.com> <443292E8.9000508@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> <4436B44F.5090407@livinglogic.de> Message-ID: <4436C476.7010802@wickedgrey.com> Guido van Rossum wrote: > On 4/7/06, Walter D?rwald wrote: >> An Interface is an abstract class that you subclass to make it a >> concrete implementation. > > That's not the view of most people who use the word interface these > days. Not in Zope (see Fred Drake's post), not in Java. > >> If you have an adapter that adapts to the >> FooInterface why shouldn't it return a FooInterface object? > > Because implementing an interface and extending a class are separate concepts. > > Did you forget duck typing? Something can be a sequence without > subclassing a common base class. I'm curious what effect overloaded functions will have on duck typing; a Something can act like a list, but from my understanding of the discussion* it won't end up matching: @pprint.register(list) def pprint_list(obj): pass It seems to me that now to get a duck-typed list, not only do you have to implement all of special methods that define "listy-ness," you also have to find the overloaded functions that are specialized for lists, and register your own implementation (or an implementation specialized for lists, if appropriate). If a consumer of your listy class is also a consumer of some separate overloaded functions that operate on lists, will they be responsible for registering your listy class into the overloaded functions? How does that work when the overloaded functions are an implementation detail of the other library they happen to use? I think there's a lot of interesting potential here; but the reliance on object type seems to imply that you can't mix it with duck typing, and I'm not sure if that's being fully considered. Eli * - I haven't been able to read all of the posts on this topic, but I've tried to skim the threads as deeply as I'm able. Sorry if this has been brought up already. From g.brandl at gmx.net Sun Apr 9 12:18:31 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 09 Apr 2006 12:18:31 +0200 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <44386C42.90803@canterbury.ac.nz> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> <44386C42.90803@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Nick Coghlan wrote: > >> If yes, then you can get to object(), and hence to pretty much whatever C >> builtins you want. > > Perhaps instead of attributes like __class__ and __subtypes__ > there should be functions like classof() and subtypes() that > could be excluded from a restricted environment. Even as properties and methods, they can be restricted, like func_code is now. Georg From vinj at alumni.rice.edu Sun Apr 9 15:32:24 2006 From: vinj at alumni.rice.edu (Vineet Jain) Date: Sun, 09 Apr 2006 09:32:24 -0400 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: References: <44373866.9070305@alumni.rice.edu> Message-ID: <44390CE8.9000403@alumni.rice.edu> > around to it before 2.5 goes final. Without thinking much, I expect > this would be a special build (ie, require a compile time flag to > activate). > python_light should be built just like any other module that you can import into a CPython application. > your needs are. If you just want to say "exit this script if it ran > more than N seconds" you can just modify the eval loop (*). But I This is all that is needed. Any more logic for time management should be added to the application. I should be able to say: import python_light pl = python_light.get_new_interpreter() pl.load_module('max_module_memory': '5M', 'time_limit':'2S', 'allow_imports':False, 'allow_introspection': False}, module_name) pl.module_name.global_variable = [1,2,3] #add variables to the user modules pl.module_name.global_function = some_custom_function #add functions to the user module return_value = pl.run_function({'time_limit':'10S', 'allow_imports':False, 'allow_introspection': False}, module_name, func, params, *args, **kwargs) pl.remove(module_name) By default: 1. Python_light will not allow_introspection will not allow the use of getattr/setattr/delattr, make any identifier using double-preceding and double-trailing underscores. 2. Will not allow use of eval or access to locals() or vars() 3. Not allow any imports 4. Restrict memory usage to a default (of say 10M) 5. Limit time_of_execution to each call to python_light of 10S Its important (at least in my use case) that the python_light application runs in the same process and you are able to initialize the user_module with global variables (which are of standard python types). Most of my users will not know python, let alone be programmers, so the scripts and functions they write will be fairly simple. It is up to the application builder to ensure that any data type (not of a standard python type: number, string, boolean, and list or tuple of standard python types) function, or module passed to python_light is secure. Couple of questions/notes: 1. It is assumed that python_light will pass on exceptions in user_scripts 2. Given all of the above restrictions, will python_light be 100% secure? If not, are there other restrictions we can add to make it 100% secure? 3. You should be able to instantiate multiple python_light interpreters in one CPython application Nice to Have: 1. Make load_time of python_light faster than that of CPython. 2. Reduce default memory_required by each instance of a python_light interpreter > difficult in total. No one has volunteered to help lead such a task. > It would be great if you found a bunch of interested people and were > able to complete all of your ideas. > My needs are immediate. Nothing would make me happier, if I could do all the above without leaving python. I'm guessing that my users would also be happy with python (than say Lua). I would be willing to work with Nick, and whoever else that is interested, to make this a reality. Vineet > n > -- > (*) here's the naive impl. You could use cpu time instead of wall > time if you wanted. Making this a real patch is left as an exercise > to the reader. > > Index: Python/ceval.c > =================================================================== > --- Python/ceval.c (revision 43738) > +++ Python/ceval.c (working copy) > @@ -16,6 +16,8 @@ > > #include > > +static time_t stop_time = 0; > + > #ifndef WITH_TSC > > #define READ_TIMESTAMP(var) > @@ -744,6 +746,9 @@ > assert(stack_pointer != NULL); > f->f_stacktop = NULL; /* remains NULL unless yield suspends frame */ > > + /* XXX: This is just here to initialize. It should be elsewhere. */ > + if (!stop_time) > + stop_time = time(NULL) + 5; > #ifdef LLTRACE > lltrace = PyDict_GetItemString(f->f_globals, "__lltrace__") != NULL; > #endif > @@ -842,6 +847,11 @@ > } > > fast_next_opcode: > + if (stop_time < time(NULL)) { > + fprintf(stderr, "Exceeded maximum time allowed.\n"); > + exit(1); > + } > + > f->f_lasti = INSTR_OFFSET(); > > /* line-by-line tracing support */ > > > From p.f.moore at gmail.com Sun Apr 9 16:47:04 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 9 Apr 2006 15:47:04 +0100 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <4436C476.7010802@wickedgrey.com> References: <44315A89.9000104@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> <4436B44F.5090407@livinglogic.de> <4436C476.7010802@wickedgrey.com> Message-ID: <79990c6b0604090747x160c58b1jd640799dccd6a4b5@mail.gmail.com> On 4/7/06, Eli Stevens (WG.c) wrote: > It seems to me that now to get a duck-typed list, not only do you have > to implement all of special methods that define "listy-ness," you also > have to find the overloaded functions that are specialized for lists, > and register your own implementation (or an implementation specialized > for lists, if appropriate). Thanks for raising this - I have similar concerns, but I hadn't managed to focus them well enough to explain. This is exactly what has been bugging me. Paul. From guido at python.org Sun Apr 9 17:27:38 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 9 Apr 2006 08:27:38 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> Message-ID: On 4/8/06, Talin wrote: > You know, on the one hand, I've been reading this thread, and I'm > excited about the possibility of generic functions (dynamic dispatch, > duckaplex-typing, whatever you want to call it), but at the same time > I kind of wonder about the eventual effect on the language. > > For example, it seems to me that most of the various magic methods, > such as __cmp__, __add__, etc. would be better done with generics, > especially since it gives you much finer control over the type of > the other argument. So instead of having to write: > > class X: > def __cmp__( self, other ): > if isinstance( other, X ): > return cmp( self.something, other.something ) > return NotImplemented # or raise TypeError or whatever > > Instead you can just write: > > @generic( X, X ) > cmp( a, b ): > return cmp( a.something, b.something ) > > Similarly, if you add an integer to a string, the overloading machinery > will report the error, instead of having to check it manually in the __add__ > method. > > The addition of a funamentally new method of control flow - especially > one that is a superset of an existing control flow method - is > something that could potentially ripple through all of the standard > libraries, transforming them beyond recognition. Given the obvious > advantages, it would be hard to justify sticking with the older methods > of doing things. > > While that's cool and I would look forward to working with them, I > recognize that there are some downsides. > > For one thing, if generic functions are used in a way that truly leverages > their potential, the structure of the code is going to be so different from > 2.x that there won't be any possibility of automated migration - unless > you decide up front that the use of generics is going to be restrained and > controlled, at least initially. > > Personally I'm fine with breaking from the past. But I wanted to point > out what some of the potential consequences would be. Right. I don't want to overdo it; I was thinking of making overloading available to frameworks that need it, by importing it from some library module, but not trying to use it for all those things for which we already have working solutions (like binary operators). Last night I looked into using overloading to replace the mess that is pickle. Unfortunately, an immediate problem is that overloading (at least my current implementation, and also e.g. Phillip Eby's dynamic dispatch code; and I believe also adaptation) is based on subclassing. However, pickle is very careful not to use the same code for pickling subclasses -- because (at least for primitive types like int) the pickled representation is so efficient that it has no way to represent the subclass information (even if it's only a class name). The pickle module could use a style overhaul to use decorators, but I don't think it should giveup its own registry mapping types to methods. I suspect many other current uses of dispatch dictionaries (copy_reg, for example) will have similar issues. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 9 17:38:25 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 9 Apr 2006 08:38:25 -0700 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <4436C476.7010802@wickedgrey.com> References: <44315A89.9000104@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> <4436B44F.5090407@livinglogic.de> <4436C476.7010802@wickedgrey.com> Message-ID: On 4/7/06, Eli Stevens (WG.c) wrote: > I'm curious what effect overloaded functions will have on duck typing; a > Something can act like a list, but from my understanding of the > discussion* it won't end up matching: > > @pprint.register(list) > def pprint_list(obj): > pass Right. Neither does it automatically pickled like a list, copy.py won't copy it as a list, etc. > It seems to me that now to get a duck-typed list, not only do you have > to implement all of special methods that define "listy-ness," you also > have to find the overloaded functions that are specialized for lists, > and register your own implementation (or an implementation specialized > for lists, if appropriate). Right. I can see some ways to make this less if an issue, but it won't go away completely: - you only have to add registrations to those overloaded functions that your application uses with this specific type of sequence - frameworks should probably build a layer of adaptation on top of overloaded functions, where you cast your object to their "sequence" interface before passing it in I believe Phillip Eby's PyProtocols solves this by letting you specify conditions including hasattr() instead of just type checks. > If a consumer of your listy class is also a consumer of some separate > overloaded functions that operate on lists, will they be responsible for > registering your listy class into the overloaded functions? How does > that work when the overloaded functions are an implementation detail of > the other library they happen to use? That's not going to work very well. > I think there's a lot of interesting potential here; but the reliance on > object type seems to imply that you can't mix it with duck typing, and > I'm not sure if that's being fully considered. A very good point. Probably "raw" overloading isn't enough; we'll still need to build a system of interfaces and adapters on top of them. It's just that building adapters out of overloading seems more elegant than having adaptation as the primitive operation. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Sun Apr 9 17:57:35 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun, 09 Apr 2006 11:57:35 -0400 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: Message-ID: <5.1.1.6.0.20060409112542.01e497e0@mail.telecommunity.com> At 12:58 PM 4/7/2006 -0700, "Eli Stevens (WG.c)" wrote: >Guido van Rossum wrote: > > Did you forget duck typing? Something can be a sequence without > > subclassing a common base class. > > > >I'm curious what effect overloaded functions will have on duck typing; a >Something can act like a list, but from my understanding of the >discussion* it won't end up matching: > >@pprint.register(list) >def pprint_list(obj): > pass Note that this doesn't work *now* with pprint, even if you actually *subclass* list. At least with this approach, subclassing list would work automatically. :) >It seems to me that now to get a duck-typed list, not only do you have >to implement all of special methods that define "listy-ness," you also >have to find the overloaded functions that are specialized for lists, >and register your own implementation (or an implementation specialized >for lists, if appropriate). It's true that generic functions do not by themselves resolve the question of what "listy-ness" is. Your comment does highlight the idea, however, that it would be useful to be able to use duck typing to target generic function methods. For example, to say that "this method is usable against any object that can have these other functions called on it". A kind of mapping from an operation to dependent operations, such that you can define a generic "operator.getslice" method for any object that can be used with the "operator.getitem" generic function. This sort of goes back to the "monkey typing" proposal I made early last year in response to the last big python-dev debate on adaptation. Monkey typing is in fact just generic functions combined with a kind of "universal adapter" type, that allows you to treat generic functions as methods of the adapter object, rather than calling them as functions. In fact, this is very close to what the Haskell programming language calls "typeclasses". A typeclass in Haskell is basically a set of generic functions that apply to some type. You declare an "instance" of a typeclass by specifying the operations that correspond to the operations in the typeclass, which then allows your new type to be used by any code that expects the typeclass. So, in Haskell you could define "listyness" by declaring a typeclass that included various listy generic functions (setitem, getitem, insert, delitem, and len, perhaps). Then, to declare your new listy type, you would merely need to specify what functions or methods of your type corresponded to the "listy" generic functions. Since other functions' methods can be defined in terms of listyness, there's no need for you to register for all possible things everywhere that take listy things. You may be wondering why typeclasses aren't the same thing as protocols or interfaces. The key difference is that a typeclass is defined in terms of *independent operations*. A typeclass is basically a collection of related generic functions, *not* a specification of a type's behavior. This means that operations can be *shared* between typeclasses. This is crucial, I think, to Python, where we have some very "fat" interfaces that can be defined using different method subsets. For example, "read file" vs. "write file": both have a "close()" method. Using traditional interface concepts found in Python and Java, you would have to declare two different interfaces, or create a "read-write" file interface in order to handle this. With typeclasses, however, the notion of "close()" can be independent, and you can assemble new typeclasses as needed by simply referring to the operations needed. And -- the most crucial point -- when somebody defines a new typeclass composed only of existing operations that your type can support, then your type gets to play without anybody making any additional declarations at all. This to me seems more Pythonic than traditional interfaces, as it allows something that's essentially just duck typing, but based on *objects* (the generic functions) rather than *names*. Where an interface is a namespace that defines operations, a typeclass is defined by the operations that comprise it. Thus, a typeclass for "duck" is just shorthand for some set of behaviors like walking and quacking. From rrr at ronadam.com Sun Apr 9 19:35:30 2006 From: rrr at ronadam.com (Ron Adam) Date: Sun, 09 Apr 2006 12:35:30 -0500 Subject: [Python-3000] Adaption & generic functions [was Generic functions] In-Reply-To: <79990c6b0604090747x160c58b1jd640799dccd6a4b5@mail.gmail.com> References: <44315A89.9000104@colorstudy.com> <4432B81F.1070903@cox.net> <44337508.2030505@livinglogic.de> <4436AC8A.50802@livinglogic.de> <4436B44F.5090407@livinglogic.de> <4436C476.7010802@wickedgrey.com> <79990c6b0604090747x160c58b1jd640799dccd6a4b5@mail.gmail.com> Message-ID: Paul Moore wrote: > On 4/7/06, Eli Stevens (WG.c) wrote: >> It seems to me that now to get a duck-typed list, not only do you have >> to implement all of special methods that define "listy-ness," you also >> have to find the overloaded functions that are specialized for lists, >> and register your own implementation (or an implementation specialized >> for lists, if appropriate). > > Thanks for raising this - I have similar concerns, but I hadn't > managed to focus them well enough to explain. This is exactly what has > been bugging me. > > Paul. I haven't looked at the actual posted examples yet so this is more an overview of issues as I see them. I tend to lag in these more technical threads, but here's the way I understand it so far. In the sane transitive adaption thread I made the comment that there may be a relationship between identifying "logical equivalence" and adaptation, although I didn't explain what I meant by that. First of all, weather your source object and your destination object have all the same methods or not isn't an issue. In fact if they did, you wouldn't need to adapt anything as duct typing would just work. What is an issue is that the data they contain is translatable. Using tuples and lists as an example: (1,2,3) == [1,2,3] -> False; They are not equivalent. But the data they contain is the same, and you can convert from one to the other without any loss. So you can say they are logically equivalent. A to B; B to C; and A == C (1,2,3) <=> [1,2,3] -> True; They can be translated. Python already treats tuples and lists as interchangeable in many cases. They have one to one data mapping and they share methods that have the same names and function. So duck typing works in many cases as is. But what if they didn't map to each others data one for one, and/or had different methods? They could still be "logically equivalent", but simple duck typing wouldn't work, so you would need something more than duck typing, ie.. an adapter, or a library factory functions, or have each type be able to read every other type. The first option adaptation could be just be a wrapper that substitutes matching method names and/or acted as a go between to allow an object to temporarily gain matching methods so duck typing would work on it. The second option actually creates object B from object A. Both of these need some sort of interface to choose the correct adapter or converter. The last one, having every object know how to read every other "logically equivalent" object just isn't practical. But that doesn't mean it can't act as if it can by calling a some common framework. The "logically equivalent" requirement only applies to implicit adaptation. With explicit (on request) adaptation, it could be assumed that it is understood the result may not be reversible or may loose some data as in converting floats to ints. The real difficulty is in how to implement these concepts in a way that can be extended and be useful in a broader range. An adapter need not implement every method, but only needs to provide mappings to methods that are missing and/or are different between two objects. Those methods could reside in a common library and be identified dynamically as they are needed or they could be pre-identified and registered as sets for known objects. Ideally once a library of adapter mappings is in place, creating and adding new adapters wouldn't be that hard as many of the preexisting methods could be reused. This is about as far as I've gotten so far, so I'll have to leave anything I missed or misunderstood to the experts and keep trying to follow along as they try out the different ideas. :-) Cheers, Ron From steven.bethard at gmail.com Sun Apr 9 20:24:00 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Sun, 9 Apr 2006 12:24:00 -0600 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) Message-ID: On 4/8/06, Phillip J. Eby wrote: > To me an AST access feature would only be useful for changing the "VM" that > Python code runs on. For example, there is an object-relational mapper out > there (I think it's Robert Brewer's DejaVu system) that takes Python > generator expressions and converts them into SQL. Right now it does it by > bytecode hacking, because the only alternative is to parse a string. I can see how reading the AST could be useful here, but Python 2.5 already has that: >>> expr = compile('(x**y for x, y in z)', '', 'eval', 0x400) >>> expr.body.elt.left.id 'x' >>> expr.body.elt.right.id 'y' >>> expr.body.elt.op <_ast.Pow object at 0x00BD0690> >>> [name.id for name in expr.body.generators[0].target.elts] ['x', 'y'] >>> expr.body.generators[0].iter.id 'z' Do the use-cases here really need to change the AST on the original function? What's wrong with creating a new function? STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From 2005a at usenet.alexanderweb.de Sun Apr 9 21:28:00 2006 From: 2005a at usenet.alexanderweb.de (Alexander Schremmer) Date: Sun, 9 Apr 2006 21:28:00 +0200 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? References: <44373866.9070305@alumni.rice.edu> Message-ID: On Sat, 8 Apr 2006 18:18:13 -0700, Neal Norwitz wrote: > #3 is easy to do a simple, naive implementation. I don't know what > your needs are. If you just want to say "exit this script if it ran > more than N seconds" you can just modify the eval loop (*). Unfortunately, it is not as easy as that. There are many code parts in Python that are implemented as functions written in C that might take very long to compute. I just remember how I could stall a well known wiki engine written in Python (;-) by entering a too complex regular expression into the search field (it needed quite a lot seconds per page). Kind regards, Alexander From fumanchu at amor.org Sun Apr 9 22:38:19 2006 From: fumanchu at amor.org (Robert Brewer) Date: Sun, 9 Apr 2006 13:38:19 -0700 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) References: Message-ID: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> Steven Bethard wrote: > On 4/8/06, Phillip J. Eby wrote: > > To me an AST access feature would only be useful for > > changing the "VM" that Python code runs on. For example, > > there is an object-relational mapper out there (I think > > it's Robert Brewer's DejaVu system) that takes Python > > generator expressions and converts them into SQL. Right > > now it does it by bytecode hacking, because the only > > alternative is to parse a string. > > I can see how reading the AST could be useful here, > but Python 2.5 already has that: > > >>> expr = compile('(x**y for x, y in z)', '', 'eval', 0x400) Part of the benefit of the bytecode-hacking is that your expression never has to be in a string. This is a win for readability, and also means you can use the usual helpers like syntax coloring, pychecker, etc. It also means you can early-bind a lot of constants from the scope where the expression is declared without resorting to string-interpolation; that cuts down on SQL-injection attacks in my case. > Do the use-cases here really need to change the AST on > the original function? What's wrong with creating a > new function? Mine don't need to modify the original AST; I'd be happy making a new function. That's what I do now (just at the bytecode level, joining two functions into one with the appropriate JUMP code). Robert Brewer System Architect Amor Ministries fumanchu at amor.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060409/b3dee374/attachment.html From tim.hochberg at cox.net Sun Apr 9 23:22:28 2006 From: tim.hochberg at cox.net (Tim Hochberg) Date: Sun, 09 Apr 2006 14:22:28 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> Message-ID: <44397B14.4060208@cox.net> """This is my attempt to rewrite pprint.py to be based on protocols. Or overloading or specialization or whatever they end up being called ;-). My first attempt only touched about 20 lines of the original pprint and probably had 90% of the usability gain that this one does, but for some reason I decided that wasn't enough work, and I went back and tried to make it completely extensible. This took far too much time, as touching one part of the code led to another part of the code until I had ripped the whole thing apart. Then I had to figure out how to put it back together sensibly. My first, simplest-thing-that-could- possibly-work, version was slow. About 20x slower than the original pprint.py for pformat, based on pprint._perfcheck. This led to another round of reorganization and optimization. The current version is slightly faster than the original for percheck, but slightly slower for saferepr. Let me add this cautionary note -- this primarily intended as for illustrative purposes. There's a good chance it has bugs. It does, however, pass test_pprint except for test_subclassing, which it can't really be expected to pass. Rather than having you scroll all through this long message/file, I'll instead place all the examples up here at the top. Assuming that you save this as pprint2.py, you should be able to run the examples just by executing the file. """ if __name__ == "__main__": import pprint2 # Here's that same old object that I stole Fred Lundh's librarybook. data = ( "this is a string", [1, 2, 3, 4], ("more tuples", 1.0, 2.3, 4.5), "this is yet another string" ) # [1] You can use pprint just as before. pprint2.pprint(data) print # ('this is a string', # [1, 2, 3, 4], # ('more tuples', 1.0, 2.3, 4.5), # 'this is yet another string') # [2] However, the core object is actually pprinter, which by default returns # a string instead of printing to stdout. print pprint2.pprinter(data) print # Same as for [1] # [3] Let's extend pprinter so that integers are printed in hex. We use # register_simple because we don't care about all of the details regarding # indentation and recursion. register_simple wants a function that takes # a single object and returns a string. mypprinter = pprint2.PrettyPrinter(pprint2.pprinter) print mypprinter(data) print # Still the same as for [1] @mypprinter.register_simple(int) @mypprinter.register_simple(long) def format_int_as_hex(obj): return hex(obj) print mypprinter(data) print # ('this is a string', # [0x1, 0x2, 0x3, 0x4], # ('more tuples', 1.0, 2.3, 4.5), # 'this is yet another string') # Pretty cool! Note that pprinter itself is unchanged. print pprint2.pprinter(data) print # Still the same as for [1]. # Ok, just to finish things off, let's override something using the full # interface. This interface is not fully documented or even fully # understood (I copied it from pprint.py, but I'm sure it warped in # transition. # # This example is pretty useless, but at least you get to see full # overriding. Note that this function must return an iterable. # For real examples, see the registrations at the bottom of the program. # (Although be warned, they can be messy!) @mypprinter.register(list) def format_list(obj, context, indent, allowance, level): yield ("" % (id(obj), indent, allowance, level)) print mypprinter(data) print # ('this is a string', # , # ('more tuples', 1.0, 2.3, 4.5), # 'this is yet another string') # I'm not sure why level is 3 instead of 2, that my be a buglet. I'm not # going to track it down right now. # OK, that's it for now. Back to my normally scheduled life! """ I include Registry and Protocol here since the Protocol implemention keeps changing. Tracking down a compatible implementation would probably be hard. I use Registries instead of dicts for my Protocol class. So far, I like this approach. It makes runtime extension of Protocols simple while exiling the increased complexity to the Registry class. While I like the semantics, the implementation is questionable. As currently implemented lookups are potentially slow while setting and deleting is fast. This is exactly backwards! This could be reimplemented with the same semantics using some sort of view observer magic as suggested by Nick Coghlan. """ _missing = object() _unknown = object() class Registry(dict): """A dict like object that forwards item requests to its parent if neeeded.""" def __init__(self, *parents): dict.__init__(self) self.parents = list(parents) def __contains__(self, key): return self.get(key, _missing) is not _missing def __getitem__(self, key): x = self._get(key, set()) if x is _missing: raise KeyError('not found') return x def _get(self, key, visited): # Get a value from self or parents. Return _missing on failure # visited contains the ids of objects searched so far. myid = _id(self) if myid in visited: return _missing visited.add(myid) x = dict.get(self, key, _missing) if x is not _missing: return x for p in self.parents: if isinstance(p, Registry): x = p._get(key, visited) else: x = p.get(key, _missing) if x is not _missing: return x return _missing def get(self, key, default=None): x = self._get(key, set()) if x is _missing: return default return x """ Yet another Protocol implementation. This one is pretty similar to most of the other recent ones except that it uses Registries. """ class Protocol(object): """Declare a protocol object that subclasses parents if provided""" def __init__(self, *parents): self.registry = Registry(*(x.registry for x in parents)) def register(self, *args): """Function decorator to register as an adapter for given keys""" if len(args) == 1: args = args[0] def helper(adapter): if adapter is None: adapter = null_adapter self.registry[args] = adapter return adapter return helper def signatures(self, arg): """Find signatures for given call arguments""" # Default behaviour dispatches on the type of the first argument return _type(arg).__mro__ def default_adapter(self, *args): """Call result when no adapter was found""" raise TypeError("Can't adapt <%s> to %s" % (_commajoin(x.__class__.__name__ for x in args), self.__class__.__name__)) def find_adapter(self, *args): """Find an adapter for args""" for key in self.signatures(*args): adapter = self.registry.get(key, _missing) if adapter is not _missing: return adapter return self.default_adapter def __call__(self, *args): """Adapt supplied arguments to this protocol""" return self.find_adapter(*args)(*args) """ Here we have the actual pprint2 stuff. The first order of business is just defining some marker and helper classes. Nothing exciting. """ import sys class Marker(object): def __init__(self, value): self.value = value # These marker classes are used to tell PrettyPrinter to use the formatter # for a long object (long in the sense that len(pformat(obj)) is long). class LongList(Marker): pass class LongTuple(Marker): pass class LongSequence(Marker): pass class LongDict(Marker): pass long_types = {list : LongList, tuple : LongTuple, dict : LongDict} # More marker classes. Recursive is used, suprise, for recursive objects and # Generic is used to format objects that are not otherwise formatable. class Recursive(Marker): pass # Generic is a little different from the other marker classes as it doesn't # wrap it's arguments. class Generic(object): pass # This is used to pass context information down to the registered formatters. class Context(object): __slots__ = ['locked', 'width', 'maxlevels', 'indent_per_level', 'readable', 'recursive', 'adapter_cache', 'long_adapter_cache', 'formatter'] def __init__(self, width, maxlevels, indent_per_level, formatter): self.locked = set() self.width = width self.maxlevels = maxlevels self.indent_per_level = indent_per_level self.readable = True self.recursive = False self.formatter = formatter self.adapter_cache = {} self.long_adapter_cache = {} # Some microoptimizations stolen from the original _id = id _len = len _type = type _commajoin = ', '.join _join = ''.join """ Now we have reached the core of the implementation. """ class PrettyPrinter(Protocol): """PrettyPrinter is a Protocol that accepts objects and either returns a pretty representation of the object or writes the representation to a stream if that is supplied. """ # Override signatures so that only superclasses with the same repr are # return. This centralizes a bunch of logic in pprint.py, in one place # simplifies things a lot farther down. Also, always yield Generic, so # that the default behavious can be overridden. def signatures(self, *args): repr = _type(args[0]).__repr__ for x in _type(args[0]).__mro__: if x.__repr__ is not repr: break yield x yield Generic # Here to ease changing formatting of simple types, like ints for example. def register_simple(self, arg): """Register a simple converter that takes a obj and returns a string""" def helper(adapter): if adapter is None: adapter = null_adapter def func(obj, context, indent, allowance, level): yield adapter(obj) self.registry[arg] = func return adapter return helper # pprint makes a distinction between objects with long representations, # which it splits between lines and others. This finds the correct formatter # for a given type. def _find_long_adapter(self, objtype, obj, context): try: return context.long_adapter_cache[objtype] except KeyError: for sig in self.signatures(obj, context): longtype = long_types.get(sig, None) if longtype: try: adapter = self.registry[longtype] break except KeyError: pass else: adapter = None context.long_adapter_cache[objtype] = adapter return adapter # This is the core of the algorithm. It's really quite simple if you look # past two complicating factors. First, there's a bunch of caching going # on, that's essential for performance reasons. Second, everything is # implemented in terms of generators. This allows the adapters to not # worry about the whole stream issue. def _format(self, obj, context, indent=0, allowance=0, level=0): max_length = context.width - indent - allowance - 1 objtype = type(obj) try: adapter = context.adapter_cache[objtype] except: adapter = context.adapter_cache[objtype] = self.find_adapter(obj) chunkmaker = adapter(obj, context, indent, allowance, level+1) rep = '' for chunk in chunkmaker: rep += chunk if (len(rep) > max_length): # If max_length gets too long, we try to use a long adapter # instead. If that works, we clear rep and break out. Other- # wise, we keep rep and still break out, but still with the # old chunkmaker. longadapter = self._find_long_adapter(objtype, obj, context) if longadapter: rep = '' context.locked.discard(_id(obj)) chunkmaker = longadapter(Marker(obj), context, indent, allowance, level+1) break # Yield rep and any remaining chunks. yield rep for chunk in chunkmaker: yield chunk # This just does some error checking, sets up the stream, and hands things # off to _format. def __call__(self, obj, stream=None, indent=1, width=80, depth=None): indent = int(indent) width = int(width) if indent < 0: raise ValueError("indent must be >= 0") if not (depth is None or depth > 0): raise ValueError("depth must be > 0") if width <= 0: raise ValueError("width must be > 0") context = Context(width, depth, indent, self._format) if stream is None: return _join(self._format(obj, context)) else: for chunk in self._format(obj, context): stream.write(chunk) # recreate the pprint interface. def isrecursive(self, obj): context = Context(80, None, 1, self._format) for chunk in self._format(obj, context): pass return context.recursive def isreadable(self, obj): context = Context(80, None, 1, self._format) for chunk in self._format(obj, context): pass return context.readable and not context.recursive """ Now that we have the Protocol object, we define two instances. One (saferepr) that doesn't do anything special with long lines, and one (pprinter) that does. pprinter extends saferepr, so any changes to saferepr will be picked up automagically by pprinter. With those two in place, it's a simple matter to recreate the pprint.py module interface. Then we're done except for actually creating and registering all of the functions. """ saferepr = PrettyPrinter() pprinter = PrettyPrinter(saferepr) # Recreate the module interface. pformat = pprinter isreadable = saferepr.isreadable isrecursive = saferepr.isrecursive def pprint(obj, indent=1, width=80, depth=None, stream=None): if stream is None: stream = sys.stdout pprinter(obj, stream, indent, width, depth) stream.write('\n') """ Way down here at the bottom is where we define and register all the behaviour of everything. Don't look too closely at these implementations -- I ripped them from pprint.py and modified them to fit here without thinking about them too much. They may have suffered in the transition. """ @saferepr.register(Generic) def _format_generic(obj, context, indent, allowance, level): rep = repr(obj) if rep.startswith('<'): context.readable = False yield rep @saferepr.register(Recursive) def _format_recursive(markerobj, context, indent, allowance, level): obj = markerobj.value context.recursive = True yield ("" % (_type(obj).__name__, _id(obj))) @saferepr.register(float) def _format_float(obj, context, indent, allowance, level): yield str(obj) @saferepr.register(str) def _format_str(obj, context, indent, allowance, level): if 'locale' not in sys.modules: yield repr(obj) return if "'" in obj and '"' not in obj: closure = '"' quotes = {'"': '\\"'} else: closure = "'" quotes = {"'": "\\'"} qget = quotes.get chars = [] write = chars.append for char in obj: if char.isalpha(): write(char) else: write(qget(char, repr(char)[1:-1])) yield ("%s%s%s" % (closure, _join(chars), closure)) @saferepr.register(dict) def _format_dict(obj, context, indent, allowance, level): objid = _id(obj) if objid in context.locked: for chunk in context.formatter(Recursive(obj), context, indent, allowance, level): yield chunk else: context.locked.add(objid) yield '{' if context.maxlevels and level > context.maxlevels: yield "..." else: if obj: format = context.formatter items = obj.iteritems() # This should really be sorted, but tests don't expect it. k, v = items.next() for chunk in format(k, context, indent, allowance, level): yield chunk yield ': ' for chunk in format(v, context, indent, allowance, level): yield chunk for k, v in items: yield ', ' for chunk in format(k, context): yield chunk yield ': ' for chunk in format(v, context): yield chunk yield '}' context.locked.remove(objid) @saferepr.register(list) @saferepr.register(tuple) def _format_sequence(obj, context, indent, allowance, level): typ = _type(obj) if issubclass(typ, list): if not obj: yield "[]" return format = "[%s]" elif _len(obj) == 1: format = "(%s,)" else: if not obj: yield "()" return format = "(%s)" objid = _id(obj) if context.maxlevels and level > context.maxlevels: yield format % "..." return if objid in context.locked: for chunk in context.formatter(Recursive(obj), context, indent, allowance, level): yield chunk return context.locked.add(objid) components = [] append = components.append formatter = context.formatter for o in obj: for orepr in formatter(o, context, indent, allowance, level): append(orepr) yield format % _commajoin(components) context.locked.remove(objid) @pprinter.register(LongList) def _format_longlist(longobj, context, indent, allowance, level): yield '[' for chunk in context.formatter(LongSequence(longobj.value), context, indent, allowance, level): yield chunk yield ']' @pprinter.register(LongTuple) def _format_longtuple(longobj, context, indent, allowance, level): obj = longobj.value yield '(' for chunk in context.formatter(LongSequence(longobj.value), context, indent, allowance, level): yield chunk if _len(obj) == 1: yield ',' yield ')' @pprinter.register(LongSequence) def _format_longsequence(longobj, context, indent, allowance, level): obj = longobj.value if context.indent_per_level > 1: yield (context.indent_per_level - 1) * ' ' length = _len(obj) if length: objid = _id(obj) if objid in context.locked: for chunk in context.formatter(Recursive(obj), context, indent, allowance, level): yield chunk return context.locked.add(objid) format = context.formatter indent += context.indent_per_level objiter = iter(obj) for chunk in format(objiter.next(), context, indent, allowance+1, level): yield chunk if length > 1: indentation = ',\n' + ' '*indent for ent in objiter: yield indentation for chunk in format(ent, context, indent, allowance+1, level): yield chunk context.locked.remove(objid) @pprinter.register(LongDict) # could break out into keys and use sequence somehow. def _format_longdict(longobj, context, indent, allowance, level): obj = longobj.value yield '{' if context.indent_per_level > 1: yield (context.indent_per_level - 1) * ' ' length = _len(obj) if length: format = context.formatter objid = _id(obj) if objid in context.locked: for chunk in context.formatter(Recursive(obj), context, indent, allowance, level): yield chunk return context.locked.add(objid) indent += context.indent_per_level items = obj.items() items.sort() itemsiter = iter(items) key, ent = itemsiter.next() valindent = indent valallow = allowance for chunk in format(key, context, indent, allowance, level): valindent += _len(chunk) yield chunk yield ': ' valindent += 2 valallow += 1 for chunk in format(ent, context): yield chunk if length > 1: indentation = ',\n'+' '*indent for key, ent in itemsiter: yield indentation for chunk in format(key, context, indent, allowance, level): yield chunk yield ': ' for chunk in format(ent, context, valindent, valallow, level): yield chunk context.locked.remove(objid) yield '}' def _perfcheck(object=None): import time if object is None: object = [("string", (1, 2), [3, 4], {5: 6, 7: 8})] * 10000 p = PrettyPrinter() t1 = time.time() saferepr(object) t2 = time.time() pformat(object) t3 = time.time() print "safe_repr:", t2 - t1 print "pformat:", t3 - t2 #~ _perfcheck() From nnorwitz at gmail.com Sun Apr 9 23:38:45 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 9 Apr 2006 14:38:45 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44397B14.4060208@cox.net> References: <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> Message-ID: On 4/9/06, Tim Hochberg wrote: > > Let me add this cautionary note -- this primarily intended as for > illustrative purposes. There's a good chance it has bugs. It does, > however, pass > test_pprint except for test_subclassing, which it can't really be > expected to pass. Py3k is a great opportunity to upgrade the existing tests. We know they are incomplete and could do a better job testing. In some cases like this, where we are looking to revamp a module. It would be great to also beef up the tests. Hopefully a lot of that work, will also benefit the 2.x series. n From janssen at parc.com Sun Apr 9 23:41:50 2006 From: janssen at parc.com (Bill Janssen) Date: Sun, 9 Apr 2006 14:41:50 PDT Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: Your message of "Sat, 08 Apr 2006 11:00:16 PDT." Message-ID: <06Apr9.144154pdt."58633"@synergy1.parc.xerox.com> > Well, I remember this was one of the key goals Mitch Kapor > outlined for Python in his keynote at PyCon. He blogged about it > too: > > http://blogs.osafoundation.org/mitch/000559.html I'm very interested in this, too. There are two things that force me into writing Java code instead of Python code, and they are: 1) The Java plug-in for Web browsers, and 2) The integrated portable fairly capable Swing GUI. These two missing pieces are the big issues for me. Everything else that gets discussed on this list is essentially fiddling with something that's already pretty good (though juicing up the syntax to Perl level sort of annoys me :-). To achieve (1), you need to have a truly effective restricted execution environment. And you may need something like (2), as well. We can talk about (2) some other time :-). Bill From tim.hochberg at ieee.org Mon Apr 10 00:17:36 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Sun, 09 Apr 2006 15:17:36 -0700 Subject: [Python-3000] Sane transitive adaptation In-Reply-To: References: <7.0.1.0.0.20060406093503.02185bd8@telecommunity.com> <7.0.1.0.0.20060406145640.020d88a8@telecommunity.com> <4436446A.5050804@iinet.net.au> <44373A46.6060900@gmail.com> Message-ID: Guido van Rossum wrote: > On 4/8/06, Talin wrote: > >>You know, on the one hand, I've been reading this thread, and I'm >>excited about the possibility of generic functions (dynamic dispatch, >>duckaplex-typing, whatever you want to call it), but at the same time >>I kind of wonder about the eventual effect on the language. >> >>For example, it seems to me that most of the various magic methods, >>such as __cmp__, __add__, etc. would be better done with generics, >>especially since it gives you much finer control over the type of >>the other argument. So instead of having to write: >> >>class X: >> def __cmp__( self, other ): >> if isinstance( other, X ): >> return cmp( self.something, other.something ) >> return NotImplemented # or raise TypeError or whatever >> >>Instead you can just write: >> >>@generic( X, X ) >>cmp( a, b ): >> return cmp( a.something, b.something ) >> >>Similarly, if you add an integer to a string, the overloading machinery >>will report the error, instead of having to check it manually in the __add__ >>method. >> >>The addition of a funamentally new method of control flow - especially >>one that is a superset of an existing control flow method - is >>something that could potentially ripple through all of the standard >>libraries, transforming them beyond recognition. Given the obvious >>advantages, it would be hard to justify sticking with the older methods >>of doing things. >> >>While that's cool and I would look forward to working with them, I >>recognize that there are some downsides. >> >>For one thing, if generic functions are used in a way that truly leverages >>their potential, the structure of the code is going to be so different from >>2.x that there won't be any possibility of automated migration - unless >>you decide up front that the use of generics is going to be restrained and >>controlled, at least initially. >> >>Personally I'm fine with breaking from the past. But I wanted to point >>out what some of the potential consequences would be. > > > Right. I don't want to overdo it; I was thinking of making overloading > available to frameworks that need it, by importing it from some > library module, but not trying to use it for all those things for > which we already have working solutions (like binary operators). > > Last night I looked into using overloading to replace the mess that is > pickle. Unfortunately, an immediate problem is that overloading (at > least my current implementation, and also e.g. Phillip Eby's dynamic > dispatch code; and I believe also adaptation) is based on subclassing. This is reason that I factored the signatures method out in my version of Protocol. With that factoring, it's a simple matter of replacing the signatures method with one that returns [type(x)] instead of x.__class__.__mro__. In one sense that may not gain you much over some custom decorators and a dictionary, but having a common interface for this sort of thing seems useful even if the implementation is not always shared. However, it's not clear to me at the moment how the signatures thing fits in with the generic function implementation where the resolution method is more complicated. It may turn out that the correct factoring is just into __call__ and find_func as you have. That would mean that things like pickle would need to override find_func with something trivial. > However, pickle is very careful not to use the same code for pickling > subclasses -- because (at least for primitive types like int) the > pickled representation is so efficient that it has no way to represent > the subclass information (even if it's only a class name). The pickle > module could use a style overhaul to use decorators, but I don't think > it should giveup its own registry mapping types to methods. I thought the point of distributed protocols / generics was that each module could have it's own (or several should it so choose). Why would it be a problem for pickles registry to use a Protocol subclass as it's registry? The main thing you'd gain is an interface in common with other clients of Protocol, but that's no small thing. > I suspect many other current uses of dispatch dictionaries (copy_reg, > for example) will have similar issues. Regards, -tim From tdelaney at avaya.com Mon Apr 10 00:56:37 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Mon, 10 Apr 2006 08:56:37 +1000 Subject: [Python-3000] Kill "generic functions"! Message-ID: <2773CAC687FD5F4689F526998C7E4E5F07436B@au3010avexu1.global.avaya.com> Guido van Rossum wrote: > What about overloaded functions, which is the term used in C++ and > Java for a similar mechanism (that happens at compile-time -- but it's > standard Python practice to move things to run-time). We could then > also talk about overloaded methods when we're using the mechanism > inside a class. Haven't seen this in the thread yet (may have missed it). There appear to be two features which these come close to - overloaded functions, and template specialisations (C++). Depending on which way you look at it, the feature could be either. If the behaviour is remarkably different depending on what type of parameters are passed, it's overloading. If the behaviour is supposed to be basically the same, but perhaps more efficient with different types of parameters, it's specialisation. It seems more akin to C++ template specialisation to me - there is a default implementation which is used if no more specialised method for those parameter types is available. So I'd suggest the term "function specialisation". Tim Delaney From greg.ewing at canterbury.ac.nz Mon Apr 10 02:28:19 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 10 Apr 2006 12:28:19 +1200 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> <44386C42.90803@canterbury.ac.nz> Message-ID: <4439A6A3.8030409@canterbury.ac.nz> Georg Brandl wrote: > Even as properties and methods, they can be restricted, > like func_code is now. Yes, but that means special-casing all the attribute accessors for them, and being careful to maintain the code which enforces the restrictions. If they're functions, the restriction is implemented by simply excluding them from the __builtins__. Another way of looking at it is that I'm suggesting a more capability-oriented way of handling this -- whether you can do something depends on whether you have access to a function or object that does it. -- Greg From greg.ewing at canterbury.ac.nz Mon Apr 10 02:32:33 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 10 Apr 2006 12:32:33 +1200 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> <44386C42.90803@canterbury.ac.nz> Message-ID: <4439A7A1.601@canterbury.ac.nz> Georg Brandl wrote: > Even as properties and methods, they can be restricted, Another thought is that if they're functions, there are more options available to the person wanting to impose restrictions. Instead of removing them altogether, he could replace them with versions that allow some subset of their functionality. This could mitigate the "crippling of the language" effect of just turning them off altogether. -- Greg From rasky at develer.com Mon Apr 10 02:35:44 2006 From: rasky at develer.com (Giovanni Bajo) Date: Mon, 10 Apr 2006 02:35:44 +0200 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? References: <44373866.9070305@alumni.rice.edu> Message-ID: <021401c65c36$b3de2f70$26452597@bagio> Neal Norwitz wrote: > #3 is easy to do a simple, naive implementation. I don't know what > your needs are. If you just want to say "exit this script if it ran > more than N seconds" you can just modify the eval loop (*). But I > suspect there are many more features that would be required and it's > not so simple. Yeah. I think of rexec as a way to safely eval expressions / exec statements provided by users without the risk of DOSing the machine executing them. And your solution doesn't handle things like 10000000000000000000**10000000000000000000. Giovanni Bajo From guido at python.org Mon Apr 10 02:47:56 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 9 Apr 2006 17:47:56 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44397B14.4060208@cox.net> References: <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> Message-ID: On 4/9/06, Tim Hochberg wrote: > Rather than having you scroll all through this long message/file, I'll instead > place all the examples up here at the top. Assuming that you save this as > pprint2.py, you should be able to run the examples just by executing the > file. Suggestion: next time use doctest for this purpose. Also, can you post a URL for this code? Unfortunately your mailer wrapped some of the longer lines. BTW, thanks a lot for working out this elaborate example! It takes working with real-world problems to validate programming innovations. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Mon Apr 10 02:49:58 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 10 Apr 2006 12:49:58 +1200 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> Message-ID: <4439ABB6.3030201@canterbury.ac.nz> Robert Brewer wrote: > Part of the benefit of the bytecode-hacking is that your expression > never has to be in a string. I'm wondering whether there should be some kind of "code literal" syntax, where you write a Python expression and the compiler transforms it as far as the AST stage, then makes it available to the program as an AST object. Or would that be too close to "programmable syntax" for Guido's liking? -- Greg From msoulier at digitaltorque.ca Mon Apr 10 02:50:15 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Sun, 9 Apr 2006 20:50:15 -0400 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <06Apr9.144154pdt."58633"@synergy1.parc.xerox.com> References: <06Apr9.144154pdt."58633"@synergy1.parc.xerox.com> Message-ID: <20060410005014.GC6659@tigger.digitaltorque.ca> On 09/04/06 Bill Janssen said: > I'm very interested in this, too. There are two things that force me > into writing Java code instead of Python code, and they are: > > 1) The Java plug-in for Web browsers, and > > 2) The integrated portable fairly capable Swing GUI. And 2 + the JRE and jar files ease distribution of apps, something else that Python needs to address. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060409/b3709f91/attachment.pgp From nnorwitz at gmail.com Mon Apr 10 03:06:48 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 9 Apr 2006 18:06:48 -0700 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: <021401c65c36$b3de2f70$26452597@bagio> References: <44373866.9070305@alumni.rice.edu> <021401c65c36$b3de2f70$26452597@bagio> Message-ID: On 4/9/06, Giovanni Bajo wrote: > Neal Norwitz wrote: > > > #3 is easy to do a simple, naive implementation. I don't know what > > your needs are. If you just want to say "exit this script if it ran > > more than N seconds" you can just modify the eval loop (*). But I > > suspect there are many more features that would be required and it's > > not so simple. > > Yeah. I think of rexec as a way to safely eval expressions / exec statements > provided by users without the risk of DOSing the machine executing them. And > your solution doesn't handle things like > 10000000000000000000**10000000000000000000. Right, that's my point and was discussed elsewhere in this thread. Though this issue is also easy to solve on most Unixes and can be done today in pure python: >>> import resource as r >>> r.setrlimit(r.RLIMIT_CPU, (5, 5)) >>> 10000000000000000000**10000000000000000000 Cputime limit exceeded To defeat this, you can do: signal.signal(signal.SIGXCPU, signal.SIG_IGN) Of course, you would presumably prevent access to the signal module. You could then install your own signal handler and handle this how you want. There are so many variations on this. I don't know what a general purpose solution would look like. I can imagine many specific solutions. If anyone wants something like this implemented, patches are always greatly appreciated. Unless if someone puts effort into solving this, it won't get done. The only people in a position to solve this are people with a need. n From guido at python.org Mon Apr 10 03:25:35 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 9 Apr 2006 18:25:35 -0700 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <4439ABB6.3030201@canterbury.ac.nz> References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> Message-ID: On 4/9/06, Greg Ewing wrote: > Robert Brewer wrote: > > Part of the benefit of the bytecode-hacking is that your expression > > never has to be in a string. > > I'm wondering whether there should be some kind of > "code literal" syntax, where you write a Python > expression and the compiler transforms it as far > as the AST stage, then makes it available to the > program as an AST object. > > Or would that be too close to "programmable > syntax" for Guido's liking? I'm more concerned about the choice of AST data structure and how it affects IronPython, PyPy, Jython and possible other Python implementations. I'd like to keep both the AST and the bytecode spec out of the language spec, otherwise those implementations will face the tough choice of either changing their parser technology to one that is probably less suited, or implementing two full parsers. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 10 03:27:04 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 9 Apr 2006 18:27:04 -0700 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <4439A6A3.8030409@canterbury.ac.nz> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> <44386C42.90803@canterbury.ac.nz> <4439A6A3.8030409@canterbury.ac.nz> Message-ID: On 4/9/06, Greg Ewing wrote: > Another way of looking at it is that I'm suggesting > a more capability-oriented way of handling this -- > whether you can do something depends on whether you > have access to a function or object that does it. I'm not sure how that would help. What would it mean to have a capability for accessing e.g. x.__class__? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.hochberg at ieee.org Mon Apr 10 03:57:06 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Sun, 09 Apr 2006 18:57:06 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> Message-ID: <4439BB72.1050203@ieee.org> Guido van Rossum wrote: > On 4/9/06, Tim Hochberg wrote: > >>Rather than having you scroll all through this long message/file, I'll instead >>place all the examples up here at the top. Assuming that you save this as >>pprint2.py, you should be able to run the examples just by executing the >>file. > > > Suggestion: next time use doctest for this purpose. Good point. That's actually my normal procedure, but for reasons that don't make any sense in retrospect it seemed better to do it this way. > > Also, can you post a URL for this code? Unfortunately your mailer > wrapped some of the longer lines. Ooops. I guess it wraps at 72 characters, not 80. I just put the code up here: http://members.cox.net/~tim.hochberg/pprint2.py > BTW, thanks a lot for working out this elaborate example! It takes > working with real-world problems to validate programming innovations. You're welcome. It was interesting, although also a ridiculous time sink. Regards, -tim From fumanchu at amor.org Mon Apr 10 05:30:22 2006 From: fumanchu at amor.org (Robert Brewer) Date: Sun, 9 Apr 2006 20:30:22 -0700 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local><4439ABB6.3030201@canterbury.ac.nz> Message-ID: <435DF58A933BA74397B42CDEB8145A86224B9B@ex9.hostedexchange.local> Guido van Rossum wrote: > On 4/9/06, Greg Ewing wrote: > > Robert Brewer wrote: > > > Part of the benefit of the bytecode-hacking is that > > > your expression never has to be in a string. > > > > I'm wondering whether there should be some kind of > > "code literal" syntax, where you write a Python > > expression and the compiler transforms it as far > > as the AST stage, then makes it available to the > > program as an AST object. > > > > Or would that be too close to "programmable > > syntax" for Guido's liking? > > I'm more concerned about the choice of AST data structure > and how it affects IronPython, PyPy, Jython and possible > other Python implementations. I'd like to keep both the > AST and the bytecode spec out of the language spec, > otherwise those implementations will face the tough choice > of either changing their parser technology to one that > is probably less suited, or implementing two full parsers. Not to mention cross-version issues with persisted AST objects or bytecode-sequences. When pickling my Expression objects, I chose to decompile them back into Python code for that reason. Although I had to upgrade the decompiler between 2.3 and 2.4 (to allow for the JUMP target changes, for example) I didn't have to upgrade a single saved Expression object. Robert Brewer System Architect Amor Ministries fumanchu at amor.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060409/2266d7a9/attachment.htm From talin at acm.org Mon Apr 10 11:44:12 2006 From: talin at acm.org (Talin) Date: Mon, 10 Apr 2006 09:44:12 +0000 (UTC) Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > I'm wondering whether there should be some kind of > "code literal" syntax, where you write a Python > expression and the compiler transforms it as far > as the AST stage, then makes it available to the > program as an AST object. > > Or would that be too close to "programmable > syntax" for Guido's liking? I have a rather odd use case for this. A while back, I decided I wanted to write an algebraic solver, similar to what Mathematica does, except that I wanted to write it in Python. Not just implement it in Python, but have Python be the language in which the rules of formula manipulation were expressed. The way that I approached this was to create something like the generic function dispatch being discussed. Although my version wasn't nearly as clean or well thought out as what other people here have done, it had one unique feature, was that it did a full unification match on the input arguments. As an example, suppose you wanted to express the transformation: x * 0 => 0 In other words, when you see the expression "x * 0", replace it with a constant zero. That's a fairly trivial transformation, a more interesting one is the rules for calculating derivatives. The general form of the derivative function was: derivative( x, formula ) ...which would produce the derivative of the formula with respect to x. Some of the transformation rules were: derivative( x, x ) => 1 derivative( x, constant( a ) ) => 0 derivative( x, -n ) => -derivative( x, n ) derivative( x, a + b ) => derivative( x, a ) + derivative( x, b ) derivative( x, constant( a ) * b ) => a * derivative( x, b ) derivative( x, a * b ) => a * derivative( x, b ) + b * derivative( x, a ) derivative( x, a ** b ) => b * derivative( x, a ) ** (b - 1) .. and so on. To express this in Python, I used a decorator function called Arity which would invoke a general dispatcher that attempted to do a recursive pattern match on the input arguments. Thus, the above rules would be written in Python as follows: @Arity( MatchAny.x, MatchAny.x ) def Derivative( x ): return 1 @Arity( MatchAny.x, MatchAny.n ) def Derivative( x, n ): return 0 @Arity( MatchAny.x, (Negate, MatchAny.y) ) def Derivative( x, y, z ): return (Negate, Derivative( x, y )) @Arity( MatchAny.x, (Add, MatchAny.y, MatchAny.z) ) def Derivative( x, y, z ): return (Add, Derivative( x, y ), Derivative( x, z ) ) @Arity( MatchAny.x, (Multiply, MatchNumber.y, MatchAny.z) ) def Derivative( x, y, z ): return (Multiply, y, Derivative( x, z )) @Arity( MatchAny.x, (Multiply, MatchAny.y, MatchAny.z) ) def Derivative( x, y, z ): return (Add, (Multiply, Derivative( x, y ), z), (Multiply, y, Derivative( x, z ))) @Arity( MatchAny.x, (Power, MatchAny.y, MatchNumber.z) ) def Derivative( x, y, z ): return (Multiply, (Multiply, z, (Power, y, z - 1)), Derivative( x, y )) Note that the input arguments to the actual Python functions are all of the unbound variables in the arity. Essentially, each match would produce a dictionary of bindings between the actual input arguments and the unbound variables. The corresponding function body would then be called by unpacking the dict. There were also special decorators which would indicate that a function has associative or commutative properties, which would alter the behavior of the dispatcher appropriately. You can immediately see the problem - since I don't have syntactical support, I have to manually enter in the expression in an AST-like form, which is clumsy to say the least. Now, bear in mind that I'm not proposing that we modify the Python language to support my obscure use case. Rather, I'm providing it as an example for people to use in their discussion :) I think that the lesson that I came away with is that if you're doing Algebra, you pretty much have to bite the bullet and write your own parser. Another interesting thing about the dispatcher, by the way, is that it was implemented as a generator -- thus, if there was an ambiguous match, it returned all of the possible matches in sequence. It attempted to order the matches so that the most specific matches were returned first (although my metric of specificity was rather loose.) -- Talin From ncoghlan at gmail.com Mon Apr 10 12:01:17 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 10 Apr 2006 20:01:17 +1000 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <20060410005014.GC6659@tigger.digitaltorque.ca> References: <06Apr9.144154pdt."58633"@synergy1.parc.xerox.com> <20060410005014.GC6659@tigger.digitaltorque.ca> Message-ID: <443A2CED.4000600@gmail.com> Michael P. Soulier wrote: > On 09/04/06 Bill Janssen said: > >> I'm very interested in this, too. There are two things that force me >> into writing Java code instead of Python code, and they are: >> >> 1) The Java plug-in for Web browsers, and >> >> 2) The integrated portable fairly capable Swing GUI. > > And 2 + the JRE and jar files ease distribution of apps, something else that > Python needs to address. Python eggs and easy_install seem to have that problem well in hand, so that's not as big of an issue. Figuring out how to build a sandbox that is: a) actually secure b) usable for the application author c) usable for the plugin writer d) portable to at least Windows, Linux, Solaris, *BSD and Mac OS X is significantly trickier, because current solutions all fail one or more of those 4 criteria, and it isn't clear which (if any) would make an appropriate starting point for doing it right. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Mon Apr 10 13:01:42 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 10 Apr 2006 23:01:42 +1200 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> Message-ID: <443A3B16.5090608@canterbury.ac.nz> Guido van Rossum wrote: > I'm more concerned about the choice of AST data structure and how it > affects IronPython, PyPy, Jython and possible other Python > implementations. I'd like to keep both the AST and the bytecode spec > out of the language spec, Well, it wouldn't be any less portable than bytecode hacking, and could potentially be made more portable. A standard non-text representation of Python code seems like a useful thing to have, even if it's implemented quite separately from the main compiler. We're actually part way there already, with the internal AST <--> python AST transformation. The other implementations would just need to provide their own versions of that transformation. -- Greg From greg.ewing at canterbury.ac.nz Mon Apr 10 13:08:37 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 10 Apr 2006 23:08:37 +1200 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> <44386C42.90803@canterbury.ac.nz> <4439A6A3.8030409@canterbury.ac.nz> Message-ID: <443A3CB5.7060809@canterbury.ac.nz> Guido van Rossum wrote: > I'm not sure how that would help. What would it mean to have a > capability for accessing e.g. x.__class__? If you can somehow get a reference to the __builtin__.classof() function, then you have that capability, otherwise you don't. The key idea is that by turning potentially dangerous things like this from attributes into functions, access to them can be much more easily controlled. Any function can be withheld without the creator of the function having had to do anything special. But you can't withhold an attribute unless its accessor has been designed with that in mind. -- Greg From eprparadocs at gmail.com Mon Apr 10 13:31:16 2006 From: eprparadocs at gmail.com (Chaz.) Date: Mon, 10 Apr 2006 07:31:16 -0400 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443A3CB5.7060809@canterbury.ac.nz> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> <44386C42.90803@canterbury.ac.nz> <4439A6A3.8030409@canterbury.ac.nz> <443A3CB5.7060809@canterbury.ac.nz> Message-ID: <443A4204.5020509@gmail.com> For a rather large application I am writing and will be running for a commercial service, I ran into the restricted environment problem. I had looked at the CPython code and concluded it would be a rather large job to secure it (if I had the time and the intimate understanding). Instead I opted for a cheap approach: 'chroot' the execution. To get around the issue of files and directories I need to access, I use the 'bind' version of mount. This allows me to mount just those specific subdirectories I need. I mount them r/o (if I need to protect them). I have thought the approach would work to make a plug-in for Firefox, though lacking the time and the knowledge has stopped me from experimenting. I know it won't work directly under Windows, but I know I don't care. FWIW, Chuck Wegrzyn From gmccaughan at synaptics-uk.com Mon Apr 10 14:03:58 2006 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Mon, 10 Apr 2006 13:03:58 +0100 Subject: [Python-3000] A few small py3k wishes In-Reply-To: References: <200604041322.02900.gmccaughan@synaptics-uk.com> Message-ID: <200604101303.59557.gmccaughan@synaptics-uk.com> On Friday 2006-04-07 20:52, Guido van Rossum wrote: > On 4/4/06, Gareth McCaughan wrote: > > I take no stand on how much not confusing Lispers matters > > to Python 3k. > > But I do: none whatsoever. I thought as much. :-) -- g From guido at python.org Mon Apr 10 17:35:54 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 08:35:54 -0700 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <443A3B16.5090608@canterbury.ac.nz> References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> <443A3B16.5090608@canterbury.ac.nz> Message-ID: On 4/10/06, Greg Ewing wrote: > Guido van Rossum wrote: > > > I'm more concerned about the choice of AST data structure and how it > > affects IronPython, PyPy, Jython and possible other Python > > implementations. I'd like to keep both the AST and the bytecode spec > > out of the language spec, > > Well, it wouldn't be any less portable than > bytecode hacking, and could potentially be > made more portable. > > A standard non-text representation of Python > code seems like a useful thing to have, > even if it's implemented quite separately > from the main compiler. > > We're actually part way there already, with > the internal AST <--> python AST transformation. > The other implementations would just need to > provide their own versions of that transformation. Well, maybe I should check in my pgen-in-Python implementation after all; that can do it out of the box except for the small detail that the symbol numbers for the grammar non-terminals don't match symbol.py. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 10 17:39:36 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 08:39:36 -0700 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> Message-ID: On 4/10/06, Talin wrote: > I have a rather odd use case for this. A while back, I decided I wanted to > write an algebraic solver, similar to what Mathematica does, except that > I wanted to write it in Python. Not just implement it in Python, but have > Python be the language in which the rules of formula manipulation were > expressed. [...] > You can immediately see the problem - since I don't have syntactical > support, I have to manually enter in the expression in an AST-like form, > which is clumsy to say the least. I've seen other packages like this (for different solvers) that solved this by overloading various operators to produce a parse tree. All you need is some primitive objects named X, Y, Z etc. and then X**2+Y**2 produces the appropriate parse tree. Then the analyzer can walk the parse tree. Anyway, I'm not sure I consider this a strong use case for needing access to Python's AST; the language you need to parse is much smaller than Python and a custom parser would probably do just as well. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kendall at monkeyfist.com Mon Apr 10 18:16:24 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Mon, 10 Apr 2006 12:16:24 -0400 Subject: [Python-3000] symbols? Message-ID: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> Folks, One thing I'd really like to see in Python 3000 is support for first- class symbols, with literal syntax. Stealing syntax from Ruby would work for me: def web_dispatch("/bank-account", :GET, retrieve_account): pass The lack thereof in Python is a wart I tend to run into in Python. I end up having to use strings, but not really using them for their stringiness (never using any of their methods, for example). Which works, but is sorta ugly. So I end up making classes and instances, but these are really just wrappers around strings, when all I really want is a symbol. (For example, in my SemWeb lab @ UMD, we built a Rete rule engine in Python that's being used in some SemWeb projects. We gave a talk about it at the last PyCon in DC. Symbols in that project would have made life a lot easier, instead of making up a bunch of classes that were just wrappers around strings. Which is: (1) really boring, (2) error-prone (because of (1)), and (3) generally not fun.) I also think, though I'm not really gonna argue the point, that having symbols would clean up lots of different sorts of APIs that use strings or class instances where symbols would work perfectly nicely. Some people claim the point of symbols is to save memory over repeated string usage, but I don't care about that, and I make claims about any kind of performance enhancement. I think symbols in Python would just make the language more fun. I don't know if the Ruby syntax for symbols, :foo, will work in Python (though I think visually it's a good thing), but it would be really nice to have symbols with *some* syntax in Python 3000. Again, this design request is based on aesthetics and fun: having symbols would make Python more fun. Cheers, Kendall Clark PS--I'd also like "?", "!", and "-" to be legal characters in function, method, and variable names, but I won't push my luck -- and I seem to remember Guido saying that would never happen, at some point back in the day. Fair enough. ;> From barry at python.org Mon Apr 10 18:24:04 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 10 Apr 2006 12:24:04 -0400 Subject: [Python-3000] symbols? In-Reply-To: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> Message-ID: <1144686244.15801.28.camel@resist.wooz.org> On Mon, 2006-04-10 at 12:16 -0400, Kendall Clark wrote: > I don't know if the Ruby syntax for symbols, :foo, will work in > Python (though I think visually it's a good thing), but it would be > really nice to have symbols with *some* syntax in Python 3000. To be fair, I think Ruby stole the idea and syntax from Lisp. It's an interesting idea, but I'd like to understand exactly what you mean by "a Python symbol". Can you give more detail about your idea, perhaps as a pseudo-PEP? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060410/2a241842/attachment.pgp From ianb at colorstudy.com Mon Apr 10 18:37:22 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 10 Apr 2006 11:37:22 -0500 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <44379062.1070004@gmail.com> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> Message-ID: <443A89C2.3040100@colorstudy.com> Nick Coghlan wrote: > And once you move to an out-of-process sandbox, then the only Python specific > issue remaining is improving the support for inter-process communications. The > security issues are then more in the domain of the OS: > - controlling file access permissions on a per-process basis > - controlling network access permissions on a per-process basis > - limiting the memory usage of a process > - limiting the CPU usage of a process This seems like an option that isn't given enough attention. Providing management routines and better IPC could provide a non-portable (but usable) restricted execution system more easily than other techniques, and serve as a starting point. From there I can imagine adding these restrictions into the bytecode interpreter directly, so that things like opening files was disallowed. Any non-restricted routines would go through some IPC code. AFAIK, this is basically the Javascript security model -- where the DOM or XPCOM or somesuch is used for everything that isn't purely sandbox code. This is a bit similar to what Vineet proposed, except that any modules added to the subinterpreter would either be run with the same restrictions as the rest of the subinterpreter, or a proxy to the module would be passed to the subinterpreter that could be used to communicate in a limited fashion to the parent interpreter. One advantage to this is that at each step something useful would be created. Better IPC would be useful for more than restricted execution. Or ways to create subinterpreters and communicate with them. Or making the interpreter startup faster, to facilitate spawning short-lived interpreters. As an example, by combining these features but leaving out restricted execution you can get something like the PHP model, which manages to protect well against buggy code (even if not so well against security bugs). There's probably a dozen distinct parts to this, but I think each is independently interesting and useful. And maybe, because these are useful subprojects, some of the people who can't commit to the time to take on this project as a whole could commit to some piece of it for which they have alternate motivations. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From jimjjewett at gmail.com Mon Apr 10 18:40:23 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 10 Apr 2006 12:40:23 -0400 Subject: [Python-3000] Adaptation and type declarations Message-ID: Every so often Guido talks about adding optional typing to python. Adaptation may offer the cleanest way to do this. Turning def fn(a, b, c="default"): ... into any of def fn(Seq a, Index b, Text c="default"): ... def fn(Seq(a), Index(b), Text(c)="default"): ... or (wrapped version) def fn(Seq a, Index b, Text c="default"): ... doesn't seem so awful. (I'm not sure it is a net positive for readability, but I'm also not sure it isn't.) I read the type information as "normally this is just an assertion, but I suppose some protocols might *make* it true for me." -jJ From kendall at monkeyfist.com Mon Apr 10 18:51:30 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Mon, 10 Apr 2006 12:51:30 -0400 Subject: [Python-3000] symbols? In-Reply-To: <1144686244.15801.28.camel@resist.wooz.org> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <1144686244.15801.28.camel@resist.wooz.org> Message-ID: <9EC6C2E8-45FB-4043-B504-CEA82A3F499E@monkeyfist.com> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Apr 10, 2006, at 12:24 PM, Barry Warsaw wrote: > On Mon, 2006-04-10 at 12:16 -0400, Kendall Clark wrote: > >> I don't know if the Ruby syntax for symbols, :foo, will work in >> Python (though I think visually it's a good thing), but it would be >> really nice to have symbols with *some* syntax in Python 3000. > > To be fair, I think Ruby stole the idea and syntax from Lisp. It's an > interesting idea, but I'd like to understand exactly what you mean > by "a > Python symbol". Can you give more detail about your idea, perhaps > as a > pseudo-PEP? I was afraid someone was going to ask that... :> FWIW, I didn't mean to suggest Ruby originated anything about symbols, just that as a close language cousin, the Ruby syntax for symbols made good sense to me for Py3K. But there are also some interesting diffs between Common Lisp, Scheme, and Ruby re: symbols. I'll try to send something pseudo-PEPish about this later today, after I've done a bit more work to get all the facts straight. Cheers, Kendall -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (Darwin) iD8DBQFEOo0VMWrdIbaAVEURAmD6AKCQXQIerODvuXbFFJ6YJCKgPZW9SACfe978 jfhBiRcXvvjrkVujNgPs0xU= =QUh8 -----END PGP SIGNATURE----- From g.brandl at gmx.net Mon Apr 10 18:54:33 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 10 Apr 2006 18:54:33 +0200 Subject: [Python-3000] symbols? In-Reply-To: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> Message-ID: Kendall Clark wrote: > PS--I'd also like "?", "!", and "-" to be legal characters in > function, method, and variable names, but I won't push my luck -- and > I seem to remember Guido saying that would never happen, at some > point back in the day. Fair enough. ;> At least "-" can never be part of any name since it's also an operator in Python (and that won't change without being noted in PEP 3099). Georg From pje at telecommunity.com Mon Apr 10 19:20:41 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 10 Apr 2006 13:20:41 -0400 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: Message-ID: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> At 08:39:36 PM 4/10/2006 -0700, "Guido van Rossum" wrote: >On 4/10/06, Talin wrote: > > I have a rather odd use case for this. A while back, I decided I wanted to > > write an algebraic solver, similar to what Mathematica does, except that > > I wanted to write it in Python. Not just implement it in Python, but have > > Python be the language in which the rules of formula manipulation were > > expressed. >[...] > > You can immediately see the problem - since I don't have syntactical > > support, I have to manually enter in the expression in an AST-like form, > > which is clumsy to say the least. > >I've seen other packages like this (for different solvers) that solved >this by overloading various operators to produce a parse tree. All you >need is some primitive objects named X, Y, Z etc. and then X**2+Y**2 >produces the appropriate parse tree. Then the analyzer can walk the >parse tree. I tried doing something like this when I was writing RuleDispatch, and gave up in disgust because there's no sane way to implement "and" and "or" operations with this approach. The bitwise operators (&, |, and ~) bind too tightly to be used with comparison expressions, so doing something like "x>y & z" would mean "x>(y&z)" instead of "(x>y)&z". Plus, since there are no "symbol" literals, you end up having to do something like "(_.x > _.y) & _.z" which looks thoroughly hideous. (The genexp to SQL use case has similar issues, since booleans are needed there as well.) So, since beautiful is better than ugly, I switched to parsing Python code in a string. As ugly as it is to bury the beautiful Python code in quotes, it's far better than writing some kind of hideous pseudo-Python outside of quotes. Even if you don't want to provide a writable func_ast attribute, it would be wonderful if there was some sort of "AST literal" syntax, especially since it would also address the request for a symbol syntax. (That is, an AST literal for a name could be used as a symbol, with no need to invent a separate symbol syntax.) Maybe in Python 3K the backquote could be used to quote code in some way, since we won't be using it for repr() anymore. :) So if you could backquote code to make an AST literal, you could spell RuleDispatch overrides like: @pprint_when(`isinstance(ob,list) and len(ob)<10`) def pprint_short_list(ob): ... And the database query use case could be done using something like: db.query(`(row for row in some_table if row.x>42)`)" And of course the symbol processing example is similar. Perhaps the backquote isn't the right character for this; I do seem to recall you commenting once that the backquote was a mistake because of issues with printing, fonts, etc., as far as not being distinguishable enough from the forward quote. But the basic idea would still be useful with some other way to spell an AST literal. From ianb at colorstudy.com Mon Apr 10 19:52:00 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 10 Apr 2006 12:52:00 -0500 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <4439ABB6.3030201@canterbury.ac.nz> References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> Message-ID: <443A9B40.1080708@colorstudy.com> Greg Ewing wrote: > Robert Brewer wrote: >> Part of the benefit of the bytecode-hacking is that your expression >> never has to be in a string. > > I'm wondering whether there should be some kind of > "code literal" syntax, where you write a Python > expression and the compiler transforms it as far > as the AST stage, then makes it available to the > program as an AST object. I also would really like this. There's a few projects I see that could use this now: Several ORMs. Dejavu (Robert's ORM) does lambda introspection. SQLObject and SQLAlchemy use operator overriding, which isn't as general but is used to similar effect. SQLComp by Peter Hunt is another lambda introspection library. I bet a non-SQL RDF system could benefit from the same thing. RuleDispatch compiles strings, maybe a bit fancier than the other projects, as I believe the other's don't pay as much attention to the scope at the position where the expression is defined. I believe Enthought's Traits is doing something vaguely similar (at least in motivation, at some level), using constraints. David Binger did a lightning talk on a system he wrote -- who's name I've forgotten -- which is also constraint-based and could benefit from a way to represent constraints. I think the various numeric packages do things along these lines, often with operator overloading and whatnot, so that you can give an expression that is efficiently implemented in a specific context. This is similar in spirit to what the ORMs are doing, except where the ORMs use the expression to generate SQL to be sent off somewhere, the numeric packages consume the expression in-process (but not in Python). In another email equation solvers were also suggested. Guido noted you can do this with operator overloading, but while that certainly does work I think we should have something better than that in Python. It also fails for the operators "and", "or", "not", "in", "is", and maybe some others I can't think of right now. So... in summary, I think there's a bunch of consumers already out there, but we could all use something a little more formalized. And maybe a little syntactic support; kludging expressions into lambdas doesn't look that pretty or make it at all clear what's going on. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From janssen at parc.com Mon Apr 10 20:05:28 2006 From: janssen at parc.com (Bill Janssen) Date: Mon, 10 Apr 2006 11:05:28 PDT Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: Your message of "Sun, 09 Apr 2006 17:50:15 PDT." <20060410005014.GC6659@tigger.digitaltorque.ca> Message-ID: <06Apr10.110535pdt."58633"@synergy1.parc.xerox.com> > And 2 + the JRE and jar files ease distribution of apps, something else that > Python needs to address.=20 I think we're very close to that with Python 2.5. Bill From msoulier at digitaltorque.ca Mon Apr 10 20:11:33 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Mon, 10 Apr 2006 14:11:33 -0400 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <06Apr10.110535pdt."58633"@synergy1.parc.xerox.com> References: <20060410005014.GC6659@tigger.digitaltorque.ca> <06Apr10.110535pdt."58633"@synergy1.parc.xerox.com> Message-ID: <20060410181133.GE476@tigger.digitaltorque.ca> On 10/04/06 Bill Janssen said: > I think we're very close to that with Python 2.5. Depends. I recently tried to install mechanize, and easytools tried to install a ton of dependencies, all requiring administrator access to install. --prefix didn't resolve dependencies, which isn't helpful if you're not root or you don't trust the install scripts. Plus, one of the resources was unavailable to my install just failed. Relying on various pieces scattered over sourceforge doesn't really seem all that close to me. That said, I should read up on 2.5. Ideally we'd have multiple distribution methods, one requiring an initial install of something like a Python runtime, and one that simply bundles everything ala PyInstaller. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060410/859ad7d7/attachment.pgp From ianb at colorstudy.com Mon Apr 10 20:30:10 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 10 Apr 2006 13:30:10 -0500 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <20060410181133.GE476@tigger.digitaltorque.ca> References: <20060410005014.GC6659@tigger.digitaltorque.ca> <06Apr10.110535pdt."58633"@synergy1.parc.xerox.com> <20060410181133.GE476@tigger.digitaltorque.ca> Message-ID: <443AA432.8050401@colorstudy.com> Michael P. Soulier wrote: > On 10/04/06 Bill Janssen said: >>I think we're very close to that with Python 2.5. > > > Depends. I recently tried to install mechanize, and easytools tried to install > a ton of dependencies, all requiring administrator access to install. --prefix > didn't resolve dependencies, which isn't helpful if you're not root or you > don't trust the install scripts. If you have specific problems you should email distutils-sig at python.org (and/or the mechanize maintainer) and describe the problems you are having, and what the desired behavior would be. And maybe try the svn trunk version of setuptools first to see if your problem is resolved. > Plus, one of the resources was unavailable to my install just failed. Relying > on various pieces scattered over sourceforge doesn't really seem all that > close to me. That's up to the people packaging their software, both how they distribute packages, and what packages they will require. It's not up to setuptools. And SF downloading just all-around sucks :-/ -- that's not setuptools' fault. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From guido at python.org Mon Apr 10 20:37:24 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 11:37:24 -0700 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443A3CB5.7060809@canterbury.ac.nz> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> <44386C42.90803@canterbury.ac.nz> <4439A6A3.8030409@canterbury.ac.nz> <443A3CB5.7060809@canterbury.ac.nz> Message-ID: On 4/10/06, Greg Ewing wrote: > Guido van Rossum wrote: > > > I'm not sure how that would help. What would it mean to have a > > capability for accessing e.g. x.__class__? > > If you can somehow get a reference to the > __builtin__.classof() function, then you > have that capability, otherwise you don't. > > The key idea is that by turning potentially > dangerous things like this from attributes > into functions, access to them can be much > more easily controlled. Any function can be > withheld without the creator of the function > having had to do anything special. But you > can't withhold an attribute unless its > accessor has been designed with that in > mind. But it does require a major cultural shift. That means backwards compatibility, and the need for users to relearn how they do things. I'm not sure I like either of those, given that these changes would affect *all* users just so that a small set of users can get the feature they want. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From adam.deprince at gmail.com Mon Apr 10 20:42:38 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Mon, 10 Apr 2006 14:42:38 -0400 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: References: <44373866.9070305@alumni.rice.edu> <021401c65c36$b3de2f70$26452597@bagio> Message-ID: <1144694558.3162.37.camel@localhost.localdomain> On Sun, 2006-04-09 at 18:06 -0700, Neal Norwitz wrote: > On 4/9/06, Giovanni Bajo wrote: > > Neal Norwitz wrote: > >>> r.setrlimit(r.RLIMIT_CPU, (5, 5)) > >>> 10000000000000000000**10000000000000000000 > Cputime limit exceeded > > To defeat this, you can do: signal.signal(signal.SIGXCPU, signal.SIG_IGN) > Of course, you would presumably prevent access to the signal module. > You could then install your own signal handler and handle this how you > want > . Instead of worrying about how to appropriately cripple CPython to provide a secure sandbox, perhaps we should be reaching towards PyPy for this answer? Cheers - Adam From msoulier at digitaltorque.ca Mon Apr 10 20:48:49 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Mon, 10 Apr 2006 14:48:49 -0400 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443AA432.8050401@colorstudy.com> References: <20060410005014.GC6659@tigger.digitaltorque.ca> <06Apr10.110535pdt."58633"@synergy1.parc.xerox.com> <20060410181133.GE476@tigger.digitaltorque.ca> <443AA432.8050401@colorstudy.com> Message-ID: <20060410184849.GF476@tigger.digitaltorque.ca> On 10/04/06 Ian Bicking said: > If you have specific problems you should email distutils-sig at python.org > (and/or the mechanize maintainer) and describe the problems you are > having, and what the desired behavior would be. > > And maybe try the svn trunk version of setuptools first to see if your > problem is resolved. > > That's up to the people packaging their software, both how they > distribute packages, and what packages they will require. It's not up > to setuptools. And SF downloading just all-around sucks :-/ -- that's > not setuptools' fault. Gotcha. Thanks. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060410/542431e7/attachment.pgp From guido at python.org Mon Apr 10 21:43:22 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 12:43:22 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/10/06, Jim Jewett wrote: > Every so often Guido talks about adding optional typing to python. > > Adaptation may offer the cleanest way to do this. > > > Turning > > def fn(a, b, c="default"): ... > > into any of > > def fn(Seq a, Index b, Text c="default"): ... > def fn(Seq(a), Index(b), Text(c)="default"): ... > > or (wrapped version) > > def fn(Seq a, > Index b, > Text c="default"): ... > > doesn't seem so awful. (I'm not sure it is a net positive for > readability, but I'm also not sure it isn't.) I read the type > information as "normally this is just an assertion, but I suppose some > protocols might *make* it true for me." I do think that we now have enough proposals on the table that require specifying types (or other metadata!) for each argument of certain function definitions that it's time to do something about this. A bit more than a year ago I blogged extensively about this. The only syntax that is acceptable to me is slightly different; the above would look like def fn(a: Seq, b: Index, c: Text = "default"): ... where Seq, Index and Text can be expressions (the main problem with the syntax you propose is that the type can't be much more than an identifier before it gets ambiguous or unreadable). A completely separate issue is what kind of objects Seq, Index and Text would be; but that's a discussion we have separate from the syntactic discussion. I would imagine that instead of the currently proposed @foo.register(list, int, str) def foo_1(a, b, c): ... we'd be allowed to write @foo.register def foo_1(a: list, b: int, c: str): ... FWIW (to ward of immediate questions) the syntax for an argument would be something like this: NAME [':' EXPR] ['=' EXPR] where currently it is NAME ['=' EXPR]. The only defined semantics would be that the name, the type and the default can all be inspected through the __signature__ attribute on the function object. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From eprparadocs at gmail.com Mon Apr 10 22:06:49 2006 From: eprparadocs at gmail.com (Chaz.) Date: Mon, 10 Apr 2006 16:06:49 -0400 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: <443ABAD9.70305@gmail.com> If you are going this far, why not also support a throws() modifier (or whatever you might call it). Right now I do something like: @throws(IOError) def foo(...) : as a way to indicate that foo() can throw a specific exception. I might suggest def foo(...) throws(...) : as a more integrated approach. On this direct topic because there is no type-specific polymorphism in Python, I occasionally write functions that do slightly different things depending on the type of input. Is there an approach to say a specific argument can take any number of different types? C. G. Guido van Rossum wrote: > On 4/10/06, Jim Jewett wrote: >> Every so often Guido talks about adding optional typing to python. >> >> Adaptation may offer the cleanest way to do this. >> >> >> Turning >> >> def fn(a, b, c="default"): ... >> >> into any of >> >> def fn(Seq a, Index b, Text c="default"): ... >> def fn(Seq(a), Index(b), Text(c)="default"): ... >> >> or (wrapped version) >> >> def fn(Seq a, >> Index b, >> Text c="default"): ... >> >> doesn't seem so awful. (I'm not sure it is a net positive for >> readability, but I'm also not sure it isn't.) I read the type >> information as "normally this is just an assertion, but I suppose some >> protocols might *make* it true for me." > > I do think that we now have enough proposals on the table that require > specifying types (or other metadata!) for each argument of certain > function definitions that it's time to do something about this. > > A bit more than a year ago I blogged extensively about this. The only > syntax that is acceptable to me is slightly different; the above would > look like > > def fn(a: Seq, b: Index, c: Text = "default"): ... > > where Seq, Index and Text can be expressions (the main problem with > the syntax you propose is that the type can't be much more than an > identifier before it gets ambiguous or unreadable). > > A completely separate issue is what kind of objects Seq, Index and > Text would be; but that's a discussion we have separate from the > syntactic discussion. > > I would imagine that instead of the currently proposed > > @foo.register(list, int, str) > def foo_1(a, b, c): ... > > we'd be allowed to write > > @foo.register > def foo_1(a: list, b: int, c: str): ... > > FWIW (to ward of immediate questions) the syntax for an argument would > be something like this: NAME [':' EXPR] ['=' EXPR] where currently it > is NAME ['=' EXPR]. The only defined semantics would be that the name, > the type and the default can all be inspected through the > __signature__ attribute on the function object. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/eprparadocs%40gmail.com > From jimjjewett at gmail.com Mon Apr 10 22:13:20 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 10 Apr 2006 16:13:20 -0400 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/10/06, Guido van Rossum wrote: > The only syntax that is acceptable to me ... would look like > def fn(a: Seq, b: Index, c: Text = "default"): ... > where Seq, Index and Text can be expressions (the main problem with > the syntax you propose is that the type can't be much more than an > identifier before it gets ambiguous or unreadable). I agree that putting the identifier after the type (and wrapping to one parameter per line) allows for a longer expression. On the other hand, it also allows for a *much* longer expression, which becomes unreadable anyhow. Whether this would be enough of a problem in practice to justify requiring identifiers ... I'm not sure. Maybe the actual usage of conditional expressions will help answer that. > A completely separate issue is what kind of objects Seq, Index and > Text would be; but that's a discussion we have separate from the > syntactic discussion. What do you mean? The only question that leaps out at me is whether they should be predicates or constructors. In other words, should def f(a:Seq): ... mean something closer to def f(a): assert Seq(a) ... or def f(a): a=Seq(a) ... If adaptation is light enough to be useful, then I can't see any reason to prefer the first. def f(a:0 References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> <443A9B40.1080708@colorstudy.com> Message-ID: > Greg Ewing wrote: > > I'm wondering whether there should be some kind of > > "code literal" syntax, where you write a Python > > expression and the compiler transforms it as far > > as the AST stage, then makes it available to the > > program as an AST object. On 4/10/06, Ian Bicking wrote: > I also would really like this. There's a few projects I see that could > use this now [...] (Also a favorable response from Phillip Eby.) I believe I've seen something like this described for an experimental version of Haskell; they had an operator that you used to prefix an expression and it returned the AST for that expression. One important feature was that they also had the opposite feature -- you could use another operator in the thus-quoted expression and it would evalulate that subtree and a normal expression, and substitute the result in the tree. They probably used it to implement a macro facility (though the example I remember was a type-safe printf facility). I'm going to place the ball firmly in the court of those who want this feature: please write a proposal *and* an implementation. All that work is *not* going to get your proposal accepted for certain, but it'll let me look at it. (You know, I really hate to be in the position of the parent always saying "no". But there's gotta be someone who ensures that Python 3000 will in fact be Python 3000 and not Perl 7. There are many, many more possible changes to the language than can possibly be accepted. We have to prioritize.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Mon Apr 10 22:34:30 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 10 Apr 2006 13:34:30 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: <443ABAD9.70305@gmail.com> References: <443ABAD9.70305@gmail.com> Message-ID: <20060410133237.A7BE.JCARLSON@uci.edu> "Chaz." wrote: > If you are going this far, why not also support a throws() modifier (or > whatever you might call it). > > Right now I do something like: > > @throws(IOError) > def foo(...) : ... > I might suggest > > def foo(...) throws(...) : > > as a more integrated approach. You are just relocating decorators. Stop. - Josiah From g.brandl at gmx.net Mon Apr 10 22:35:01 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Mon, 10 Apr 2006 22:35:01 +0200 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: <443ABAD9.70305@gmail.com> References: <443ABAD9.70305@gmail.com> Message-ID: Chaz. wrote: > If you are going this far, why not also support a throws() modifier (or > whatever you might call it). > > Right now I do something like: > > @throws(IOError) > def foo(...) : > > as a way to indicate that foo() can throw a specific exception. > > I might suggest > > def foo(...) throws(...) : Oh, and as we go, we could require that code that calls such a function explicitly handles that exceptions. Changing the language's name to Javthon seems favorable, too ;) Georg From guido at python.org Mon Apr 10 22:43:50 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 13:43:50 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: <443ABAD9.70305@gmail.com> References: <443ABAD9.70305@gmail.com> Message-ID: On 4/10/06, Chaz. wrote: > If you are going this far, why not also support a throws() modifier (or > whatever you might call it). > > Right now I do something like: > > @throws(IOError) > def foo(...) : > > as a way to indicate that foo() can throw a specific exception. > > I might suggest > > def foo(...) throws(...) : > > as a more integrated approach. -1 without more (and I mean A LOT MORE) motivation. IOW unless you write a whole PEP I'm going to ignore this proposal. > On this direct topic because there is no type-specific polymorphism in > Python, What does that mean? I always think of Python as a language with excellent support for polymorphism (due to duck typing). So you probably are using words in a way that I don't understand (don't worry, that happens all the time). > I occasionally write functions that do slightly different things > depending on the type of input. Is there an approach to say a specific > argument can take any number of different types? Yes, I proposed using the '|' operator to separate alternatives. E.g. def foo(x: int|float, y: str|list[str]): ... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From eprparadocs at gmail.com Mon Apr 10 22:43:54 2006 From: eprparadocs at gmail.com (Chaz.) Date: Mon, 10 Apr 2006 16:43:54 -0400 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: <20060410133237.A7BE.JCARLSON@uci.edu> References: <443ABAD9.70305@gmail.com> <20060410133237.A7BE.JCARLSON@uci.edu> Message-ID: <443AC38A.1070900@gmail.com> Is that what adding some typing to the system will do? Josiah Carlson wrote: > "Chaz." wrote: >> If you are going this far, why not also support a throws() modifier (or >> whatever you might call it). >> >> Right now I do something like: >> >> @throws(IOError) >> def foo(...) : > ... >> I might suggest >> >> def foo(...) throws(...) : >> >> as a more integrated approach. > > You are just relocating decorators. Stop. > > > - Josiah > > From eprparadocs at gmail.com Mon Apr 10 22:47:38 2006 From: eprparadocs at gmail.com (Chaz.) Date: Mon, 10 Apr 2006 16:47:38 -0400 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: <443ABAD9.70305@gmail.com> Message-ID: <443AC46A.2030900@gmail.com> Personally I don't care what you call it. If you demean it because you disagree, you might as well fore go the whole issue of typing arguments. After all why turn Python in Cobol? C. G. Georg Brandl wrote: > Chaz. wrote: >> If you are going this far, why not also support a throws() modifier (or >> whatever you might call it). >> >> Right now I do something like: >> >> @throws(IOError) >> def foo(...) : >> >> as a way to indicate that foo() can throw a specific exception. >> >> I might suggest >> >> def foo(...) throws(...) : > > Oh, and as we go, we could require that code that calls such a function > explicitly handles that exceptions. > > Changing the language's name to Javthon seems favorable, too ;) > > Georg > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/eprparadocs%40gmail.com > From guido at python.org Mon Apr 10 22:48:56 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 13:48:56 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/10/06, Jim Jewett wrote: > On 4/10/06, Guido van Rossum wrote: > > A completely separate issue is what kind of objects Seq, Index and > > Text would be; but that's a discussion we have separate from the > > syntactic discussion. > > What do you mean? The only question that leaps out at me is whether > they should be predicates or constructors. In other words, should > > def f(a:Seq): ... > > mean something closer to > > def f(a): > assert Seq(a) > ... > > or > > def f(a): > a=Seq(a) > ... It should mean neither. As I tried to say, the *only* semantics is that the value of the expression is accessible through the __signature__ attribute of the function. It is then up to the decorator to decide what it means. > If adaptation is light enough to be useful, then I can't see any > reason to prefer the first. > > def f(a:0 > isn't *that* much cleaner than whichever of > > def f(a:clip_to_int_range(0,10)): ... > def f(a:int_in_range(0,10)): ... > def f(a:clip_to_number_range(0,10)): ... > def f(a:number_in_range(0,10)): ... > > was actually intended. That's not something I had considered. It would have to turn the expression into an implicit lambda (or an AST), which isn't something we should do lightly, and which conflicts with other proposed uses, where the value of the expression is made available to the decorator -- the value of '0 References: Message-ID: <443AC6AA.8040303@ieee.org> Guido van Rossum wrote: > On 4/10/06, Jim Jewett wrote: > >>Every so often Guido talks about adding optional typing to python. >> >>Adaptation may offer the cleanest way to do this. >> >> >>Turning >> >> def fn(a, b, c="default"): ... >> >>into any of >> >> def fn(Seq a, Index b, Text c="default"): ... >> def fn(Seq(a), Index(b), Text(c)="default"): ... >> >>or (wrapped version) >> >> def fn(Seq a, >> Index b, >> Text c="default"): ... >> >>doesn't seem so awful. (I'm not sure it is a net positive for >>readability, but I'm also not sure it isn't.) I read the type >>information as "normally this is just an assertion, but I suppose some >>protocols might *make* it true for me." > > > I do think that we now have enough proposals on the table that require > specifying types (or other metadata!) for each argument of certain > function definitions that it's time to do something about this. > > A bit more than a year ago I blogged extensively about this. The only > syntax that is acceptable to me is slightly different; the above would > look like > > def fn(a: Seq, b: Index, c: Text = "default"): ... > > where Seq, Index and Text can be expressions (the main problem with > the syntax you propose is that the type can't be much more than an > identifier before it gets ambiguous or unreadable). > > A completely separate issue is what kind of objects Seq, Index and > Text would be; but that's a discussion we have separate from the > syntactic discussion. > > I would imagine that instead of the currently proposed > > @foo.register(list, int, str) > def foo_1(a, b, c): ... > > we'd be allowed to write > > @foo.register > def foo_1(a: list, b: int, c: str): ... > > FWIW (to ward of immediate questions) the syntax for an argument would > be something like this: NAME [':' EXPR] ['=' EXPR] where currently it > is NAME ['=' EXPR]. The only defined semantics would be that the name, > the type and the default can all be inspected through the > __signature__ attribute on the function object. > I think we'd still want some way to explicitly register a function for a signature. Either because the function had no types declared or because we wanted to register it to different types than the signature implied. Say, for example: @foo.register_for(list, int, unicode) def foo_2(a, b, c): ... I imagine register would then be just syntactic sugar for register_for. Something like: def register(obj): signature = [x.type for x in obj.__signature__] self.register_for(*signature)(obj) Or something like that. Regards, -tim From tim.hochberg at ieee.org Mon Apr 10 23:03:14 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Mon, 10 Apr 2006 14:03:14 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: <443AC812.30609@ieee.org> Guido van Rossum wrote: > On 4/10/06, Jim Jewett wrote: > >>On 4/10/06, Guido van Rossum wrote: >> >>>A completely separate issue is what kind of objects Seq, Index and >>>Text would be; but that's a discussion we have separate from the >>>syntactic discussion. >> >>What do you mean? The only question that leaps out at me is whether >>they should be predicates or constructors. In other words, should >> >> def f(a:Seq): ... >> >>mean something closer to >> >> def f(a): >> assert Seq(a) >> ... >> >>or >> >> def f(a): >> a=Seq(a) >> ... > > > It should mean neither. As I tried to say, the *only* semantics is > that the value of the expression is accessible through the > __signature__ attribute of the function. It is then up to the > decorator to decide what it means. > > >>If adaptation is light enough to be useful, then I can't see any >>reason to prefer the first. >> >> def f(a:0> >>isn't *that* much cleaner than whichever of >> >> def f(a:clip_to_int_range(0,10)): ... >> def f(a:int_in_range(0,10)): ... >> def f(a:clip_to_number_range(0,10)): ... >> def f(a:number_in_range(0,10)): ... >> >>was actually intended. > > > That's not something I had considered. It would have to turn the > expression into an implicit lambda (or an AST), which isn't something > we should do lightly, and which conflicts with other proposed uses, > where the value of the expression is made available to the decorator > -- the value of '0 References: Message-ID: On 4/10/06, Guido van Rossum wrote: > On 4/10/06, Jim Jewett wrote: > > On 4/10/06, Guido van Rossum wrote: > > > A completely separate issue is what kind of objects > > > Seq, Index and Text would be > > What do you mean? The only question that leaps > > out at me is whether they should be predicates or > > constructors. In other words, should > > def f(a:Seq): ... > > mean something closer to > > def f(a): assert Seq(a) ... > > or > > def f(a): a=Seq(a) ... > It should mean neither. As I tried to say, the *only* > semantics is that the value of the expression is > accessible through the __signature__ attribute of > the function. It is then up to the decorator to decide > what it means. So type annotation is only for use with decorators, not for general documentation or compiler speedups? > > If adaptation is light enough to be useful, then I > > can't see any reason to prefer the first. > > def f(a:0 > isn't *that* much cleaner than whichever of > > def f(a:clip_to_int_range(0,10)): ... > > def f(a:int_in_range(0,10)): ... > > def f(a:clip_to_number_range(0,10)): ... > > def f(a:number_in_range(0,10)): ... > > was actually intended. > That's not something I had considered. It would have to > turn the expression into an implicit lambda (or an AST), > which isn't something we should do lightly, and which > conflicts with other proposed uses, where the value of > the expression is made available to the decorator > -- the value of '0 References: Message-ID: <443AC6AA.8040303@ieee.org> Guido van Rossum wrote: > On 4/10/06, Jim Jewett wrote: > >>Every so often Guido talks about adding optional typing to python. >> >>Adaptation may offer the cleanest way to do this. >> >> >>Turning >> >> def fn(a, b, c="default"): ... >> >>into any of >> >> def fn(Seq a, Index b, Text c="default"): ... >> def fn(Seq(a), Index(b), Text(c)="default"): ... >> >>or (wrapped version) >> >> def fn(Seq a, >> Index b, >> Text c="default"): ... >> >>doesn't seem so awful. (I'm not sure it is a net positive for >>readability, but I'm also not sure it isn't.) I read the type >>information as "normally this is just an assertion, but I suppose some >>protocols might *make* it true for me." > > > I do think that we now have enough proposals on the table that require > specifying types (or other metadata!) for each argument of certain > function definitions that it's time to do something about this. > > A bit more than a year ago I blogged extensively about this. The only > syntax that is acceptable to me is slightly different; the above would > look like > > def fn(a: Seq, b: Index, c: Text = "default"): ... > > where Seq, Index and Text can be expressions (the main problem with > the syntax you propose is that the type can't be much more than an > identifier before it gets ambiguous or unreadable). > > A completely separate issue is what kind of objects Seq, Index and > Text would be; but that's a discussion we have separate from the > syntactic discussion. > > I would imagine that instead of the currently proposed > > @foo.register(list, int, str) > def foo_1(a, b, c): ... > > we'd be allowed to write > > @foo.register > def foo_1(a: list, b: int, c: str): ... > > FWIW (to ward of immediate questions) the syntax for an argument would > be something like this: NAME [':' EXPR] ['=' EXPR] where currently it > is NAME ['=' EXPR]. The only defined semantics would be that the name, > the type and the default can all be inspected through the > __signature__ attribute on the function object. > I think we'd still want some way to explicitly register a function for a signature. Either because the function had no types declared or because we wanted to register it to different types than the signature implied. Say, for example: @foo.register_for(list, int, unicode) def foo_2(a, b, c): ... I imagine register would then be just syntactic sugar for register_for. Something like: def register(obj): signature = [x.type for x in obj.__signature__] self.register_for(*signature)(obj) Or something like that. Regards, -tim From guido at python.org Mon Apr 10 23:57:09 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 14:57:09 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: [Guido] > > It should mean neither. As I tried to say, the *only* > > semantics is that the value of the expression is > > accessible through the __signature__ attribute of > > the function. It is then up to the decorator to decide > > what it means. [Jim Jewett] > So type annotation is only for use with decorators, not for general > documentation or compiler speedups? Those are two entirely different purposes. It's fine for documentation, for use by IDEs, and for use by e.g. pychecker. However, using it for compiler speedups is unlikely given the run-time semantics and Python's extremely dynamic nature. > > > If adaptation is light enough to be useful, then I > > > can't see any reason to prefer the first. > > > > def f(a:0 > > > isn't *that* much cleaner than whichever of > > > > def f(a:clip_to_int_range(0,10)): ... > > > def f(a:int_in_range(0,10)): ... > > > def f(a:clip_to_number_range(0,10)): ... > > > def f(a:number_in_range(0,10)): ... > > > > was actually intended. Oops. I guess you were arguing *against* the former; I (and perhaps others) misread your mail as arguing *for* the "a:0 > That's not something I had considered. It would have to > > turn the expression into an implicit lambda (or an AST), > > which isn't something we should do lightly, and which > > conflicts with other proposed uses, where the value of > > the expression is made available to the decorator > > -- the value of '0 > I suppose I should have supplied definitions, just to be explicit. For example: > > def number_in_range(min, max): > def checker(val): > if min < val < max: > return val > raise ValueError("%s not between %s and %s", > (val, min, max)) > return checker > > Then checker (the actual predicate) is available at compile time; the > only thing waiting for run time is the result of checker(a) for that > particular call. This is perfectly reasonable. > Or were you just saying that even the predicate-only simple expression > version should really have been written as > > def f(a:lambda a:(0 > so that there would be a complete predicate available independent of > the function itself? I'm still confused; I'm tempted to say "yes of course" because '0 If so, then I can't see any reason not to require that the type > markers be callables whose return value is used in place of the > argument. Again a double negative, possibly confusing me. And again, I don't want to assign *any* semantics to the type markers; requiring them to be "adaptation" functions vastly reduces their usefulness IMO. E.g. I'd like to be able to have a decorator that requires the convention of using concrete types (e.g. list, int, file) as type markers; calling these would be a mistake, they should be used in isinstance() calls for example. > There is nothing to prevent the type markers from returning the > original object (possibly with extra annotations). There is if you want to use existing concrete types. list(x) is defined as returning a new list. Etc. > For efficiency, > that should be the normal case. Nor is there anything preventing a > decorator (if you assume there will always be one) from changing the > signature anyhow. The only downside is an additional call per > annotated argument *if* neither the decorator nor the compiler has > used the information to optimize something away. You're thinking of the type markers as adapters exclusively. I'm trying to promote the idea that they could be anything and that they are only constrained by the implicit contract of the decorator used (if any). That's what "used for documentation purposes only" means. Several people (in a previous round of discussion) have agreed that they'd be totally happy if def f(x: int): print x*2 f("abc") executed as if the ": int" part was not present (hence printing "abcabc"). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 11 00:19:37 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 15:19:37 -0700 Subject: [Python-3000] Did I miss anything? Message-ID: I've been intermittently following and skipping the Python-3000 list; my personal bandwidth just wasn't up to following everything. I know a few PEPs have been checked in (thanks Georg!). I know Adam DePrince has four different proto-PEPs up at http://www.deprince.net/ideas/peps.html but I don't know to what extent these are his personal ideas or reflect the outcome of discussions here; I'm confused by his striking through the first two. Are there other proto-PEPs being worked on? I would appreciate if the authors would send me a note (or reply here) with the URL and the status. Unfortunately I'm not going to be able to keep up in the next two weeks either -- tomorrow night I'm flying to London for a workshop with the Shuttleworth Foundation; then, after spending the easter weekend with my relatives in Holland, I'm off to Oxford for the ACCU conference. I hope to see some of you there! (I'll be giving a keynote about Python 3000 there, which I hope to make as current as humanly possible, hence in part my request here.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From steven.bethard at gmail.com Tue Apr 11 00:59:53 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 10 Apr 2006 16:59:53 -0600 Subject: [Python-3000] Did I miss anything? In-Reply-To: References: Message-ID: On 4/10/06, Guido van Rossum wrote: > Are there other proto-PEPs being worked on? I would appreciate if the > authors would send me a note (or reply here) with the URL and the > status. This is the Backwards Incompatibility PEP discussed earlier. I've submitted it for a PEP number, but haven't heard back yet: http://ucsu.colorado.edu/~bethard/py/pep_backwards_incompatible.txt http://ucsu.colorado.edu/~bethard/py/pep_backwards_incompatible.html This is potentially a Python 2.6 PEP, but it has some optional extensions for Python 3000 and may be relevant to the adaptation/overloading/interfaces discussion. It proposes a make statement such that: make : would be translated into the assignment: = ("", , ) much in the same way that the class statement works. I've posted it to comp.lang.python and had generally positive feedback. I've submitted it for a PEP number, but I haven't heard back yet: http://ucsu.colorado.edu/~bethard/py/pep_make_statement.txt http://ucsu.colorado.edu/~bethard/py/pep_make_statement.html Steve -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From jcarlson at uci.edu Tue Apr 11 01:58:28 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 10 Apr 2006 16:58:28 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: <443AC38A.1070900@gmail.com> References: <20060410133237.A7BE.JCARLSON@uci.edu> <443AC38A.1070900@gmail.com> Message-ID: <20060410165152.A7C7.JCARLSON@uci.edu> "Chaz." wrote: > Is that what adding some typing to the system will do? Not necessarily, but this particular instance isn't necessarily a typing-system issue. All sane or insane implementations of the @throws decorator that I can think of (annotation, exception translation, unittest, etc.) offer no improvement to the language that would warrant the necessity of a "throws" syntax to replace the @throws decorator. Remember, not all X line functions should become builtins or syntax. - Josiah > Josiah Carlson wrote: > > "Chaz." wrote: > >> If you are going this far, why not also support a throws() modifier (or > >> whatever you might call it). > >> > >> Right now I do something like: > >> > >> @throws(IOError) > >> def foo(...) : > > ... > >> I might suggest > >> > >> def foo(...) throws(...) : > >> > >> as a more integrated approach. > > > > You are just relocating decorators. Stop. > > > > > > - Josiah > > > > From greg.ewing at canterbury.ac.nz Tue Apr 11 02:00:54 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 12:00:54 +1200 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> Message-ID: <443AF1B6.7080702@canterbury.ac.nz> Guido van Rossum wrote: > Anyway, I'm not sure I consider this a strong use case for needing > access to Python's AST; the language you need to parse is much smaller > than Python and a custom parser would probably do just as well. So maybe this is an argument for having some kind of simple parsing package in the stdlib, for knocking up little specialised languages like this. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 11 02:05:16 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 12:05:16 +1200 Subject: [Python-3000] symbols? In-Reply-To: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> Message-ID: <443AF2BC.4050702@canterbury.ac.nz> Kendall Clark wrote: > Folks, > > One thing I'd really like to see in Python 3000 is support for first- > class symbols, > > def web_dispatch("/bank-account", :GET, retrieve_account): pass class SymGen: def __getattr__(self, name): return intern(name) sym = SymGen() web_dispatch("/bank-account", sym.GET, retrieve_account) -- Greg From pje at telecommunity.com Tue Apr 11 02:09:44 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon, 10 Apr 2006 20:09:44 -0400 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <443AECA3.2040601@latte.ca> References: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060410200628.05b37440@mail.telecommunity.com> At 07:39 PM 4/10/2006 -0400, Blake Winton wrote: >Phillip J. Eby wrote: >>I tried doing something like this when I was writing RuleDispatch, and >>gave up in disgust because there's no sane way to implement "and" and >>"or" operations with this approach. The bitwise operators (&, |, and ~) >>bind too tightly to be used with comparison expressions, so doing >>something like "x>y & z" would mean "x>(y&z)" instead of "(x>y)&z". > >How would symbols, or ast access have helped you with that problem? Would >you really go around manually mangling the parse tree to move the "&" >higher in the branches somehow? No - I'd have used 'and', 'or' and 'not': they're what I wanted to use in the first place. >What do you think about the oft-maligned curly brace? I think it should remain the syntax for dictionary literals, and not get involved here. >And it should be possible to tell the difference between it and a dict >with a bit of a lookahead... (Really it's the conflict with dict >initialization that I expect to kill this proposal.) For both implementation and readability reasons, Python syntax should require no more than 1 token of lookahead - by which I mean you should be able to parse it in purely token-by-token fashion without ever peeking ahead. From greg.ewing at canterbury.ac.nz Tue Apr 11 02:11:32 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 12:11:32 +1200 Subject: [Python-3000] symbols? In-Reply-To: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> Message-ID: <443AF434.7000503@canterbury.ac.nz> Kendall Clark wrote: > One thing I'd really like to see in Python 3000 is support for first- > class symbols, with literal syntax. Actually I think enumerations would be more useful than symbols. There's no essential difference between a symbol and an interned string. The only real disadvantage to using strings as symbols is that you don't get an immediate NameError if you misspell one. A symbol type wouldn't solve that problem; enumerations would. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 11 02:17:00 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 12:17:00 +1200 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> Message-ID: <443AF57C.1020806@canterbury.ac.nz> Georg Brandl wrote: >>PS--I'd also like "?", "!", and "-" to be legal characters in >>function, method, and variable names, but I won't push my luck -- and >>I seem to remember Guido saying that would never happen, at some >>point back in the day. And I hope he never changes his mind! Identifiers with random punctuation in them look ugly to me. -- Greg From bwinton at latte.ca Tue Apr 11 01:39:15 2006 From: bwinton at latte.ca (Blake Winton) Date: Mon, 10 Apr 2006 19:39:15 -0400 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> References: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> Message-ID: <443AECA3.2040601@latte.ca> Phillip J. Eby wrote: > I tried doing something like this when I was writing RuleDispatch, and gave > up in disgust because there's no sane way to implement "and" and "or" > operations with this approach. The bitwise operators (&, |, and ~) bind > too tightly to be used with comparison expressions, so doing something like > "x>y & z" would mean "x>(y&z)" instead of "(x>y)&z". How would symbols, or ast access have helped you with that problem? Would you really go around manually mangling the parse tree to move the "&" higher in the branches somehow? > So if you could backquote code to make an AST literal, you could spell > RuleDispatch overrides like: > @pprint_when(`isinstance(ob,list) and len(ob)<10`) > def pprint_short_list(ob): > ... > And the database query use case could be done using something like: > db.query(`(row for row in some_table if row.x>42)`)" > > Perhaps the backquote isn't the right character for this; What do you think about the oft-maligned curly brace? @pprint_when({isinstance(ob,list) and len(ob)<10}) def pprint_short_list(ob): db.query( {(row for row in some_table if row.x>42)} ) ? Not too heinous, in my view. And it should be possible to tell the difference between it and a dict with a bit of a lookahead... (Really it's the conflict with dict initialization that I expect to kill this proposal.) Later, Blake. From guido at python.org Tue Apr 11 02:26:35 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 10 Apr 2006 17:26:35 -0700 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <443AF1B6.7080702@canterbury.ac.nz> References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> <443AF1B6.7080702@canterbury.ac.nz> Message-ID: On 4/10/06, Greg Ewing wrote: > Guido van Rossum wrote: > > Anyway, I'm not sure I consider this a strong use case for needing > > access to Python's AST; the language you need to parse is much smaller > > than Python and a custom parser would probably do just as well. > > So maybe this is an argument for having some > kind of simple parsing package in the stdlib, > for knocking up little specialised languages > like this. Now I'll *really* try to get pgen in. Unfortunately I can't make the April 17 deadline. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Tue Apr 11 02:32:33 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 12:32:33 +1200 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> References: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> Message-ID: <443AF921.1090508@canterbury.ac.nz> Phillip J. Eby wrote: > I tried doing something like this when I was writing RuleDispatch, and gave > up in disgust because there's no sane way to implement "and" and "or" > operations with this approach. The bitwise operators (&, |, and ~) bind > too tightly to be used with comparison expressions, Perhaps you'd like to lend your support to PEP 335, then?-) http://www.python.org/dev/peps/pep-0335/ > Maybe in Python 3K the backquote could be used to quote code in some way, > since we won't be using it for repr() anymore. :) I really hope something better could be found. Code in backquotes doesn't look much less ugly than code in normal quotes. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 11 02:50:19 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 12:50:19 +1200 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: <1144694558.3162.37.camel@localhost.localdomain> References: <44373866.9070305@alumni.rice.edu> <021401c65c36$b3de2f70$26452597@bagio> <1144694558.3162.37.camel@localhost.localdomain> Message-ID: <443AFD4B.1060201@canterbury.ac.nz> Adam DePrince wrote: > Instead of worrying about how to appropriately cripple CPython to > provide a secure sandbox, perhaps we should be reaching towards PyPy for > this answer? This thought might be relevant to another issue that came to my mind recently. It concerns using Python as an extension language for an application. If you have an application implemented in some traditional way such as compiled C or C++, embedding a Python interpreter in it is a nice way of providing users with a nice playground they can program in without fear of messing up the main application, which is "hard coded" and able to protect itself from most things that the Python code can do. But what happens if the application itself is implemented in Python? Then it's very difficult to keep a separation between the application proper and the scripting environment. So I think this is another, distinct use case for wanting a "nested" Python environment. Here the goal isn't to protect against a malicious user, since the user running the application and the user doing the scripting are the same person. Rather, it's just to reduce the likelihood of accidentally messing things up, and to present the user with a much simpler programming environment than the whole application. So the requirements are far less strict, and if there were a few holes in places, it wouldn't matter so much. -- Greg From ianb at colorstudy.com Tue Apr 11 05:01:04 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 10 Apr 2006 22:01:04 -0500 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: <443AFD4B.1060201@canterbury.ac.nz> References: <44373866.9070305@alumni.rice.edu> <021401c65c36$b3de2f70$26452597@bagio> <1144694558.3162.37.camel@localhost.localdomain> <443AFD4B.1060201@canterbury.ac.nz> Message-ID: <443B1BF0.80301@colorstudy.com> Greg Ewing wrote: > So I think this is another, distinct use case for > wanting a "nested" Python environment. Here the goal > isn't to protect against a malicious user, since the > user running the application and the user doing the > scripting are the same person. Rather, it's just > to reduce the likelihood of accidentally messing > things up, and to present the user with a much > simpler programming environment than the whole > application. So the requirements are far less > strict, and if there were a few holes in places, > it wouldn't matter so much. So I hear there's long been the ability to make multiple interpreters at the C level -- used by mod_python and presumably used in some other embedding situations -- but this has never been exposed at the Python level. I'm curious why that never happened? Merely inertia, or something more significant? -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From talin at acm.org Tue Apr 11 05:31:51 2006 From: talin at acm.org (Talin) Date: Tue, 11 Apr 2006 03:31:51 +0000 (UTC) Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) References: <435DF58A933BA74397B42CDEB8145A86224B96@ex9.hostedexchange.local> <4439ABB6.3030201@canterbury.ac.nz> <443AF1B6.7080702@canterbury.ac.nz> Message-ID: Guido van Rossum python.org> writes: > > So maybe this is an argument for having some > > kind of simple parsing package in the stdlib, > > for knocking up little specialised languages > > like this. > > Now I'll *really* try to get pgen in. Unfortunately I can't make the > April 17 deadline. Have a look at reflex.py (in the cheese shop), a very tiny, bare-bones lexer which is intended for just this sort of thing. (Well, it can be used for bigger things too - I've managed to implement a complete hand-written C++ parser on top of it...) -- Talin From talin at acm.org Tue Apr 11 05:41:18 2006 From: talin at acm.org (Talin) Date: Tue, 11 Apr 2006 03:41:18 +0000 (UTC) Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) References: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> <443AF921.1090508@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > Perhaps you'd like to lend your support to PEP 335, then?-) > > http://www.python.org/dev/peps/pep-0335/ I think that would give me *most* of what I would need for my solver. What remains is a consise way to specify bound vs. unbound variables I would prefer not to have to restrict people to using a limited set of pre-declared undefined variables, i.e. X, Y, Z, etc.; plus I would want to be able to associate additional constraints with those unbound variables. At the risk of violating Guido's prohibition against 'wild ideas' -- it seems to me that a lot could be done by reserving a special operator - say '?' - that has no built-in semantics. Thus, people who wanted to create alternative notations, expressed in Python code, could use this 'user operator' to attach any language specific metadata to terms of the expression. Just a thought, not to be taken too seriously... -- Talin From kendall at monkeyfist.com Tue Apr 11 06:58:47 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 11 Apr 2006 00:58:47 -0400 Subject: [Python-3000] symbols? In-Reply-To: <443AF57C.1020806@canterbury.ac.nz> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF57C.1020806@canterbury.ac.nz> Message-ID: On Apr 10, 2006, at 8:17 PM, Greg Ewing wrote: > Georg Brandl wrote: > >>> PS--I'd also like "?", "!", and "-" to be legal characters in >>> function, method, and variable names, but I won't push my luck -- >>> and >>> I seem to remember Guido saying that would never happen, at some >>> point back in the day. > > And I hope he never changes his mind! Identifiers > with random punctuation in them look ugly to me. Hmm. Tastes vary, obviously. I don't consider using ? to signal boolean returning functions or methods to be random. And "-" has always seemed a better word separator (or at least as good) as "_". But whatever. I happily accept the limitations. :> Cheers, Kendall From kendall at monkeyfist.com Tue Apr 11 07:00:14 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 11 Apr 2006 01:00:14 -0400 Subject: [Python-3000] symbols? In-Reply-To: <443AF2BC.4050702@canterbury.ac.nz> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF2BC.4050702@canterbury.ac.nz> Message-ID: <12216E36-F0BF-4E90-841A-2A45300F5D5D@monkeyfist.com> On Apr 10, 2006, at 8:05 PM, Greg Ewing wrote: > Kendall Clark wrote: >> Folks, >> >> One thing I'd really like to see in Python 3000 is support for first- >> class symbols, >> >> def web_dispatch("/bank-account", :GET, retrieve_account): pass > > class SymGen: > > def __getattr__(self, name): > return intern(name) > > sym = SymGen() > > web_dispatch("/bank-account", sym.GET, retrieve_account) Yep, of course. I suspect anyone who's used Python in anger has written some such thing. The question is whether it's worth effort to have something standardized in the language. I think it would be worth the effort, but that's my 2 cents. Cheers, Kendall From kendall at monkeyfist.com Tue Apr 11 07:06:31 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 11 Apr 2006 01:06:31 -0400 Subject: [Python-3000] symbols? In-Reply-To: <443AF434.7000503@canterbury.ac.nz> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF434.7000503@canterbury.ac.nz> Message-ID: On Apr 10, 2006, at 8:11 PM, Greg Ewing wrote: > Kendall Clark wrote: > >> One thing I'd really like to see in Python 3000 is support for first- >> class symbols, with literal syntax. > > Actually I think enumerations would be more useful than > symbols. I don't see why we couldn't have both or neither. No *real* connection between them, not really. > There's no essential difference between a symbol > and an interned string. Well, some differences. In Ruby, for example, symbols are global in scope and strings aren't. (Well, also, in Ruby, symbols are immutable, but Python strings are, so that's not really relevant.) The other difference (which you may not think essential) is that strings have lots of methods that are *never* relevant to the uses one would put a symbol. That may or may not bother anyone, but it is a difference. Symbols wouldn't have methods like strings have. And there is a *bit* more work in using strings bound to variables (or using instances of classes, like yr SymGen example in the other thread) as opposed to using symbols with a literal syntax. I think in many cases symbols win on points for convenience and also express intent more clearly (similarly to the way tuples express some intent vis-a-vis using lists instead). I agree, however, that none of these alone are decisive differences or advantages. I think the sum of the differences warrants having symbols in Python, but YMMV. Though I don't see why Py3k couldn't have symbols and enumerations. > The only real disadvantage to > using strings as symbols is that you don't get an > immediate NameError if you misspell one. Well, as I said above, there is a difference in scope between a symbol (as I would like to see it added to Py) and a string. And I think there's more work to using strings as symbols. > A symbol type > wouldn't solve that problem; enumerations would. Nope, :foo and :Foo are different symbols, no doubt about that. Cheers, Kendall From kendall at monkeyfist.com Tue Apr 11 07:12:06 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 11 Apr 2006 01:12:06 -0400 Subject: [Python-3000] Did I miss anything? In-Reply-To: References: Message-ID: <6E31BB9C-A58D-48BE-849D-35BFDCF4AC0B@monkeyfist.com> On Apr 10, 2006, at 6:19 PM, Guido van Rossum wrote: > Are there other proto-PEPs being worked on? I would appreciate if the > authors would send me a note (or reply here) with the URL and the > status. I told Barry I'd work on a proto-PEP for adding symbols (global, immutable things (or, if you prefer, "names" or "dataless labels") a la Ruby's :symbol) to Py3k, but I don't yet have anything worth sharing. Symbols would be a pretty picayune addition to the language, but the Ruby folks seem to get v. good mileage out of them for metaprogramming, in projects like Rails, and in using Ruby as a host for DSLs, an area where it'd be a shame for Python to fall behind. Though if you have a strong feeling one way or the other, that would either save me some time working on something that hasn't a chance or motivate me to finish something that might. :> Cheers, Kendall Clark From nnorwitz at gmail.com Tue Apr 11 08:51:33 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 10 Apr 2006 23:51:33 -0700 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443A4204.5020509@gmail.com> References: <44373866.9070305@alumni.rice.edu> <44379062.1070004@gmail.com> <4437C398.9040600@alumni.rice.edu> <4437F697.2010606@gmail.com> <44386C42.90803@canterbury.ac.nz> <4439A6A3.8030409@canterbury.ac.nz> <443A3CB5.7060809@canterbury.ac.nz> <443A4204.5020509@gmail.com> Message-ID: On 4/10/06, Chaz. wrote: > > I have thought the approach would work to make a plug-in for Firefox, > though lacking the time and the knowledge has stopped me from experimenting. Knowledge should not be an impediment, many of us would be interested in helping you learn what you need to learn. OTOH, we can't help you get more time. :-) n From phd at mail2.phd.pp.ru Tue Apr 11 08:47:08 2006 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Tue, 11 Apr 2006 10:47:08 +0400 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF57C.1020806@canterbury.ac.nz> Message-ID: <20060411064707.GB1291@phd.pp.ru> On Tue, Apr 11, 2006 at 12:58:47AM -0400, Kendall Clark wrote: > Hmm. Tastes vary, obviously. I don't consider using ? to signal > boolean returning functions or methods to be random. They look ugly in expressions: if python.has_punctuation?() and someone.uses_it_often?: print "Please don't do it!" PS. .uses_it_often? is an attribute, not a function here... Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From greg.ewing at canterbury.ac.nz Tue Apr 11 09:55:45 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 19:55:45 +1200 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: <443B1BF0.80301@colorstudy.com> References: <44373866.9070305@alumni.rice.edu> <021401c65c36$b3de2f70$26452597@bagio> <1144694558.3162.37.camel@localhost.localdomain> <443AFD4B.1060201@canterbury.ac.nz> <443B1BF0.80301@colorstudy.com> Message-ID: <443B6101.1010903@canterbury.ac.nz> Ian Bicking wrote: > So I hear there's long been the ability to make multiple interpreters at > the C level -- but this has never been exposed at the Python > level. I'm not sure it would help in this case, unless there was no sharing of objects between the interpreters, in which case you might as well be running it in a separate process. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 11 10:04:38 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 20:04:38 +1200 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF57C.1020806@canterbury.ac.nz> Message-ID: <443B6316.3040701@canterbury.ac.nz> Kendall Clark wrote: > I don't consider using ? to signal boolean > returning functions or methods to be random. Picking those two characters out of all the punctuation symbols to be legal in identifiers is a fairly arbitrary choice, given that most natural languages only use letters in their words. > And "-" has always seemed > a better word separator (or at least as good) as "_". I wouldn't object to "-" if it weren't already taken for subtraction, since hyphenation of words is a well-established practice. And "_" is the next best thing to "-". -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 11 10:18:19 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 20:18:19 +1200 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF434.7000503@canterbury.ac.nz> Message-ID: <443B664B.4060501@canterbury.ac.nz> Kendall Clark wrote: > Well, some differences. In Ruby, for example, symbols are global in > scope and strings aren't. In Python, interning is global in scope, so intern('foo') done in one module gives the same object as intern('foo') done in another module. And since CPython automatically interns any string literal that looks like an identifier, using the literal 'foo' gives you exactly the same benefits as a symbol would, as far as I can see. The only difference is a syntactic one -- whether you write it as 'foo' or something else like :foo. > The other > difference (which you may not think essential) is that strings have > lots of methods that are *never* relevant to the uses one would put a > symbol. I don't see this as a problem. Strings have a lot of methods that I never use on the vast majority of strings in my programs, and I don't lose any sleep over it. >> A symbol type >> wouldn't solve that problem; enumerations would. > > Nope, :foo and :Foo are different symbols, no doubt about that. You miss the point. If you type :Foo when you meant :foo, no error occurs at that point. But if you are using a value bound to the name foo, referring to it as Foo gives an immediate NameError. That's the benefit that enumerations have which symbols don't -- because you have to explicitly define them, you get an extra degree of error checking. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 11 10:21:09 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 11 Apr 2006 20:21:09 +1200 Subject: [Python-3000] Did I miss anything? In-Reply-To: <6E31BB9C-A58D-48BE-849D-35BFDCF4AC0B@monkeyfist.com> References: <6E31BB9C-A58D-48BE-849D-35BFDCF4AC0B@monkeyfist.com> Message-ID: <443B66F5.2040809@canterbury.ac.nz> Kendall Clark wrote: > Symbols would be a pretty picayune addition to the language, but the > Ruby folks seem to get v. good mileage out of them for > metaprogramming, Because Ruby strings are mutable, you can't use them as symbols the way you can in Python. The languages are different enough in this area that you can't transfer experience directly from one to the other. -- Greg From ncoghlan at gmail.com Tue Apr 11 13:42:41 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 11 Apr 2006 21:42:41 +1000 Subject: [Python-3000] The 'make' statement PEP (was Re: Did I miss anything?) In-Reply-To: References: Message-ID: <443B9631.5090807@gmail.com> Steven Bethard wrote: > This is potentially a Python 2.6 PEP, but it has some optional > extensions for Python 3000 and may be relevant to the > adaptation/overloading/interfaces discussion. It proposes a make > statement such that: > make : > > would be translated into the assignment: > = ("", , ) > much in the same way that the class statement works. I've posted it > to comp.lang.python and had generally positive feedback. I've > submitted it for a PEP number, but I haven't heard back yet: > > http://ucsu.colorado.edu/~bethard/py/pep_make_statement.txt > http://ucsu.colorado.edu/~bethard/py/pep_make_statement.html Pretty nice, but I think it would be even better if instead of focusing on the rare use case of a callable that needs to know the name it is being bound to, we looked at the more general idea of "use a trailing suite to define the keyword arguments to a function". And rather than a new keyword, I'd suggest permitting a "**" without a following expression to denote 'fill in these keyword arguments based on the following suite'. This would be permitted in simple statements only (so there's no other suite to get in the way), with all occurrences being passed the namespace of the following suite as the keyword dictionary. Cases that wanted the callable to know the name would still need to pass it in explicitly as a string, but the upside of the tradeoff is that all existing callables that take keyword arguments would *just work* with the new syntax. (Also see the ElementTree example at the end for the case where not being able to provide the name explicitly would be a big problem). # Populate a switch dictionary options = dict(**): def option1(*args, **kwds): # Do option 1 def option2(*args, **kwds): # Do option 2 # etc. . . # Define a property class C(object): foo = property(**): doc = "The foo property" def fget(self): # Get foo def fset(self): # Set foo def fdel(self): # Delete foo # Sort an iterable by an arbitrary key results = sorted(result_gen, **): def key(item): return (item.kind, item.result[0]) I'd be perfectly happy to trade the need to occasionally repeat myself in the rare cases where I'd want the callable to know the name for the benefit of a syntax that let me push the keywords of an arbitrary callable into a subordinate suite. And here's a trick if you really wanted to be able to define an 'anonymous' function: def anon(**kwds): return kwds['anon'] result = widget.callback(anon(**)): def anon(*args, **kwds): # Define the callback for the widget And to finish, an element tree example (based on the one on the front page of Effbot's overview, rather than the abbreviated one in the PEP). I think this example shows that requiring that an element's tag be the same as its name would in fact be a *bad* idea (how would you build the two paragraphs in the body?) # The supporting act import xml.etree.ElementTree as ET def _make_element(tag, text, tail, namespace): """Build an XML element from a namespace""" element = ET.Element(tag) element.text = text element.tail = tail for name, value in namespace.iteritems(): if isinstance(value, ET.Element): if value.tag is None: value.tag = name element.append(value) else: element.set(name, value) def element(tag=None, text=None, tail=None, **kwds): """Create a new element""" return _make_element(tag, text, tail, kwds) def make_tree(tag, text=None, tail=None, **kwds): root = _make_element(tag, text, tail, kwds) return ET.ElementTree(root) # The performance tree = make_tree("html", **): head = element(**): title = element(**): text = "Page Title" body = element(**): bgcolor = "#ffffff" para1 = element("p", **): text = "Hello, World!" para2 = element("p", **): text = "And hello, again!" # wrap it in an ElementTree instance, and save as XML tree.write("page.xhtml") Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From barry at python.org Tue Apr 11 14:56:33 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 11 Apr 2006 08:56:33 -0400 Subject: [Python-3000] symbols? In-Reply-To: <20060411064707.GB1291@phd.pp.ru> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF57C.1020806@canterbury.ac.nz> <20060411064707.GB1291@phd.pp.ru> Message-ID: <1144760193.26621.5.camel@geddy.wooz.org> On Tue, 2006-04-11 at 10:47 +0400, Oleg Broytmann wrote: > On Tue, Apr 11, 2006 at 12:58:47AM -0400, Kendall Clark wrote: > > Hmm. Tastes vary, obviously. I don't consider using ? to signal > > boolean returning functions or methods to be random. > > They look ugly in expressions: > > if python.has_punctuation?() and someone.uses_it_often?: > print "Please don't do it!" > > PS. .uses_it_often? is an attribute, not a function here... That's what _p suffix is for. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060411/239c060d/attachment.pgp From phd at mail2.phd.pp.ru Tue Apr 11 15:00:00 2006 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Tue, 11 Apr 2006 17:00:00 +0400 Subject: [Python-3000] symbols? In-Reply-To: <1144760193.26621.5.camel@geddy.wooz.org> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF57C.1020806@canterbury.ac.nz> <20060411064707.GB1291@phd.pp.ru> <1144760193.26621.5.camel@geddy.wooz.org> Message-ID: <20060411130000.GA18403@phd.pp.ru> On Tue, Apr 11, 2006 at 08:56:33AM -0400, Barry Warsaw wrote: > > if python.has_punctuation?() and someone.uses_it_often?: > > print "Please don't do it!" > > > That's what _p suffix is for. :) T (: Hello, Lispers! ;) Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From kendall at monkeyfist.com Tue Apr 11 16:06:02 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 11 Apr 2006 10:06:02 -0400 Subject: [Python-3000] symbols? In-Reply-To: <443B664B.4060501@canterbury.ac.nz> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF434.7000503@canterbury.ac.nz> <443B664B.4060501@canterbury.ac.nz> Message-ID: <575B0300-213E-4836-97F8-9C950DF5873D@monkeyfist.com> On Apr 11, 2006, at 4:18 AM, Greg Ewing wrote: > The only difference is a syntactic one -- whether you > write it as 'foo' or something else like :foo. Yeah, my point about signaling intent. (I forgot about automatic interning, so thanks for that reminder.) >> The other >> difference (which you may not think essential) is that strings have >> lots of methods that are *never* relevant to the uses one would >> put a >> symbol. > > I don't see this as a problem. Strings have a lot of > methods that I never use on the vast majority of strings > in my programs, and I don't lose any sleep over it. Well, there's a diff between majority I mostly don't use and *all that are never used*, but YMMV. I don't lose *much* sleep over this. :> >> Nope, :foo and :Foo are different symbols, no doubt about that. > > You miss the point. If you type :Foo when you meant :foo, > no error occurs at that point. Right, nor should they since those are different symbols. > But if you are using a value > bound to the name foo, referring to it as Foo gives an > immediate NameError. Ah, of course. I don't really see that as a benefit here, since the joy of symbols is how lightweight they are. Just write one in place and go. Enumerations and using variables bound to strings as symbols both lose some of that joy since you have to do more work, which equates to me to having less fun. But, obviously, this is one of those things that strikes different people differently. Cheers, Kendall From ianb at colorstudy.com Tue Apr 11 17:52:16 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 11 Apr 2006 10:52:16 -0500 Subject: [Python-3000] The 'make' statement PEP (was Re: Did I miss anything?) In-Reply-To: <443B9631.5090807@gmail.com> References: <443B9631.5090807@gmail.com> Message-ID: <443BD0B0.5060508@colorstudy.com> Nick Coghlan wrote: > Steven Bethard wrote: > >>This is potentially a Python 2.6 PEP, but it has some optional >>extensions for Python 3000 and may be relevant to the >>adaptation/overloading/interfaces discussion. It proposes a make >>statement such that: >> make : >> >>would be translated into the assignment: >> = ("", , ) >>much in the same way that the class statement works. I've posted it >>to comp.lang.python and had generally positive feedback. I've >>submitted it for a PEP number, but I haven't heard back yet: >> >>http://ucsu.colorado.edu/~bethard/py/pep_make_statement.txt >>http://ucsu.colorado.edu/~bethard/py/pep_make_statement.html > > > Pretty nice, but I think it would be even better if instead of focusing on the > rare use case of a callable that needs to know the name it is being bound to, > we looked at the more general idea of "use a trailing suite to define the > keyword arguments to a function". I don't think an object wanting to know its name is a particularly rare use case. Functions know their name, and classes know their name. Certainly the ElementTree-building example is better in Steven's version. When building descriptors, they often need to know their name -- if, for instance, the descriptor represents any state, that state needs to be attached to the object, and attaching the state to a fixed location means that the descriptor can only be used once on that class. Or you choose an arbitrary (and probably unstable) name, and make the object unpickleable. property() manages to avoid this by not actually storing anything and just being a rather simple container for user-provided functions; but other more interesting descriptors can't do that. I think this syntax has particular potential for descriptors, a currently underused feature in Python. Certainly the decorator pattern saw a strong upsurge after the decorator syntax was introduced, so syntax can have a substantial impact. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Tue Apr 11 18:07:38 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 11 Apr 2006 11:07:38 -0500 Subject: [Python-3000] symbols? In-Reply-To: <443AF2BC.4050702@canterbury.ac.nz> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF2BC.4050702@canterbury.ac.nz> Message-ID: <443BD44A.7080704@colorstudy.com> Greg Ewing wrote: > Kendall Clark wrote: >>One thing I'd really like to see in Python 3000 is support for first- >>class symbols, >> >> def web_dispatch("/bank-account", :GET, retrieve_account): pass > > > class SymGen: > > def __getattr__(self, name): > return intern(name) > > sym = SymGen() > > web_dispatch("/bank-account", sym.GET, retrieve_account) The advantage of an enumerator would be more if it was namespace. E.g.: class Sym: def __init__(self, parent, name): self.parent = parent self.name = name def __repr__(self): return '' % (self.parent, self.name) class SymGen: def __init__(self, name=None): if name is None: name = self.__class__.__name__ self.name = name def __getattr__(self, attr): sym = Sym(self.name, attr) setattr(self, attr, sym) return sym Considering the namespace, the difference between enumerations and symbols becomes a little more clear (though certainly symbols are used for enumerations in languages that have symbols). You can't use an enumeration from a namespace for a method name, for example. Steven's proposal in another thread for a 'make' statement (http://ucsu.colorado.edu/~bethard/py/pep_make_statement.html) would alleviate some of the motivation for symbols. E.g., currently: class MyTable(MyORM): mycol = Column('mycol', type=Int) That is, barring metaclass tricks that require cooperation with Column, you have to pass the column's name into it. So 'mycol' is really a Python identifier without a namespace, represented as a string. This is what they seem to do a lot of in Ruby. For instance (assuming class decorators were adopted) this is how they might set up a property: class Foo: @class attr('bar') Which would add a 'bar' attribute (which doesn't really make sense in Python since we have public instance variables, and Ruby only has private). But the make syntax offers an alternative: class MyTable(MyORM): make Column mycol: type = Int Now there's no strings, the identifier isn't repeated, and its clear that "mycol" is an identifier, not a string or a mere variable binding. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From steven.bethard at gmail.com Tue Apr 11 18:16:46 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 11 Apr 2006 10:16:46 -0600 Subject: [Python-3000] The 'make' statement PEP (was Re: Did I miss anything?) In-Reply-To: <443B9631.5090807@gmail.com> References: <443B9631.5090807@gmail.com> Message-ID: On 4/11/06, Nick Coghlan wrote: > Steven Bethard wrote: > > This is potentially a Python 2.6 PEP, but it has some optional > > extensions for Python 3000 and may be relevant to the > > adaptation/overloading/interfaces discussion. It proposes a make > > statement such that: > > make : > > > > would be translated into the assignment: > > = ("", , ) > > much in the same way that the class statement works. I've posted it > > to comp.lang.python and had generally positive feedback. I've > > submitted it for a PEP number, but I haven't heard back yet: > > > > http://ucsu.colorado.edu/~bethard/py/pep_make_statement.txt > > http://ucsu.colorado.edu/~bethard/py/pep_make_statement.html > > Pretty nice, but I think it would be even better if instead of focusing on the > rare use case of a callable that needs to know the name it is being bound to, > we looked at the more general idea of "use a trailing suite to define the > keyword arguments to a function". I started with something like that, and at the suggestion of folks from the python-list tried to add some emphasis to the DRY part of it too. I think they're both important, and they're both a reason to declare a class instead of calling type (and thus a reason to use the make statement instead of calling the callable). > And rather than a new keyword, I'd suggest permitting a "**" without a > following expression to denote 'fill in these keyword arguments based on the > following suite'. I think this is a reasonable proposal too, but it's different enough that I think it should have its own PEP. Note that unless you really change how metaclasses are called, this strongly breaks the parallel between the new statement and class statements. On the python-list at least, people really didn't want to break that parallel. > And to finish, an element tree example (based on the one on the front page of > Effbot's overview, rather than the abbreviated one in the PEP). I think this > example shows that requiring that an element's tag be the same as its name > would in fact be a *bad* idea (how would you build the two paragraphs in the > body?) Yeah, this was brought up on the python-list: http://mail.python.org/pipermail/python-list/2006-April/335829.html I've concluded that the best thing is to just pull the example (and have done so). I don't think you really want to use the make statement for creating XML. STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From martin at v.loewis.de Tue Apr 11 18:45:39 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 18:45:39 +0200 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: <443B1BF0.80301@colorstudy.com> References: <44373866.9070305@alumni.rice.edu> <021401c65c36$b3de2f70$26452597@bagio> <1144694558.3162.37.camel@localhost.localdomain> <443AFD4B.1060201@canterbury.ac.nz> <443B1BF0.80301@colorstudy.com> Message-ID: <443BDD33.4060109@v.loewis.de> Ian Bicking wrote: > So I hear there's long been the ability to make multiple interpreters at > the C level -- used by mod_python and presumably used in some other > embedding situations -- but this has never been exposed at the Python > level. I'm curious why that never happened? Merely inertia, or > something more significant? Part of the problem is that it doesn't really work. Some objects *are* shared across interpreters, such as global objects in extension modules (extension modules are initialized only once). I believe that the GIL management code (for acquiring the GIL out of nowhere) breaks if there are multiple interpreters. Regards, Martin From martin at v.loewis.de Tue Apr 11 18:49:59 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 18:49:59 +0200 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: <443B6101.1010903@canterbury.ac.nz> References: <44373866.9070305@alumni.rice.edu> <021401c65c36$b3de2f70$26452597@bagio> <1144694558.3162.37.camel@localhost.localdomain> <443AFD4B.1060201@canterbury.ac.nz> <443B1BF0.80301@colorstudy.com> <443B6101.1010903@canterbury.ac.nz> Message-ID: <443BDE37.6070207@v.loewis.de> Greg Ewing wrote: > Ian Bicking wrote: > >> So I hear there's long been the ability to make multiple interpreters at >> the C level -- but this has never been exposed at the Python >> level. > > I'm not sure it would help in this case, unless > there was no sharing of objects between the > interpreters, in which case you might as well > be running it in a separate process. Well, if you were to create a hierarchy of proxy objects for "remoting" objects across different interpreters, you would get much better performance than if you used IPC. This would be similar to .NET AppDomains or Java Isolates. Regards, Martin From shponglespore at gmail.com Tue Apr 11 18:45:58 2006 From: shponglespore at gmail.com (John Williams) Date: Tue, 11 Apr 2006 11:45:58 -0500 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/10/06, Guido van Rossum wrote: > A bit more than a year ago I blogged extensively about this. The only > syntax that is acceptable to me is slightly different; the above would > look like > > def fn(a: Seq, b: Index, c: Text = "default"): ... > > where Seq, Index and Text can be expressions (the main problem with > the syntax you propose is that the type can't be much more than an > identifier before it gets ambiguous or unreadable). I appologize if this has been suggested before, but what about this instead? @mydecorator def fn(a @Seq, b @Index, c @Text = default): ... My rationale is that since type annotations are primarily useful with decorators, they should have a syntax that is reminiscent of decorators. The downside is that type annotations don't have anything like the semantics of decorators, but I think the risk of confusion is very small if decorators and type annotations are documented together (as IMHO they should be in any case). The "@" stands out far too much in a variable-width font, so I've attached an HTML file to show how both syntaxes look with syntax highlighting and a fixed-width font. -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060411/9923b921/attachment.html From guido at python.org Tue Apr 11 20:01:57 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Apr 2006 11:01:57 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/11/06, John Williams wrote: > I appologize if this has been suggested before, but what about this instead? No. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Tue Apr 11 20:05:59 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 20:05:59 +0200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization Message-ID: <443BF007.9080401@v.loewis.de> Abstract: Module initialization currently has a few deficiencies. There is no cleanup for modules, the entry point name might give naming conflicts, the entry functions don't follow the usual calling convention, and multiple interpreters are not supported well. This PEP addresses these issues. Module Finalization ----------------- Currently, C modules are initialized usually once and then "live" forever. The only exception is when Py_Finalize() is called: then the initialization routine is invoked a second time. This is bad from a resource management point of view: memory and other resources might get allocated each time initialization is called, but there is no way to reclaim them. As a result, there is currently no way to completely release all resources Python has allocated. Entry point name conflicts -------------------------- The entry point is currently called init. This might conflict with other symbols also called init. In particular, initsocket is known to have conflicted in the past (this specific problem got resolved as a side effect of renaming the module to _socket). Entry point signature --------------------- The entry point is currently a procedure (returning void). This deviates from the usual calling conventions; callers can find out whether there was an error during initialization only by checking PyErr_Occurred. The entry point should return a PyObject*, which will be the module created, or NULL in case of an exception. Multiple Interpreters --------------------- Currently, extension modules share their state across all interpreters. This allows for undesirable information leakage across interpreters: one script could permanently corrupt objects in an extension module, possibly breaking all scripts in other interpreters. Specification ------------- The module initialization routines change their signature to PyObject *PyInit_(PyInterpreterState*) The initialization routine will be invoked once per interpreter, when the module is imported. It should return a new module object each time. In addition, the module MAY implement a finalizer PyObject *PyFinalize_(PyInterpreterState*) which returns None on success. In order to store per-module state in C variables, the following API is introduced: struct PyModule_Slot; void PyInterpreter_AllocateSlot(PyInterpreterState*, PyModule_Slot*, size_t); void* PyInterpreter_AccessSlot(PyInterpreterState*, PyModule_Slot*); Each module should declare a single global variable of struct PyModule_Slot. This will get initialized to some unique value on the first call of PyInterpreter_AllocateSlot; this and each subsequent call also allocate and zero-initialize a block of memory (per interpreter and module). To simplify access, the module code can put the lines PyModule_Slot module_slot; struct state{ /* members, e.g. PyObject *member; */ }; #define STATE PyModule_STATE(module_slot, struct state) after including Python.h, and then access the module's state simply with STATE->member. This macro expands to ((struct state*)PyInterpreter_AccessSlot( PyInterpreter_Current(), &module_slot)) Discussion ---------- It would be possible to support the existing init functions if that is desirable; in that case, nothing would change. From jimjjewett at gmail.com Tue Apr 11 20:06:44 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 11 Apr 2006 14:06:44 -0400 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/10/06, Guido van Rossum wrote: > Oops. I guess you were arguing *against* the former; > I (and perhaps others) misread your mail as arguing *for* > the "a:0 proposed that before (unless I missed it). > ... My confusion is why you would bring this up only to > shoot it down. Consider it rejected, as I had misunderstood you. Reason for my initial confusion: You had said there was some question of what kind of objects the type markers should be. I couldn't (and still can't) see any good reason for them to be anything but a way to ensure [to some level] that the argument is [in some way] appropriate. Since you suggested there was still a question, I wanted to know what you saw as the alternative. The only alternative I had seen was a more general pre-condition from DBC decorators. I now see that your actual alternative was to have them be a sort of annotation rather than a checker, and to leave the actual checking up to a decorator. -jJ From martin at v.loewis.de Tue Apr 11 20:18:30 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 20:18:30 +0200 Subject: [Python-3000] symbols? In-Reply-To: <1144686244.15801.28.camel@resist.wooz.org> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <1144686244.15801.28.camel@resist.wooz.org> Message-ID: <443BF2F6.6060605@v.loewis.de> Barry Warsaw wrote: > To be fair, I think Ruby stole the idea and syntax from Lisp. It's an > interesting idea, but I'd like to understand exactly what you mean by "a > Python symbol". Can you give more detail about your idea, perhaps as a > pseudo-PEP? I think it's as simple as the the LISP symbols, except when it comes to name binding. What the OP apparently wants is this: class Symbol(object): def __init__(self, str): self.name = name symbols = {} def make_symbol(str): try: return symbols[str] except KeyError: symbols[str] = result = Symbol(str) In addition, he wants literals for the symbol type, namely :[A-Za-z_][A-Za-z0-9_] As you can put them into source code, you also need support for marshalling them. That's about all that he has specified so far. Open issues then are: - does .name include the colon or not? - can you pass them to getattr, instead of strings? Everything else follows from this spec (I hope). Symbols compare for identity, are hashable, meant to be immutable (Bug: the specification allows for mutation; the implementation shouldn't). Regards, Martin From guido at python.org Tue Apr 11 20:21:41 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Apr 2006 11:21:41 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/11/06, Jim Jewett wrote: > You had said there was some question of what kind of objects the type > markers should be. I couldn't (and still can't) see any good reason > for them to be anything but a way to ensure [to some level] that the > argument is [in some way] appropriate. You just can't avoid using a double negative can't you? :-) I'd like to stretch the definition further, so that *any* argument metadata can be placed in that syntactic position, as long as a decorator is used that can interpret the metadata, and as long as the metadata can be expressed as a single expression. (If the expression gets too long we can always lift it out and move it to a preceding position in the file, as the expression is evaluated at the same time the function definition and the default values are -- I probably forgot to make that explicit.) Just like decorators have no defined semantics (beyond "the decorator gets called this way and what it returns gets assigned to that variable") but have found a few well-defined conventions for their most common usage (call wrappers and registration), I expect that we'll develop a convention for how to express various useful metadata soon enough -- but I don't want the language to enforce that. The syntactic position is much too precious to assign it a single specific meaning. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 11 20:29:37 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Apr 2006 11:29:37 -0700 Subject: [Python-3000] Did I miss anything? In-Reply-To: References: Message-ID: On 4/10/06, Steven Bethard wrote: > On 4/10/06, Guido van Rossum wrote: > > Are there other proto-PEPs being worked on? I would appreciate if the > > authors would send me a note (or reply here) with the URL and the > > status. > > This is the Backwards Incompatibility PEP discussed earlier. I've > submitted it for a PEP number, but haven't heard back yet: > > http://ucsu.colorado.edu/~bethard/py/pep_backwards_incompatible.txt > http://ucsu.colorado.edu/~bethard/py/pep_backwards_incompatible.html I like this! I hope it can be checked in soon. > This is potentially a Python 2.6 PEP, but it has some optional > extensions for Python 3000 and may be relevant to the > adaptation/overloading/interfaces discussion. It proposes a make > statement such that: > make : > > would be translated into the assignment: > = ("", , ) > much in the same way that the class statement works. I've posted it > to comp.lang.python and had generally positive feedback. I've > submitted it for a PEP number, but I haven't heard back yet: > > http://ucsu.colorado.edu/~bethard/py/pep_make_statement.txt > http://ucsu.colorado.edu/~bethard/py/pep_make_statement.html I don't like this. It's been proposed many times before and it always ends up being stretched until it breaks. Also, I don't like the property declaration use case; IMO defining explicit access method and explicitly defining a property makes more sense. In particular it bugs me that the proposed syntax indents the access methods and places them in their own scope, while in fact they become (unnamed) methods. Also, I expect that the requirement that the accessor methods have fixed names will make debugging harder, since now the function name in the traceback doesn't tell you which property was being accessed. I expect that the PEP will go forward despite my passive aggressive negativism; there are possible rebuttals for all of my objections. But I don't have to like it. I wish the community efforts for Python 3000 were focused more on practical things like the effects of making all strings unicode, designing a bytes datatype, a new I/O stack, and the view objects to be returned by keys() etc. These things need thorough design as well as serious prototyping efforts in the next half year. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Tue Apr 11 20:55:14 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 11 Apr 2006 14:55:14 -0400 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/10/06, Guido van Rossum wrote: > [Guido] > > > It should mean neither. As I tried to say, the *only* > > > semantics is that the value of the expression is > > > accessible through the __signature__ attribute of > > > the function. It is then up to the decorator to decide > > > what it means. Rephrasing for verification: In and of itself, the type specifier does nothing. It does add a (virtual) annotation to the argument, so that anything which *does* act (such as a decorator) can be written in a more generic manner. > [Jim Jewett] > > So type annotation is only for use with decorators, > > not for general documentation or compiler speedups? > Those are two entirely different purposes. It's fine for > documentation, for use by IDEs, and for use by e.g. > pychecker. So (besides decorators), it is for (exactly) introspective uses? > However, using it for compiler speedups is unlikely > given the run-time semantics and Python's extremely > dynamic nature. There are several PEPs (and some PyPy progress) towards more stable namespaces. As PEP 266 points out, these namespaces typically never change, and it might be reasonable to make these changes more expensive if they sped up the common "from now on, it is read-only" case. I can easily imagine a slightly modified __import__ that sealed the module after creation. Even for a seal that *can* be broken, it might be reasonable to say "but if you do, these precompiled objects that relied on the seal will take the slow path until they're recompiled." -jJ From jimjjewett at gmail.com Tue Apr 11 20:58:08 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 11 Apr 2006 14:58:08 -0400 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/10/06, Guido van Rossum wrote: > I don't want to assign *any* semantics to the type markers; > ... I'd like to be able to have a decorator that > requires the convention of using concrete types > (e.g. list, int, file) as type markers; calling these > would be a mistake, they should be used > in isinstance() calls for example. list is indeed a corner case, because of the copying. To me, the answer is to instead create a similar List that doesn't copy unless it has to. def List(val): return val if isinstance(val, list) else list(val) It would be nicer to use list directly, but I'm not sure it is worth having to also add a decorator every time I use a type marker. That starts to look too much like boilerplate. Alternatively, if you're assuming a decorator anyhow, then a decorator for only-these-odd-cases could wrap the function with a different signature and its own isinstance check. > You're thinking of the type markers as adapters > exclusively. No, but I am saying that they should be constrained to be callables which can be used as (possibly faulty) adapters. > Several people (in a previous round of discussion) > have agreed that they'd be totally happy if > [the type annotations were ignored], so ... >>> def f(x: int): ... print x*2 >>> f("abc") 'abcabc' I think "satisfied" is more accurate that "totally happy". That call is arguably an error, and there are certainly execution contexts where it isn't worthwhile checking for errors. I think the same people would be just as happy if the call raised a TypeError (particularly in debug mode). In optimized mode, they might even be willing to live with undefined results, to avoid the try-except overhead. What they don't want is for (a) Every type marker to be an annoyance to create, particularly if (b) Every parameter has to be marked up (in a likely useless fashion), because of style expectations imported from other languages. So the fact that users might want any of: def MyType(val): return val # no run-time overhead class MyType(RealType): # copyless adapter def __new__(self, other): if isinstance(other, MyType): return other return RealType(other) def MyType(val): # checker if not isinstance(val, _MySecretType): raise TypeError("%s is not a MyType" % (val,)) return val shouldn't mean that type providers have to write all of the above. Having to provide "a callable" isn't nearly so restrictive. Even meeting a full "type annotation API" isn't so bad, if inheritance lets them mostly not bother. -jJ From guido at python.org Tue Apr 11 21:01:19 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Apr 2006 12:01:19 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/11/06, Jim Jewett wrote: > On 4/10/06, Guido van Rossum wrote: > > [Guido] > > > > It should mean neither. As I tried to say, the *only* > > > > semantics is that the value of the expression is > > > > accessible through the __signature__ attribute of > > > > the function. It is then up to the decorator to decide > > > > what it means. > > Rephrasing for verification: > > In and of itself, the type specifier does nothing. It does add a > (virtual) annotation to the argument, so that anything which *does* > act (such as a decorator) can be written in a more generic manner. Right. > > [Jim Jewett] > > > So type annotation is only for use with decorators, > > > not for general documentation or compiler speedups? > > > Those are two entirely different purposes. It's fine for > > documentation, for use by IDEs, and for use by e.g. > > pychecker. > > So (besides decorators), it is for (exactly) introspective uses? It is for no specific purpose. :-) > > However, using it for compiler speedups is unlikely > > given the run-time semantics and Python's extremely > > dynamic nature. > > There are several PEPs (and some PyPy progress) towards more stable > namespaces. As PEP 266 points out, these namespaces typically never > change, and it might be reasonable to make these changes more > expensive if they sped up the common "from now on, it is read-only" > case. > > I can easily imagine a slightly modified __import__ that sealed the > module after creation. Even for a seal that *can* be broken, it might > be reasonable to say "but if you do, these precompiled objects that > relied on the seal will take the slow path until they're recompiled." That is all pretty much uncharted territory (for Python). If and when it comes to fruition I hope it will come with a set of recommended best practices for using argument types and decorators to tell the compiler more about a function. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From steven.bethard at gmail.com Tue Apr 11 21:02:27 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 11 Apr 2006 13:02:27 -0600 Subject: [Python-3000] Did I miss anything? In-Reply-To: References: Message-ID: On 4/11/06, Guido van Rossum wrote: > I wish the community efforts for Python 3000 were focused more on > practical things like the effects of making all strings unicode, > designing a bytes datatype, a new I/O stack, and the view objects to > be returned by keys() etc. These things need thorough design as well > as serious prototyping efforts in the next half year. Sorry for the noise then. I only meant to point to it since it potentially addresses the issue of (at least the syntax of) creating interface objects. I'm going to avoid responding to any more comments on it on the Python-3000 list to keep the noise level down, and redirect the discussion to the python-dev list once I get a PEP number. Steve -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From mcherm at mcherm.com Tue Apr 11 21:02:35 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Tue, 11 Apr 2006 12:02:35 -0700 Subject: [Python-3000] Separating out CPython and core Python tests Message-ID: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com> Neal Norwitz writes: > Py3k is a great opportunity to upgrade the existing tests. We know > they are incomplete and could do a better job testing. In some cases > like this, where we are looking to revamp a module. It would be great > to also beef up the tests. Hopefully a lot of that work, will also > benefit the 2.x series. There's a related change that I would love to see. Currently, we have a large number of unit tests, but we have made no attempt to distinguish between python language tests, and CPython implementation tests. If we made such a distinction, then PyPy, Jython, Iron Python, and perhaps others would benefit. Python 3000 might be a good time to introduce the distinction. In fact, I don't think "we" even need to do the hard work (where "we" is defined as the CPython developers). I am confident that if the CPython team offered to make such a split, then either the PyPy or Jython developers (maybe both) would happily provide us with a list of which tests they thought were core language tests and which were CPython-specific. Guido could review and pronounce on any controversial cases (I'm guessing there would be remarkably few that weren't obvious). All that the CPython team would need to do would be to make the offer, use separate directories or a naming convention to mark which are which, and modify the top-level test driver to run both sets of tests. What'da'ya'think? -- Michael Chermside From guido at python.org Tue Apr 11 21:12:38 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Apr 2006 12:12:38 -0700 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/11/06, Jim Jewett wrote: > On 4/10/06, Guido van Rossum wrote: > > > I don't want to assign *any* semantics to the type markers; > > > ... I'd like to be able to have a decorator that > > requires the convention of using concrete types > > (e.g. list, int, file) as type markers; calling these > > would be a mistake, they should be used > > in isinstance() calls for example. > > list is indeed a corner case, because of the copying. > > To me, the answer is to instead create a similar List that doesn't > copy unless it has to. > > def List(val): return val if isinstance(val, list) else list(val) > > It would be nicer to use list directly, but I'm not sure it is worth > having to also add a decorator every time I use a type marker. That > starts to look too much like boilerplate. Beware that this (which has been discussed many times in the context of adaptation) is a very treacherous path. Saying it is a "List" doesn't say what you intend to do to it; modifying a copy and modifying the original have very different effects. > Alternatively, if you're assuming a decorator anyhow, then a decorator > for only-these-odd-cases could wrap the function with a different > signature and its own isinstance check. Every framework should probably decide for itself how it wants to handle these things. Implicit adaptation isn't exactly a feature which has been tried a lot before in Python. > > You're thinking of the type markers as adapters > > exclusively. > > No, but I am saying that they should be constrained to be callables > which can be used as (possibly faulty) adapters. And to the contrary I think they should be allowed to be whatever is acceptable to the decorator you use. Argument annotations (I guess we should drop the "type" part altogether :-) should be a matter of agreement between the annotation and the decorator -- if any. Or implied introspection, of course. > > Several people (in a previous round of discussion) > > have agreed that they'd be totally happy if > > [the type annotations were ignored], so ... > > >>> def f(x: int): > ... print x*2 > >>> f("abc") > 'abcabc' > > I think "satisfied" is more accurate that "totally happy". > > That call is arguably an error, and there are certainly execution > contexts where it isn't worthwhile checking for errors. I think the > same people would be just as happy if the call raised a TypeError > (particularly in debug mode). In optimized mode, they might even be > willing to live with undefined results, to avoid the try-except > overhead. > > What they don't want is for > > (a) Every type marker to be an annoyance to create, > > particularly if > > (b) Every parameter has to be marked up (in a likely useless > fashion), because of style expectations imported from other languages. I don't think anyone suggested that? > So the fact that users might want any of: > > def MyType(val): return val # no run-time overhead What do you mean by no run-time overhead? What about the call to MyType(val) itself? > class MyType(RealType): # copyless adapter > def __new__(self, other): > if isinstance(other, MyType): > return other > return RealType(other) > > def MyType(val): # checker > if not isinstance(val, _MySecretType): > raise TypeError("%s is not a MyType" % (val,)) > return val > > shouldn't mean that type providers have to write all of the above. Well what's your proposed alternative? We seem to be going around in circles. > Having to provide "a callable" isn't nearly so restrictive. Even > meeting a full "type annotation API" isn't so bad, if inheritance lets > them mostly not bother. I'm lost. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From msoulier at digitaltorque.ca Tue Apr 11 21:21:42 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Tue, 11 Apr 2006 15:21:42 -0400 Subject: [Python-3000] Separating out CPython and core Python tests In-Reply-To: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com> References: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com> Message-ID: <20060411192142.GO476@tigger.digitaltorque.ca> On 11/04/06 Michael Chermside said: > There's a related change that I would love to see. Currently, we > have a large number of unit tests, but we have made no attempt to > distinguish between python language tests, and CPython implementation > tests. If we made such a distinction, then PyPy, Jython, Iron Python, > and perhaps others would benefit. Python 3000 might be a good time > to introduce the distinction. I for one would love to see a simple test that can be run to verify compliance to the Python language standard, if such a thing exists, to help prevent Jython, IronPython and similar efforts from diverging and causing a Lisp/Scheme kind of schism in Python. Alarmist perhaps, but it would still be a nice test suite to have. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060411/9424621b/attachment.pgp From guido at python.org Tue Apr 11 21:28:26 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 11 Apr 2006 12:28:26 -0700 Subject: [Python-3000] Separating out CPython and core Python tests In-Reply-To: <20060411192142.GO476@tigger.digitaltorque.ca> References: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com> <20060411192142.GO476@tigger.digitaltorque.ca> Message-ID: On 4/11/06, Michael P. Soulier wrote: > On 11/04/06 Michael Chermside said: > > > There's a related change that I would love to see. Currently, we > > have a large number of unit tests, but we have made no attempt to > > distinguish between python language tests, and CPython implementation > > tests. If we made such a distinction, then PyPy, Jython, Iron Python, > > and perhaps others would benefit. Python 3000 might be a good time > > to introduce the distinction. I agree that such a test suite would be helpful. I don't think it should wait for Python 3000. > I for one would love to see a simple test that can be run to verify compliance > to the Python language standard, if such a thing exists, to help prevent > Jython, IronPython and similar efforts from diverging and causing a > Lisp/Scheme kind of schism in Python. I'm not worried about that. In fact, PyPy has contributed much to the existing test suite, and we could probably get Jim Hugunin to give us some hints on tests that are CPython specific. > Alarmist perhaps, but it would still be a nice test suite to have. Yes, nice to have. No, no reason for alarm. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kendall at monkeyfist.com Tue Apr 11 21:30:13 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 11 Apr 2006 15:30:13 -0400 Subject: [Python-3000] symbols? In-Reply-To: <443BF2F6.6060605@v.loewis.de> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <1144686244.15801.28.camel@resist.wooz.org> <443BF2F6.6060605@v.loewis.de> Message-ID: <53D14D58-1BE5-477E-B003-1CBB489DDE0C@monkeyfist.com> On Apr 11, 2006, at 2:18 PM, Martin v. L?wis wrote: > I think it's as simple as the the LISP symbols, except when it comes > to name binding. Common Lisp symbols are a lot more complex in that they are not globally unique, can be uninterned, can carry data, get automagically upcased, have property lists, etc. etc. > What the OP apparently wants is this: > > class Symbol(object): > def __init__(self, str): > self.name = name > > symbols = {} > def make_symbol(str): > try: > return symbols[str] > except KeyError: > symbols[str] = result = Symbol(str) > > In addition, he wants literals for the symbol type, namely > > :[A-Za-z_][A-Za-z0-9_] Yes, I wrote exactly this regex in the message I was working on. > As you can put them into source code, you also need support > for marshalling them. Yes. > That's about all that he has specified so far. Correct. > Open issues > then are: > > - does .name include the colon or not? No. (And its repr has no quote marks, either.) > - can you pass them to getattr, instead of strings? Haven't thought about that at all. > Everything else follows from this spec (I hope). Symbols > compare for identity, are hashable, meant to be immutable > (Bug: the specification allows for mutation; the implementation > shouldn't). Thanks, Martin. That is, in fact, pretty much what I'd like to see in Py3K, modulo the issue about getattr. Cheers, Kendall From ianb at colorstudy.com Tue Apr 11 21:40:01 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 11 Apr 2006 14:40:01 -0500 Subject: [Python-3000] Interfaces for views and extended collections Message-ID: <443C0611.1060108@colorstudy.com> So Guido asked for more concrete discussion of things like views. A richer set of collections also fits in here, as for instance dict.keys() would be a view with a collection interface not exactly like other collections. I wrote up some notes, which I'll pass on. So, I can imagine 3 attributes which seem orthogonal to me: a) Is it ordered? b) Can it contain multiple equal items? c) Is it associative? In addition there is mutability, though anything that can be mutable could also have an immutable form. In general the only difference I see is the removal of methods, and the collection becomes hashable. There's 8 combinations. Associative collections: dict: no multiple items, unordered (missing an immutable form) multidict: multiple items, unordered ordered multidict: multiple items, ordered ordered dict: no multiple items, ordered (useful?) Non-associative: Set/ImmutableSet: no multiple items, unordered bag: multiple items, unordered ordered set: no multiple items, ordered (useful? -- priority queue?) list/tuple: multiple items, ordered So, ordered sets and ordered dicts seem questionable. Also, it is unclear to me if there's a compelling reason for both ordered multidict and multidict, though the performance is likely to be different for the two (underneath multidict likely being a dictionary of sets, ordered multidict being a list of tuples). So, at least, we are missing bags, (ordered) multidicts. In terms of views, .keys() is a set, multidict.keys() is a bag, and orderedmultidict.keys() is a list (or tuple, or immutable list-like object). There's some other attributes. For instance, we have defaultdict. Another option that could be applied to almost any of these is a key function. E.g., KeyedDict(key=lambda x: x.lower()) would be a case-insensitive dictionary. Because views introduce an relationship between these collections, if we add a keyed dictionary we should also have a keyed set (returned by KeyedDict.keys()). If we build this directly into the collection, that's easy, if not then we double the number of collections to be implemented; which is perhaps fine. OK... so now we're this far; what will the interfaces be for these objects? What follows is a long list of, I think, all the major methods and operators collections define. This is mostly just source material, there's no conclusions or proposals. I hope this will be useful to consider what the interfaces might look like, or to compare proposed interfaces against. Methods that don't imply mutability =================================== Combine two collections: + Currently only applies to lists and tuples. Lists cannot be added to tuples. This only applies to objects with order, that can contain multiple items. Combine two collections: | For sets, produces the union. Applies to objects without order and cannot contain multiple items. Should dicts implement this? Which side would take precedence? Combine two collections: & For sets, produces the intersection. Should dicts implement this? Combine two collections: ^ For sets, produces the symmetric difference. Dicts could implement this; neither side would have to take precedence. Difference: - For sets, a-b produces a new set with items from b removed. Could apply to bags. Maybe also to dicts? Repeat collection: *int For lists and tuples, repeats the contents. Could apply to ordered multidicts. Maybe also to bags and multidicts? Get item at index: [index] Returns the item at that index. Doesn't apply to dicts, even ordered dicts, because it conflicts with another meaning of __getitem__. Doesn't apply to unordered collections Get item for association: [key] Returns the value for that key. Applies to all dictionaries. Do multidicts return all values or the first value? Or the last value? Dictionaries essentially return the *last* value (where all previous values were disposed of). An additional method is needed for the other option. If multiple values, in what kind of container are those values? Mutable? A view (yuck)? Get an item, with not-found handling: .get() Returns the value for that key. For multidicts, does it return None or an empty collection if nothing is found? For whatever method added in addition to __getitem__, is there an equivalent get*() form? Count items: .count(item) This works on lists. For some reason not on tuples. Should work on bags. Would this apply to multidicts? Is item a key or (key, value) in that case? Would there be a way to count values? Value counting can apply to all associations. Get index: .index(item) This works on lists. Also for ordered dicts? Would item be key, or (key, value)? Count items: len(collection) This works on everything. For multidicts, does it count values or keys? For bags does it count unique items or all items? Test if an item is in collection: item in collection Works for keys and items. No way to tell if a value is in a dictionary, or a (key, value). Test if a collection is a superset: .issuperset(collection) For sets; could apply to bags. Could be defined in terms of __contains__ and __iter__. Could be a function. .issubset(collection) is related, of course. Get keys: .keys() (.iterkeys()) Returns all the keys in a dictionary. In py3k as a view. .iterkeys() goes away? For multidicts, do the keys show up multiple times? For ordered multidicts, this seems necessary. Get values: .values() (.itervalues()) Returns all values. Get items: .items() (.iteritems()) Returns all (key, value). Or (key, values) for multidict? Mutable methods =============== Delete index: del col[index] For lists only. Delete key: del col[key] For dicts. For multidicts, does this delete the first value or all values? Remove first matching: .remove(item) For lists, sets, bags. Could be used as a single-value-delete-only version of __delitem__ for multidicts. Discard: .discard(item) For sets, bags. Removes, but doesn't signal error if missing. Could be implement anywhere .remove() is implemented. Pop index: .pop(index=-1) Removes item at index, and returns that. Pop key: .pop(key) Removes an value at key, and returns. For multidicts, removes just the first value? No default key. Pop key/value: .popitem() Removes an arbitrary key/value and returns. For multidicts, returns values one at a time? Set item/index: col[index_or_key] = value Sets the value, overwriting previous value (if any). Cannot extend lists. For multidicts, does this append a new value, or overwrite the value? Append: .append(item) For lists. Would this apply to ordered dicts? .append(key, value)? Extend from sequence: .extend(seq) Defined in terms of .append(). Extend without sequence: .update(mapping) Defined for dict and set, should be for bag. Defined in terms of __setitem__; has special behavior for sequences of tuples, vs. dict-like objects. Add item: .add() Applies to sets, bags. Could apply to non-ordered multidicts; .add(key, value)? Rotate: .rotate() Currently implemented only in queues. Could apply to any ordered collection. Copy: .copy() Applies to dicts, sets, bags. Missing in lists. Should all mutable collections have this? Clear: .clear() Removes all items. Not implemented for lists for some reason. In-place versions of &|^-: .union(other), x.intersection(other), x.difference(other), x.symmetric_difference(other) Defined in terms of the operators (or vice versa). In-place sort: .sort(cmp=None, key=None, reverse=None) Does a sort. Could apply to any ordered collection. Sort on (key, value) or just key? Reverse: .reverse() Reverses; applies to all ordered collections. Missing ======= There's no method for getting the key or keys for a value (similar to list.index). There's no way to invert a dictionary (change key->value to value->key). With more types of dictionaries proper collections will exist to represent the inverse. E.g., multidict is the inverse of dict. This maybe is an argument for the full range of dictionaries. The inverted form could also be a view. somedict.invert()['foo'] would return all the keys which had the value of 'foo'. This is a clever way of adding lots of new concise lookups by leveraging existing methods, but I suspect very hard to read in practice. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From jjl at pobox.com Tue Apr 11 21:12:33 2006 From: jjl at pobox.com (John J Lee) Date: Tue, 11 Apr 2006 19:12:33 +0000 (UTC) Subject: [Python-3000] Separating out CPython and core Python tests In-Reply-To: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com> References: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com> Message-ID: On Tue, 11 Apr 2006, Michael Chermside wrote: [...] > There's a related change that I would love to see. Currently, we > have a large number of unit tests, but we have made no attempt to > distinguish between python language tests, and CPython implementation > tests. If we made such a distinction, then PyPy, Jython, Iron Python, > and perhaps others would benefit. Python 3000 might be a good time > to introduce the distinction. > > In fact, I don't think "we" even need to do the hard work (where "we" > is defined as the CPython developers). I am confident that if the > CPython team offered to make such a split, then either the PyPy or > Jython developers (maybe both) would happily provide us with a list of > which tests they thought were core language tests and which were > CPython-specific. Guido could review and pronounce on any controversial [...] +1, but why wait for Python 3000? Maybe add a "resource name" to regrtest.py, so e.g. regrtest.py-u -cpython ran everything except the CPython-specific tests? RESOURCE_NAMES = ('audio', 'curses', 'largefile', 'network', 'bsddb', - 'decimal', 'compiler', 'subprocess', 'urlfetch') + 'decimal', 'compiler', 'subprocess', 'urlfetch', + 'cpython') (tests then use test_support.is_resouce_enabled('cpython') when building test suites) John From martin at v.loewis.de Tue Apr 11 21:52:29 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Tue, 11 Apr 2006 21:52:29 +0200 Subject: [Python-3000] symbols? In-Reply-To: <53D14D58-1BE5-477E-B003-1CBB489DDE0C@monkeyfist.com> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <1144686244.15801.28.camel@resist.wooz.org> <443BF2F6.6060605@v.loewis.de> <53D14D58-1BE5-477E-B003-1CBB489DDE0C@monkeyfist.com> Message-ID: <443C08FD.9010401@v.loewis.de> Kendall Clark wrote: > Thanks, Martin. That is, in fact, pretty much what I'd like to see in > Py3K, modulo the issue about getattr. The question is still: why? In Smalltalk, symbols are used to denote names (methods/selector names and class names); this apparently isn't your intention (and I do believe strings serve fine as names, no need for a new datatype - except that it should support Unicode). So the only advantage I can see is the simplified typing: instead of typing two quotation marks, you type a single colon. It might be a saving on your keyboard that you don't have to press the shift key (is colon available unshifted on US keyboards?); I have to press the shift key either way (: is shift-.; " is shift-2). Regards, Martin From rrr at ronadam.com Tue Apr 11 22:03:43 2006 From: rrr at ronadam.com (Ron Adam) Date: Tue, 11 Apr 2006 15:03:43 -0500 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: Guido van Rossum wrote: > On 4/11/06, Jim Jewett wrote: >> You had said there was some question of what kind of objects the type >> markers should be. I couldn't (and still can't) see any good reason >> for them to be anything but a way to ensure [to some level] that the >> argument is [in some way] appropriate. > > You just can't avoid using a double negative can't you? :-) > > I'd like to stretch the definition further, so that *any* argument > metadata can be placed in that syntactic position, as long as a > decorator is used that can interpret the metadata, and as long as the > metadata can be expressed as a single expression. (If the expression > gets too long we can always lift it out and move it to a preceding > position in the file, as the expression is evaluated at the same time > the function definition and the default values are -- I probably > forgot to make that explicit.) > > Just like decorators have no defined semantics (beyond "the decorator > gets called this way and what it returns gets assigned to that > variable") but have found a few well-defined conventions for their > most common usage (call wrappers and registration), I expect that > we'll develop a convention for how to express various useful metadata > soon enough -- but I don't want the language to enforce that. The > syntactic position is much too precious to assign it a single specific > meaning. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) I almost posted this suggestion yesterday but didn't because I wasn't sure it fit the context. But maybe it does. There is a possibility of parsing selected meta data items from doc strings. Doc strings already are a type of meta data that gets set to an attribute, so this would be extending that precedence. If the __signature__ is parsed from the first line, it puts it right below the function definition. def pattern_search(seq, pattern, start=0): """ fn(str, str, int) -> (str, int) Get a sub-string and index using a pattern. """ This doesn't have as strong a relationship as adding syntax in the definition. Having the syntax in the definition seems (to me) like it should always be enforced. While having it in the doc string seems more like suggested use, and it may, or may not be enforced. I also like that it is self documenting. If the signature doesn't get recognized and parsed correctly, then an exception would probably occur as soon as something tries to use __signature__, so it should be reasonably easy to debug. Depending on how freely the signature line is parsed it could fit a pattern or be anything that someone finds to be useful. Other meta data such as __version__, __author__, and/or __date__ could also be parsed from doc strings. Currently those are used very inconsistently in the standard library. Cheers, Ron From ianb at colorstudy.com Tue Apr 11 22:09:46 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 11 Apr 2006 15:09:46 -0500 Subject: [Python-3000] symbols? In-Reply-To: <443C08FD.9010401@v.loewis.de> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <1144686244.15801.28.camel@resist.wooz.org> <443BF2F6.6060605@v.loewis.de> <53D14D58-1BE5-477E-B003-1CBB489DDE0C@monkeyfist.com> <443C08FD.9010401@v.loewis.de> Message-ID: <443C0D0A.1040904@colorstudy.com> Martin v. L?wis wrote: > Kendall Clark wrote: > >>Thanks, Martin. That is, in fact, pretty much what I'd like to see in >>Py3K, modulo the issue about getattr. > > > The question is still: why? In Smalltalk, symbols are used to denote > names (methods/selector names and class names); this apparently isn't > your intention (and I do believe strings serve fine as names, no need > for a new datatype - except that it should support Unicode). > > So the only advantage I can see is the simplified typing: instead of > typing two quotation marks, you type a single colon. It might be > a saving on your keyboard that you don't have to press the shift key > (is colon available unshifted on US keyboards?); I have to press > the shift key either way (: is shift-.; " is shift-2). Symbols make it clear you are talking about some more-constrained value (like an enumeration), or an in-process name (like a method name). Strings can just be data, which is generally used very differently. In addition to immutability, I think this is an important part of how symbols are used in Lisp and Ruby (they don't seem to actually be used as much in Smalltalk, more of an implementation detail). Also, when passed a string or symbol, you can get a better idea of what it was intended for. And, for example, having to cast the string to a symbol before using it with getattr() will make the security implications a little clearer. At the same time, I think it is more in line with Python's design to avoid talking about the names of things, and instead talk about the things themselves. After all, you don't do: make(:var, 10) You do: var = 10 In most cases this is how Python works now. Literal strings show up sometimes. I see them in descriptors (or complex tricks to avoid them). In hasattr() they are common, in getattr() occasionally to make use of the third getattr() argument -- either can be avoided with try:except AttributeError: if you choose. That's most of what I can think of now. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From brett at python.org Tue Apr 11 22:20:39 2006 From: brett at python.org (Brett Cannon) Date: Tue, 11 Apr 2006 13:20:39 -0700 Subject: [Python-3000] Separating out CPython and core Python tests In-Reply-To: References: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com> Message-ID: On 4/11/06, John J Lee wrote: > On Tue, 11 Apr 2006, Michael Chermside wrote: > [...] > > There's a related change that I would love to see. Currently, we > > have a large number of unit tests, but we have made no attempt to > > distinguish between python language tests, and CPython implementation > > tests. If we made such a distinction, then PyPy, Jython, Iron Python, > > and perhaps others would benefit. Python 3000 might be a good time > > to introduce the distinction. > > > > In fact, I don't think "we" even need to do the hard work (where "we" > > is defined as the CPython developers). I am confident that if the > > CPython team offered to make such a split, then either the PyPy or > > Jython developers (maybe both) would happily provide us with a list of > > which tests they thought were core language tests and which were > > CPython-specific. Guido could review and pronounce on any controversial > [...] > > +1, but why wait for Python 3000? > > Maybe add a "resource name" to regrtest.py, so e.g. regrtest.py-u -cpython > ran everything except the CPython-specific tests? > > RESOURCE_NAMES = ('audio', 'curses', 'largefile', 'network', 'bsddb', > - 'decimal', 'compiler', 'subprocess', 'urlfetch') > + 'decimal', 'compiler', 'subprocess', 'urlfetch', > + 'cpython') > > > (tests then use test_support.is_resouce_enabled('cpython') when building > test suites) But many module tests have mixed spec/implementation tests. Wouldn't a decorator be a better solution? We keep talking about adding decorators that signal test known to fail and we could probably stand to flag tests known to leak (all of this could lead to a list of tests with such decorators at the end of regrtest so we are constantly reminded where things are broken). -Brett From kendall at monkeyfist.com Tue Apr 11 22:37:10 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 11 Apr 2006 16:37:10 -0400 Subject: [Python-3000] symbols? In-Reply-To: <443C0D0A.1040904@colorstudy.com> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <1144686244.15801.28.camel@resist.wooz.org> <443BF2F6.6060605@v.loewis.de> <53D14D58-1BE5-477E-B003-1CBB489DDE0C@monkeyfist.com> <443C08FD.9010401@v.loewis.de> <443C0D0A.1040904@colorstudy.com> Message-ID: <063F9248-3D70-49A0-B688-EEEDBEC76FE8@monkeyfist.com> On Apr 11, 2006, at 4:09 PM, Ian Bicking wrote: > Martin v. L?wis wrote: >> Kendall Clark wrote: >>> Thanks, Martin. That is, in fact, pretty much what I'd like to >>> see in >>> Py3K, modulo the issue about getattr. >> the shift key either way (: is shift-.; " is shift-2). > > Symbols make it clear you are talking about some more-constrained > value (like an enumeration), or an in-process name (like a method > name). Strings can just be data, which is generally used very > differently. In addition to immutability, I think this is an > important part of how symbols are used in Lisp and Ruby (they don't > seem to actually be used as much in Smalltalk, more of an > implementation detail). Also, when passed a string or symbol, you > can get a better idea of what it was intended for. Yeah, my primary motivation is summed up nicely here; in some cases a symbol expresses intent better (IMO) than a string. > And, for example, having to cast the string to a symbol before > using it with getattr() will make the security implications a > little clearer. Yep. > At the same time, I think it is more in line with Python's design > to avoid talking about the names of things, and instead talk about > the things themselves. After all, you don't do: > > make(:var, 10) > > You do: > > var = 10 > > In most cases this is how Python works now. Literal strings show > up sometimes. They get used as terms in Domain Specific Languages in Python. Or, more commonly, one writes some king of generic symbol-like class that just wraps the string. But then everyone has to repeatedly reinvent that wheel. I guess it depends on how often Python is used as a host language for DSLs. But I tend to do a lot of that kind of programming (or, rather, a high percentage of the programming I do is of that type). So embedding a rule engine (with syntax for rules & facts) in Python is harder w/out symbols than it would be with them. > I see them in descriptors (or complex tricks to avoid them). Yeah, Pylons just got this syntax for web-dispatching: @pylons.rest.restrict("GET", "HEAD") def get_instance(self, id): pass Okay, so HTTP methods are so well know that the intent there is very clear. But it's less so when yr talking about some less known DSL, perhaps one written for a specific application. The other choice, of course, is for the library or API to define some variables bound to strings and then use them like constants, except that they can get redefined: @pylons.rest.restrict(GET, HEAD) ... A symbol has the advantage that it can't be assigned to, so can't change like a variable, but expresses more clearly than a string the intent to refer to a name or term, not string data: @pylon.rest.restrct(:GET, :HEAD) > In hasattr() they are common, in getattr() occasionally to make > use of the third getattr() argument -- either can be avoided with > try:except AttributeError: if you choose. That's most of what I > can think of now. I think Ian put his finger on a good point, as usual: in Python we refer to things, not to their names, more often. But if you want to embed little languages in Python, you refer to the names of things from some other domain pretty often. Yeah, this is more of a Lisp or Ruby or Haskell application, but I prefer Python for lots of reason and want to use it for writing DSLs. In that kind of app, strings get used to represent terms and names pretty often, and it just doesn't *feel* right. :> Cheers, Kendall From thomas at python.org Tue Apr 11 23:43:07 2006 From: thomas at python.org (Thomas Wouters) Date: Tue, 11 Apr 2006 23:43:07 +0200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <443BF007.9080401@v.loewis.de> References: <443BF007.9080401@v.loewis.de> Message-ID: <9e804ac0604111443l73f7401bge821c9c3b2b8560f@mail.gmail.com> On 4/11/06, "Martin v. L?wis" wrote: [ Bold out-of-the-box thinking on modules by Martin ] #define STATE PyModule_STATE(module_slot, struct state) I wonder if standardization of the struct layout of 'struct state', and the naming scheme of 'struct state', 'module_slot' and 'STATE' would be desireable. The state and module_slot structs would be c-file-static, so naming conflicts shouldn't exist, and it would improve readability the way Py_VISIT does (IMHO.) In fact, maybe the struct state should just be the module object, as a C-subclass of the module type? It means a bit more work, like defining a module struct and a list of members Python classes get direct access to, but it does close the gap between modules and normal objects a bit. (At least from C; Python code still has to replace modules by lookalikes in sys.modules.) In fact... perhaps module functions should then get the module object as the first argument (currently unused but always defined)? Then we just need conceptually-new magic to fetch the right module object when dealing with methods of types defined in the module (which, I believe, should be pretty rare.) It would be possible to support the existing init > functions if that is desirable; in that case, nothing > would change. I think, for Py3K, we should bite the bullet and not check init. Extension writers will have to update their init-functions anyway, because PyImport_InitModule* all changes (I hope!) While we're at it, should generate the init name (and maybe prototype) by macro instead of relying on the user to get the naming right? It'd allow easier changing of the naming and such in the future, but I don't know if that outweighs the discomfort of hiding function prototypes. Probably not. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060411/84838a6c/attachment.html From martin at v.loewis.de Wed Apr 12 00:20:48 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 12 Apr 2006 00:20:48 +0200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <9e804ac0604111443l73f7401bge821c9c3b2b8560f@mail.gmail.com> References: <443BF007.9080401@v.loewis.de> <9e804ac0604111443l73f7401bge821c9c3b2b8560f@mail.gmail.com> Message-ID: <443C2BC0.4080407@v.loewis.de> Thomas Wouters wrote: > I wonder if standardization of the struct layout of 'struct state', and > the naming scheme of 'struct state', 'module_slot' and 'STATE' would be > desireable. The state and module_slot structs would be c-file-static, so > naming conflicts shouldn't exist, and it would improve readability the > way Py_VISIT does (IMHO.) I was thinking about that: a fixed naming should work. Not sure what you mean by a standardized layout: you certainly want to put arbitrary data in the module state (and, in particular, not a Python dictionary - people could already lookup exception objects in the module's dictionary, but they prefer "real pointers"). > In fact, maybe the struct state should just be > the module object, as a C-subclass of the module type? That would also work. > In fact... perhaps module functions should then get the module object as > the first argument (currently unused but always defined)? That wouldn't work: the first argument currently *is* used, atleast if you pass a non-NULL value for self to Py_InitModule4. Of course, we could drop that in Py3k, and declare that you always get the module object. However, that still wouldn't fully work, since you might need access to module objects in a method of a type you implemented, where self is the object. So I think you need some way to go through the current interpreter. > I think, for Py3K, we should bite the bullet and not check init. > Extension writers will have to update their init-functions anyway, > because PyImport_InitModule* all changes (I hope!) While we're at it, > should generate the init name (and maybe prototype) by macro > instead of relying on the user to get the naming right? That would also work. But, if we also have finalizers, we already need two macros to generate the function name. > It'd allow > easier changing of the naming and such in the future, but I don't know > if that outweighs the discomfort of hiding function prototypes. Probably > not. Right. Martin From janssen at parc.com Wed Apr 12 02:13:44 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 11 Apr 2006 17:13:44 PDT Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: Your message of "Tue, 11 Apr 2006 09:49:59 PDT." <443BDE37.6070207@v.loewis.de> Message-ID: <06Apr11.171352pdt."58633"@synergy1.parc.xerox.com> > Well, if you were to create a hierarchy of proxy objects for "remoting" > objects across different interpreters, you would get much better > performance than if you used IPC. This would be similar to .NET > AppDomains or Java Isolates. Or you could use ILU, which works in-process... Bill From greg.ewing at canterbury.ac.nz Wed Apr 12 02:29:41 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Apr 2006 12:29:41 +1200 Subject: [Python-3000] Will we have a true restricted exec environmentfor python-3000? In-Reply-To: <443BDE37.6070207@v.loewis.de> References: <44373866.9070305@alumni.rice.edu> <021401c65c36$b3de2f70$26452597@bagio> <1144694558.3162.37.camel@localhost.localdomain> <443AFD4B.1060201@canterbury.ac.nz> <443B1BF0.80301@colorstudy.com> <443B6101.1010903@canterbury.ac.nz> <443BDE37.6070207@v.loewis.de> Message-ID: <443C49F5.8040906@canterbury.ac.nz> Martin v. L?wis wrote: > Well, if you were to create a hierarchy of proxy objects for "remoting" > objects across different interpreters, Then you have the problem of bulletproofing the proxy objects, and somehow convincing yourself that nothing other than proxy objects can leak across. -- Greg From tjreedy at udel.edu Wed Apr 12 05:49:05 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 11 Apr 2006 23:49:05 -0400 Subject: [Python-3000] Separating out CPython and core Python tests References: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com><20060411192142.GO476@tigger.digitaltorque.ca> Message-ID: "Guido van Rossum" wrote in message > I agree that such a test suite would be helpful. I don't think it > should wait for Python 3000. Possible plan: Add 'spec' or 'specification' or 'language', 'implementation', and possibly 'support' subdirs to ...Lib/test. Put most test files in 'spec', possibly support files in 'support' and any implementation-only tests in 'implementation'. Test_peepholer.py at least would go in the latter. Should tests for C-coded modules go here too? Unless done immediately for 2.5, do this for trunk (future 2.6) only after 2.5 maintenance branch established. Adjust 'from test.something import ...' statements to 'from test.subdir.something import ...' with Python script. Split a few files known to have implementation tests, such as the test somewhere that small ints up to a certain size are preallocated. This could break even for a customized CPython build. Would having tests like this segregated in an implementation test directory help even CPython customizers? As Michael suggested, let Jython, IronPython, and PyPy people suggest additional file splits or movements. Terry Jan Reedy From tim.peters at gmail.com Wed Apr 12 06:47:33 2006 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 12 Apr 2006 00:47:33 -0400 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <443BF007.9080401@v.loewis.de> References: <443BF007.9080401@v.loewis.de> Message-ID: <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> [Martin v. L?wis] > Abstract: Module initialization currently has a few deficiencies. > There is no cleanup for modules, the entry point name might give > naming conflicts, the entry functions don't follow the usual > calling convention, and multiple interpreters are not supported > well. This PEP addresses these issues. > ... This sounds like a good time to reproduce the body of an email that's been sitting in my inbox since March of 2001. Since I didn't ask for permission to reproduce it, I'll only reveal that it was written by the Guido you're thinking of: """ OK, so we need to define a way for modules to specify a function to be called at cleanup time. I propose that we bring this out in the open, at least in python-dev. Note that there are three or four potential cleanup times: - when the module object is deleted from sys.modules - when reload() is used on an extension - when an interpreter created with PyInterpreter_New() is finalized - when Py_Finalize() is called - when Python exits - when the Python DLL is unloaded (Windows only) Good luck! """ PythonLabs never found the bandwidth for this, as we all spent the next year trying to teach Barry how to use Unicode. I don't even remember what the thread above was about, but it would be great if the PEP contained enough relevant words that I could finally clean that message out of my inbox ;-) From martin at v.loewis.de Wed Apr 12 09:30:38 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed, 12 Apr 2006 09:30:38 +0200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> References: <443BF007.9080401@v.loewis.de> <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> Message-ID: <443CAC9E.9070504@v.loewis.de> Tim Peters wrote: > PythonLabs never found the bandwidth for this, as we all spent the > next year trying to teach Barry how to use Unicode. I don't even > remember what the thread above was about, but it would be great if the > PEP contained enough relevant words that I could finally clean that > message out of my inbox ;-) Sure: I think many of these cleanup points overlap, in the sense that cleanup should get invoked if the module object is discarded. I can put that in the discussion of the PEP (and it's in the python-3000 archives meanwhile to remind me): > - when the module object is deleted from sys.modules This may not have immediate effect, since somebody might still holding a reference to the module. A new import will create a new module object, so I guess it would be best if indeed the state lives in the module object. > - when reload() is used on an extension This is trickier: I guess reload should invoke the finalize function, discard the memory for the state, and then invoke the initialize function again, which will allocate a new state block. > - when an interpreter created with PyInterpreter_New() is finalized This shouldn't cause any module finalization directly. Instead, sys.modules of the interpreter gets cleared, which should cause the module objects to be released. > - when Py_Finalize() is called Likewise. Of course, a forced clearing of all modules might be desirable, since the modules might keep a cycle alive. This raises the question whether modules should be able to participate in cyclic GC... > - when Python exits That should not cause action: Py_Finalize should clear it all. > - when the Python DLL is unloaded (Windows only) IMO, the DLL should only get unloaded when all modules referring to it have been unloaded. So, when the DLL is unloaded, no action should be taken. I also think it's not Windows only: on dl, we should invoke dlclose when all modules have been finalized. Regards, Martin From thomas at python.org Wed Apr 12 11:22:43 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 12 Apr 2006 11:22:43 +0200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <443CAC9E.9070504@v.loewis.de> References: <443BF007.9080401@v.loewis.de> <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> <443CAC9E.9070504@v.loewis.de> Message-ID: <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> On 4/12/06, "Martin v. L?wis" wrote: > > Tim Peters quoted Guido: > > - when reload() is used on an extension > > This is trickier: I guess reload should invoke the finalize function, > discard the memory for the state, and then invoke the initialize > function again, which will allocate a new state block. Actually, no, reload() has to remove the module from sys.modules and load it anew. The reason for this is that any instances of types defined in the module can still live, after the reload, and they would still refer to their old-module-object-referring types. (And, as you said, this is a reason to keep the 'state' in module objects.) This raises the question whether modules should be able to > participate in cyclic GC... Don't they already? They have a traverse proc, just not a clear proc. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060412/44ffd02b/attachment.html From thomas at python.org Wed Apr 12 11:31:38 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 12 Apr 2006 11:31:38 +0200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <443C2BC0.4080407@v.loewis.de> References: <443BF007.9080401@v.loewis.de> <9e804ac0604111443l73f7401bge821c9c3b2b8560f@mail.gmail.com> <443C2BC0.4080407@v.loewis.de> Message-ID: <9e804ac0604120231o35feda09u20c0e9753f07c5e0@mail.gmail.com> On 4/12/06, "Martin v. L?wis" wrote: > > Thomas Wouters wrote: > > In fact... perhaps module functions should then get the module object as > > the first argument (currently unused but always defined)? > > That wouldn't work: the first argument currently *is* used, atleast > if you pass a non-NULL value for self to Py_InitModule4. Of course, > we could drop that in Py3k, and declare that you always get the module > object. Right, forgot about that. I've never seen anyone use it, though. However, that still wouldn't fully work, since you might need access > to module objects in a method of a type you implemented, where self > is the object. So I think you need some way to go through the current > interpreter. Well, I said that, it just wasn't part of what you quoted: "Then we just need conceptually-new magic to fetch the right module object when dealing with methods of types defined in the module (which, I believe, should be pretty rare.)" Your point about importing a module the second time in the same interpreter (by removing it from sys.modules, or by using reload()) also requires this. We'd need a convenient way to do that (I'm not sure I consider casting the result of PyObject_GetAttrString(self, "__module__") 'convenient'), and we'd need to educate all extension writers (including the ones working on CPython proper) when to use the module object instead of direct C variable access. Using a C global instead of a module object attribute would lead to subtle bugs in subtle cases, like when multiple interpreters are involved. Maybe, when we're done, we should be running regrtest in multiple interpreters concurrently, too. ;P I do think it's worth it, though. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060412/7c2a2c9e/attachment.htm From mcherm at mcherm.com Wed Apr 12 14:15:14 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 12 Apr 2006 05:15:14 -0700 Subject: [Python-3000] symbols? Message-ID: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> Kendall Clark writes: > The other choice, of course, is for the library or API to define some > variables bound to strings and then use them like constants, except > that they can get redefined: > > @pylons.rest.restrict(GET, HEAD) > ... > > A symbol has the advantage that it can't be assigned to, so can't > change like a variable, but expresses more clearly than a string the > intent to refer to a name or term, not string data When I see a variable name in all-caps, I don't assign to it. I don't even need a tool like PyChecker to remind me that this is a constant because I've been familiar with the "all-caps == constant" convention from shortly after I got a computer with lower-case letters on it. The other programmers I work with seem to behave the same way. I must be unusually lucky in this regard, because I meet lots of people who are very concerned about the fact that it *is* possible to change these values. I can only surmise that they work with people who make a habit of modifying all-caps variables randomly just for fun. I've found it even more curious that no one ever seems to be worried about the ability to write infinite loops ("while True: pass") in pretty much any piece of code. Of course, I suppose it might have something to do with the fact that the haulting problem makes it theoretically impossible to prevent all infinite loops. But it's funny how no one considers this capability to be a dangerous flaw in the language... perhaps because it just doesn't happen often. But I will say that I've encountered more bugs due to infinite loops than bugs due to modification of "constants". Later, Kendall continues: > In that kind of app, strings get used to represent terms and names > pretty often, and it just doesn't *feel* right. Now that argument I *understand*, even if I don't find myself convinced by it. I used to think that way too, but I guess "practicality" has sunk in over the years. And there's also: > if you want to > embed little languages in Python, you refer to the names of things > from some other domain pretty often. Yeah, this is more of a Lisp or > Ruby or Haskell application, but I prefer Python for lots of reason > and want to use it for writing DSLs. I don't write enough Python-based DSLs in Python to hold an informed opinion on this one. -- Michael Chermside From martin at v.loewis.de Wed Apr 12 18:43:27 2006 From: martin at v.loewis.de (=?UTF-8?B?Ik1hcnRpbiB2LiBMw7Z3aXMi?=) Date: Wed, 12 Apr 2006 18:43:27 +0200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> References: <443BF007.9080401@v.loewis.de> <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> <443CAC9E.9070504@v.loewis.de> <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> Message-ID: <443D2E2F.501@v.loewis.de> Thomas Wouters wrote: > Actually, no, reload() has to remove the module from sys.modules and > load it anew. That's not what reload currently does for Python modules. The module object stays the same (so that anybody who imported it will also see the reload's effect). > The reason for this is that any instances of types defined > in the module can still live, after the reload, and they would still > refer to their old-module-object-referring types. (And, as you said, > this is a reason to keep the 'state' in module objects.) Are you expecting that reload of an extension module will actually load the current code from disk? This can't really work: on some systems, you can't update the DLL while it is in use; on others, it might not be possible to load the same shared library twice. > This raises the question whether modules should be able to > participate in cyclic GC... > > Don't they already? They have a traverse proc, just not a clear proc. Sure: for the state that is common to all modules. For the per-module state, traverse functionality might also be needed, which would mean that the module needs to implement it. Regards, Martin From thomas at python.org Wed Apr 12 20:46:28 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 12 Apr 2006 20:46:28 +0200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <443D2E2F.501@v.loewis.de> References: <443BF007.9080401@v.loewis.de> <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> <443CAC9E.9070504@v.loewis.de> <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> <443D2E2F.501@v.loewis.de> Message-ID: <9e804ac0604121146m5be89280ua0bf2b899e1f2e33@mail.gmail.com> On 4/12/06, "Martin v. L?wis" wrote: > > Thomas Wouters wrote: > > Actually, no, reload() has to remove the module from sys.modules and > > load it anew. > > That's not what reload currently does for Python modules. The module > object stays the same (so that anybody who imported it will also see > the reload's effect). > > > The reason for this is that any instances of types defined > > in the module can still live, after the reload, and they would still > > refer to their old-module-object-referring types. (And, as you said, > > this is a reason to keep the 'state' in module objects.) > > Are you expecting that reload of an extension module will actually > load the current code from disk? This can't really work: on some > systems, you can't update the DLL while it is in use; on others, > it might not be possible to load the same shared library twice. Well, in the case of Python modules, a reload recreates all classes, so existing instances remain instances of the old class (and typechecks by name fail in unexpected ways; you've got to love those "Excected 'Foo' instance, got 'Foo' instance" errors ;) I guess I was thinking we need to work around that for C modules as well, but I guess we really don't. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060412/bc188f5d/attachment.html From adam.deprince at gmail.com Wed Apr 12 21:02:37 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Wed, 12 Apr 2006 15:02:37 -0400 Subject: [Python-3000] symbols? In-Reply-To: <443AF434.7000503@canterbury.ac.nz> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443AF434.7000503@canterbury.ac.nz> Message-ID: <1144868557.7581.15.camel@localhost.localdomain> On Tue, 2006-04-11 at 12:11 +1200, Greg Ewing wrote: > Kendall Clark wrote: > > > One thing I'd really like to see in Python 3000 is support for first- > > class symbols, with literal syntax. > > Actually I think enumerations would be more useful than > symbols. There's no essential difference between a symbol > and an interned string. The only real disadvantage to > using strings as symbols is that you don't get an > immediate NameError if you misspell one. A symbol type > wouldn't solve that problem; enumerations would. You can get all of the benefits of a symbol, including the immediate spelling correction, by encouraging users to use module members instead of string literals. Drawing on the os as an example ... ... os.O_EXCL os.O_LARGEFILE ... I don't see the benefit. > > -- > Greg > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/adam.deprince%40gmail.com From tim.hochberg at ieee.org Wed Apr 12 22:18:05 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 12 Apr 2006 13:18:05 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <44397B14.4060208@cox.net> References: <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> Message-ID: This is just a little update on my rewrite of pprint. I've continued to clean up and optimize this. If anyone wants to look at it, I've updated the code here: http://members.cox.net/~tim.hochberg/pprint2.py [The changes are probably not signifigant enough to justify another posting of the code to the group.] The most interesting thing is that the code is now marginally faster for saferepr and over twice as fast for pformat than the original pprint.py. That's according to _perftest, a benchmark I make no great claims for. Still, I think it shows that, with a little caching, there's no reason that even a pure python protocol implementation need be slow. Regards, -tim From tim.hochberg at ieee.org Wed Apr 12 23:17:37 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Wed, 12 Apr 2006 14:17:37 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44343917.4060503@cox.net> <44343E24.9010906@livinglogic.de> <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> Message-ID: <443D6E71.9090807@ieee.org> Here's a little thought experiment: What would happen if 'a+b' was just syntactic sugar for 'operator.add(a,b)', where operator.add was a generic function, instead of the current magic dance involving __add__ and __radd__. Neglecting the fact that it would break all Python code now in existence, would this be a good thing, a bad thing or can we even tell? To get the current effect of __add__ and __radd__, class definitions would look something like: class Weeble: #.... @operator.add.register(Weeble, object) def add(a, b): #... @operator.add.register(object, Weeble) def radd(b, a): #... That's not too different from today, although it is a little weird that add and radd are outside of Weeble. One would also need to need to be more flexible about strict dominance, allowing some sorts of ties and trying the tied functions one at a time until one worked. (Hmmm... perhaps this is the kind of thing Guido was referring to in his blog. If so, it bounced off my thick head ;-) I suppose one could add some metaclass magic to type so that __add__ and __radd__ were picked up and registered as above. That would make things look superficially the same as now and would be more or less backwards compatible. On the plus side, there's a lot more flexibility. You could teach two types that don't know anything about each other how to play nice together simply by registering a couple more adapters to operator.add. On the minus side, there's perhaps too much flexibility. Any function call or module importation or even attribute access could suddenly change the behaviour of an unrelated type. >>> a = 1 >>> b = 2 >>> # this calls operater.add.register(int,int)(lambda a,b:str(a+b)) >>> c = frobulate() >>> a + b "3" We're all consenting adults here, so should I even worry about this. I don't know. I suspect this falls into the "wild ideas' category, but it's an interesting thought experiment anyway. Perhaps with relevance to other uses of generic functions. Regards, -tim From brett at python.org Wed Apr 12 23:44:00 2006 From: brett at python.org (Brett Cannon) Date: Wed, 12 Apr 2006 14:44:00 -0700 Subject: [Python-3000] Interfaces for views and extended collections In-Reply-To: <443C0611.1060108@colorstudy.com> References: <443C0611.1060108@colorstudy.com> Message-ID: On 4/11/06, Ian Bicking wrote: > So Guido asked for more concrete discussion of things like views. A > richer set of collections also fits in here, as for instance dict.keys() > would be a view with a collection interface not exactly like other > collections. I wrote up some notes, which I'll pass on. > > So, I can imagine 3 attributes which seem orthogonal to me: > > a) Is it ordered? > b) Can it contain multiple equal items? > c) Is it associative? > > In addition there is mutability, though anything that can be mutable > could also have an immutable form. In general the only difference I see > is the removal of methods, and the collection becomes hashable. > > There's 8 combinations. > > Associative collections: > > dict: no multiple items, unordered (missing an immutable form) > multidict: multiple items, unordered > ordered multidict: multiple items, ordered > ordered dict: no multiple items, ordered (useful?) > > Non-associative: > > Set/ImmutableSet: no multiple items, unordered > bag: multiple items, unordered > ordered set: no multiple items, ordered (useful? -- priority queue?) > list/tuple: multiple items, ordered > > > So, ordered sets and ordered dicts seem questionable. Also, it is > unclear to me if there's a compelling reason for both ordered multidict > and multidict, though the performance is likely to be different for the > two (underneath multidict likely being a dictionary of sets, ordered > multidict being a list of tuples). > > So, at least, we are missing bags, (ordered) multidicts. In terms of > views, .keys() is a set, multidict.keys() is a bag, and > orderedmultidict.keys() is a list (or tuple, or immutable list-like object). > Making sure views work with bags and multidicts is good, but order, I think, does not fall within this scope. If people want them ordered they can get the iterator and pass it to sorted(). > There's some other attributes. For instance, we have defaultdict. > Another option that could be applied to almost any of these is a key > function. E.g., KeyedDict(key=lambda x: x.lower()) would be a > case-insensitive dictionary. Because views introduce an relationship > between these collections, if we add a keyed dictionary we should also > have a keyed set (returned by KeyedDict.keys()). If we build this > directly into the collection, that's easy, if not then we double the > number of collections to be implemented; which is perhaps fine. > If we make them mutable, then yes, having a way to automatically transform all passed-in values would be handy. > OK... so now we're this far; what will the interfaces be for these > objects? What follows is a long list of, I think, all the major methods > and operators collections define. This is mostly just source material, > there's no conclusions or proposals. I hope this will be useful to > consider what the interfaces might look like, or to compare proposed > interfaces against. > > > > Methods that don't imply mutability > =================================== > > Combine two collections: + > Currently only applies to lists and tuples. Lists cannot be added to > tuples. This only applies to objects with order, that can contain > multiple items. > > Combine two collections: | > For sets, produces the union. Applies to objects without order and > cannot contain multiple items. Should dicts implement this? Which side > would take precedence? > > Combine two collections: & > For sets, produces the intersection. Should dicts implement this? > > Combine two collections: ^ > For sets, produces the symmetric difference. Dicts could implement > this; neither side would have to take precedence. > > Difference: - > For sets, a-b produces a new set with items from b removed. Could apply > to bags. Maybe also to dicts? > > Repeat collection: *int > For lists and tuples, repeats the contents. Could apply to ordered > multidicts. Maybe also to bags and multidicts? > > Get item at index: [index] > Returns the item at that index. Doesn't apply to dicts, even ordered > dicts, because it conflicts with another meaning of __getitem__. > Doesn't apply to unordered collections > > Get item for association: [key] > Returns the value for that key. Applies to all dictionaries. Do > multidicts return all values or the first value? Or the last value? > Dictionaries essentially return the *last* value (where all previous > values were disposed of). An additional method is needed for the other > option. If multiple values, in what kind of container are those values? > Mutable? A view (yuck)? > I would say multidicts return all values. > Get an item, with not-found handling: .get() > Returns the value for that key. For multidicts, does it return None or > an empty collection if nothing is found? For whatever method added in > addition to __getitem__, is there an equivalent get*() form? > > Count items: .count(item) > This works on lists. For some reason not on tuples. Should work on > bags. Would this apply to multidicts? Is item a key or (key, value) in > that case? Would there be a way to count values? Value counting can > apply to all associations. > > Get index: .index(item) > This works on lists. Also for ordered dicts? Would item be key, or > (key, value)? > > Count items: len(collection) > This works on everything. For multidicts, does it count values or keys? > For bags does it count unique items or all items? > Multidicts: keys Bags: unique items. > Test if an item is in collection: item in collection > Works for keys and items. No way to tell if a value is in a dictionary, > or a (key, value). > > Test if a collection is a superset: .issuperset(collection) > For sets; could apply to bags. Could be defined in terms of > __contains__ and __iter__. Could be a function. .issubset(collection) > is related, of course. > > Get keys: .keys() (.iterkeys()) > Returns all the keys in a dictionary. In py3k as a view. .iterkeys() > goes away? For multidicts, do the keys show up multiple times? For > ordered multidicts, this seems necessary. > > Get values: .values() (.itervalues()) > Returns all values. > > Get items: .items() (.iteritems()) > Returns all (key, value). Or (key, values) for multidict? > > Mutable methods > =============== > > Delete index: del col[index] > For lists only. > > Delete key: del col[key] > For dicts. For multidicts, does this delete the first value or all values? > > Remove first matching: .remove(item) > For lists, sets, bags. Could be used as a single-value-delete-only > version of __delitem__ for multidicts. > > Discard: .discard(item) > For sets, bags. Removes, but doesn't signal error if missing. Could be > implement anywhere .remove() is implemented. > > Pop index: .pop(index=-1) > Removes item at index, and returns that. > > Pop key: .pop(key) > Removes an value at key, and returns. For multidicts, removes just the > first value? No default key. > > Pop key/value: .popitem() > Removes an arbitrary key/value and returns. For multidicts, returns > values one at a time? > > > Set item/index: col[index_or_key] = value > Sets the value, overwriting previous value (if any). Cannot extend > lists. For multidicts, does this append a new value, or overwrite the > value? > > Append: .append(item) > For lists. Would this apply to ordered dicts? .append(key, value)? > > Extend from sequence: .extend(seq) > Defined in terms of .append(). > > Extend without sequence: .update(mapping) > Defined for dict and set, should be for bag. Defined in terms of > __setitem__; has special behavior for sequences of tuples, vs. dict-like > objects. > > Add item: .add() > Applies to sets, bags. Could apply to non-ordered multidicts; .add(key, > value)? > > Rotate: .rotate() > Currently implemented only in queues. Could apply to any ordered > collection. > > Copy: .copy() > Applies to dicts, sets, bags. Missing in lists. Should all mutable > collections have this? > > Clear: .clear() > Removes all items. Not implemented for lists for some reason. > > In-place versions of &|^-: .union(other), x.intersection(other), > x.difference(other), x.symmetric_difference(other) > Defined in terms of the operators (or vice versa). > > In-place sort: .sort(cmp=None, key=None, reverse=None) > Does a sort. Could apply to any ordered collection. Sort on (key, > value) or just key? > > Reverse: .reverse() > Reverses; applies to all ordered collections. > > > > Missing > ======= > > There's no method for getting the key or keys for a value (similar to > list.index). > > There's no way to invert a dictionary (change key->value to value->key). > With more types of dictionaries proper collections will exist to > represent the inverse. E.g., multidict is the inverse of dict. This > maybe is an argument for the full range of dictionaries. > > The inverted form could also be a view. somedict.invert()['foo'] would > return all the keys which had the value of 'foo'. This is a clever way > of adding lots of new concise lookups by leveraging existing methods, > but I suspect very hard to read in practice. > For immutable views, the basics I think that are needed are __len__(), __contains__(), and __iter__(). I would normally say __getitem__(), but that doesn't really work for sets unless we just say they always return None and for lists it would need to return indexes to make sense in terms of what the iterator returns. We can add more methods if we can agree on the bar minimum first. I am ignoring mutable views for now to keep the discussion simple. I think that should provide enough information to query about the variaous data structures (sequences and collections) and what they contain. The iterator can be used to access the data or the original data structure itself. -Brett From guido at python.org Thu Apr 13 01:06:27 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 13 Apr 2006 00:06:27 +0100 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <443D6E71.9090807@ieee.org> References: <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> <443D6E71.9090807@ieee.org> Message-ID: On 4/12/06, Tim Hochberg wrote: > What would happen if 'a+b' was just syntactic sugar for > 'operator.add(a,b)', where operator.add was a generic > function, instead of the current magic dance involving > __add__ and __radd__. Funny, exactly the same thought occured to me while flying across the Atlantic. (But actually I believe I've heard someone else mention it in this thread.) Around the same time I also had this scary thought: What if, instead of adding registries mapping types to functions left and right (in pickle, copy, pprint, who knows where else), we had a convention of adding __foo_bar__ methods directly to the class dict? This would require giving up the effective immutability of built-in types, which has widespread repercussions (mostly in the area of multiple interpreters) but nevertheless if this was officially endorsed, we could clean up a lot of type registries... (I notice that I haven't really found any use for *multiple* dispatch in my few explorations of @overloaded; that is, until the operator.add idea came along.) > Neglecting the fact that it would break all Python code now in > existence, would this be a good thing, a bad thing or can we even tell? We could make it break less code if it fell back on the old way of doing things (looking for __add__ and __radd__). I'm not convinced that that is a particularly *bad* convention; it's just not *enough* in case you want to make an X and a Y addable where you control neither X nor Y. One problem (which isn't limited to this use case) is that there's no guarantee that you're the only personl who came up with the idea of making an X addable to a Y, but you may have a different implementation in mind than someone else. If there are conflicting registrations, what happens? (My current implementation just lets the last one to register win, but that's not very user-friendly.) What if the desired effect depends on who caused the call to be made? I'm not saying "who made the call" because if you control the call site you don't need operator overloading; you can code whatever add() function you like. But if the + call itself is inside the package defining X or Y, and the other argument happens to be obtained indirectly, the ability to define what X()+Y() means without modifying X or Y is the most desirable. But this is also when conflicting definitions are most likely to happen. So maybe a better strategy would be to wrap your Y instance in a wrapper that knows how to add itself to an X? That way someone else with a different need to add an X to a Y can provide their own wrapper and both will live happily independent from each other, at least until the two needs meet. > To get the current effect of __add__ and __radd__, class definitions > would look something like: > > class Weeble: > #.... > @operator.add.register(Weeble, object) > def add(a, b): #... > @operator.add.register(object, Weeble) > def radd(b, a): #... > > That's not too different from today, although it is a little weird that > add and radd are outside of Weeble. It doesn't look attractive as long as you control Weeble (hence my proposal not to drop the old way but to grandfather it in as a last resort). > One would also need to need to be > more flexible about strict dominance, allowing some sorts of ties and > trying the tied functions one at a time until one worked. (Hmmm... > perhaps this is the kind of thing Guido was referring to in his blog. If > so, it bounced off my thick head ;-) I kind of doubt it; I don't know exactly which passage you're referring to but I've been kind of skeptical of the whole "next method" concept in general here because of the possibility of conflicts at all. But yes, the conventional way of treating __add__ etc. has a different need for resolving conflicts than Phillip Eby's idea of requiring strict dominance of the selected solution over all other candidates. So now that you mention it I wonder if the whole strict dominance requirement isn't a red herring? And perhaps we should in general use "returns NotImplemented" as a signal to try the next best candidate, strict dominance be damned... (I guess we're entering a different application domain than where Phillip first developed his ideas.) > I suppose one could add some metaclass magic to type so that __add__ and > __radd__ were picked up and registered as above. That would make things > look superficially the same as now and would be more or less backwards > compatible. Or do this in the default method. > On the plus side, there's a lot more flexibility. You could teach two > types that don't know anything about each other how to play nice > together simply by registering a couple more adapters to operator.add. Right. That's the clear advantage. > On the minus side, there's perhaps too much flexibility. Any function > call or module importation or even attribute access could suddenly > change the behaviour of an unrelated type. Well, all those operations could have undesirable side effects anyway; this sounds like an overly broad fear of side effects. I worry more about the specific issue of two different libraries trying to add the same overloading with different semantics. > >>> a = 1 > >>> b = 2 > >>> # this calls operater.add.register(int,int)(lambda a,b:str(a+b)) > >>> c = frobulate() > >>> a + b > "3" > > We're all consenting adults here, so should I even worry about this. I > don't know. Not about this particular issue. frobulate() could use sys._getframe() and assign new values to a and b. > I suspect this falls into the "wild ideas' category, but it's an > interesting thought experiment anyway. Perhaps with relevance to other > uses of generic functions. Definitely wild. Reader beware. Now is the time to generate lots of wild ideas and let them sink in. If it still seems a good idea 3 months from now we may select it for a code experiment (as opposed to a thought experiment). BTW I want to encourage lots of code experiments for Python 3.0 -- though not directly in the p3yk (sic) branch. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Apr 13 03:58:59 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 13 Apr 2006 13:58:59 +1200 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> References: <443BF007.9080401@v.loewis.de> <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> <443CAC9E.9070504@v.loewis.de> <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> Message-ID: <443DB063.5020808@canterbury.ac.nz> Thomas Wouters wrote: > > On 4/12/06, *"Martin v. L?wis"* > wrote: > > This raises the question whether modules should be able to > participate in cyclic GC... > > Don't they already? They have a traverse proc, just not a clear proc. It's been suggested before that the explicit clearing of a module at interpreter shutdown should be replaced with just letting cyclic GC do its thing. Is there any reason why that wouldn't work? It might lead to less of the unintuitive behaviour that sometimes occurs because of this. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From greg.ewing at canterbury.ac.nz Thu Apr 13 04:07:51 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 13 Apr 2006 14:07:51 +1200 Subject: [Python-3000] symbols? In-Reply-To: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> References: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> Message-ID: <443DB277.5040606@canterbury.ac.nz> Michael Chermside wrote: > I must > be unusually lucky in this regard, because I meet lots of people who > are very concerned about the fact that it *is* possible to change > these values. Occasionally I wonder whether it would be useful to have some form of const declaration in the language, not to prevent accidental change, but so that access to them can be optimised. E.g. in const GET = 'GET' def f(address): html_fungle(address, GET) the access to GET in the function could be compiled into some sort of array lookup instead of a global dict lookup. Although if a way is found to optimise global accesses in general, this might become unnecessary. -- Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | Carpe post meridiam! | Christchurch, New Zealand | (I'm not a morning person.) | greg.ewing at canterbury.ac.nz +--------------------------------------+ From tim.peters at gmail.com Thu Apr 13 04:21:07 2006 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 12 Apr 2006 22:21:07 -0400 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <443DB063.5020808@canterbury.ac.nz> References: <443BF007.9080401@v.loewis.de> <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> <443CAC9E.9070504@v.loewis.de> <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> <443DB063.5020808@canterbury.ac.nz> Message-ID: <1f7befae0604121921u2192baaaua152d727481006a2@mail.gmail.com> [Greg Ewing] > It's been suggested before that the explicit clearing > of a module at interpreter shutdown should be replaced > with just letting cyclic GC do its thing. Is there any > reason why that wouldn't work? It might lead to less of > the unintuitive behaviour that sometimes occurs because > of this. Armin started moving in that direction 2.5 years ago, in patch 812369 (which is still open, but out of date). Note that for some reason, he still found it necessary there to None-out some module globals. From guido at python.org Thu Apr 13 09:48:02 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 13 Apr 2006 08:48:02 +0100 Subject: [Python-3000] Draft PEP: Module Initialization and finalization In-Reply-To: <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> References: <443BF007.9080401@v.loewis.de> <1f7befae0604112147w53543044k86de00ec7d82a9f9@mail.gmail.com> <443CAC9E.9070504@v.loewis.de> <9e804ac0604120222pa65212fwc88695254e670681@mail.gmail.com> Message-ID: On 4/12/06, Thomas Wouters wrote: > > On 4/12/06, "Martin v. L?wis" wrote: > > Tim Peters quoted Guido: > > > - when reload() is used on an extension > > > This is trickier: I guess reload should invoke the finalize function, > > discard the memory for the state, and then invoke the initialize > > function again, which will allocate a new state block. > > Actually, no, reload() has to remove the module from sys.modules and load it > anew. The reason for this is that any instances of types defined in the > module can still live, after the reload, and they would still refer to their > old-module-object-referring types. (And, as you said, this is a reason to > keep the 'state' in module objects.) Maybe I'm misunderstanding, but the *identity* of the object ought to be retained across reload(). Consider two modules A and B each of which imports C. Now B says reload(C). Then A's reference to C should be updated automatically (but of course all instances of classes defined in C have references to the old class object, not the new one). Come to think of it, B's reference to C should also be updated automatically. And by "updated automatically" I don't mean that somehow the reference is patched (which would be very fragile -- there could be any number of aliases) but that the old module object continues to work. I don't care about preserving the old module's dict -- although there are some Python modules that care about preserving contents; I don't know if that feature should be extended to C extensions though. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 13 09:54:15 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 13 Apr 2006 08:54:15 +0100 Subject: [Python-3000] symbols? In-Reply-To: <443DB277.5040606@canterbury.ac.nz> References: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> <443DB277.5040606@canterbury.ac.nz> Message-ID: On 4/13/06, Greg Ewing wrote: > Occasionally I wonder whether it would be useful > to have some form of const declaration in the > language, not to prevent accidental change, but > so that access to them can be optimised. E.g. in > > const GET = 'GET' > > def f(address): > html_fungle(address, GET) > > the access to GET in the function could be > compiled into some sort of array lookup instead > of a global dict lookup. But have you ever wondered if the microscopic speed-up would make enough of a difference to bother with changing the language? (Not to mention the problem of making the const-ness be preserved across import -- in practice, the GET in this example would likely be exported by the same module that exports the function that uses it. And, FWIW, that function is also likely a constant!) > Although if a way is found to optimise global > accesses in general, this might become unnecessary. This will eventually happen one way or another, if only to shut up complaints from people who believe it's a major cause of slowdowns. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mcherm at mcherm.com Thu Apr 13 14:34:26 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 13 Apr 2006 05:34:26 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions Message-ID: <20060413053426.7y82alyyyyo0wcsg@login.werra.lunarpages.com> Guido writes: > the conventional way of treating __add__ > etc. has a different need for resolving conflicts than Phillip Eby's > idea of requiring strict dominance of the selected solution over all > other candidates. So now that you mention it I wonder if the whole > strict dominance requirement isn't a red herring? And perhaps we > should in general use "returns NotImplemented" as a signal to try the > next best candidate, strict dominance be damned... Maybe I'm getting confused here, but I disagree... as I look at the __add__()/__radd__() example it convinces me that Phillip's strict cominance requirement is very wise. I'll talk through my reasoning with what I consider a realistic use case and someone else can write in and tell me what obvious point I'm missing. Okay, Guido fires up the time machine and goes back to early Python and instead of introducing __add__()/__radd__() in the first place, he uses operator.add(a,b) with overloading / multiple-dispatch / generic-functions / whatever-we-call-it. So then the mpz library introduces arbitrary precision numbers based on the Gnu library. They create a type "mpz" and define the following: @operator.add.register(mpz, mpz) def add(mpz1, mpz2): ... # efficient mpz add @operator.add.register(mpz, object) def leftAdd(mpz1, x): ... # convert x and add, returning a mpz @operator.add.register(object, mpz) def rightAdd(x, mpz1): ... # convert x and add, returning a mpz These work properly and all is good. Meanwhile, somewhere else the NumPy folks (in this history they named it "NumPy" first then changed it to "Numeric" later... it's just this sort of thing that makes it so confusing to mess around with the time machine), are inventing some nifty arrays, which are able to be added to each other (if the sizes are appropriate) or to integers. They create an type named "array" and define the following: @operator.add.register(NumPy.array, NumPy.array) def add(array1, array2): ... # add component-wise or raise ShapeMismatchException @operator.add.register(NumPy.array, object) def leftAdd(array1, x): ... # add x to each component @operator.add.register(object, NumPy.array) def rightAdd(x, array1): ... # add x to each component Again, all is well. Everything a user wants to do has a clear dominance. ... Until the day when Suzy Programmer chooses to import *both* mpz AND NumPy. Actually, that still doesn't cause any problems until she chooses to add the two types, with this line: my_x = my_mpz + my_array THIS then raises an exception because there is no one dominant definition. To me, that seems like a Good Thing. It seems rather Pythonic that there is no attempt to guess what to do, either the programmer makes it very clear and indicates the correct behavior by registering a function, or they don't, and Python refuses to guess. "Explicit is better than Implicit", "In the Face of Ambiguity, Refuse to Guess". If Suzy really knows what she wants to do, she can register her own operator.add(), which shouldn't take more than a couple of lines -- I can't imagine that a couple of lines is too much of a burden to impose on someone trying to integrate two unrelated libraries! But in a more realistic situation, the NumPy folks realize that many of their users are doing scientific work and a noticable number make use of mpz. (The mpz folks are just wrapping an external library so they don't think of these things, but that's OK since only one of them needs to.) The NumPy folks add the following lines of code: try: import mpz @operator.add.register(NumPy.array, mpz.mpz) def leftAdd(array1, mpz1): ... # invoke mpz.rightAdd() on each component @operator.add.register(mpz.mpz, NumPy.array) def rightAdd(mpz1, array1): ... # invoke mpz.leftAdd() on each component except ImportError: pass And now Suzy can effortlessly integrate the libraries. I realize that it is *possible* for two different integrators to invent competing definitions of what to do when combining a mpz and NumPy.array, and it would be nice to avoid having the registry be global... but somehow I can't get _too_ excited about it -- somehow it seems like it wouldn't arise too often in practice. Perhaps there's a clever solution similar to Decimal contexts, involving use of a "with" statement... but it doesn't feel like a roadblock. And the inability of the two libraries to interface *automatically* in a protocol designed for adding numbers, which never quite expected either library feels like an advantage, not a weakness. -- Michael Chermside From martin at v.loewis.de Thu Apr 13 15:40:22 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu, 13 Apr 2006 15:40:22 +0200 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <20060413053426.7y82alyyyyo0wcsg@login.werra.lunarpages.com> References: <20060413053426.7y82alyyyyo0wcsg@login.werra.lunarpages.com> Message-ID: <443E54C6.6050600@v.loewis.de> > Guido writes: >> the conventional way of treating __add__ >> etc. has a different need for resolving conflicts than Phillip Eby's >> idea of requiring strict dominance of the selected solution over all >> other candidates. So now that you mention it I wonder if the whole >> strict dominance requirement isn't a red herring? And perhaps we >> should in general use "returns NotImplemented" as a signal to try the >> next best candidate, strict dominance be damned... Michael Chermside wrote: > my_x = my_mpz + my_array > > THIS then raises an exception because there is no one dominant > definition. Or, to make a long story short: there might not be a single next best candidate, but multiple, which are mutually equally-good: both mpz+object and object+array would match, and neither is better than the other. This is because they only form a partial order. Of course, it might be possible to impose a total order on top of it (e.g. giving the left-more operands higher precedence, or considering the order of registration). This all sound arbitrary, though. Regards, Martin From mcherm at mcherm.com Thu Apr 13 16:54:48 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 13 Apr 2006 07:54:48 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions Message-ID: <20060413075448.d5665pw6b34goco8@login.werra.lunarpages.com> Martin v. L?wis replies to my story: > Or, to make a long story short: there might not be a single next > best candidate, but multiple, which are mutually equally-good: > both mpz+object and object+array would match, and neither is > better than the other. > > This is because they only form a partial order. Of course, it > might be possible to impose a total order on top of it (e.g. > giving the left-more operands higher precedence, or considering > the order of registration). This all sound arbitrary, though. That's a really good way of looking at it. Our current __add__()/ __radd__() system prefers the left argument. A multi-dispatch function with the strict dominance rule refuses to prefer either, raising an exception unless you register something to tell it what to do. I believe that's a point in FAVOR of the multi-dispatch function approach. Adding multi-dispatch functions to Python is a BIG, DEEP change. Similar, perhaps, to adding descriptors, not similar to adding a "with" statement. That means great care should be exercised, but also that great things are possible. -- Michael Chermside From p.f.moore at gmail.com Thu Apr 13 17:04:47 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 13 Apr 2006 16:04:47 +0100 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <443E54C6.6050600@v.loewis.de> References: <20060413053426.7y82alyyyyo0wcsg@login.werra.lunarpages.com> <443E54C6.6050600@v.loewis.de> Message-ID: <79990c6b0604130804y42a28018o2bfb787af8b9c72c@mail.gmail.com> On 4/13/06, "Martin v. L?wis" wrote: > Michael Chermside wrote: > > my_x = my_mpz + my_array > > > > THIS then raises an exception because there is no one dominant > > definition. > > Or, to make a long story short: there might not be a single next > best candidate, but multiple, which are mutually equally-good: > both mpz+object and object+array would match, and neither is > better than the other. While this is true, I'd argue that mpz+object is over-general. After all, mpz+list is not expected to work, presumably, so the library authors are promising more than they actually deliver. The mpz library authors should have registered signatures for all applicable types in the core and standard library (quite possibly only int/long - I'm not sure that even mpz+float makes sense) and left all other cases for people using mpz with non-standard types. Of course, this exposes a potential usability issue - there may be a reasonable number of cases where an "obvious" derivation works. But there's no way of specifying this, so there's a temptation to specify an over-general signature, like mpz+object. Paul. From ianb at colorstudy.com Thu Apr 13 17:34:16 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 10:34:16 -0500 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <20060413053426.7y82alyyyyo0wcsg@login.werra.lunarpages.com> References: <20060413053426.7y82alyyyyo0wcsg@login.werra.lunarpages.com> Message-ID: <443E6F78.7020306@colorstudy.com> Michael Chermside wrote: > But in a more realistic situation, the NumPy folks realize that > many of their users are doing scientific work and a noticable > number make use of mpz. (The mpz folks are just wrapping an > external library so they don't think of these things, but that's > OK since only one of them needs to.) The NumPy folks add the > following lines of code: > > try: > import mpz > > @operator.add.register(NumPy.array, mpz.mpz) > def leftAdd(array1, mpz1): > ... # invoke mpz.rightAdd() on each component > > @operator.add.register(mpz.mpz, NumPy.array) > def rightAdd(mpz1, array1): > ... # invoke mpz.leftAdd() on each component > > except ImportError: > pass A niggling detail here, but with setuptools and package activation try:except ImportError: doesn't work very well for these things. If mpz is available through pkg_resources.require(), but not directly importable, this may fail until mpz is required. You don't have to be using pkg_resources.require() directly either to see this, though you'll only really see it when features are dynamically activated at runtime (instead of being activated entirely through requirements statically provided in your setup.py/entry_points.txt). And even if that wasn't a problem, it means mpz would be eagerly loaded along with NumPy even if it wasn't needed. Or if you had to manually import this operator.add-glue, then if you forget it is likely to be difficult to debug, because there's no *bad* code, just code that is missing. This is somewhat tangential to generic functions, but I guess happens whenever you have any kind of registry that is modified on import. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From steven.bethard at gmail.com Thu Apr 13 18:07:27 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 13 Apr 2006 10:07:27 -0600 Subject: [Python-3000] PEP 3002: Procedure for Backwards-Incompatible Changes Message-ID: This should be pretty similar to the last time I posted it, but now it's got an official PEP number. Let me know what you think! At the moment, I'm only posting this to the Python 3000 list. If you think I should be posting it to python-dev or python-list (as I would with other PEPs), let me know. PEP: 3002 Title: Procedure for Backwards-Incompatible Changes Version: $Revision: 45366 $ Last-Modified: $Date: 2006-04-13 07:36:24 -0600 (Thu, 13 Apr 2006) $ Author: Steven Bethard Status: Draft Type: Process Content-Type: text/x-rst Created: 03-Mar-2006 Post-History: 03-Mar-2006 Abstract ======== This PEP describes the procedure for changes to Python that are backwards-incompatible between the Python 2.X series and Python 3000. All such changes must be documented by an appropriate Python 3000 PEP and must be accompanied by code that can identify when pieces of Python 2.X code may be problematic in Python 3000. Rationale ========= Python 3000 will introduce a number of backwards-incompatible changes to Python, mainly to streamline the language and to remove some previous design mistakes. But Python 3000 is not intended to be a new and completely different language from the Python 2.X series, and it is expected that much of the Python user community will make the transition to Python 3000 when it becomes available. To encourage this transition, it is crucial to provide a clear and complete guide on how to upgrade Python 2.X code to Python 3000 code. Thus, for any backwards-incompatible change, two things are required: * An official Python Enhancement Proposal (PEP) * Code that can identify pieces of Python 2.X code that may be problematic in Python 3000 Python Enchancement Proposals ============================= Every backwards-incompatible change must be accompanied by a PEP. This PEP should follow the usual PEP guidelines and explain the purpose and reasoning behind the backwards incompatible change. In addition to the usual PEP sections, all PEPs proposing backwards-incompatible changes must include an additional section: Compatibility Issues. This section should describe what is backwards incompatible about the proposed change to Python, and the major sorts of breakage to be expected. While PEPs must still be evaluated on a case-by-case basis, a PEP may be inappropriate for Python 3000 if its Compatibility Issues section implies any of the following: * Most or all instances of a Python 2.X construct are incorrect in Python 3000, and most or all instances of the Python 3000 construct are incorrect in Python 2.X. So for example, changing the meaning of the for-loop else-clause from "executed when the loop was not broken out of" to "executed when the loop had zero iterations" would mean that all Python 2.X for-loop else-clauses would be broken, and there would be no way to use a for-loop else-clause in a Python-3000-appropriate manner. Thus a PEP for such an idea would likely be rejected. * Many instances of a Python 2.X construct are incorrect in Python 3000 and the PEP fails to demonstrate real-world use-cases for the changes. Backwards incompatible changes are allowed in Python 3000, but not to excess. A PEP that proposes backwards-incompatible changes should provide good examples of code that visibly benefits from the changes. PEP-writing is time-consuming, so when a number of backwards-incompatible changes are closely related, they should be proposed in the same PEP. Such PEPs will likely have longer Compatibility Issues sections however, since they must now describe the sorts of breakage expected from *all* the proposed changes. Identifying Problematic Code ============================ In addition to the PEP requirement, backwards incompatible changes to Python must also be accompanied by code that can identify pieces of Python 2.X code that may be problematic in Python 3.0. This PEP proposes to house this code in tools/scripts/python3warn.py. Thus PEPs for backwards incompatible changes should include a patch to this file that produces the appropriate warnings. Code in python3warn.py should be written to the latest version of Python 2.X (not Python 3000) so that Python 2.X users will be able to run the program without having Python 3000 installed. Currently, it seems too stringent to require that the code in python3warn.py identify all changes perfectly. Thus it is permissable if a backwards-incompatible PEP's python3warn.py code produces a number of false-positives (warning that a piece of code might be invalid in Python 3000 when it's actually still okay). However, false-negatives (not issuing a warning for code that will do the wrong thing in Python 3000) should be avoided whenever possible -- users of python3warn.py should be reasonably confident that they have been warned about the vast majority of incompatibilities. So for example, a PEP proposing that ``dict.items()`` be modified to return an iterator instead of a list might add code like the following to python3warn.py:: items_in_for = re.compile(r'for\s+\w+\s+in\s+\w+\.items\(\):') ... for i, line in enumerate(file_lines): ... if '.items()' in line and not items_in_for.search(line): message = 'dict.items() call may expect list at line %i' warnings.warn(message % i) This would issue a warning any time a ``.items()`` method was called and not immediately iterated over in a for-loop. Clearly this will issue a number of false-positive warnings (e.g. ``d2 = dict(d.items())``), but the number of false-negative warnings should be relatively low. Optional Extensions =================== Instead of the python3warn.py script, a branch of Python 3000 could be maintained that added warnings at all the appropriate points in the code-base. PEPs proposing backwards-incompatible changes would then provide patches to the Python-3000-warn branch instead of to python3warn.py. With such a branch, the warnings issued could be near-perfect and Python users could be confident that their code was correct Python 3000 code by first running it on the Python-3000-warn branch and fixing all the warnings. At the moment, however, this PEP opts for the weaker measure (python3warn.py) as it is expected that maintaining a Python-3000-warn branch will be too much of a time drain. References ========== TBD Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From jimjjewett at gmail.com Thu Apr 13 18:23:10 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 13 Apr 2006 12:23:10 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> <443D6E71.9090807@ieee.org> Message-ID: On 4/12/06, Guido van Rossum wrote: > On 4/12/06, Tim Hochberg wrote: > > What would happen if 'a+b' was just syntactic sugar for > > 'operator.add(a,b)', where operator.add was a generic > > function, instead of the current magic dance involving > > __add__ and __radd__. Conceptually, I had already thought of it that way in existing python, and had to spend a few days thinking about why it isn't quite true. As I see it, (1) Returning NotImplemented is a clumsier way of restricting the signature for which you want to register. (2) The leftovers of coerce are arguably extra, but could certainly be part of the generic default. (3) 3rd parties could register new add signatures; the equivalent of modifying __add__ and __radd__ methods. Only number (3) is really a change, and realistically, it would be a (cleaner) version of letting you monkeypatch even builtins. Now that Guido is thinking about it ... > What if, instead of adding registries mapping types to functions left > and right (in pickle, copy, pprint, who knows where else), we had a > convention of adding __foo_bar__ methods directly to the class dict? If you can only add (not replace), then the first user wins. If you can replace, that is a gaping security issue when someone replaces part of dict. (Which would be a quite reasonable thing to do, for debugging or read-only proxies, or ...) -jJ From ianb at colorstudy.com Thu Apr 13 18:38:01 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 11:38:01 -0500 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: References: <44344432.5030704@livinglogic.de> <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> <443D6E71.9090807@ieee.org> Message-ID: <443E7E69.9000900@colorstudy.com> Guido van Rossum wrote: > Around the same time I also had this scary thought: > > What if, instead of adding registries mapping types to functions left > and right (in pickle, copy, pprint, who knows where else), we had a > convention of adding __foo_bar__ methods directly to the class dict? > This would require giving up the effective immutability of built-in > types, which has widespread repercussions (mostly in the area of > multiple interpreters) but nevertheless if this was officially > endorsed, we could clean up a lot of type registries... (I notice that > I haven't really found any use for *multiple* dispatch in my few > explorations of @overloaded; that is, until the operator.add idea came > along.) One advantage of using a function as the registry is that the function lives in a particular place. A method only has a name, there is no namespace for it. In addition to possible name collisions, functions are possible to backtrack. If you want to know what some generic function specialization is about, you look at the original function definition, which you can find by following the imports. If you want to know what a particular magic method is about, there's no particular technique to accomplish that. >>I suspect this falls into the "wild ideas' category, but it's an >>interesting thought experiment anyway. Perhaps with relevance to other >>uses of generic functions. > > > Definitely wild. Reader beware. Now is the time to generate lots of > wild ideas and let them sink in. If it still seems a good idea 3 > months from now we may select it for a code experiment (as opposed to > a thought experiment). BTW I want to encourage lots of code > experiments for Python 3.0 -- though not directly in the p3yk (sic) > branch. And I could have sworn you asked for less wild ideas just the other day... -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Apr 13 18:41:43 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 11:41:43 -0500 Subject: [Python-3000] Removing 'self' from method definitions Message-ID: <443E7F47.9060308@colorstudy.com> OK, so first Guido says he wants people to speculate less, then he says he wants them to speculate more. Who knows. So, now I speculate more. I propose that the self argument be removed from method definitions. It would not be removed from other places, e.g., you'd still use "self.foo()", but the definition for that method would be "def foo():". One data point: after years of doing this, I still make this mistake nearly every day: def doit(self): blah blah x = result of blah blah more stuff Then refactor to: def doit(self): x = self.get_x(self) more stuff def get_x(self): return blah blah Why do I keep doing this? I don't know, mabye it's because I usually copy the "blah blah" to a new function definition, write the signature there, then mentally copy that signature back into the original function. I'm not the only one; this is one of the most common criticisms of Python, probably on par with significant whitespace. People might be wrong on whitespace, but not on this. What are the advantages of the current behavior? * You can name the first argument whatever you want. I could say something snarky about this, but that's probably not necessary. * Functions are functions wherever you define them; only access determines if a function is a "method". * Scoping is explicit. It becomes hard to determine where "self" is defined if any function could be invoked as a method. * Decorator wrappers work more easily on both functions and methods when the wrapping is just passing through its arguments. If the decorator has to understand the arguments in any then it becomes more difficult. I propose: 'self' and 'cls' implicit arguments. Their presence as the first argument to a function or method will be ignored, for backward compatibility. Problems: This wouldn't work: def get_writer(self): def write(item): self.write_item(item) return write Right now you can tell that 'self' comes from the enclosing function; if self is an implicit argument, then this won't work anymore. In Javascript they solve this same issue with the pattern: def get_writer(): this = self def write(item): this.write_item(item) return write This is admittedly lame, but I can't think of any resolution. I think this should still work fine: def some_method(): return self.foo()+1 SomeClass.some_method = some_method However, some_method() could not be called outside of a method-like context (if it was you'd get a NameError or subclass thereof). At the least you could still do some_method.__get__(self_like_object)(); maybe a nicer method could be provided as well. The traditional way of calling superclass methods and getting an unbound method should still work. It just won't be symmetric with the way the functions are defined. Since this is not the typical way methods are called, it seems reasonable. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From tim.peters at gmail.com Thu Apr 13 18:51:39 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 13 Apr 2006 12:51:39 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <443E7E69.9000900@colorstudy.com> References: <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> <443D6E71.9090807@ieee.org> <443E7E69.9000900@colorstudy.com> Message-ID: <1f7befae0604130951x5fe99efbg5514222256207b41@mail.gmail.com> [Guido] >> ... >> Definitely wild. Reader beware. Now is the time to generate lots of >> wild ideas and let them sink in. If it still seems a good idea 3 >> months from now we may select it for a code experiment (as opposed to >> a thought experiment). BTW I want to encourage lots of code >> experiments for Python 3.0 -- though not directly in the p3yk (sic) >> branch. [Ian Bicking] > And I could have sworn you asked for less wild ideas just the other day... Ah, good. For a while, I was afraid that someone else may usurp my mysterious role as Guido-channeler for Py3K. But if seeming contradictions this simple throw you, you're sure no threat there ;-) the-path-to-perfect-pythonicity-is-paved-with-pythonic-wild-ideas-ly y'rs - tim From tim.hochberg at ieee.org Thu Apr 13 19:03:31 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 13 Apr 2006 10:03:31 -0700 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <443E54C6.6050600@v.loewis.de> References: <20060413053426.7y82alyyyyo0wcsg@login.werra.lunarpages.com> <443E54C6.6050600@v.loewis.de> Message-ID: Let me update my original thought experiment in a way that (I think) preserves the current behaviour while also requiring strict dominance. Let's imagine for a second that we have two new types at the top of the hierarchy. I'll call them Ultimate and Penultimate. So the inheritance tree goes like: Ultimate->Penultimate->type->... It's weird to have types above type, and Ultimate and Penultimate wouldn't necessarily have to be real types, they could just be markers that generic treats like types. Still, for the moment let's pretend they're types. Now, class Weeble: #... def __add__(a, b): # ... def __radd__(a, b): # ... gets automagically translated into something like: class Weeble: #.... @operator.add.register(Weeble, Penultimate) def add(a, b): #... @operator.add.register(Ultimate, Weeble) def radd(b, a): #... Now add will be tried before radd naturally and we can still preserve strict dominance. There are issues with this: Ultimate and Penultimate are weird, it doesn't extend naturally to more than two args and we still don't know how to treat NotImplemented. Plus more that I'm not seeing right now I'm sure. One possibility is that instead of superclasses we have a marker object that generic treats specially; let's call it Any. The constructor for this object takes an argument that determines the order in which it is tried. Thus Penultimate becomes Any(0) and Ultimate becomes Any(1). This generalizes to any number of arguments and removes the extra classes above type in exchanges for extra complexity inside of generic. To support NotImplemented you need some way to get the next method. And before you do that, you need to agree on what next method means. A procedure I floated on Guido's blog to get the next method is to use the same machinery used for finding the primary method, but to exclude any methods already tried. I believe that this works in the simple case above although I'm unsure about its utility in more complex cases. Regards, -tim From jack at performancedrivers.com Thu Apr 13 19:08:00 2006 From: jack at performancedrivers.com (Jack Diederich) Date: Thu, 13 Apr 2006 13:08:00 -0400 Subject: [Python-3000] PEP 3002: Procedure for Backwards-Incompatible Changes In-Reply-To: References: Message-ID: <20060413170759.GO5760@performancedrivers.com> On Thu, Apr 13, 2006 at 10:07:27AM -0600, Steven Bethard wrote: > This should be pretty similar to the last time I posted it, but now > it's got an official PEP number. Let me know what you think! > > Identifying Problematic Code > ============================ > > This PEP proposes to house this code in tools/scripts/python3warn.py. > Thus PEPs for backwards incompatible changes should include a patch to > this file that produces the appropriate warnings. Code in > python3warn.py should be written to the latest version of Python 2.X > (not Python 3000) so that Python 2.X users will be able to run the > program without having Python 3000 installed. > > So for example, a PEP proposing that ``dict.items()`` be modified to > return an iterator instead of a list might add code like the following > to python3warn.py:: > > items_in_for = re.compile(r'for\s+\w+\s+in\s+\w+\.items\(\):') > ... > for i, line in enumerate(file_lines): > ... > if '.items()' in line and not items_in_for.search(line): > message = 'dict.items() call may expect list at line %i' > warnings.warn(message % i) Could we mandate that all warnings use warnings.showwarning() and provide the right filename and line number? That would make it very easy to run the script in an editor and step through all the lines that might need changes. Any editor with a unix tradition should be able to handle the format 'filename:line number:message' - I don't know if windows has any tradition here. -Jack From pje at telecommunity.com Thu Apr 13 19:13:01 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 13 Apr 2006 13:13:01 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: Message-ID: <5.1.1.6.0.20060413130459.01fded50@mail.telecommunity.com> At 10:34 AM 4/13/2006 -0500, Ian Bicking wrote: >And even if that wasn't a problem, it means mpz would be eagerly loaded >along with NumPy even if it wasn't needed. Or if you had to manually >import this operator.add-glue, then if you forget it is likely to be >difficult to debug, because there's no *bad* code, just code that is >missing. PEAK addresses this problem with a decorator: @whenImported('mpz') def setup_for_ mpz(mpz): @operator.add.register(...) def ... The 'whenImported' function checks to see if mpz is already present in sys.modules. If so, it calls the function immediately, passing in the named module. If not, it puts an object in sys.modules whose __getattribute__ tries to actually import the module and then calls back any registered functions, so that anybody else importing mpz will get the real module. I've actually been using this for a few years now, as the eager-vs.-lazy problem is independent of setuptools and package activation. It comes up any time you want to offer support for something without creating a dependency on it. From gustavo at niemeyer.net Thu Apr 13 19:13:01 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Thu, 13 Apr 2006 14:13:01 -0300 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443E7F47.9060308@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> Message-ID: <20060413171301.GA9869@localhost.localdomain> (...) > The traditional way of calling superclass methods and getting an unbound > method should still work. It just won't be symmetric with the way the > functions are defined. Since this is not the typical way methods are > called, it seems reasonable. Using "obj.some_method(blah)" when something is defined as "def some_method(self, blah):" is easy to understand since someone may be filling the "self" parameter for you (partial functions anyone?), but using "Class.some_method(self, blah)" when something is defined as "def some_method(blah):" looks bizarre. I've programmed for a long time in other languages where "self" is implicit, and have also programmed in Python for many years using the explicit "self". The explicit "self" in Python completely matches with the features of the language, IMO. Making it implicit would be a huge mistake, and one that would most probably seem absurd if you have used Python for a while and haven't had contact with other languages that introduce a very magic "self" in some contexts of your program. -- Gustavo Niemeyer http://niemeyer.net From steven.bethard at gmail.com Thu Apr 13 19:23:50 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 13 Apr 2006 11:23:50 -0600 Subject: [Python-3000] PEP 3002: Procedure for Backwards-Incompatible Changes In-Reply-To: <20060413170759.GO5760@performancedrivers.com> References: <20060413170759.GO5760@performancedrivers.com> Message-ID: On 4/13/06, Jack Diederich wrote: > On Thu, Apr 13, 2006 at 10:07:27AM -0600, Steven Bethard wrote: > > This should be pretty similar to the last time I posted it, but now > > it's got an official PEP number. Let me know what you think! > > > > Identifying Problematic Code > > ============================ > > > > This PEP proposes to house this code in tools/scripts/python3warn.py. > > Thus PEPs for backwards incompatible changes should include a patch to > > this file that produces the appropriate warnings. Code in > > python3warn.py should be written to the latest version of Python 2.X > > (not Python 3000) so that Python 2.X users will be able to run the > > program without having Python 3000 installed. > > > > So for example, a PEP proposing that ``dict.items()`` be modified to > > return an iterator instead of a list might add code like the following > > to python3warn.py:: > > > > items_in_for = re.compile(r'for\s+\w+\s+in\s+\w+\.items\(\):') > > ... > > for i, line in enumerate(file_lines): > > ... > > if '.items()' in line and not items_in_for.search(line): > > message = 'dict.items() call may expect list at line %i' > > warnings.warn(message % i) > > Could we mandate that all warnings use warnings.showwarning() and > provide the right filename and line number? Yeah, that sounds like a good idea. I'll update the PEP to require that. Steve -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From ianb at colorstudy.com Thu Apr 13 19:59:47 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 12:59:47 -0500 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <20060413171301.GA9869@localhost.localdomain> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> Message-ID: <443E9193.7000908@colorstudy.com> Gustavo Niemeyer wrote: >>The traditional way of calling superclass methods and getting an unbound >>method should still work. It just won't be symmetric with the way the >>functions are defined. Since this is not the typical way methods are >>called, it seems reasonable. > > > Using "obj.some_method(blah)" when something is defined as > "def some_method(self, blah):" is easy to understand since someone > may be filling the "self" parameter for you (partial functions > anyone?), but using "Class.some_method(self, blah)" when something > is defined as "def some_method(blah):" looks bizarre. Class.some_method(self, blah) seems like a corner case. How often do you do that? If it is calling a superclass method, then super(Class, self).some_method() should be used (or even better, a more pleasant syntax for super -- which seems more feasible to implement given out-of-bad communication into the function about how it is being called). > I've programmed for a long time in other languages where "self" is > implicit, and have also programmed in Python for many years > using the explicit "self". The explicit "self" in Python completely > matches with the features of the language, IMO. Making it implicit > would be a huge mistake, and one that would most probably seem > absurd if you have used Python for a while and haven't had contact > with other languages that introduce a very magic "self" in some > contexts of your program. I have been using Python for a while, and it does not feel absurd to me. Note that I am not suggesting that self be removed anywhere but from the signature of the method definition. My own interpretation of self-related criticism is that people get bothered with all the "self"s from their experience with other languages, but the self in the signature (and the miscount of arguments in TypeError exceptions) keeps them from feeling completely as ease with the new way of working. They project that unease onto all use of "self", and then we dismiss them for just trying to make Python into another language when they propose removing many of the uses of self. That is, their interpretation of *why* they are dissatisfied may not be entirely correct, but underlying that dissatisfaction are real issues. I think this is a common occurance when studying usability -- users can identify problem areas, but when they reflect on what exactly those problems are they often misinterpret their experience. Anyway, maybe it's possible to experiment with this right now with some metaclass cleverness. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Apr 13 20:01:04 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 13:01:04 -0500 Subject: [Python-3000] Making strings non-iterable Message-ID: <443E91E0.5020304@colorstudy.com> I propose that strings (unicode/text) shouldn't be iterable. Seeing this:
  • i
  • t
  • e
  • m
  • 1
a few too many times... it's annoying. Instead, I propose that strings get a list-like view on their characters. Oh synergy! Thus you would do: for c in a_string.chars(): print c This view would have the full complement of list methods (like .count(), .index(), etc), and would not have string methods (like .upper()). Iterating over strings causes frequent hard bugs (bad data, as opposed to exceptions which make for easy bugs), as the bug can manifest itself far from its origination. Also strings aren't containers. Because Python has no characters, only strings, as a result strings look like they contain strings, and those strings in turn contain themselves. It just doesn't make sense. And it is because a string and the characters it contains are interchangeable (they are both strings) that the resulting bugs can persist without exceptions. Should bytes be iterable as well? Because bytes (the container) and integers are not interchangeable, the problems that occur with strings seem much less likely, and the container-like nature of bytes is clearer. So I don't propose this effect bytes in any way. Questions: * .chars() doesn't return characters; should it be named something else? * Should it be a method that is called? dict.keys() has a legacy, but this does not. There is presumably very little overhead to getting this view. However, symmetry with the only other views we are considering (dictionary views) would indicate it should be a method. Also, there are no attributes on strings currently. * Are there other views on strings? Can string->byte encoding be usefully seen as a view in some cases? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From steven.bethard at gmail.com Thu Apr 13 20:16:27 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 13 Apr 2006 12:16:27 -0600 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <443E91E0.5020304@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> Message-ID: On 4/13/06, Ian Bicking wrote: > I propose that strings (unicode/text) shouldn't be iterable. Seeing this: > >
    >
  • i >
  • t >
  • e >
  • m >
  • >
  • 1 >
> > a few too many times... it's annoying. Instead, I propose that strings > get a list-like view on their characters. +1. I don't know how many times I wished I'd gotten an error here instead. I do need to iterate over the characters of a string occasionally, but not often enough to worry about typing a few extra characters. > * .chars() doesn't return characters; should it be named something else? I say leave this one up to Guido. I'm happy with chars because it does return characters, as long as we define characters as length-one-strings. ;-) > * Should it be a method that is called? I think this is actually a general question about views. Should views be created with methods or properties? I'm inclined towards the former, but I think this discussion should be carried out in a different thread. > * Are there other views on strings? Can string->byte encoding be > usefully seen as a view in some cases? I refuse to answer this question on the grounds that it may incriminate me. ;-) STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From jcarlson at uci.edu Thu Apr 13 20:36:43 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 13 Apr 2006 11:36:43 -0700 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443E7F47.9060308@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> Message-ID: <20060413112429.A7E8.JCARLSON@uci.edu> Ian Bicking wrote: > > OK, so first Guido says he wants people to speculate less, then he says > he wants them to speculate more. Who knows. So, now I speculate more. > > I propose that the self argument be removed from method definitions. It > would not be removed from other places, e.g., you'd still use > "self.foo()", but the definition for that method would be "def foo():". -1 Why? Because right now the various methods on a class have obvious signature-specific clues as to what they are: instance method (has a leading self), class method (has a leading cls or klass generally), static method (has no leading self, cls, etc.) . Removing self (and possibly even cls if one goes a step farther) ambiguates what kind of method a function definition is, from a signature standpoint. As was said in the decorator discussion; signatures are important and should be grepable (that along with readability is why we didn't see decorators _inside_ the signature). By removing the _explicit_ self, we make instance methods generally undifferentiatable (is that a word?) from a grep standpoint. - Josiah From jimjjewett at gmail.com Thu Apr 13 20:35:50 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 13 Apr 2006 14:35:50 -0400 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <443E91E0.5020304@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> Message-ID: On 4/13/06, Ian Bicking wrote: > I propose that strings (unicode/text) shouldn't be iterable. I understand that there are use cases, but I can't remember seeing one in real life. I have seen many uses for the container version x in string and of course many good uses of iteration over the results of a string method (such as string.split()), but no good uses for iterating over a string directly for x in string A special method to get a tuple (or immutable list) view would be a good thing, as it would document that I was doing so intentionally. (I wouldn't want to give up slicing, though, which might make the no-iteration trickier.) -jJ From ianb at colorstudy.com Thu Apr 13 20:39:43 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 13:39:43 -0500 Subject: [Python-3000] Is reload() feasible? Message-ID: <443E9AEF.7040902@colorstudy.com> In another thread ("Module Initialization and finalization") reload() has come up, and some suggestions to improve the reloadability of modules. I would like to ask: is reloading feasible at all? Right now reload() works poorly. The builtin reload() can be improved some, but there are problems that cannot ever be resolved in general. For instance, generic functions involve a central registry; reloading a module will reload any registrations. This can probably be handled, though I suspect special code for this condition will be required. Reloading the model function (damn, we need more names for this stuff -- the original function that contains the the registry) will lose all the registrations, and I don't see a good way to resolve that. Reloading a module that removes a registration that was previously there is also hard to handle. This is just for one example, these kinds of problems are all over the place. Notably, Zope 3 does not include reloading (though it was included in Zope 2, in a more general form than just what reload() does). It was removed because of these corner cases where it doesn't work, and because when these corner cases happen the results are inscrutable and frustrating errors. To focus energy I think we should give up on reloading. At the same time, long running processes that require manual restarts just *feel* really uncomfortable. It's like a compile step. So the underlying problem shouldn't be ignored. I would propose that light, disposable interpreters are a better way of handling the problem. When a file changes, you throw away the interpreter and start over. Several projects are doing this now, but making it more standardized would be very helpful to improving the usability of the technique. For interactive environments some of the same issues apply, though the resolution is more difficult. Can you safely re-evaluate the entire history and get to the same state that you started at? You can imagine using restart() at the interactive prompt, and having the process die, restart, run the entire history (showing output?) and leaving you with a potentially equivalent environment. Replaying history also might fail, or have side effects that cannot or should not be repeated. So, a restart strategy opens up lots of questions; but I think abandoning reload() would better bring those questions into focus. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Apr 13 20:43:25 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 13:43:25 -0500 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <20060413112429.A7E8.JCARLSON@uci.edu> References: <443E7F47.9060308@colorstudy.com> <20060413112429.A7E8.JCARLSON@uci.edu> Message-ID: <443E9BCD.6050405@colorstudy.com> Josiah Carlson wrote: > Ian Bicking wrote: > >>OK, so first Guido says he wants people to speculate less, then he says >>he wants them to speculate more. Who knows. So, now I speculate more. >> >>I propose that the self argument be removed from method definitions. It >>would not be removed from other places, e.g., you'd still use >>"self.foo()", but the definition for that method would be "def foo():". > > > -1 Why? Because right now the various methods on a class have obvious > signature-specific clues as to what they are: instance method (has a > leading self), class method (has a leading cls or klass generally), > static method (has no leading self, cls, etc.) . Removing self (and > possibly even cls if one goes a step farther) ambiguates what kind of > method a function definition is, from a signature standpoint. As was > said in the decorator discussion; signatures are important and should be > grepable (that along with readability is why we didn't see decorators > _inside_ the signature). By removing the _explicit_ self, we make > instance methods generally undifferentiatable (is that a word?) from a > grep standpoint. Because all kinds of methods live in the same namespace and are *called* in the same way, the name alone is sufficient to identify the method. That is, searching for "def foo" will find instance, class, and static methods, and there can't be both a class and instance method with the same name on a class. "x.foo()" could be an instance, class, or static method, so an inclusive search is called for anyway. Actually reading the method in context, the decorator makes it clear what kind of method is being written; @classmethod and the cls argument are expressing the same thing. So outside of grepping it doesn't seem that big of an issue. Ideally introspective tools like pydoc would indicate what kind of function you were looking at, as that would be identifiable at runtime. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From rasky at develer.com Thu Apr 13 20:51:02 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 13 Apr 2006 20:51:02 +0200 Subject: [Python-3000] Is reload() feasible? References: <443E9AEF.7040902@colorstudy.com> Message-ID: <08b201c65f2b$361adb40$bf03030a@trilan> Ian Bicking wrote: > I would like to ask: is reloading feasible at all? Right now reload() > works poorly. The builtin reload() can be improved some, but there are > problems that cannot ever be resolved in general. Sure, but there are many simple cases where reload() does its job pretty well. I had a stupid program with more stupid plugins, and it was doing its job remarkably. -- Giovanni Bajo From jimjjewett at gmail.com Thu Apr 13 20:54:11 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 13 Apr 2006 14:54:11 -0400 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443E9193.7000908@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> Message-ID: (Cc to python dev, as my question is really about 2.x) On 4/13/06, Ian Bicking wrote: > ... the self in the signature (and the miscount of arguments > in TypeError exceptions) ... Even in the 2.x line, the TypeError messages should be improved. >>> # No syntax errors when creating m() >>> class C: def m(): pass but the method can't actually be called, and won't quite say why. >>> # This message is good >>> C.m() Traceback (most recent call last): File "", line 1, in -toplevel- C.m() TypeError: unbound method m() must be called with C instance as first argument (got nothing instead) but the obvious fix of using an instance is just confusing >>> C().m() Traceback (most recent call last): File "", line 1, in -toplevel- C().m() TypeError: m() takes no arguments (1 given) Could it at least say something like "(1 given, including self)"? I suppose the argument might be named something other than self, particularly in C code, but ... that strikes me as a smaller problem than the mysteriously appearing invisible argument. -jJ From rasky at develer.com Thu Apr 13 20:57:02 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 13 Apr 2006 20:57:02 +0200 Subject: [Python-3000] Making strings non-iterable References: <443E91E0.5020304@colorstudy.com> Message-ID: <08c601c65f2c$0cd95fd0$bf03030a@trilan> Ian Bicking wrote: > I propose that strings (unicode/text) shouldn't be iterable. Seeing this: > >
    >
  • i >
  • t >
  • e >
  • m >
  • >
  • 1 >
> > a few too many times... it's annoying. Instead, I propose that strings > get a list-like view on their characters. Oh synergy! > > Thus you would do: > > for c in a_string.chars(): > print c Would this remove __getitem__? I believe I wouldn't want to give up s[0] as "the first character in s". That would make a string a thing where you can use [3] to access the third character, but still not iterable. -- Giovanni Bajo From barry at python.org Thu Apr 13 21:04:20 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 13 Apr 2006 15:04:20 -0400 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443E9193.7000908@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> Message-ID: <1144955060.7368.23.camel@resist.wooz.org> -1 on removing self from method defs. I always make the opposite mistake from Ian when moving from Python to C++ or Java. :) On Thu, 2006-04-13 at 12:59 -0500, Ian Bicking wrote: > My own interpretation of self-related criticism is that people get > bothered with all the "self"s from their experience with other > languages, but the self in the signature (and the miscount of arguments > in TypeError exceptions) I'd rather fix this nit than remove self. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060413/a2428816/attachment.pgp From ianb at colorstudy.com Thu Apr 13 21:06:29 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 14:06:29 -0500 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <08c601c65f2c$0cd95fd0$bf03030a@trilan> References: <443E91E0.5020304@colorstudy.com> <08c601c65f2c$0cd95fd0$bf03030a@trilan> Message-ID: <443EA135.1040506@colorstudy.com> Giovanni Bajo wrote: >>Thus you would do: >> >> for c in a_string.chars(): >> print c > > > Would this remove __getitem__? I believe I wouldn't want to give up s[0] as > "the first character in s". That would make a string a thing where you can > use [3] to access the third character, but still not iterable. Definitely __getitem__ should stay, there's nothing wrong with that part. str.__iter__ would raise NotImplemented. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From s.percivall at chello.se Thu Apr 13 21:07:45 2006 From: s.percivall at chello.se (Simon Percivall) Date: Thu, 13 Apr 2006 21:07:45 +0200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <443E91E0.5020304@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> Message-ID: On 13 apr 2006, at 20.01, Ian Bicking wrote: > I propose that strings (unicode/text) shouldn't be iterable. > Seeing this: > >
    >
  • i >
  • t >
  • e >
  • m >
  • >
  • 1 >
> > a few too many times... it's annoying. Instead, I propose that > strings > get a list-like view on their characters. Oh synergy! > > Thus you would do: > > for c in a_string.chars(): > print c The best scenario would be if strings were like they are now, except that to iterate you'd have to write iter(strobj). This would obviously require some changes to how iterability is determined, though. //Simon From barry at python.org Thu Apr 13 21:09:21 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 13 Apr 2006 15:09:21 -0400 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <443E91E0.5020304@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> Message-ID: <1144955361.7368.29.camel@resist.wooz.org> On Thu, 2006-04-13 at 13:01 -0500, Ian Bicking wrote: > I propose that strings (unicode/text) shouldn't be iterable. +1. I think I've mentioned before that we have a particular choke point that may get scalars or sequences, but that everything coming out of that choke point is guaranteed to be a sequence. You can use iter() on the thing, but you have to typecheck that it's not a basestring first. > Should bytes be iterable as well? Because bytes (the container) and > integers are not interchangeable, the problems that occur with strings > seem much less likely, and the container-like nature of bytes is > clearer. So I don't propose this effect bytes in any way. That seems right. bytes probably won't suffer from the Python String Photonic Effect (that they're both waves and particles, er, I mean sequences and scalars). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060413/9b1b487b/attachment.pgp From ianb at colorstudy.com Thu Apr 13 21:17:39 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 14:17:39 -0500 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443E7F47.9060308@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> Message-ID: <443EA3D3.4090407@colorstudy.com> Ian Bicking wrote: > I propose: 'self' and 'cls' implicit arguments. Their presence as the > first argument to a function or method will be ignored, for backward > compatibility. It occurs to me that some of the problems with doing this involve the fact that you can't know how if a function is a function or a method; adding an implicit self argument that is injected only when the function is called causes some potential problems. Maybe another way to implement this would be for the metaclass (type) to change functions when they were added to the class. Then classes wouldn't just contain functions with a wrapper applied when an attribute is accessed; instead they would flip a switch on the function that turns it into a method. This seems like it removes some of the abiguity of an implicit argument. This seems like it might be possible to implement in Python right now, so perhaps an experiment along those lines would be useful to consider the full extent of what such a change. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From mcherm at mcherm.com Thu Apr 13 21:58:31 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 13 Apr 2006 12:58:31 -0700 Subject: [Python-3000] Removing 'self' from method definitions Message-ID: <20060413125831.8lonimb5yvwggwcw@login.werra.lunarpages.com> Ian Bicking writes: > I propose that the self argument be removed from method definitions. This is not a new proposal. The fundamental problem is a perceived mismatch between the argument list used for calling and the argument list used in defining the method. Of the many times that it has been proposed, the best syntax I have seen was as follows: def self.my_method(arg1, arg2): # body goes here In other words, make the definition look like the invocation. In this proposal "self" is not magical, it is simply declared in a slightly different location. The syntax would be used by sensible people only when defining methods, but there would be no such rule enforced by the interpreter, nor would the existing syntax be broken or prohibited. To be abundantly specific, the proposal is that if the next two tokens after a "def" were "self" and "." then the result of compiling such code would be exactly the same as if they were not present but "self" and "," were inserted immediately after the opening parenthesis. Whether to allow any identifier or only the particular identifier "self" is an interesting question that I would leave for Guido. Personally, I favor doing this. I think that making declarations look similar to invocations is very useful. -- Michael Chermside From jack at performancedrivers.com Thu Apr 13 22:13:37 2006 From: jack at performancedrivers.com (Jack Diederich) Date: Thu, 13 Apr 2006 16:13:37 -0400 Subject: [Python-3000] Is reload() feasible? In-Reply-To: <443E9AEF.7040902@colorstudy.com> References: <443E9AEF.7040902@colorstudy.com> Message-ID: <20060413201337.GQ5760@performancedrivers.com> On Thu, Apr 13, 2006 at 01:39:43PM -0500, Ian Bicking wrote: > In another thread ("Module Initialization and finalization") reload() > has come up, and some suggestions to improve the reloadability of modules. > > I would like to ask: is reloading feasible at all? Right now reload() > works poorly. [snip] > To focus energy I think we should give up on reloading. At the same > time, long running processes that require manual restarts just *feel* > really uncomfortable. It's like a compile step. So the underlying > problem shouldn't be ignored. +1 reload() is OK for the interactive shell but worthless in real situations (e.g. anything that uses classes or custom exceptions). > I would propose that light, disposable interpreters are a better way of > handling the problem. When a file changes, you throw away the > interpreter and start over. Several projects are doing this now, but > making it more standardized would be very helpful to improving the > usability of the technique. A standard way to hide objects across interpreter restarts would be great. For anything with an open socket if you could hide it from GC, serialize your current state (if the app is stateful) and then start fresh with a new interpreter, and start using the socket again. Will an object that has been PyObject_GC_UnTrack() survive across a Py_Finalize()? If so a hidey-hole for objects would be fairly easy to implement. Stateful applications would still have to serialize most of their state and reload it across interpreters. Trying to hide instances of python-defined objects in the hidey-hole has all the same problems as reload(). > For interactive environments some of the same issues apply, though the > resolution is more difficult. Can you safely re-evaluate the entire > history and get to the same state that you started at? You can imagine > using restart() at the interactive prompt, and having the process die, > restart, run the entire history (showing output?) and leaving you with a > potentially equivalent environment. Replaying history also might fail, > or have side effects that cannot or should not be repeated. So, a > restart strategy opens up lots of questions; but I think abandoning > reload() would better bring those questions into focus. Some apps might not have internal state that easily lends itself to being serialized and deserialized but those same apps don't work with reload either. Mostly stateless apps like web servers would have an easy time saving and restoring what little state they have. The name "hidey-hole" could use a bit of work, though. -Jack From thomas at python.org Thu Apr 13 22:40:47 2006 From: thomas at python.org (Thomas Wouters) Date: Thu, 13 Apr 2006 22:40:47 +0200 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443E7F47.9060308@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> Message-ID: <9e804ac0604131340g74b3ec86s69534794b087f0b5@mail.gmail.com> On 4/13/06, Ian Bicking wrote: > I propose that the self argument be removed from method definitions. It > would not be removed from other places, e.g., you'd still use > "self.foo()", but the definition for that method would be "def foo():". Guido already shot this idea down. (It's one of the items in the PEP-13 draft Brett put in my mailbox.) You're free to make a PEP, of course, especially if it covers every pro and con. But you'll have to work hard, very hard, to convince Guido (and me, and many others, but we don't count.) If you are going to make a real PEP, don't forget to ask for feedback on python-list a few times. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060413/dd356ddb/attachment-0001.htm From pje at telecommunity.com Thu Apr 13 23:19:06 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 13 Apr 2006 17:19:06 -0400 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: Message-ID: <5.1.1.6.0.20060413170800.00a04748@mail.telecommunity.com> At 12:58 PM 4/13/2006 -0700, Michael Chermside wrote: >To be abundantly specific, the proposal is that if the next two tokens >after a "def" were "self" and "." then the result of compiling such >code would be exactly the same as if they were not present but "self" >and "," were inserted immediately after the opening parenthesis. >Whether to allow any identifier or only the particular identifier >"self" is an interesting question that I would leave for Guido. > >Personally, I favor doing this. I think that making declarations look >similar to invocations is very useful. You could make them look even closer if it worked like this: def .aMethod(arg1, arg2): return .otherMethod(arg1*2+arg2) In other words, 'self' here is uniformly replaced by an empty string. :) Interestingly, I believe this syntax could get everybody off our backs about explicit self, as although it is still explicit, it's in an implicit way. :) Of course, for this to really work properly, you would have to be able to use a bare '.' to mean the hidden argument, for cases like "getattr(.,foo)" and ".[27]". From ianb at colorstudy.com Thu Apr 13 23:23:24 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 13 Apr 2006 16:23:24 -0500 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <5.1.1.6.0.20060413170800.00a04748@mail.telecommunity.com> References: <5.1.1.6.0.20060413170800.00a04748@mail.telecommunity.com> Message-ID: <443EC14C.8010003@colorstudy.com> Phillip J. Eby wrote: > At 12:58 PM 4/13/2006 -0700, Michael Chermside wrote: > >>To be abundantly specific, the proposal is that if the next two tokens >>after a "def" were "self" and "." then the result of compiling such >>code would be exactly the same as if they were not present but "self" >>and "," were inserted immediately after the opening parenthesis. >>Whether to allow any identifier or only the particular identifier >>"self" is an interesting question that I would leave for Guido. >> >>Personally, I favor doing this. I think that making declarations look >>similar to invocations is very useful. > > > You could make them look even closer if it worked like this: > > def .aMethod(arg1, arg2): > return .otherMethod(arg1*2+arg2) When talking about a method people often use ".method()", not just "method()", to distinguish a method-on-some-object (which from the outside can't be called "self") from a function. This syncs with that. OTOH, while the definition doesn't bother me, ".otherMethod(arg1*2+arg2)" looks pretty scary to me. Anyway, both this and "def self.aMethod(...)" solve the problems of an implicit argument by making the method-like nature of the function explicit. More explicit than currently, really, since currently the method-like nature is only inferred through a convention of naming the first argument "self". If we commit to the name "self" being more than mere convention, then "def .aMethod()" could really cause "self" to be bound. > In other words, 'self' here is uniformly replaced by an empty string. :) > > Interestingly, I believe this syntax could get everybody off our backs > about explicit self, as although it is still explicit, it's in an implicit > way. :) > > Of course, for this to really work properly, you would have to be able to > use a bare '.' to mean the hidden argument, for cases like "getattr(.,foo)" > and ".[27]". But "." is just *so* small. At least @ is sufficiently large that you can't visually miss it. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From tjreedy at udel.edu Fri Apr 14 00:44:09 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 13 Apr 2006 18:44:09 -0400 Subject: [Python-3000] PEP 3002: Procedure for Backwards-Incompatible Changes References: Message-ID: "Steven Bethard" wrote in message news:d11dcfba0604130907y73d57c14o7929ea405c4ef48b at mail.gmail.com... > Thus, for any backwards-incompatible change, two things are required: > > * An official Python Enhancement Proposal (PEP) > * Code that can identify pieces of Python 2.X code that may be > problematic in Python 3000 I should hope and like it specified that the code is required for final acceptance rather than submission. For some PEPs, triggering a warning will only require matching parsed names to the affected name(s). Example, moving filter from builtins to functools. I should hope that python3warn.py will at least parse the target code into tokens and have a dict of trigger_names:message_templates (or possible just a set if one template will do) and code something like if name in changed_names: warn(name, lineno, changed_names[name]) executed for each parsed name. Then the incremental code change needed is a line in the definition of changed_names. For some simple changes, we can do auto-edits (with programmer okays) instead of just warn. For the movement of filter, the change is either add 'from functools import filter' at the top or add 'import functools' if not there alread and change 'filter' to 'functools.filter'. So a python3edit could be added as a desireable possibility. (I initially thought filter replacement could be unconditional until I remembered the possibility of 'def filter...'. On the other hand, I have the impression that one of pylint/pychecker can detect builtin overrides already.) Terry Jan Reedy From tjreedy at udel.edu Fri Apr 14 01:15:16 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 13 Apr 2006 19:15:16 -0400 Subject: [Python-3000] Adaptation vs. Generic Functions References: <44344432.5030704@livinglogic.de><4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> <443D6E71.9090807@ieee.org> <443E7E69.9000900@colorstudy.com> Message-ID: "Ian Bicking" wrote in message news:443E7E69.9000900 at colorstudy.com... >> Definitely wild. Reader beware. Now is the time to generate lots of >> wild ideas and let them sink in. If it still seems a good idea 3 >> months from now we may select it for a code experiment (as opposed to >> a thought experiment). BTW I want to encourage lots of code >> experiments for Python 3.0 -- though not directly in the p3yk (sic) >> branch. > > And I could have sworn you asked for less wild ideas just the other > day... Well, there are surface wild ideas that amount to rearranging deck chairs, and there are deeper wild ideas that are more like changing the propeller drive from piston engines to turbines. Less of the former and more of the latter. tjr From brett at python.org Fri Apr 14 01:18:25 2006 From: brett at python.org (Brett Cannon) Date: Thu, 13 Apr 2006 16:18:25 -0700 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <443E91E0.5020304@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> Message-ID: On 4/13/06, Ian Bicking wrote: > I propose that strings (unicode/text) shouldn't be iterable. Seeing this: > >
    >
  • i >
  • t >
  • e >
  • m >
  • >
  • 1 >
> > a few too many times... it's annoying. Instead, I propose that strings > get a list-like view on their characters. Oh synergy! > > Thus you would do: > > for c in a_string.chars(): > print c > > This view would have the full complement of list methods (like .count(), > .index(), etc), and would not have string methods (like .upper()). > +1 from me. I almost suggest a words() view, but with split() I don't know if that is needed. Then again split() could possibly go away if there was a words() view. -Brett > Iterating over strings causes frequent hard bugs (bad data, as opposed > to exceptions which make for easy bugs), as the bug can manifest itself > far from its origination. Also strings aren't containers. Because > Python has no characters, only strings, as a result strings look like > they contain strings, and those strings in turn contain themselves. It > just doesn't make sense. And it is because a string and the characters > it contains are interchangeable (they are both strings) that the > resulting bugs can persist without exceptions. > > Should bytes be iterable as well? Because bytes (the container) and > integers are not interchangeable, the problems that occur with strings > seem much less likely, and the container-like nature of bytes is > clearer. So I don't propose this effect bytes in any way. > > Questions: > > * .chars() doesn't return characters; should it be named something else? > > * Should it be a method that is called? dict.keys() has a legacy, but > this does not. There is presumably very little overhead to getting this > view. However, symmetry with the only other views we are considering > (dictionary views) would indicate it should be a method. Also, there > are no attributes on strings currently. > > * Are there other views on strings? Can string->byte encoding be > usefully seen as a view in some cases? > > -- > Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/brett%40python.org > From adam.deprince at gmail.com Fri Apr 14 05:55:05 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Thu, 13 Apr 2006 23:55:05 -0400 Subject: [Python-3000] Did I miss anything? In-Reply-To: References: Message-ID: <1144986906.3259.13.camel@localhost.localdomain> On Mon, 2006-04-10 at 15:19 -0700, Guido van Rossum wrote: > I've been intermittently following and skipping the Python-3000 list; > my personal bandwidth just wasn't up to following everything. I know a > few PEPs have been checked in (thanks Georg!). > > I know Adam DePrince has four different proto-PEPs up at > http://www.deprince.net/ideas/peps.html but I don't know to what Forgive me, I should have made that page more clear. The first two were related to our earlier views discussion. As they went out of favor and the conversation drifted to adaptation, I struck them out. The later two I haven't submitted yet for consideration to this forum because I have not yet figured out if they are appropriate for P3K, or something further in the future. Anyhow, I've improved the organization my Proto-PEPS a bit. - Adam DePrince > extent these are his personal ideas or reflect the outcome of > discussions here; I'm confused by his striking through the first two. > > Are there other proto-PEPs being worked on? I would appreciate if the > authors would send me a note (or reply here) with the URL and the > status. > > Unfortunately I'm not going to be able to keep up in the next two > weeks either -- tomorrow night I'm flying to London for a workshop > with the Shuttleworth Foundation; then, after spending the easter > weekend with my relatives in Holland, I'm off to Oxford for the ACCU > conference. I hope to see some of you there! (I'll be giving a keynote > about Python 3000 there, which I hope to make as current as humanly > possible, hence in part my request here.) > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/adam.deprince%40gmail.com From ncoghlan at iinet.net.au Fri Apr 14 06:41:52 2006 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri, 14 Apr 2006 14:41:52 +1000 Subject: [Python-3000] Another generic functions/adaptation use case Message-ID: <443F2810.3060301@iinet.net.au> Another potential generic function use case that occurred to me is the open() builtin. If that was extensible, the discussion about providing a nice filesystem path abstraction in the standard lib wouldn't need to waste any time on the question of whether or not to inherit from unicode - it would just register a new open() signature for path objects. Ditto for if we later added objects to represent URLs/URIs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Apr 14 07:22:43 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Apr 2006 15:22:43 +1000 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <443E91E0.5020304@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> Message-ID: <443F31A3.5050907@gmail.com> Ian Bicking wrote: > I propose that strings (unicode/text) shouldn't be iterable. Seeing this: > >
    >
  • i >
  • t >
  • e >
  • m >
  • >
  • 1 >
> > a few too many times... it's annoying. Instead, I propose that strings > get a list-like view on their characters. Oh synergy! Another +1 here. Some other details: __contains__ would still be there, so "substr in string" would still work __getitem__ would still be there, so slicing would work To remove the iterable behaviour either iter() would have to change so that the default "all sequences are iterable" behaviour goes away (which Guido has suggested previously) or else the __iter__ method of strings would need to explicitly raise TypeError. My preference is the latter (as it's less likely to break other classes), but the former could also work if python3warn flagged classes which defined __getitem__ without also defining __iter__. > Iterating over strings causes frequent hard bugs (bad data, as opposed > to exceptions which make for easy bugs), as the bug can manifest itself > far from its origination. Also strings aren't containers. Because > Python has no characters, only strings, as a result strings look like > they contain strings, and those strings in turn contain themselves. It > just doesn't make sense. And it is because a string and the characters > it contains are interchangeable (they are both strings) that the > resulting bugs can persist without exceptions. And far too many uses of itertools end up having to either check an "atomic_types" set (usually consisting of str and unicode), or else a self-recursion check (iter(x).next() is x) to avoid breaking in horrible ways. Away with them, I say! > Should bytes be iterable as well? Because bytes (the container) and > integers are not interchangeable, the problems that occur with strings > seem much less likely, and the container-like nature of bytes is > clearer. So I don't propose this effect bytes in any way. Agreed - there's no self-recursive iteration here, so no real problems. > Questions: > > * .chars() doesn't return characters; should it be named something else? Why do you say it doesn't return characters? Python's chars are just strings of length 1, and that's what this view will contain. > * Should it be a method that is called? dict.keys() has a legacy, but > this does not. There is presumably very little overhead to getting this > view. However, symmetry with the only other views we are considering > (dictionary views) would indicate it should be a method. Also, there > are no attributes on strings currently. Using methods for view creation is fine by me. The various ideas for turning them into attributes instead are cute, but not particularly compelling. > * Are there other views on strings? Can string->byte encoding be > usefully seen as a view in some cases? Given that all Py3k strings will be Unicode, I think providing a view that exposed the code points of the characters would be good. Being able to write mystr.codes() instead of [ord(c) for c in mystr.chars()] would be a good thing. Interestingly, ord(c) would then be little more than an abbreviation of c.codes()[0]. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From adam.deprince at gmail.com Fri Apr 14 07:29:34 2006 From: adam.deprince at gmail.com (Adam DePrince) Date: Fri, 14 Apr 2006 01:29:34 -0400 Subject: [Python-3000] symbols? In-Reply-To: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> References: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> Message-ID: <1144992574.3259.58.camel@localhost.localdomain> On Wed, 2006-04-12 at 05:15 -0700, Michael Chermside wrote: > Kendall Clark writes: > > The other choice, of course, is for the library or API to define some > > variables bound to strings and then use them like constants, except > > that they can get redefined: > > > > @pylons.rest.restrict(GET, HEAD) > > ... > > > > A symbol has the advantage that it can't be assigned to, so can't > > change like a variable, but expresses more clearly than a string the > > intent to refer to a name or term, not string data > > When I see a variable name in all-caps, I don't assign to it. I don't > even need a tool like PyChecker to remind me that this is a constant > because I've been familiar with the "all-caps == constant" convention > from shortly after I got a computer with lower-case letters on it. > The other programmers I work with seem to behave the same way. I must > be unusually lucky in this regard, because I meet lots of people who > are very concerned about the fact that it *is* possible to change > these values. I can only surmise that they work with people who make > a habit of modifying all-caps variables randomly just for fun. One thing we should consider is how euro-centric this convention is. I'm not certain that all of the characters in all eastern languages can be readily divided into upper and lower cases. IIRC one of the goals with P3K is to allow Python to compile unicode source files, with that this convention will eventually stop working. I remember asking a similar question about a similar topic when I was a wee-lad; why does C have enumerations when it has #define. Aside from answers like "because gdb can return symbols instead of numbers", the real motivation is compiler encouraged discipline. The real question isn't about the merit of this new symbol data type, but rather a philosophical question about the direction the language should go. On the spectrum of "linguistic liberty", we have "B&D" languages like Java and C# on one side and the Rebelaisian utopia of Perl on the other. Python is sort of in the middle. The question being asked here is "Does using plain strings for symbols make us too much like Perl, or will fixing it by introducing a symbol type make us too much like Java. Personally, I rather like the direction of the symbols idea, but am unsure if I like the :symbol form for the literal. I give it a +0.5. > I've found it even more curious that no one ever seems to be worried > about the ability to write infinite loops ("while True: pass") in > pretty much any piece of code. Of course, I suppose it might have > something to do with the fact that the haulting problem makes it > theoretically impossible to prevent all infinite loops. But it's Just to be a nit pick, modern computers are not Turing complete. They are finite state machines; albeit large FSMs. Proper Turing machines have one ended mythical tapes of infinite length. Your computer is a finite state machine. But I can understand the confusion, old open reel tapes, when dropped with a copy of your thesis on them tend to feel a lot like Turing tapes. There is a correlation between the number of copies you had, and the number of tape ends you find. Solving the halting problem for a Turing machine isn't hard at all; hop on a plane to Greece and wander about the ruins of the temple of Delphi ... On a serious note, nobody thinks about it as a dangerous flaw because it is a provably inescapable part of the computing experience. But as vocal as our friend Alan was about the dangers of his auto-referential one sided tapes, he was rather quiet about the issue of case we face here. The two issues are not at all comparable. Cheers - Adam DePrince From ncoghlan at gmail.com Fri Apr 14 07:45:59 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Apr 2006 15:45:59 +1000 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <20060413125831.8lonimb5yvwggwcw@login.werra.lunarpages.com> References: <20060413125831.8lonimb5yvwggwcw@login.werra.lunarpages.com> Message-ID: <443F3717.3050207@gmail.com> Michael Chermside wrote: > Ian Bicking writes: >> I propose that the self argument be removed from method definitions. > > This is not a new proposal. The fundamental problem is a perceived > mismatch between the argument list used for calling and the argument > list used in defining the method. Of the many times that it has been > proposed, the best syntax I have seen was as follows: > > def self.my_method(arg1, arg2): > # body goes here > > In other words, make the definition look like the invocation. In > this proposal "self" is not magical, it is simply declared in a > slightly different location. The syntax would be used by sensible > people only when defining methods, but there would be no such > rule enforced by the interpreter, nor would the existing syntax > be broken or prohibited. > > To be abundantly specific, the proposal is that if the next two tokens > after a "def" were "self" and "." then the result of compiling such > code would be exactly the same as if they were not present but "self" > and "," were inserted immediately after the opening parenthesis. > Whether to allow any identifier or only the particular identifier > "self" is an interesting question that I would leave for Guido. > > Personally, I favor doing this. I think that making declarations look > similar to invocations is very useful. As you say, we could do this just as a grammar tweak, with: def NAME1.NAME2(ARG_SPEC): BODY being syntactic sugar for: def NAME2(NAME1, ARG_SPEC): BODY Doing this would, however, rule out the possibility of allowing that syntax to be used for monkey-patching by interpreting it as: def _f(ARG_SPEC): BODY NAME1.NAME2 = _f del _f I'd also be concerned that this second possible interpretation could easily lead to confusion if the syntax were ever used outside a class definition. So what if this syntax was permitted *only* within the body of a class definition? Aside from monkey-patching (which this doesn't do) is there a use case for supporting it anywhere else? Anyway, below is an example of method definitions of a real class (Queue.Queue) updated to use this method syntax. I like it, I really, really like it - the methods are defined exactly the way they're typically meant to be called. Currently, I have to think when reading a class's source in order to figure out what the method signatures actually are - with this syntax, it just leaps out at you. Cheers, Nick. ------------- class Queue: def self.__init__(maxsize=0): # ... def self.task_done(): # ... def self.join(): # ... def self.qsize(): # ... def self.empty(): # ... def self.full(): # ... def self.put(item, block=True, timeout=None): # ... def self.put_nowait(item): # ... def self.get(block=True, timeout=None): # ... def self.get_nowait(): # ... # Override these methods to implement other queue organizations # (e.g. stack or priority queue). # These will only be called with appropriate locks held # Initialize the queue representation def self._init(maxsize): # ... def self._qsize(self): # ... # Check whether the queue is empty def self._empty(): # ... # Check whether the queue is full def self._full(): # ... # Put a new item in the queue def self._put(item): # ... # Get an item from the queue def self._get(): # ... -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ian.bollinger at gmail.com Fri Apr 14 08:28:25 2006 From: ian.bollinger at gmail.com (Ian D. Bollinger) Date: Fri, 14 Apr 2006 02:28:25 -0400 Subject: [Python-3000] symbols? In-Reply-To: <1144992574.3259.58.camel@localhost.localdomain> References: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> <1144992574.3259.58.camel@localhost.localdomain> Message-ID: <443F4109.7040700@gmail.com> Adam DePrince wrote: > On Wed, 2006-04-12 at 05:15 -0700, Michael Chermside wrote: > >> When I see a variable name in all-caps, I don't assign to it. I don't >> even need a tool like PyChecker to remind me that this is a constant >> because I've been familiar with the "all-caps == constant" convention >> from shortly after I got a computer with lower-case letters on it. >> The other programmers I work with seem to behave the same way. I must >> be unusually lucky in this regard, because I meet lots of people who >> are very concerned about the fact that it *is* possible to change >> these values. I can only surmise that they work with people who make >> a habit of modifying all-caps variables randomly just for fun. >> > One thing we should consider is how euro-centric this convention is. > I'm not certain that all of the characters in all eastern languages can > be readily divided into upper and lower cases. IIRC one of the goals > with P3K is to allow Python to compile unicode source files, with that > this convention will eventually stop working. > All identifiers have a standard naming convention involving case, so we should throw them out because maybe in the future someone will be able to write identifiers in Mandarin Chinese? I hope I never have to decipher such code. Also, at first I cared about giving constants all-capital names as a carry over from statically-typed languages, but eventually I dropped the convention. In the majority of cases where I might use a constant in a statically typed language, the constancy doesn't seem significant enough to denote in a Python variable name. And if I really need a unique object, I use the same convention the built-ins use (None, Ellipsis, etc.) From ncoghlan at gmail.com Fri Apr 14 08:41:07 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Apr 2006 16:41:07 +1000 Subject: [Python-3000] symbols? In-Reply-To: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> Message-ID: <443F4403.8090202@gmail.com> Kendall Clark wrote: > I don't know if the Ruby syntax for symbols, :foo, will work in > Python (though I think visually it's a good thing), but it would be > really nice to have symbols with *some* syntax in Python 3000. > > Again, this design request is based on aesthetics and fun: having > symbols would make Python more fun. I'd like to link a few threads together here. . . Guido proposed a while back on python-dev to support the idea of overridable properties - properties that were implemented in terms of other methods of the class, and were looked up when the property was invoked rather than when it was defined. Something like: class C: x = property("_get_x", "_set_x", "_del_x", "This is the x property") def _get_x(self): # Retrieve x def _set_x(self): # Set x def _del_x(self): # Delete x This is a PITA to type (particularly if programming in C has ingrained in you the habit of using double-quotes for strings) and isn't all that easy to read, either. What objections there were at the time seemed to stem mostly from aesthetics - the strings for attribute names in the call to property() are just plain ugly. However, if a leading dot in an expression simply indicated "this is a string that is also a legal Python identifier", the above could be written: class C: x = property(._get_x, ._set_x, ._del_x, "This is the x property") def _get_x(self): # Retrieve x def _set_x(self): # Set x def _del_x(self): # Delete x By using a separate syntax, you get the following benefits: 1. It makes the code easier to write as it's not a bracketed syntax and if your keyboard makes '.' inconvenient writing Python code would already be hellish 2. It lets the reader know that these values are identifiers rather than arbitrary strings 3. It also lets the compiler know to enforce the rules for identifiers rather than the rules for strings (which are far more lax) Easier to write, easier to read and better error checking adds up to a big win in my book :) Add in the method syntax idea from the thread about method definitions, and you get the final result: class C: x = property(._get_x, ._set_x, ._del_x, "This is the x property") def self._get_x(): # Retrieve x def self._set_x(): # Set x def self._del_x(): # Delete x YMMV, but that's a property declaration syntax I could definitely live with - it makes it clear that the first three arguments passed to property are Python identifiers, it makes it easy for subclasses to control some or all of the aspects of the way the property is handled, and it doesn't require any metaclass magic or new statements like 'make'. This meaning for a leading dot also aligns well with normal attribute access: x.attr <=> getattr(x, .attr) <=> getattr(x, 'attr') x.attr = y <=> setattr(x, .attr, y) <=> setattr(x, 'attr', y) del x.attr <=> delattr(x, .attr) <=> delattr(x, 'attr') Cheers, Nick. P.S. I did try this with a real object in front of the dot with a __getattribute__ that simply returns an interned string of the attribute name. This just ended up being confusing because the symbol definition looks too much like normal attribute access. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Apr 14 10:20:00 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Apr 2006 18:20:00 +1000 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443A89C2.3040100@colorstudy.com> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <443A89C2.3040100@colorstudy.com> Message-ID: <443F5B30.9050209@gmail.com> Ian Bicking wrote: > One advantage to this is that at each step something useful would be > created. Better IPC would be useful for more than restricted execution. > Or ways to create subinterpreters and communicate with them. Or making > the interpreter startup faster, to facilitate spawning short-lived > interpreters. As an example, by combining these features but leaving > out restricted execution you can get something like the PHP model, which > manages to protect well against buggy code (even if not so well against > security bugs). There's probably a dozen distinct parts to this, but I > think each is independently interesting and useful. > > And maybe, because these are useful subprojects, some of the people who > can't commit to the time to take on this project as a whole could commit > to some piece of it for which they have alternate motivations. This is why I think the first step in a solid Python restricted execution framework isn't an implementation activity but a research/scoping activity, looking at the various systems already out there, and the various trade-offs involved in each. The requirements for a subinterpreter are completely different from those for the main Python interpreter which actually runs a Python application - the initial state of a subinterpreter should be controlled entirely by the invoking interpreter, so much of the normal interpreter startup sequence should be eliminated. Subinterpreters are designed to manipulate the state of the main application rather than the platform they are executing on, so direct access to the underlying OS isn't needed. The meaning of module imports in the subinterpreter is also up to the main application. My initial inclination is that aiming straight for in-process restricted execution is premature optimisation at its worst - worrying about IPC overhead before there's even anything on the table that is functionally complete (i.e. both usable and demonstrably secure) seems to be putting the cart before the horse. The liberal use of C static variables by both the interpreter core and third party extensions means that in-process restricted execution will be necessarily invasive - every static variable will need to be checked for security implications, and some mechanism provided to support partitioning between different interpreters in the same process. No C extensions could ever be trusted, as they would have full access to the main interpreter's C API. Initially targeting an out-of-process sandbox allows all those issues to be dealt with later, rather than having them be blockers for the initial implementation of a restricted execution mechanism that is actually able to keep its promises. If the OS provides sufficient support (such as chroot jails) then it may even be possible to provide secure execution of C extension modules. An out-of-process sandbox frees up the technology options - it may turn out to make more sense to base a restricted interpreter on PyPy rather than CPython. It also allows a strategy similar to that used by Lua - start the subinterpreter with a stripped core that can't read or write files directly at all, and don't do anything on startup unless explicitly requested by the main interpreter. If we can get something that's secure when operating in a different process, *then* thread-local storage may permit that subinterpreter to be migrated from an out-of-process object to a separate OS-level thread inside the main process. The subinterpreter would still be entirely independent of the main interpreter's state, and main interpreter objects would still need to be converted to subinterpreter objects in order for code running in the subinterpreter to manipulate them (and vice versa) but any OS-level IPC overhead would be gone. Making such subinterpreters easy to create is also, IMO, the best way to deal with criticism of the GIL - as such interpreters would have their own state, they'd be free to run on as many different processors as you wanted. The only time they'd block on the GIL is when they needed to send information to, or receive information from, the main interpreter. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Fri Apr 14 10:43:33 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Apr 2006 09:43:33 +0100 Subject: [Python-3000] symbols? In-Reply-To: <443F4403.8090202@gmail.com> References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: > Kendall Clark wrote: > > I don't know if the Ruby syntax for symbols, :foo, will work in > > Python (though I think visually it's a good thing), but it would be > > really nice to have symbols with *some* syntax in Python 3000. > > > > Again, this design request is based on aesthetics and fun: having > > symbols would make Python more fun. I don't see how this particular feature adds more fun; I'd rather see it argued on the basis of use cases. The aesthetics of the earlier proposal to use :foo to indicate symbols are poor; using random punctuation is unpythonic (as is the use of ? or ! in identifiers BTW) but more importantly there are a few syntactic positions where it would be ambiguous, e.g. foo[:bar] already means foo[0:(bar)]. I'd also like to point out (again?) that the "const-ness" of ALLCAPS is a red herring; in practice, we treat almost all imported names as constants, and in fact class and function names are generally considered constant. (I've also seen plenty of code that used ALLCAPS to indicate "configuration parameter" rather than "constant" and which freely assigned to ALLCAPS variables in configuration code.) Then, on 4/14/06, Nick Coghlan wrote: [...] class C: x = property("_get_x", "_set_x", "_del_x", "This is the x property") [...] (BTW I thought I implemented this but can't find any evidence.) > However, if a leading dot in an expression simply indicated "this is a string > that is also a legal Python identifier", the above could be written: [...] class C: x = property(._get_x, ._set_x, ._del_x, "This is the x property") [...] This is slightly more Pythonic, and unambiguous, but I'd rather reserve leading dot for a more powerful feature, perhaps .foo as an abbreviation for self.foo, or some other scope-ish operator. (I also note that in a proportional font the . is easily missed.) > By using a separate syntax, you get the following benefits: > 1. It makes the code easier to write as it's not a bracketed syntax and if > your keyboard makes '.' inconvenient writing Python code would already be hellish > 2. It lets the reader know that these values are identifiers rather than > arbitrary strings > 3. It also lets the compiler know to enforce the rules for identifiers > rather than the rules for strings (which are far more lax) I'm not sure (1) weighs much but I buy (2) and some of (3), so perhaps we ought to think of some other prefix character (or overcome my objection to '.'). Somewhat unrelated: there's one advantage to enums (which have been suggested as an alternative to symbols) which symbols don't share: if you have a typo in an enum name, you presumably get an early, hard NameError or AttributeError (and pychecker could easily check this); but if you misspell a symbol, you just pass a different symbol, which is likely to trigger a later error or no error at all (depending on how the symbol is used). Of course, enums really serve a different use case; they wouldn't help at all for the property-defining use case Nick described here. > Easier to write, easier to read and better error checking adds up to a big win > in my book :) Is that also my book? :-) > Add in the method syntax idea from the thread about method definitions, and > you get the final result: > > class C: > x = property(._get_x, ._set_x, ._del_x, > "This is the x property") > > def self._get_x(): > # Retrieve x Oh, but I don't like this at all. Anyway, it's pretty much unrelated to the other idea. > x.attr <=> getattr(x, .attr) <=> getattr(x, 'attr') > x.attr = y <=> setattr(x, .attr, y) <=> setattr(x, 'attr', y) > del x.attr <=> delattr(x, .attr) <=> delattr(x, 'attr') So this begs the question: would the following assertion pass or fail? assert .foo == "foo" What about this one? assert type(.foo) == str (I'm just trying to probe for implementation details of the proposal; I'm something like -0 on the whole idea.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From krstic at fas.harvard.edu Fri Apr 14 10:35:15 2006 From: krstic at fas.harvard.edu (Ivan Krstic) Date: Fri, 14 Apr 2006 04:35:15 -0400 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443F5B30.9050209@gmail.com> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <443A89C2.3040100@colorstudy.com> <443F5B30.9050209@gmail.com> Message-ID: <443F5EC3.2010204@fas.harvard.edu> Nick Coghlan wrote: > This is why I think the first step in a solid Python restricted execution > framework isn't an implementation activity but a research/scoping activity, > looking at the various systems already out there, and the various trade-offs > involved in each. This might be a good point to invite Ben Laurie (of Apache-SSL fame) to jump into the conversation. He researched restricted Python execution when working on what eventually became CaPerl: http://www.links.org/?p=28 See also the responses to Ted Leung's blog post about this (http://www.sauria.com/blog/2005/Nov/27), with comments by Phillip Eby and Ian. -- Ivan Krstic | GPG: 0x147C722D From ncoghlan at gmail.com Fri Apr 14 11:38:23 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Apr 2006 19:38:23 +1000 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443F5EC3.2010204@fas.harvard.edu> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <443A89C2.3040100@colorstudy.com> <443F5B30.9050209@gmail.com> <443F5EC3.2010204@fas.harvard.edu> Message-ID: <443F6D8F.3080007@gmail.com> Ivan Krstic wrote: > Nick Coghlan wrote: >> This is why I think the first step in a solid Python restricted >> execution framework isn't an implementation activity but a >> research/scoping activity, looking at the various systems already out >> there, and the various trade-offs involved in each. > > This might be a good point to invite Ben Laurie (of Apache-SSL fame) to > jump into the conversation. He researched restricted Python execution > when working on what eventually became CaPerl: > > http://www.links.org/?p=28 > > See also the responses to Ted Leung's blog post about this > (http://www.sauria.com/blog/2005/Nov/27), with comments by Phillip Eby > and Ian. Ah, interesting. Especially this: http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto So one avenue definitely worth investigating would be extracting Zope's untrusted code execution security mechanisms into a form that can be used independently of the rest of Zope. That would probably be a lot nicer than having to start from scratch, but there's the question of how tightly integrated that mechanism is with the rest of Zope's infrastructure. It may prove more effective to build something *modelled* on Zope's system, without actually reusing their code. I'll also note the similarities between this model and what Ian and I were suggesting - the restricted subinterpreter would contain only 'safe' builtins, with any unsafe operations being proxied back to the original application (which could then allow or deny them as it saw fit). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Fri Apr 14 13:31:56 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 14 Apr 2006 21:31:56 +1000 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: <443F882C.9000604@gmail.com> Guido van Rossum wrote: > I'd also like to point out (again?) that the "const-ness" of ALLCAPS > is a red herring; in practice, we treat almost all imported names as > constants, and in fact class and function names are generally > considered constant. (I've also seen plenty of code that used ALLCAPS > to indicate "configuration parameter" rather than "constant" and which > freely assigned to ALLCAPS variables in configuration code.) I tend to use all caps that way, too (i.e. they refer to data structures that won't be modified after the module has been imported, but they're fair game while the module itself is being initialised). > Then, on 4/14/06, Nick Coghlan wrote: > [...] > class C: > x = property("_get_x", "_set_x", "_del_x", > "This is the x property") > [...] > > (BTW I thought I implemented this but can't find any evidence.) I thought you'd implemented it too, so I was surprised to find it didn't work in 2.5a1. Could it be languishing in a working copy somewhere? >> However, if a leading dot in an expression simply indicated "this is a string >> that is also a legal Python identifier", the above could be written: > [...] > class C: > x = property(._get_x, ._set_x, ._del_x, > "This is the x property") > [...] > > This is slightly more Pythonic, and unambiguous, but I'd rather > reserve leading dot for a more powerful feature, perhaps .foo as an > abbreviation for self.foo, or some other scope-ish operator. (I also > note that in a proportional font the . is easily missed.) The leap from "Starts with ." to "It's a string" is a pretty substantial one too, so scratch that idea. >> By using a separate syntax, you get the following benefits: >> 1. It makes the code easier to write as it's not a bracketed syntax and if >> your keyboard makes '.' inconvenient writing Python code would already be hellish >> 2. It lets the reader know that these values are identifiers rather than >> arbitrary strings >> 3. It also lets the compiler know to enforce the rules for identifiers >> rather than the rules for strings (which are far more lax) > > I'm not sure (1) weighs much but I buy (2) and some of (3), so perhaps > we ought to think of some other prefix character (or overcome my > objection to '.'). Crazy idea time. . . since we won't be using it for repr() anymore, what about using backtick quoting? That wouldn't make much difference for (1), but would provide both (2) and (3). class C: x = property(`_get_x`, `_set_x`, `_del_x`, "This is the x property") Since the results actually *are* strings with the given contents, the typographic ambiguity isn't as big of an issue as it is with the current meaning of backticks - `foo`, 'foo' and "foo" would all result in the exact same object, so if the current font happened to display `foo` and 'foo' the same, you'd still be able to understand the code correctly. The difference between backtick quoting and normal quoting would then be that the compiler would just be far pickier about what was allowed between backticks, with only legal identifiers permitted (i.e. must start with a letter or underscore, may contain only letters, numbers and underscores, no string escape sequences). The result would still just be a normal string. From a 2.x compatibility standpoint: - python3warn would pick up any usage of backticks and recommend replacing `EXPR` with repr(EXPR) (or backticks could just be fully deprecated in 2.x) - identifier strings written within either '' or "" would work in both versions Programs written expecting the Python 3 semantics would fail on 2.x, either due to backticks being fully deprecated or else failing with a NameError when attempting to eval() the expression between the backticks. > Somewhat unrelated: there's one advantage to enums (which have been > suggested as an alternative to symbols) which symbols don't share: if > you have a typo in an enum name, you presumably get an early, hard > NameError or AttributeError (and pychecker could easily check this); > but if you misspell a symbol, you just pass a different symbol, which > is likely to trigger a later error or no error at all (depending on > how the symbol is used). Of course, enums really serve a different use > case; they wouldn't help at all for the property-defining use case > Nick described here. Yes, this idea doesn't really overlap all that much with enumerations. The existence of the enumeration definition as an object in its own right means normal attribute access generally works fine for those. >> x.attr <=> getattr(x, .attr) <=> getattr(x, 'attr') >> x.attr = y <=> setattr(x, .attr, y) <=> setattr(x, 'attr', y) >> del x.attr <=> delattr(x, .attr) <=> delattr(x, 'attr') > > So this begs the question: would the following assertion pass or fail? > > assert .foo == "foo" > > What about this one? > > assert type(.foo) == str I would expect both of those assertions to be accurate. This is really just about a bit of syntactic sugar to more clearly convey intent when programming (and to get the compiler to help out a bit with sanity checking). Something that might be related is an additional string introspection method, isidentifier(), which returned true only if the string was a legal Python identifier. This would then always be true for a string literal using backticks, but it could be false for other string literals. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Fri Apr 14 14:16:03 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Apr 2006 13:16:03 +0100 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443F3717.3050207@gmail.com> References: <20060413125831.8lonimb5yvwggwcw@login.werra.lunarpages.com> <443F3717.3050207@gmail.com> Message-ID: > Michael Chermside wrote: > > Ian Bicking writes: > >> I propose that the self argument be removed from method definitions. > > > > This is not a new proposal. The fundamental problem is a perceived > > mismatch between the argument list used for calling and the argument > > list used in defining the method. Of the many times that it has been > > proposed, the best syntax I have seen was as follows: > > > > def self.my_method(arg1, arg2): > > # body goes here -1 to this version and to Ian's original proposal. You can save your breath, or write up a PEP with the goal of explaining why this proposal (which keeps coming back) is a bad idea -- such a PEP would be a good idea but it won't get the feature accepted. Capsule argument Ian's original proposal: you're proposing to hide a fundamental truth in Python, that methods are "just" functions whose first argument can be supplied using syntactic sugar in the call (x.foo() instead of x.__class__.foo(x)). And against the "def self.foo(x):" syntax: that's a lot of new syntax for no gain in readability. You just need to get your head around the fundamental truth (see above), just like you should get used to reading " ".join(list_of_strings). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 14 14:25:15 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Apr 2006 13:25:15 +0100 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <443F31A3.5050907@gmail.com> References: <443E91E0.5020304@colorstudy.com> <443F31A3.5050907@gmail.com> Message-ID: On 4/14/06, Nick Coghlan wrote: > Ian Bicking wrote: > > I propose that strings (unicode/text) shouldn't be iterable. Seeing this: > > > >
    > >
  • i > >
  • t > >
  • e > >
  • m > >
  • > >
  • 1 > >
> > > > a few too many times... it's annoying. Instead, I propose that strings > > get a list-like view on their characters. Oh synergy! > > Another +1 here. And a moderate +0.1 here (we need to research the consequences more). > Some other details: > > __contains__ would still be there, so "substr in string" would still work > __getitem__ would still be there, so slicing would work Right. > To remove the iterable behaviour either iter() would have to change so that > the default "all sequences are iterable" behaviour goes away (which Guido has > suggested previously) or else the __iter__ method of strings would need to > explicitly raise TypeError. I'm for simplifying iter() so that it *only* looks for __iter__(). (There's another problem with falling back on __getitem__(): if someone implements a dict-ish type from scratch without defining __iter__, iteration tries to use keys 0, 1, ... instead of cleanly failing or iterating over the actual keys.) > My preference is the latter (as it's less likely to break other classes), but > the former could also work if python3warn flagged classes which defined > __getitem__ without also defining __iter__. The breakage argument isn't very strong for Python 3000. [...] > > Should bytes be iterable as well? Because bytes (the container) and > > integers are not interchangeable, the problems that occur with strings > > seem much less likely, and the container-like nature of bytes is > > clearer. So I don't propose this effect bytes in any way. Agreed; I doubt that bytes will cause users to fall in the same trap often. But we should review this after bytes is implemented and has been used some. > > Questions: > > > > * .chars() doesn't return characters; should it be named something else? > > Why do you say it doesn't return characters? Python's chars are just strings > of length 1, and that's what this view will contain. I'm not too keen on the name chars(), but I can't think of an alternative just now. The argument above doesn't hold IMO; it *does* return characters. Perhaps it should be named characters()? (Some folks don't like abbrevs.) I'm not sure of the functionality of chars()/characters() -- perhaps it should just return an iterator? I don't think that s.count('a') is a trap. > > * Should it be a method that is called? dict.keys() has a legacy, but > > this does not. There is presumably very little overhead to getting this > > view. However, symmetry with the only other views we are considering > > (dictionary views) would indicate it should be a method. Also, there > > are no attributes on strings currently. > > Using methods for view creation is fine by me. The various ideas for turning > them into attributes instead are cute, but not particularly compelling. -1 on using attributes to return views. > > * Are there other views on strings? Can string->byte encoding be > > usefully seen as a view in some cases? That's not a view but an operation that returns a new object! > Given that all Py3k strings will be Unicode, I think providing a view that > exposed the code points of the characters would be good. Being able to write > mystr.codes() instead of [ord(c) for c in mystr.chars()] would be a good thing. Feels like YAGNI territory to me. > Interestingly, ord(c) would then be little more than an abbreviation of > c.codes()[0]. That doesn't make it a good idea. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas at python.org Fri Apr 14 14:33:20 2006 From: thomas at python.org (Thomas Wouters) Date: Fri, 14 Apr 2006 14:33:20 +0200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: References: <443E91E0.5020304@colorstudy.com> <443F31A3.5050907@gmail.com> Message-ID: <9e804ac0604140533q47e480eahd47dbc5c4efa3be6@mail.gmail.com> On 4/14/06, Guido van Rossum wrote: > > > a few too many times... it's annoying. Instead, I propose that > strings > > > get a list-like view on their characters. Oh synergy! > > > > Another +1 here. > > And a moderate +0.1 here (we need to research the consequences more). We can fairly easily make strings non-iterable in the p3yk branch (and remove the default-iter in general), see how many tests it breaks ;P -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060414/fac0b299/attachment.htm From guido at python.org Fri Apr 14 14:33:59 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Apr 2006 13:33:59 +0100 Subject: [Python-3000] Another generic functions/adaptation use case In-Reply-To: <443F2810.3060301@iinet.net.au> References: <443F2810.3060301@iinet.net.au> Message-ID: On 4/14/06, Nick Coghlan wrote: > Another potential generic function use case that occurred to me is the open() > builtin. > > If that was extensible, the discussion about providing a nice filesystem path > abstraction in the standard lib wouldn't need to waste any time on the > question of whether or not to inherit from unicode - it would just register a > new open() signature for path objects. It needn't waste time on that question anyway -- the only sensible answer is no. > Ditto for if we later added objects to represent URLs/URIs. I'm not sure I follow this whole use case. Different VFSs probably all represent their pathnames as strings (at some level) so the type wouldn't be a useful differentiator. If you already have a path object specific to a VFS (or one for the URL space), the open operation should just be a method on the path object. Generic/overloaded functions (of the kind dispatched on the argument type(s)) mainly have a place when the operation is defined separate from the object. That's not a reasonable assumption in your use case -- the path object is defined in close coordination with the open() implementation so there's no decoupling requirement; a regular method is fine. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 14 14:36:33 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Apr 2006 13:36:33 +0100 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <9e804ac0604140533q47e480eahd47dbc5c4efa3be6@mail.gmail.com> References: <443E91E0.5020304@colorstudy.com> <443F31A3.5050907@gmail.com> <9e804ac0604140533q47e480eahd47dbc5c4efa3be6@mail.gmail.com> Message-ID: On 4/14/06, Thomas Wouters wrote: > We can fairly easily make strings non-iterable in the p3yk branch (and > remove the default-iter in general), see how many tests it breaks ;P I would do this in two steps: first remove the default-iter, see how many tests it breaks. But first we need to make all tests pass! If anyone wants to help out in the p3yk (sic) branch, please pick one of the failing tests, figure out why it fails, and propose a fix. (You can the fix in SF, assign it to me, and mail me about it. I'll assign it back to you to signal that it's okay to check it in.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 14 14:55:45 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 14 Apr 2006 13:55:45 +0100 Subject: [Python-3000] Adaptation vs. Generic Functions In-Reply-To: <443E7E69.9000900@colorstudy.com> References: <4434560C.8030203@cox.net> <443481EE.6060303@ieee.org> <44397B14.4060208@cox.net> <443D6E71.9090807@ieee.org> <443E7E69.9000900@colorstudy.com> Message-ID: > Guido van Rossum wrote: > > What if, instead of adding registries mapping types to functions left > > and right (in pickle, copy, pprint, who knows where else), we had a > > convention of adding __foo_bar__ methods directly to the class dict? > > This would require giving up the effective immutability of built-in > > types, which has widespread repercussions (mostly in the area of > > multiple interpreters) but nevertheless if this was officially > > endorsed, we could clean up a lot of type registries... (I notice that > > I haven't really found any use for *multiple* dispatch in my few > > explorations of @overloaded; that is, until the operator.add idea came > > along.) On 4/13/06, Ian Bicking wrote: > One advantage of using a function as the registry is that the function > lives in a particular place. A method only has a name, there is no > namespace for it. In addition to possible name collisions, functions > are possible to backtrack. If you want to know what some generic > function specialization is about, you look at the original function > definition, which you can find by following the imports. If you want to > know what a particular magic method is about, there's no particular > technique to accomplish that. I'm expecting that the probability of name collisions here would be at the same order of the probability of module name collisions. But I'm not married to the idea; it's pretty wild as ideas go; and I'm not sure we can solve the issues with sharing of builtins. > >>I suspect this falls into the "wild ideas' category, but it's an > >>interesting thought experiment anyway. Perhaps with relevance to other > >>uses of generic functions. > > > > Definitely wild. Reader beware. Now is the time to generate lots of > > wild ideas and let them sink in. If it still seems a good idea 3 > > months from now we may select it for a code experiment (as opposed to > > a thought experiment). BTW I want to encourage lots of code > > experiments for Python 3.0 -- though not directly in the p3yk (sic) > > branch. > > And I could have sworn you asked for less wild ideas just the other day... Um, yes. I want wild ideas quickly proposed so they can quickly be shot down, or not. I do *not* want extended arguments (like we've seen too many) where proponents of a proposal attempt to defend it at all cost. I'm still very skeptical of new statement forms or new paradigms. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From benji at benjiyork.com Fri Apr 14 15:28:17 2006 From: benji at benjiyork.com (Benji York) Date: Fri, 14 Apr 2006 09:28:17 -0400 Subject: [Python-3000] Is reload() feasible? In-Reply-To: <443E9AEF.7040902@colorstudy.com> References: <443E9AEF.7040902@colorstudy.com> Message-ID: <443FA371.4060401@benjiyork.com> Ian Bicking wrote: > Notably, Zope 3 does not include reloading (though it was > included in Zope 2, in a more general form than just what reload() > does). It was removed because of these corner cases where it doesn't > work, and because when these corner cases happen the results are > inscrutable and frustrating errors. A bit more info: Gintautas Miliauskas has done a "view reloader" for Z3 (http://gintas.pov.lt/darcs/z3reload/README.txt) that only reloads a very specific (but most often wanted) type of module. We've considered taking a similar approach and defining a very narrow set of requirements to make a module reloadable. Perhaps something along those lines would be useful for Py3K. -- Benji York From msoulier at digitaltorque.ca Fri Apr 14 15:38:19 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Fri, 14 Apr 2006 09:38:19 -0400 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443E7F47.9060308@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> Message-ID: <20060414133819.GX6659@tigger.digitaltorque.ca> On 13/04/06 Ian Bicking said: > I propose that the self argument be removed from method definitions. It > would not be removed from other places, e.g., you'd still use > "self.foo()", but the definition for that method would be "def foo():". What happened to... "Explicit is better than implicit." ? Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060414/d389e202/attachment.pgp From mail at manuzhai.nl Fri Apr 14 15:44:31 2006 From: mail at manuzhai.nl (Manuzhai) Date: Fri, 14 Apr 2006 15:44:31 +0200 Subject: [Python-3000] Another generic functions/adaptation use case In-Reply-To: <443F2810.3060301@iinet.net.au> References: <443F2810.3060301@iinet.net.au> Message-ID: Nick Coghlan wrote: > Another potential generic function use case that occurred to me is the open() > builtin. I think this is a great idea. For example, I'd like to use open() to do simple HTTP fetching (for simple GET requests). This would allow for things somewhat similar to PHP's streams API. http://nl.php.net/manual/en/streams.php Regards, Manuzhai From ianb at colorstudy.com Fri Apr 14 18:01:01 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 14 Apr 2006 11:01:01 -0500 Subject: [Python-3000] Is reload() feasible? In-Reply-To: <443FA371.4060401@benjiyork.com> References: <443E9AEF.7040902@colorstudy.com> <443FA371.4060401@benjiyork.com> Message-ID: <443FC73D.10207@colorstudy.com> Benji York wrote: > Ian Bicking wrote: > >> Notably, Zope 3 does not include reloading (though it was included in >> Zope 2, in a more general form than just what reload() does). It was >> removed because of these corner cases where it doesn't work, and >> because when these corner cases happen the results are inscrutable and >> frustrating errors. > > > A bit more info: Gintautas Miliauskas has done a "view reloader" for Z3 > (http://gintas.pov.lt/darcs/z3reload/README.txt) that only reloads a > very specific (but most often wanted) type of module. We've considered > taking a similar approach and defining a very narrow set of requirements > to make a module reloadable. Perhaps something along those lines would > be useful for Py3K. Yes; having gone the restart direction, for some class of files a reload is also nice. E.g., in another web framework it might be the controller files, presuming those files only contain controllers, and the system that accesses those files is smart about reloads -- and these are files that are being incrementally edited. So a way of detecting "this module is really reloadable" is nice. On the other end, sometimes there's non-Python files that need to trigger a full restart, like a configuration change (assuming you haven't built config reloading into your app). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From kendall at monkeyfist.com Fri Apr 14 18:36:29 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Fri, 14 Apr 2006 12:36:29 -0400 Subject: [Python-3000] symbols? In-Reply-To: <1144992574.3259.58.camel@localhost.localdomain> References: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> <1144992574.3259.58.camel@localhost.localdomain> Message-ID: <7C1C3765-4187-4519-A764-BFBCBE33BEAE@monkeyfist.com> On Apr 14, 2006, at 1:29 AM, Adam DePrince wrote: > Perl on the other. Python is sort of in the middle. The question > being > asked here is "Does using plain strings for symbols make us too much > like Perl, or will fixing it by introducing a symbol type make us too > much like Java. That's a reasonable way of looking at it, I suppose. But note how Common Lisp, which defies the Perl-Java language spectrum in some sense, uses symbols in lots of ways, none of which I'm suggesting for Python. That is, symbols are a really fundamental type in CL, and I'm not suggesting that radical an injection of them into Python. > Personally, I rather like the direction of the symbols idea, but am > unsure if I like the :symbol form for the literal. I give it a +0.5. Yeah, I don't think that syntax really works either. I'm not sure *which* syntax works best, but I think the notion of literal syntax is less important than whether people are interested in this as a language addition at all. (I decided I don't like the :foo syntax after starting at this for a while: if arg == :GET: dispath() The :foo: is rather ghastly.) Cheers, Kendall From kendall at monkeyfist.com Fri Apr 14 18:46:01 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Fri, 14 Apr 2006 12:46:01 -0400 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On Apr 14, 2006, at 4:43 AM, Guido van Rossum wrote: >> Kendall Clark wrote: >>> I don't know if the Ruby syntax for symbols, :foo, will work in >>> Python (though I think visually it's a good thing), but it would be >>> really nice to have symbols with *some* syntax in Python 3000. >>> >>> Again, this design request is based on aesthetics and fun: having >>> symbols would make Python more fun. > > I don't see how this particular feature adds more fun; I'd rather see > it argued on the basis of use cases. Fair enough. My primary use case is using Python as a host for a Domain Specific Language. This may not be a core Python use case, though I don't know if that's true. It is something that happens a lot in Ruby (Rails has been described as a DSL for Web programming), Haskell, and Common Lisp. I think having a symbol type makes writing DSLs in Python easier (hence, my claim about "more fun"), since you can use symbols to reflect terms from the domain of the DSL in Python in a relatively easy way. Easier than creating a module full of constants or variables. Perhaps not really easier than using strings, but it expresses one's intent more clearly, IMO -- in a way I find analogous to using tuples instead of lists in some cases. > The aesthetics of the earlier proposal to use :foo to indicate symbols > are poor; using random punctuation is unpythonic (as is the use of ? > or ! in identifiers BTW) but more importantly there are a few > syntactic positions where it would be ambiguous, e.g. foo[:bar] > already means foo[0:(bar)]. Good points. (Though I don't concede that "?" and "!" in method or function identifiers is random. But that's beside the point.) I worried that :foo might make the grammar ambiguous, and I should have thought about that more. I'm more interested in having a symbol type than any particular concrete syntax, FWIW. So I'm happy to consider the :foo variant dead in the water. > I'd also like to point out (again?) that the "const-ness" of ALLCAPS > is a red herring; in practice, we treat almost all imported names as > constants, and in fact class and function names are generally > considered constant. (I've also seen plenty of code that used ALLCAPS > to indicate "configuration parameter" rather than "constant" and which > freely assigned to ALLCAPS variables in configuration code.) Agreed. I don't read ALLCAPS in Python as meaning constant. I almost always use it as configuration parameter, for cases where a separate configuration mechanism isn't warranged. > Somewhat unrelated: there's one advantage to enums (which have been > suggested as an alternative to symbols) which symbols don't share: if > you have a typo in an enum name, you presumably get an early, hard > NameError or AttributeError (and pychecker could easily check this); Yes, absolutely. That's entirely true. I assume that enums will be more heavyweight to create (that is, more keystrokes than simply writing the symbol straight away), which is perhaps an advantage for symbols. I'd like to see both added to Py3k. > but if you misspell a symbol, you just pass a different symbol, which > is likely to trigger a later error or no error at all (depending on > how the symbol is used). Of course, enums really serve a different use > case; they wouldn't help at all for the property-defining use case > Nick described here. Agreed; I think they are distinct and should be considered orthogonally. Cheers, Kendall From ianb at colorstudy.com Fri Apr 14 19:09:58 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 14 Apr 2006 12:09:58 -0500 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: <443FD766.90609@colorstudy.com> Guido van Rossum wrote: > Then, on 4/14/06, Nick Coghlan wrote: > [...] > class C: > x = property("_get_x", "_set_x", "_del_x", > "This is the x property") > [...] > > (BTW I thought I implemented this but can't find any evidence.) Name conventions are a bit bothersome to me, but then explicit names like this are even more bothersome. Anyway, I might imagine: x = property('x') def x__get(self): ... def x__set(self): ... def x__del(self): ... Where the __get/__set/__del versions are all implied simply by 'x'. (I've come to prefer x__* over _*_x, because it doesn't clash with the notion of private, but still looks obviously magic) Even better, though, would be: x = property() Where it just magically knew that it was being called 'x'. This is what I have done in SQLObject (and is done similarly in several other ORMs) using metaclass futzing, where on class instantiation you check every class variable to see if it is interested in its name and what class it is bound to. PEP 359 (the 'make' syntax) addresses this to a degree, since you can do: make property x: ... And it intrinsically knows its name. Which is no longer important if the property *contains* the items. In fact, this might indicate a flaw or other problem with the syntax; or maybe it just means that it should be extended to a non-block form, so you can say *just* make property x And it is invoked with an empty dictionary. Hrm. As another use case is a set-once attribute. This is basically a read-only attribute in descriptor form, but set once, since read-only is useless if you can't write to at least once in __init__ ;) You can do that just with descriptors right now, but it doesn't work well because the descriptor doesn't know its name. Because it doesn't know its name the descriptor has to store its value in an arbitrary attribute. Once that has happened, the object is essentially unpickleable. I describe the descriptor here: http://blog.ianbicking.org/easy-readonly-attributes.html and the problem here: http://blog.ianbicking.org/descriptor-nit.html Again, while it wants to know its name, it doesn't have any "body", and so PEP 359 as it is doesn't quite address the problem. This doesn't address any of the issues of a descriptor knowing what *class* it is bound to, only what name. But I've found the class part to be much more difficult than the name, due to subclassing. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From thomas at python.org Fri Apr 14 22:57:43 2006 From: thomas at python.org (Thomas Wouters) Date: Fri, 14 Apr 2006 22:57:43 +0200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: References: <443E91E0.5020304@colorstudy.com> <443F31A3.5050907@gmail.com> <9e804ac0604140533q47e480eahd47dbc5c4efa3be6@mail.gmail.com> Message-ID: <9e804ac0604141357v25366a84o9303b097d4236395@mail.gmail.com> On 4/14/06, Guido van Rossum wrote: > > > But first we need to make all tests pass! If anyone wants to help out > in the p3yk (sic) branch, please pick one of the failing tests, figure > out why it fails, and propose a fix. (You can the fix in SF, assign it > to me, and mail me about it. I'll assign it back to you to signal that > it's okay to check it in.) I had a bunch already fixed (although some quick-and-dirty like), so I cleaned those up and submitted them as separate patches. Then I figured I'd do some more, so you have a couple of patches assigned now ;P If I'm going to be the only one fixing tests this weekend, it may be more efficient to have me make a single big patch and upload that, instead, though :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060414/56948458/attachment.htm From fumanchu at amor.org Sat Apr 15 02:43:42 2006 From: fumanchu at amor.org (Robert Brewer) Date: Fri, 14 Apr 2006 17:43:42 -0700 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) References: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com><443AF921.1090508@canterbury.ac.nz> Message-ID: <435DF58A933BA74397B42CDEB8145A86224BA1@ex9.hostedexchange.local> Talin wrote: > What remains is a consise way to specify bound vs. > unbound variables I would prefer not to have to > restrict people to using a limited set of pre-declared > undefined variables, i.e. X, Y, Z, etc.; plus I > would want to be able to associate additional > constraints with those unbound variables. In Dejavu I did that with a separate "kwargs" attribute on the Expression object. You can set the kwargs attribute and it will be used when the Expression is evaluated: >>> e = logic.Expression(lambda a, b, **kw: a + b + kw['c']) >>> e.kwargs.update({'c': 16}) >>> e.evaluate(1, 8) 25 The "evaluate" method is a synonym for __call__ as well, and you can override previously-bound kwargs during evaluation: >>> e(1, 8, c=256) 265 Robert Brewer System Architect Amor Ministries fumanchu at amor.org -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060414/e0a32dbf/attachment.html From ian.bollinger at gmail.com Sat Apr 15 05:32:17 2006 From: ian.bollinger at gmail.com (Ian D. Bollinger) Date: Fri, 14 Apr 2006 23:32:17 -0400 Subject: [Python-3000] Allowing underscores in numeric constants. Message-ID: <44406941.3090700@gmail.com> Personally I like the idea (stolen from Perl, and others) of allowing underscores in numbers as to group digits, for instance: one_million = 1_000_000 maximum = 0xFF_FF_FF_FF Once four or more identical digits appear in a row, I find a number starts to become unwieldy, and separators would be a useful visual aid. -- Ian D. Bollinger From brett at python.org Sat Apr 15 11:45:10 2006 From: brett at python.org (Brett Cannon) Date: Sat, 15 Apr 2006 02:45:10 -0700 Subject: [Python-3000] Will we have a true restricted exec environment for python-3000? In-Reply-To: <443F6D8F.3080007@gmail.com> References: <44373866.9070305@alumni.rice.edu> <443789FC.3000802@v.loewis.de> <44379062.1070004@gmail.com> <443A89C2.3040100@colorstudy.com> <443F5B30.9050209@gmail.com> <443F5EC3.2010204@fas.harvard.edu> <443F6D8F.3080007@gmail.com> Message-ID: On 4/14/06, Nick Coghlan wrote: > Ivan Krstic wrote: > > Nick Coghlan wrote: > >> This is why I think the first step in a solid Python restricted > >> execution framework isn't an implementation activity but a > >> research/scoping activity, looking at the various systems already out > >> there, and the various trade-offs involved in each. > > > > This might be a good point to invite Ben Laurie (of Apache-SSL fame) to > > jump into the conversation. He researched restricted Python execution > > when working on what eventually became CaPerl: > > > > http://www.links.org/?p=28 > > > > See also the responses to Ted Leung's blog post about this > > (http://www.sauria.com/blog/2005/Nov/27), with comments by Phillip Eby > > and Ian. > > Ah, interesting. Especially this: > http://svn.zope.org/Zope3/trunk/src/zope/security/untrustedinterpreter.txt?view=auto > > So one avenue definitely worth investigating would be extracting Zope's > untrusted code execution security mechanisms into a form that can be used > independently of the rest of Zope. That would probably be a lot nicer than > having to start from scratch, but there's the question of how tightly > integrated that mechanism is with the rest of Zope's infrastructure. It may > prove more effective to build something *modelled* on Zope's system, without > actually reusing their code. > > I'll also note the similarities between this model and what Ian and I were > suggesting - the restricted subinterpreter would contain only 'safe' builtins, > with any unsafe operations being proxied back to the original application > (which could then allow or deny them as it saw fit). Anyone have any other links to stuff, especially research papers? My dissertation topic is adding Python into HTML to be at the same usage level as JavaScript (have not started work on it yet; just decided on the topic this past week). This means sandboxing Python like JavaScript is so that Python code doesn't hurt the browser or the system it is running on. Won't cover C code, but it should still be at least a good starting point. -Brett From talin at acm.org Sat Apr 15 12:32:21 2006 From: talin at acm.org (Talin) Date: Sat, 15 Apr 2006 10:32:21 +0000 (UTC) Subject: [Python-3000] More wishful thinking Message-ID: Another grab-bag of language ideas / wishes. Some of these are items that have been raised before on other lists - I want to get them written down so that they can be rejected quickly :) 1) I don't know if this is already on the table, but I sure would like to be able to have more options as far as mixing positional and keyword arguments. A common case that I run into is where I have a function that takes a variable number of positional arguments, but also takes a couple of optional keyword arguments. What I would like to be able to write is: def function( *args, option=False ): ... # Puts (1, 2, 3) into args, option is False function( 1, 2, 3 ) # Puts (1, 2, 3) into args, option is True function( 1, 2, 3, option=True ) In other words, any non-named arguments would be put into 'args', and the 'option' argument would only be used if explicitly named. However, Python gives me a syntax error when I try this - it wants the *args to appear after all other arguments. But I can't write: def function( option=False, *args ) because now option will be assigned the value '1', and *args will get (2, 3). Generally what I end up having to do is: def function( *args, **kwargs ): option = kwargs.get( 'option', False ) ...which is both clumsy and ugly. 2) A suggestion which I've seen others bring up before is the use of the * operator for tuple packing / unpacking operations, i.e.: a, *b = (1, 2, 3) Yes, I realize that you can also do: a, b = v[ 0 ], v[ 1: ] or other variations. But it does seem like a logical extension. Note that I'm only proposing this for lists, not dicts -- generalizing the **syntax seems rather less useful. 2a) One particularly useful variation of *list (again, one which I have brought up before, as have others), is the "yield *" syntax. The primary use case is for chained generators, that is a generator that is yielding the output of multiple sub-generators. Here is an example where we have a list of patterns which we want to check against all of the nodes of a tree, and yield all matching results. Each tree node has a 'left' and 'right' attribute. def search( pattern_list, tree ): for pattern in pattern_list: if pattern.match( tree.name ): yield tree if tree.left for result in search( pattern_list, tree.left ): yield result if tree.rght: for result in search( pattern_list, tree.right ): yield result What's pessimal about this is that each result has to bubble up the stack - so if you are checking a node that is 6 levels down the tree, each value gets yielded 6 times, and the execution context gets changed 12 times - 6 times going up the stack and 6 times coming back down again. What I'd like to write instead would be: def search( pattern_list, tree ): for pattern in pattern_list: if pattern.match( tree.name ): yield tree if tree.left yield *search( pattern_list, tree.left ) yield *search( pattern_list, tree.right ) The 'yield *' syntax basically means 'yield all of the values from the specified iterator, one at a time.' Ideally, this would be implemented via a "short-circuit", such that the output of the lowest-level generator was feeding results directly into the final consumer. In other words, the consumer would be temporarily redirected to consume values directly from the innermost iterator, and when that iteration was complete, the consumer's connection to the original iterator would be restored. (I suspect that the new yield behavior in 2.5 could be used to solve this problem in a different way, however at this point I'm not clever enough to be able to figure out what that might be.) 3) str.join should automatically stringify its arguments. I'm constantly having to write: ' '.join( [ str( x ) for x in items ] ) when I ought to be able to write: ' '.join( items ) I suppose I could do something like: ' '.join( string_iter( items ) ) but having a string function automatically convert its input arguments to strings seems intuitive to me. 4) I still want a prettier lambda. 'nuff said :) 5) At one point, I was a big fan of reduce(), untill I realized that 99% of my use cases were just variations of str.join() :) Even so, I've always wondered if it would be possible to invent a "comprehension- style" syntax for reduce. I have absolutely no good ideas on this one, even though I've been pondering it for years... 6) Although I would not want to propose that Python adopt C's 'assignment is an expression' behavior, I find myself constantly wanting to combine the following two statements into one: m = re.match( ... ) if m: # do something with m -- Talin From ferringb at gmail.com Sat Apr 15 13:09:21 2006 From: ferringb at gmail.com (Brian Harring) Date: Sat, 15 Apr 2006 04:09:21 -0700 Subject: [Python-3000] More wishful thinking In-Reply-To: References: Message-ID: <20060415110920.GA11063@nightcrawler> On Sat, Apr 15, 2006 at 10:32:21AM +0000, Talin wrote: > 2a) One particularly useful variation of *list (again, one which I have > brought up before, as have others), is the "yield *" syntax. The primary > use case is for chained generators, that is a generator that is yielding > the output of multiple sub-generators. Choice of operator there isn't all that grand; at least to me, *sequence means expand the sequence out as positional args, eg i = (1,2,3,4) yield *i reads as yield 1,2,3,4 when you're proposing it behave more like map(yield, i) # and yes, I realize that isn't valid python. Personally want something similar, but -1 for the syntax there. Alternatives? ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060415/eee0f9e3/attachment.pgp From jjl at pobox.com Sat Apr 15 22:54:54 2006 From: jjl at pobox.com (John J Lee) Date: Sat, 15 Apr 2006 20:54:54 +0000 (UTC) Subject: [Python-3000] Allowing underscores in numeric constants. In-Reply-To: <44406941.3090700@gmail.com> References: <44406941.3090700@gmail.com> Message-ID: On Fri, 14 Apr 2006, Ian D. Bollinger wrote: > Personally I like the idea (stolen from Perl, and others) of allowing > underscores in numbers as to group digits, for instance: > one_million = 1_000_000 > maximum = 0xFF_FF_FF_FF > > Once four or more identical digits appear in a row, I find a number > starts to become unwieldy, and separators would be a useful visual aid. This was discussed at length on python-dev not long ago. John From ian.bollinger at gmail.com Sun Apr 16 00:24:58 2006 From: ian.bollinger at gmail.com (Ian D. Bollinger) Date: Sat, 15 Apr 2006 18:24:58 -0400 Subject: [Python-3000] Allowing underscores in numeric constants. In-Reply-To: References: <44406941.3090700@gmail.com> Message-ID: <444172BA.5040302@gmail.com> John J Lee wrote: > On Fri, 14 Apr 2006, Ian D. Bollinger wrote: > > >> Personally I like the idea (stolen from Perl, and others) of allowing >> underscores in numbers as to group digits, for instance: >> one_million = 1_000_000 >> maximum = 0xFF_FF_FF_FF >> >> Once four or more identical digits appear in a row, I find a number >> starts to become unwieldy, and separators would be a useful visual aid. >> > > This was discussed at length on python-dev not long ago. > > To my my knowledge this was discussed on c.l.p, not python-dev, and there appeared to be little consensus as to whether it would be a good idea or not. -- Ian D. Bollinger From rasky at develer.com Sun Apr 16 00:31:12 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 16 Apr 2006 00:31:12 +0200 Subject: [Python-3000] More wishful thinking References: Message-ID: <008701c660dc$4d141070$2452fea9@bagio> Talin wrote: > 6) Although I would not want to propose that Python adopt C's > 'assignment is an expression' behavior, I find myself constantly > wanting to combine the following two statements into one: > > m = re.match( ... ) > if m: > # do something with m I'm sure you were talking more generically, but as for this specific case (which is indeed very common), I always wondered why re couldn't export wrappers around bound methods of the last match object, so to allow: if re.match(...): x,y = re.group(1), re.group(2) elif re.match(..): ... There are many other modules in standard library which exposes a function-based interface where the functions are actually bound methods of a static object (logging, for instance). If "re" gained this, it'd be great. In fact, this could probably be done in 2.x as well. Giovanni Bajo From skip at pobox.com Sun Apr 16 01:04:19 2006 From: skip at pobox.com (skip at pobox.com) Date: Sat, 15 Apr 2006 18:04:19 -0500 Subject: [Python-3000] More wishful thinking In-Reply-To: <008701c660dc$4d141070$2452fea9@bagio> References: <008701c660dc$4d141070$2452fea9@bagio> Message-ID: <17473.31731.38680.400480@montanaro.dyndns.org> Giovanni> I'm sure you were talking more generically, but as for this Giovanni> specific case (which is indeed very common), I always wondered Giovanni> why re couldn't export wrappers around bound methods of the Giovanni> last match object, so to allow: Giovanni> if re.match(...): Giovanni> x,y = re.group(1), re.group(2) Giovanni> elif re.match(..): Giovanni> ... Because then the re module wouldn't be thread safe. Skip From greg.ewing at canterbury.ac.nz Sun Apr 16 06:11:16 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 16 Apr 2006 16:11:16 +1200 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <443E9193.7000908@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> Message-ID: <4441C3E4.4070504@canterbury.ac.nz> Ian Bicking wrote: > Class.some_method(self, blah) seems like a corner case. How often do > you do that? Very frequently, when calling inherited __init__ methods. > If it is calling a superclass method, then super(Class, > self).some_method() should be used That's a matter of opinion. In the case of __init__ methods, most of the time it simply doesn't work, because different __init__ methods rarely have the same signature. In other cases, so far in my code I've found exactly zero use cases for super(). I've considered using it a couple of times, but changed my mind when I realised that it wouldn't do what I wanted, and it was better to redesign my inheritance hierarchy so that I could use plain inherited calls instead. So I would never tell anyone that they "should" be using super(). -- Greg From greg.ewing at canterbury.ac.nz Sun Apr 16 06:46:58 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 16 Apr 2006 16:46:58 +1200 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <20060413125831.8lonimb5yvwggwcw@login.werra.lunarpages.com> References: <20060413125831.8lonimb5yvwggwcw@login.werra.lunarpages.com> Message-ID: <4441CC42.5000300@canterbury.ac.nz> Michael Chermside wrote: > def self.my_method(arg1, arg2): > # body goes here Unfortunately, this would clash with another potential future feature, that of allowing defs into arbitrary lvalues instead of just names. I think I'd rather it was reserved for that. -- Greg From greg.ewing at canterbury.ac.nz Sun Apr 16 07:18:55 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 16 Apr 2006 17:18:55 +1200 Subject: [Python-3000] symbols? In-Reply-To: <1144992574.3259.58.camel@localhost.localdomain> References: <20060412051514.1p0h6tg0v0ecksoc@login.werra.lunarpages.com> <1144992574.3259.58.camel@localhost.localdomain> Message-ID: <4441D3BF.8090108@canterbury.ac.nz> Adam DePrince wrote: > IIRC one of the goals > with P3K is to allow Python to compile unicode source files ... > Personally, I rather like the direction of the symbols idea, but am > unsure if I like the :symbol form for the literal. I give it a +0.5. Putting these thoughts together, we could use different alphabets to distinguish symbols from other identifers. For example, ordinary names -- Roman and Greek; symbols -- Cyrillic, Hebrew and Arabic, possibly others. A single lambda, of course, would be reserved for replacing the keyword "lambda", thus making lambda expressions more concise and freeing up a word for use as a name. -- Greg From greg.ewing at canterbury.ac.nz Sun Apr 16 07:20:04 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 16 Apr 2006 17:20:04 +1200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: References: <443E91E0.5020304@colorstudy.com> Message-ID: <4441D404.6030802@canterbury.ac.nz> Jim Jewett wrote: > (I wouldn't want to give up slicing, though, which might make the > no-iteration trickier.) You'd want to allow slicing but not indexing -- i.e. s[i:j] is okay but not s[i]. -- Greg From aahz at pythoncraft.com Sun Apr 16 07:22:59 2006 From: aahz at pythoncraft.com (Aahz) Date: Sat, 15 Apr 2006 22:22:59 -0700 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <4441D404.6030802@canterbury.ac.nz> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> Message-ID: <20060416052258.GA12059@panix.com> On Sun, Apr 16, 2006, Greg Ewing wrote: > Jim Jewett wrote: >> >> (I wouldn't want to give up slicing, though, which might make the >> no-iteration trickier.) > > You'd want to allow slicing but not indexing -- i.e. > s[i:j] is okay but not s[i]. Ewww. I've been getting steadily less comfortable with this idea, despite the fact that I've been bitten a few times by iterating over strings. This, however, is enough to make me a firm thumbs-down on the idea: I think it is gratuitous code breakage. I think it is perfectly reasonable to want to look at a single character of a string. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From talin at acm.org Sun Apr 16 07:37:16 2006 From: talin at acm.org (Talin) Date: Sun, 16 Apr 2006 05:37:16 +0000 (UTC) Subject: [Python-3000] More wishful thinking References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> Message-ID: pobox.com> writes: > Giovanni> I'm sure you were talking more generically, but as for this > Giovanni> specific case (which is indeed very common), I always wondered > Giovanni> why re couldn't export wrappers around bound methods of the > Giovanni> last match object, so to allow: > > Giovanni> if re.match(...): > Giovanni> x,y = re.group(1), re.group(2) > Giovanni> elif re.match(..): > Giovanni> ... > > Because then the re module wouldn't be thread safe. > > Skip True, although I suppose that you could add a "last match" to regular expression objects: re_pattern = re.compile( ... ) if re_pattern.match( string ): text = re_pattern.lastmatch.group( 1 ) This violates the stateless nature of regular expression objects, which is a minor wart. But other than that, the code seems cleaner to me - in other words, we concern ourselves only test part of the match first, and only after we've determined that there is a match, we start to look at the specific attributes of the match. -- Talin From talin at acm.org Sun Apr 16 07:49:37 2006 From: talin at acm.org (Talin) Date: Sun, 16 Apr 2006 05:49:37 +0000 (UTC) Subject: [Python-3000] More wishful thinking References: Message-ID: Talin acm.org> writes: > 2) A suggestion which I've seen others bring up before is the use of > the * operator for tuple packing / unpacking operations, i.e.: > > a, *b = (1, 2, 3) I wanted to add another case that I run across in my code a lot. I often times want to split off a single leading word or token from a string, but I don't know in advance whether or not the split will actually succeed. For example, suppose you want to render the first word of a paragraph in a different style: first, rest = paragraph.split( ' ', 1 ) Unfortunately, if the paragraph only contains a single word, this blows up. So what you end up having to do is: parts = paragraph.split( ' ', 1 ) if len( parts ) > 1: first, rest = parts else: first = parts[ 0 ] rest = "" My objection here is that the intent of the code is cluttered up by the error-handling logic. If I could do an argument-style unpack, however, I could instead write: first, *rest = paragraph.split( ' ', 1 ) It does mean that 'rest' is wrapped in a tuple, but that's not a terrible hardship. If there's no second word, then 'rest' is an empty tuple, which is easy to test with "if rest" and such. -- Talin From guido at python.org Sun Apr 16 13:06:01 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 13:06:01 +0200 Subject: [Python-3000] More wishful thinking In-Reply-To: References: Message-ID: On 4/15/06, Talin wrote: > Another grab-bag of language ideas / wishes. In the future, please use one idea/wish per email and use a subject line indicative of the subject (so *don't* use "Another wish" :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 16 14:02:29 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 14:02:29 +0200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <20060416052258.GA12059@panix.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> Message-ID: On 4/16/06, Aahz wrote: > On Sun, Apr 16, 2006, Greg Ewing wrote: > > Jim Jewett wrote: > >> > >> (I wouldn't want to give up slicing, though, which might make the > >> no-iteration trickier.) > > > > You'd want to allow slicing but not indexing -- i.e. > > s[i:j] is okay but not s[i]. (I'm surprised by this. Why would s[i] be wrong?) > Ewww. I've been getting steadily less comfortable with this idea, > despite the fact that I've been bitten a few times by iterating over > strings. This, however, is enough to make me a firm thumbs-down on the > idea: I think it is gratuitous code breakage. I think it is perfectly > reasonable to want to look at a single character of a string. I think you may be right. I implemented this (it was really simple to do) but then found I had to fix tons of places that iterate over strings. For example: - The sre parser and compiler use things like set("0123456789") and also iterate over the characters of the input regexp to parse it. - difflib has an API defined for either two lists of strings (a typical line-by-line diff of a file), or two strings (a typical intra-line diff), or even two lists of anything (for a generalized sequence diff). - small changes in optparse.py, textwrap.py, string.py. And I'm not even at the point where the regrtest.py framework even works (due to the difflib problem). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Sun Apr 16 14:26:56 2006 From: skip at pobox.com (skip at pobox.com) Date: Sun, 16 Apr 2006 07:26:56 -0500 Subject: [Python-3000] More wishful thinking In-Reply-To: References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> Message-ID: <17474.14352.844015.16479@montanaro.dyndns.org> >> Because then the re module wouldn't be thread safe. talin> True, although I suppose that you could add a "last match" to talin> regular expression objects: talin> re_pattern = re.compile( ... ) talin> if re_pattern.match( string ): talin> text = re_pattern.lastmatch.group( 1 ) No, it would still be thread-unsafe. The re_pattern object can be shared by multiple threads. Skip From guido at python.org Sun Apr 16 15:45:52 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 15:45:52 +0200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> Message-ID: On 4/16/06, Guido van Rossum wrote: > I think you may be right. I implemented this (it was really simple to > do) but then found I had to fix tons of places that iterate over > strings. I'm abandoning this project; the patch is SF patch 1471291. I'm no longer in favor of this idea; it's just not practical, and the premise that there are few good reasons to iterate over a string has been refuted by the use cases I found in both sre and difflib. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Sun Apr 16 16:40:05 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 16 Apr 2006 07:40:05 -0700 Subject: [Python-3000] More wishful thinking In-Reply-To: References: Message-ID: <20060416073755.A7F9.JCARLSON@uci.edu> Talin wrote: > > Talin acm.org> writes: > > > 2) A suggestion which I've seen others bring up before is the use of > > the * operator for tuple packing / unpacking operations, i.e.: > > > > a, *b = (1, 2, 3) > > I wanted to add another case that I run across in my code a lot. > > I often times want to split off a single leading word or token > from a string, but I don't know in advance whether or not the > split will actually succeed. > > For example, suppose you want to render the first word of > a paragraph in a different style: > > first, rest = paragraph.split( ' ', 1 ) [snip] See str.partition in Python 2.5 (I don't know if it is implemented yet or not). 'Python is fun!'.partition(' ') -> 'Python', ' ', 'is fun' - Josiah From rasky at develer.com Sun Apr 16 17:31:23 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 16 Apr 2006 17:31:23 +0200 Subject: [Python-3000] More wishful thinking References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> Message-ID: <014501c6616a$d169b030$2452fea9@bagio> skip at pobox.com wrote: > Giovanni> I'm sure you were talking more generically, but as for > this Giovanni> specific case (which is indeed very common), I > always wondered Giovanni> why re couldn't export wrappers around > bound methods of the Giovanni> last match object, so to allow: > > Giovanni> if re.match(...): > Giovanni> x,y = re.group(1), re.group(2) > Giovanni> elif re.match(..): > Giovanni> ... > > Because then the re module wouldn't be thread safe. Can't you put the global last-match object into TLS? Giovanni Bajo From skip at pobox.com Sun Apr 16 19:26:42 2006 From: skip at pobox.com (skip at pobox.com) Date: Sun, 16 Apr 2006 12:26:42 -0500 Subject: [Python-3000] More wishful thinking In-Reply-To: <014501c6616a$d169b030$2452fea9@bagio> References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> <014501c6616a$d169b030$2452fea9@bagio> Message-ID: <17474.32338.297184.343720@montanaro.dyndns.org> Giovanni> Can't you put the global last-match object into TLS? What if you're not using threads? Skip From rasky at develer.com Sun Apr 16 20:36:41 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sun, 16 Apr 2006 20:36:41 +0200 Subject: [Python-3000] More wishful thinking References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> <014501c6616a$d169b030$2452fea9@bagio> <17474.32338.297184.343720@montanaro.dyndns.org> Message-ID: <02d201c66184$b4a1ae70$2452fea9@bagio> skip at pobox.com wrote: > Giovanni> Can't you put the global last-match object into TLS? > > What if you're not using threads? Hey, that was *my* point. For all the Python programs which don't use multi-threading, having a global match object is a very important shortcut. It allows two main things: having multiple if/elif regexp chains which do not add an indent level each time, and have one less object around to care of. I think that in 99% percent of the times I have been using the re module, I either wanted to use the match object immediatly, or simply didn't care about it (just wanted to know whether there was a match or not). I can't think a *single* case where I stored the match object aside and used it later. Maybe because it's easier to immediately extract the matched substrings and store *those* aside (since, after all, it's the data you wanted to extract). Multi-threading programs could be warned that the global match object is unsafe and they should not use it. If you are really concerned of adding the global match objects for threading programs too, the TLS thingie could be activated on request, but I don't think it's worth. Really, having the match objects is cute and good, but having the fast-path shortcut (which would be taken 99% of times) is good as well. Giovanni Bajo From guido at python.org Sun Apr 16 21:37:37 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 21:37:37 +0200 Subject: [Python-3000] More wishful thinking In-Reply-To: <02d201c66184$b4a1ae70$2452fea9@bagio> References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> <014501c6616a$d169b030$2452fea9@bagio> <17474.32338.297184.343720@montanaro.dyndns.org> <02d201c66184$b4a1ae70$2452fea9@bagio> Message-ID: On 4/16/06, Giovanni Bajo wrote: > For all the Python programs which don't use multi-threading, having a global > match object is a very important shortcut. It allows two main things: having > multiple if/elif regexp chains which do not add an indent level each time, and > have one less object around to care of. Well, sorry, but this won't happen, so you can save your breath. If you want the shortcut, write a module that implements that API yourself. Should be about 3 lines of code. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 16 21:54:59 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 21:54:59 +0200 Subject: [Python-3000] More wishful thinking In-Reply-To: References: Message-ID: On 4/15/06, Talin wrote: > Another grab-bag of language ideas / wishes. Some of these are > items that have been raised before on other lists - I want to get them > written down so that they can be rejected quickly :) > > 1) I don't know if this is already on the table, but I sure would like > to be able to have more options as far as mixing positional and > keyword arguments. I agree. Please produce a patch that implements what you want. > 2) A suggestion which I've seen others bring up before is the use of > the * operator for tuple packing / unpacking operations, i.e.: > > a, *b = (1, 2, 3) I'm not keen on this one, so I'm not inviting you to write a patch. But it's not terrible either; if some continent of old-timers agrees to add it, it could happen. > 2a) One particularly useful variation of *list (again, one which I have > brought up before, as have others), is the "yield *" syntax. The primary > use case is for chained generators, that is a generator that is yielding > the output of multiple sub-generators. This doesn't fly so well for the new generator API introduced in 2.5. Also, the shortcut that you are hoping for won't happen; there are too many reasons to keep the stack frames involved (like tracebacks). A solution that would work for the new API extensions might be to add an optional expression to the continue statement, which would pass that value into the iterator using send(). If someone wants to produce a patch for this we could do it in 2.6 or 3.0. > 3) str.join should automatically stringify its arguments. I'm on the fence about this one. In 2.x, a problem is that if a later object is a unicode string, the earlier objects should be converted using unicode() instead of str(). But in 3.0 we won't have this. I suggest that someone write up a patch and then we'll look at how well it works. > 5) At one point, I was a big fan of reduce(), untill I realized that 99% of > my use cases were just variations of str.join() :) > > Even so, I've always wondered if it would be possible to invent a > "comprehension- style" syntax for reduce. I have absolutely no good > ideas on this one, even though I've been pondering it for years... There isn't one. But generator expressions actually were invented to make this more feasible. The sum() builtin (like str().join(), max() and any() etc.) is an example of an *accumulator* function which takes an iterable and returns a single value. This pattern is easily extended to other common accumulator idioms. Beyond that, a for loop looks attractive... > 6) [global regex matching state] I already rejected this in a previous response to a later message in this thread (trying to squash churn). PS. IMPORTANT! You may ask, "why does he ask for patches instead of PEPs suddenly? why the reversal?" This is because for some issues, code just speaks so much clearer than words. The IETF requires working implementations before accepting an RFC into the standards track. We chose not to do this for PEPs. But I believe that PEPs written after the code has been written and tested can be much clearer on corner cases. We've had several important PEPs accepted recently that had to be modified extensively after implementation attempts showed either holes in the specs or choices that were less than optimal from an implementation POV. (E.g. both PEP 342 and PEP 343 were adjusted after the implementation.) Most recently, we had at least half a dozen folks agree that it would be a good idea to disallow iterating over strings in 3.0. But after spending no more than an hour and a half of implementation effort I came to realize that the approach is doomed. Reaching this conclusing during PEP discussion would have cost much more time. So, especially for things that I think *may* be worth while, I'm proposing to write a patch first. The patches should be written againt the p3yk (sic) branch, BTW. We should still write PEPs too (unless the feature is so simple that it can be described in a single line added to PEP 3100). Python 3000 PEPs need to be especially clear on the backwards compatibility issues. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 16 22:06:45 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 22:06:45 +0200 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On 4/14/06, Kendall Clark wrote: > Fair enough. My primary use case is using Python as a host for a > Domain Specific Language. [...] Then I suggest that having a good syntax is important; you don't want your DSL to look ugly because that would not contribute to having fun. Let's assume that !foo can be used as a symbol; it's ugly but perhaps not too ugly; and we can pick a better character later. I propose you (or someone else who would like to see this happen) an experiment: write a lexer + parser extension that implement this; write a symbol object type implementation that behave the way you want it to; and then use it to implement a mid-size DSL (two or three would be even better). Then, with this experience under your belt, you can report back and we can decide whether this really is a valuable addition. I know this is a lot of work to ask the proponents to do. But hey, if you don't want to do the work, well, I don't either, so it won't happen, so you certainly won't get your feature. If you do the work, you might have to quit at any time because there's a flaw to the idea; that would be a useful result and you will have learned something (and hopefully you'll pass back the information so we'll all learn). Or you might end up with something so Pythonic and useful that I would be a dick not to accept it as a new feature. Do we have a deal? PS. On my recent tendency to request code instead of PEPs, see my PS recently to another thread in this list. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 16 22:10:34 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 22:10:34 +0200 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <4441C3E4.4070504@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> Message-ID: On 4/16/06, Greg Ewing wrote: > Ian Bicking wrote: > > > Class.some_method(self, blah) seems like a corner case. How often do > > you do that? > > Very frequently, when calling inherited __init__ methods. > > > If it is calling a superclass method, then super(Class, > > self).some_method() should be used > > That's a matter of opinion. In the case of __init__ > methods, most of the time it simply doesn't work, > because different __init__ methods rarely have the > same signature. How does that prevent you from using super()? (Hint: it doesn't. :-) (Not that I disagree strongly with your sentiment against super.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 16 22:16:32 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 22:16:32 +0200 Subject: [Python-3000] Allowing underscores in numeric constants. In-Reply-To: <44406941.3090700@gmail.com> References: <44406941.3090700@gmail.com> Message-ID: On 4/15/06, Ian D. Bollinger wrote: > Personally I like the idea (stolen from Perl, and others) of allowing > underscores in numbers as to group digits, for instance: > one_million = 1_000_000 > maximum = 0xFF_FF_FF_FF > > Once four or more identical digits appear in a row, I find a number > starts to become unwieldy, and separators would be a useful visual aid. I do recall this whas brought up before; I don't recall whether a good reason was given to kill it apart from that it doesn't add a whole lot of joy either. There are probably lots of niggling decisions to be made about whether int("1_000") should work, and a formatting code that inserts _ every three characters, and whether to accept abominations like 12__ or 0_x_ff or 3.14_e_1_0_0. But if someone cares enough to write a PEP that covers all the corner cases with reasonable rules *and* a complete implementation, it would be a lot harder to say no. (See my PS on code vs. PEPs in a previous post in a different thread on this list.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 16 22:24:23 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 22:24:23 +0200 Subject: [Python-3000] AST access (WAS: Adaptation vs. Generic Functions) In-Reply-To: <435DF58A933BA74397B42CDEB8145A86224BA1@ex9.hostedexchange.local> References: <5.1.1.6.0.20060410130428.01e2a298@mail.telecommunity.com> <443AF921.1090508@canterbury.ac.nz> <435DF58A933BA74397B42CDEB8145A86224BA1@ex9.hostedexchange.local> Message-ID: If someone really wants AST access in Python 3000, they should design and implement an API and propose it. I see it as a borderline feature from the POV of my own agenda for Python 3000, but not as something I'd likely reject if it were to be offered on a silver platter; especially if there was a clear path towards providing the same feature for IronPython, Jython and PyPy. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 16 22:26:43 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 22:26:43 +0200 Subject: [Python-3000] Python 3000 and the Google Summer of Code Message-ID: Some of the coding projects I've recently been proposing as challenges to certain wild feature proposals might make good topics for the Google Summer of Code (see http://code.google.com/soc/ ). We'd need someone to volunteer as a mentor, and a student to do the work over the summer; the student would get $4500 and the mentor $500 (or a donation to the PSF worth that much). (I can't volunteer to be a mentor myself; I already have a project I'd like to mentor and an intern residing at Google.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kendall at monkeyfist.com Sun Apr 16 22:34:37 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Sun, 16 Apr 2006 16:34:37 -0400 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On Apr 16, 2006, at 4:06 PM, Guido van Rossum wrote: > On 4/14/06, Kendall Clark wrote: >> Fair enough. My primary use case is using Python as a host for a >> Domain Specific Language. [...] > > Then I suggest that having a good syntax is important; you don't want > your DSL to look ugly because that would not contribute to having fun. Hmm, can't tell whether yr putting me on here or not, but I agree. Ugly syntax sucks. > Let's assume that !foo can be used as a symbol; it's ugly but perhaps > not too ugly; and we can pick a better character later. Seems fine. > I propose you > (or someone else who would like to see this happen) an experiment: > write a lexer + parser extension that implement this; write a symbol > object type implementation that behave the way you want it to; and > then use it to implement a mid-size DSL (two or three would be even > better). A perfectly reasonable suggestion. I like this better than writing a PEP. Alas, that doesn't mean I have time to do it, which sucks. I guess I shouldn't have proposed a feature for Py3K w/out the time to back it up with code. > I know this is a lot of work to ask the proponents to do. But hey, if > you don't want to do the work, well, I don't either, so it won't > happen, so you certainly won't get your feature. Not a matter of not wanting to, but rather not having time. I don't think this is too much to ask, actually. It's a good way to make these sorts of decisions; at least, it's better than endless arguing abstractly on this list. > If you do the work, > you might have to quit at any time because there's a flaw to the idea; > that would be a useful result and you will have learned something (and > hopefully you'll pass back the information so we'll all learn). Or you > might end up with something so Pythonic and useful that I would be a > dick not to accept it as a new feature. Do we have a deal? I think it's a good idea. I'm unlikely to take you up on it due to lack of time. Alas. Perhaps someone else cares enough. > PS. On my recent tendency to request code instead of PEPs, see my PS > recently to another thread in this list. I missed it; but I have no problem with this approach. Cheers, Kendall From guido at python.org Sun Apr 16 22:40:13 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 22:40:13 +0200 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On 4/16/06, Kendall Clark wrote: > A perfectly reasonable suggestion. I like this better than writing a > PEP. Alas, that doesn't mean I have time to do it, which sucks. I > guess I shouldn't have proposed a feature for Py3K w/out the time to > back it up with code. Perhaps you have the time to mentor someone else doing it for the Google Summer of Code program? The PSF is looking for projects and mentors. (Even though there's nothing on the soc page yet.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sun Apr 16 22:57:14 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 16 Apr 2006 22:57:14 +0200 Subject: [Python-3000] Is reload() feasible? In-Reply-To: <443FC73D.10207@colorstudy.com> References: <443E9AEF.7040902@colorstudy.com> <443FA371.4060401@benjiyork.com> <443FC73D.10207@colorstudy.com> Message-ID: I'm not against a better reload() if a reasonable set of heuristics can be defined (e.g. many people have asked for class objects to be *edited* instead of replaced; and of course there's the dependencies graph problem). I'm even OK with changing the definition of the language (slightly) to make e.g. class editing work better. If someone really wants to do this, the Google Summer of Code (Google for it :-) might be a place to get it done. We'd need a mentor and a student. It is customary (though not necessary) that a mentor be found first. See my other posts on this. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas at python.org Sun Apr 16 23:34:02 2006 From: thomas at python.org (Thomas Wouters) Date: Sun, 16 Apr 2006 23:34:02 +0200 Subject: [Python-3000] More wishful thinking In-Reply-To: References: Message-ID: <9e804ac0604161434p4c4b961cu3d40c82e122975ba@mail.gmail.com> On 4/16/06, Guido van Rossum wrote: > > On 4/15/06, Talin wrote: > > Another grab-bag of language ideas / wishes. Some of these are > > items that have been raised before on other lists - I want to get them > > written down so that they can be rejected quickly :) > > > > 1) I don't know if this is already on the table, but I sure would like > > to be able to have more options as far as mixing positional and > > keyword arguments. > > I agree. Please produce a patch that implements what you want. > > > 2) A suggestion which I've seen others bring up before is the use of > > the * operator for tuple packing / unpacking operations, i.e.: > > > > a, *b = (1, 2, 3) > > I'm not keen on this one, so I'm not inviting you to write a patch. > But it's not terrible either; if some continent of old-timers agrees > to add it, it could happen. I actually have half a patch for this (and the keyword-only argument thing, and some other generalizations of *seq and **mapping) lying around. Since it isn't going in before 2.6 anyway, I wasn't planning on doing anything with it until 2.5 is well and done. (I mostly started it to get to know the AST; PEP 308 and 328 have since done the job.) I dunno if I count as an 'old-timer', but I'm sure I can convince some of you real old fogies that it's a good idea. :) The IETF requires working implementations before accepting an > RFC into the standards track. We chose not to do this for PEPs. But I > believe that PEPs written after the code has been written and tested > can be much clearer on corner cases. Absolutely a good idea (and also how I've generally done it in the past myself.) Besides, code is easy, a PEP is a heck of a lot of work. It still has to be done, it just won't have to be wasted so often. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060416/8f278729/attachment-0001.html From skip at pobox.com Sun Apr 16 23:42:40 2006 From: skip at pobox.com (skip at pobox.com) Date: Sun, 16 Apr 2006 16:42:40 -0500 Subject: [Python-3000] More wishful thinking In-Reply-To: <02d201c66184$b4a1ae70$2452fea9@bagio> References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> <014501c6616a$d169b030$2452fea9@bagio> <17474.32338.297184.343720@montanaro.dyndns.org> <02d201c66184$b4a1ae70$2452fea9@bagio> Message-ID: <17474.47696.600874.855528@montanaro.dyndns.org> Giovanni> Can't you put the global last-match object into TLS? >> What if you're not using threads? Giovanni> Hey, that was *my* point. Giovanni> For all the Python programs which don't use multi-threading, Giovanni> having a global match object is a very important shortcut. And it breaks as soon as you decide to use threads. Suppose I'm writing a library and want to use regular expressions. I don't know how my library will be used (threaded or not). How should my re code be written, with TLS, shortcuts or the status quo? The only safe way to write is the last option. This really isn't Python 3000 material, folks. I thought my first response would trigger an "Oh yeah, I hadn't thought of that", not continual ill-conceived "yeah, but you can do this instead when the situation is right". Please take it c.l.py, hash it out there, and if a reasonable idea floats to the top of the soup, raise it on python-dev unless the idea is somehow backward-incompatible and has to wait until Python 3000. Skip From talin at acm.org Sun Apr 16 23:43:29 2006 From: talin at acm.org (Talin) Date: Sun, 16 Apr 2006 21:43:29 +0000 (UTC) Subject: [Python-3000] More wishful thinking References: Message-ID: Guido van Rossum python.org> writes: > On 4/15/06, Talin acm.org> wrote: > > Another grab-bag of language ideas / wishes. Some of these are > > items that have been raised before on other lists - I want to get them > > written down so that they can be rejected quickly :) > > > > 1) I don't know if this is already on the table, but I sure would like > > to be able to have more options as far as mixing positional and > > keyword arguments. > > I agree. Please produce a patch that implements what you want. All right, in that case, I'd like to post a general implementation strategy for criticism before I end up going too far down the wrong path. As I see it, the problem breaks down into 3 parts: -- parsing -- representing the argument list -- executing a call The parsing part is fairly straightforward. The new syntax would allow keyword arguments to appear anywhere in the argument list, intermixed with positional arguments, and after the varargs argument. (It would be possible to have non-keyword arguments after the varargs argument as well, but that's harder to implement, and I don't really see the value in it.) The kwargs argument would remain unchanged - there's no real reason to allow it to be moved around in the argument list. So the argument list syntax would be (in psuedo-grammar): (keyword | positional )* [*varargs] keyword* [**kwargs] (This is of course omitting all of the complexities of generator expressions for the moment.) Representing the argument list. The current _arguments struct looks like this: struct _arguments { asdl_seq *args; identifier vararg; identifier kwarg; asdl_seq *defaults; }; The approach that I would take would be to remove the 'vararg', and instead put the vararg argument into args. A special sentinel value in the *defaults array (say, (void*)-1) would be used to indicate that this argument was a varargs argument. Thus, for each argument (other than kwargs), you would have both an argument name and a default, where a NULL value indicated a positional argument, a non-NULL value indicated a keyword argument, and a sentinel value indicated a varargs argument. Executing a call: This is the tricky part. Specifically, we want to insure that the mapping of values to formal parameters remains the same for existing code. Start with a list of unfilled slots, one per formal parameter, including the varargs parameter. for each argument: if it is a keyword argument: find a the named slot; if found: if it is already filled, raise TypeError otherwise fill the slot with the argument else if there's a kwargs, add it there else raise TypeError if it is a positional argument if the next slot is a varargs slot append to the vararg list else fill the slot with the argument increment the slot index There might be a more optimal implementation of this, in particular it should be possible to create the varargs tuple in a single allocation rather than having to build it up incrementally. Anyway, if this seems unreasonable, or if you find any bugs let me know. I have a lot on my plate at the moment, so I don't know when I will get around to making the patch, but hopefully soon. -- Talin From talin at acm.org Sun Apr 16 23:59:24 2006 From: talin at acm.org (Talin) Date: Sun, 16 Apr 2006 21:59:24 +0000 (UTC) Subject: [Python-3000] Test and Assign [was: More wishful thinking] References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> <014501c6616a$d169b030$2452fea9@bagio> <17474.32338.297184.343720@montanaro.dyndns.org> <02d201c66184$b4a1ae70$2452fea9@bagio> <17474.47696.600874.855528@montanaro.dyndns.org> Message-ID: pobox.com> writes: > This really isn't Python 3000 material, folks. I thought my first response > would trigger an "Oh yeah, I hadn't thought of that", not continual > ill-conceived "yeah, but you can do this instead when the situation is > right". Please take it c.l.py, hash it out there, and if a reasonable idea > floats to the top of the soup, raise it on python-dev unless the idea is > somehow backward-incompatible and has to wait until Python 3000. It's also gotten off track. You're right, the discussion of the implementation details of re.match isn't Python-3000 material, but that's not what I was really talking about. I'm really more interested in the general case where you have a test, and then you have the results of the test. It would be nice to put the conditional statement first, and then once you know that the test succeeded, start to look at the specific details. Its like coming home and finding someone has left you a birthday present, so you decide to open it and see what's inside, as opposed to coming home, attempting to see what's in the birthday present, and then noticing that there isn't one. Now I agree, that this isn't a huge issue, I only bring it up because it occurs so frequently. The number of lines of code isn't affected - the only thing that would be affected is the readability of the code IMHO. I'm happy to drop the subject if that's what folks want ::) (One of the things I've been doing is going through my codebase looking for coding patterns that are awkward attempts to get around language limitations.) PS. I meant to change the title of that last post as well - could whoever replies to the "More Wishful Thinking" topic please change the title of their next posting as well? Thanks :) -- Talin From nnorwitz at gmail.com Mon Apr 17 01:41:57 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 16 Apr 2006 16:41:57 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: On 4/16/06, Guido van Rossum wrote: > Some of the coding projects I've recently been proposing as challenges > to certain wild feature proposals might make good topics for the > Google Summer of Code (see http://code.google.com/soc/ ). > > (I can't volunteer to be a mentor myself; I already have a project I'd > like to mentor and an intern residing at Google.) Awwww, I was planning to rope you in as mentor. :-) I plan to mentor, I'm not sure how many projects I can handle though. I hope all the mentors from last year will mentor again. I'm still not clear on exactly how mentors request access. I know I can accept them once they are put into the system though. n From brett at python.org Mon Apr 17 01:50:07 2006 From: brett at python.org (Brett Cannon) Date: Sun, 16 Apr 2006 16:50:07 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: On 4/16/06, Neal Norwitz wrote: > On 4/16/06, Guido van Rossum wrote: > > Some of the coding projects I've recently been proposing as challenges > > to certain wild feature proposals might make good topics for the > > Google Summer of Code (see http://code.google.com/soc/ ). > > > > (I can't volunteer to be a mentor myself; I already have a project I'd > > like to mentor and an intern residing at Google.) > > Awwww, I was planning to rope you in as mentor. :-) I plan to mentor, > I'm not sure how many projects I can handle though. > > I hope all the mentors from last year will mentor again. I'm still > not clear on exactly how mentors request access. I know I can accept > them once they are put into the system though. I can mentor on the restricted execution topic since that is going to be covered by my dissertation. I just will need to ask that the student not use it to base further research that is meant to lead to publication since that would lead to publication competition between me and the student. But if no student takes that one up I can still mentor one student, but I didn't feel I did a spectacular job last summer either. Then again, if my host at Google wants me to spend my working hours being a good mentor I could possibly be convinced to make sure I get assigned one student (two turned out to be too much). =) -Brett From nnorwitz at gmail.com Mon Apr 17 02:01:12 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Sun, 16 Apr 2006 17:01:12 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: On 4/16/06, Brett Cannon wrote: > > again, if my host at Google wants me to spend my working hours being a > good mentor I could possibly be convinced to make sure I get assigned > one student (two turned out to be too much). =) I heard your host is a slave driver and an asshole. You should start getting some extra sleep now so you can work 200 hours a week during the summer. Your first task will be to figure out how to find the extra 32 hours for each week. :-) I think it's reasonable to take a couple hours here and there to help mentor. How much time did it take to mentor last year per student? How could you do a better job (invest less time and get more out of the student)? ISTM like it's mostly just keeping in touch via email. n From tjreedy at udel.edu Mon Apr 17 02:02:34 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 16 Apr 2006 20:02:34 -0400 Subject: [Python-3000] More wishful thinking References: Message-ID: "Talin" wrote in message news:loom.20060416T073739-283 at post.gmane.org... > Unfortunately, if the paragraph only contains a single word, > this blows up. So what you end up having to do is: > > parts = paragraph.split( ' ', 1 ) > if len( parts ) > 1: > first, rest = parts > else: > first = parts[ 0 ] > rest = "" > > My objection here is that the intent of the code is cluttered > up by the error-handling logic. So wrap it all up in a function, especially if you need the logic more than once. > If I could do an argument-style unpack, however, I could instead > write: > > first, *rest = paragraph.split( ' ', 1 ) first, rest = my_para_split(s,1) tjr From brett at python.org Mon Apr 17 02:06:35 2006 From: brett at python.org (Brett Cannon) Date: Sun, 16 Apr 2006 17:06:35 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: On 4/16/06, Neal Norwitz wrote: > On 4/16/06, Brett Cannon wrote: > > > > again, if my host at Google wants me to spend my working hours being a > > good mentor I could possibly be convinced to make sure I get assigned > > one student (two turned out to be too much). =) > > I heard your host is a slave driver and an asshole. You should start > getting some extra sleep now so you can work 200 hours a week during > the summer. Your first task will be to figure out how to find the > extra 32 hours for each week. :-) > Glad I am taking a two week vacation before I start then! Gives me time to plot ways to duck out of responsibility and to make shoddy work look good as as to make sure those 32 extra hours aren't quite so painful. =) > I think it's reasonable to take a couple hours here and there to help > mentor. How much time did it take to mentor last year per student? Hardly any, but that's because I didn't set aside enough time to actually spend one-on-one time with the student. I should have gotten them on IM or IRC and had weekly meetings with them. Also didn't just stare at their code enough to make sure they were coming along okay; just assumed they knew what they were doing. It all still worked out in the end, but I was lucky. > How could you do a better job (invest less time and get more out of > the student)? ISTM like it's mostly just keeping in touch via email. I think keeping up with email, having weekly real-time meetings over IM or IRC, and making sure I do thorough spot checks on the results. I also think making sure you to test-driven development so that the needed goals are very clearly spelled out is really important with SoC since it lets the student know exactly what is expected of them and gives them a clear picture of how close or far away they are from meeting their goal (and thus getting paid). -Brett From jparlar at cogeco.ca Mon Apr 17 05:14:36 2006 From: jparlar at cogeco.ca (Jay Parlar) Date: Sun, 16 Apr 2006 20:14:36 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: <2c6213fabd568d49a197e492b50037b2@cogeco.ca> On Apr 16, 2006, at 2:34 PM, Guido van Rossum wrote: > Some of the coding projects I've recently been proposing as challenges > to certain wild feature proposals might make good topics for the > Google Summer of Code (see http://code.google.com/soc/ ). > > We'd need someone to volunteer as a mentor, and a student to do the > work over the summer; the student would get $4500 and the mentor $500 > (or a donation to the PSF worth that much). > > (I can't volunteer to be a mentor myself; I already have a project I'd > like to mentor and an intern residing at Google.) > Well, I have zero experience hacking the Python internals, but I've been an active Python coder for about 6 years now (with a *lot* of C experience in various industry roles, in that time). I'm also now a grad student, working towards a Masters in Software Engineering. The SOC is something I've been looking forward to since last year. Haven't been able to think up any ideas on my own yet, but I'm definitely interested in something Python related. So if any Py3K ideas come up that I'll be able to have a good run at, I'll definitely apply. Jay P. From greg.ewing at canterbury.ac.nz Mon Apr 17 02:53:46 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 17 Apr 2006 12:53:46 +1200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> Message-ID: <4442E71A.2090501@canterbury.ac.nz> Guido van Rossum wrote: > (I'm surprised by this. Why would s[i] be wrong?) Because if we're to be consistent about the notion that a string isn't a sequence of characters, s[i] is really a slice of length one, and should be expressed that way. Also, if len(s) and s[i] are both legal, it becomes rather surprising if iter(s) doesn't work. > I think you may be right. I implemented this (it was really simple to > do) but then found I had to fix tons of places that iterate over > strings. I wonder if the stdlib might be a bit unusual here, since it's the place where low-level things are implemented. I don't think there would be many places in my code that would be affected, but others' experiences may be different. -- Greg From aahz at pythoncraft.com Mon Apr 17 03:36:11 2006 From: aahz at pythoncraft.com (Aahz) Date: Sun, 16 Apr 2006 18:36:11 -0700 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <4442E71A.2090501@canterbury.ac.nz> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> Message-ID: <20060417013611.GA19922@panix.com> On Mon, Apr 17, 2006, Greg Ewing wrote: > Guido van Rossum wrote: >> >> I think you may be right. I implemented this (it was really simple >> to do) but then found I had to fix tons of places that iterate over >> strings. > > I wonder if the stdlib might be a bit unusual here, since it's the > place where low-level things are implemented. I don't think there > would be many places in my code that would be affected, but others' > experiences may be different. My post was prompted by a recent dealing with someone who was not a Python programmer but needed to check strings for punctuation (user-mode code in a larger app). After thinking for a bit, the simplest solution that I could think of for easy explanation was this: PUNCT = '.,!;' for char in PUNCT: if char in field: raise ValueError(char + " not allowed in field") Anything that would make this solution more difficult to explain would make my life more difficult. I know there are many better and more efficient mechanisms (particularly if one is not restricted to Python 2.2), but it seems to me that we cannot afford to let Python 3.0 become the experts' playground. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From greg.ewing at canterbury.ac.nz Mon Apr 17 03:41:56 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 17 Apr 2006 13:41:56 +1200 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> Message-ID: <4442F264.7030307@canterbury.ac.nz> Guido van Rossum wrote: > On 4/16/06, Greg Ewing wrote: > > In the case of __init__ > > methods, most of the time it simply doesn't work, > > because different __init__ methods rarely have the > > same signature. > > How does that prevent you from using super()? (Hint: it doesn't. :-) It's okay as long as you know which __init__ method the super() is going to call, so you know its signature. But if you know that, there's no need to use super() -- just call it directly. The use case for super() is where you have a chain of methods that are sufficiently compatible that you don't need to know or care which one you're calling next. That's not usually the case with __init__ methods. The other possible reason for using super() is so you don't have to write the name of the base class into all your inherited method calls. But that's a separate issue that would be better addressed by a different mechanism, rather than conflating the two in super(). -- Greg From msoulier at digitaltorque.ca Mon Apr 17 03:59:19 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Sun, 16 Apr 2006 21:59:19 -0400 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <20060417013611.GA19922@panix.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <20060417013611.GA19922@panix.com> Message-ID: <20060417015918.GD28891@tigger.digitaltorque.ca> On 16/04/06 Aahz said: > Anything that would make this solution more difficult to explain would > make my life more difficult. I know there are many better and more > efficient mechanisms (particularly if one is not restricted to Python > 2.2), but it seems to me that we cannot afford to let Python 3.0 become > the experts' playground. Agreed. Personally, I've always hated how I could not do some of the simplest things with strings in Perl when Perl was supposed to be good at managing strings. Leaving in the ability to iterate strings in Python solved that for me, and I don't see why anyone with a pragmatic view of the language would wish to remove that feature. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060416/cbf99412/attachment.pgp From msoulier at digitaltorque.ca Mon Apr 17 04:04:15 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Sun, 16 Apr 2006 22:04:15 -0400 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <4442F264.7030307@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> Message-ID: <20060417020415.GE28891@tigger.digitaltorque.ca> On 17/04/06 Greg Ewing said: > The other possible reason for using super() is so > you don't have to write the name of the base class > into all your inherited method calls. But that's a > separate issue that would be better addressed by a > different mechanism, rather than conflating the > two in super(). Although you do have to put the current class name in the method calls, as super() requires it as the first argument. I never understood that. Why would I wish to use super(Bar) if I'm in class Foo? Cannot Foo be implied here? Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060416/97517bc1/attachment.pgp From aahz at pythoncraft.com Mon Apr 17 05:18:58 2006 From: aahz at pythoncraft.com (Aahz) Date: Sun, 16 Apr 2006 20:18:58 -0700 Subject: [Python-3000] auto-super() In-Reply-To: <20060417020415.GE28891@tigger.digitaltorque.ca> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> Message-ID: <20060417031858.GA354@panix.com> On Sun, Apr 16, 2006, Michael P. Soulier wrote: > On 17/04/06 Greg Ewing said: >> >> The other possible reason for using super() is so you don't have >> to write the name of the base class into all your inherited method >> calls. But that's a separate issue that would be better addressed by >> a different mechanism, rather than conflating the two in super(). > > Although you do have to put the current class name in the method > calls, as super() requires it as the first argument. I never > understood that. Why would I wish to use super(Bar) if I'm in class > Foo? Cannot Foo be implied here? Remember that in its current form super() is a regular function; it cannot be implied without stack hackery. The autosuper metaclass should probably become part of the regular type metaclass in 3.0, but I think that requires a PEP -- and in keeping with Guido's recent admonishment, someone should probably try implementing it first and see what happens with the test suite. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From tim.peters at gmail.com Mon Apr 17 05:25:12 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 16 Apr 2006 23:25:12 -0400 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <4442E71A.2090501@canterbury.ac.nz> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> Message-ID: <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> [Guido] >> ... >> I implemented this (it was really simple to do) but then found I had >> to fix tons of places that iterate over strings. [Greg Ewing] > I wonder if the stdlib might be a bit unusual here, > since it's the place where low-level things are > implemented. I don't think there would be many places > in my code that would be affected, but others' > experiences may be different. I expect it depends on whether someone writes text-processing algorithms. If you do, it's quite natural to iterate over strings. I have lots of code doing that, although I had no objection to needing to ask for a view in order to iterate, so didn't object to the proposal. Now that I think of it, though, I've been burned perhaps twice in my total Python life by recursing on a string when I didn't intend to. Apart from flatten()-ish functions, I'm not sure I've written anything vulnerable to that. The only example of "a trap" I noticed in this thread was: """ Seeing this:
  • i
  • t
  • e
  • m
  • 1
a few too many times... it's annoying. """ But I've never seen that, and didn't see the connection to strings being iterable. I'm sure some careless web code somewhere does that, though ;-) From aahz at pythoncraft.com Mon Apr 17 05:29:30 2006 From: aahz at pythoncraft.com (Aahz) Date: Sun, 16 Apr 2006 20:29:30 -0700 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> Message-ID: <20060417032930.GB25577@panix.com> On Sun, Apr 16, 2006, Tim Peters wrote: > > Now that I think of it, though, I've been burned perhaps twice in my > total Python life by recursing on a string when I didn't intend to. > Apart from flatten()-ish functions, I'm not sure I've written anything > vulnerable to that. I've been burned a few times, but only because I was expecting a list input and got a string. I think that improving code analysis tools and making better use of unit tests is the proper cure for that. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From ncoghlan at gmail.com Mon Apr 17 05:34:13 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Apr 2006 13:34:13 +1000 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: Message-ID: <44430CB5.5020907@gmail.com> Talin wrote: > Guido van Rossum python.org> writes: > >> On 4/15/06, Talin acm.org> wrote: >>> Another grab-bag of language ideas / wishes. Some of these are >>> items that have been raised before on other lists - I want to get them >>> written down so that they can be rejected quickly :) >>> >>> 1) I don't know if this is already on the table, but I sure would like >>> to be able to have more options as far as mixing positional and >>> keyword arguments. >> I agree. Please produce a patch that implements what you want. A question for Guido: Is this flexibility for argument passing only, or does it apply to parameter declarations as well? In this approach, parameters declared after *args would be "keyword-only" parameters, so should a single "*" be permitted in the parameter list to allow functions with a fixed number of positional arguments to also declare keyword-only arguments? Currently, the only way to do this is to declare **kwds and go poking into the dictionary manually (the same way C code does it), which means important information is missing from the function signature. > All right, in that case, I'd like to post a general implementation > strategy for criticism before I end up going too far down the > wrong path. > > As I see it, the problem breaks down into 3 parts: > > -- parsing > -- representing the argument list > -- executing a call > > The parsing part is fairly straightforward. The new syntax would > allow keyword arguments to appear anywhere in the argument > list, intermixed with positional arguments, and after the varargs > argument. I think that's making things a bit *too* flexible - I believe its worthwhile keeping the positional argument at the beginning of the call. The only thing we're trying to fix here is that the current syntax requires the varargs argument to be separated from the normal positional arguments by any specified keyword arguments. > (It would be possible to have non-keyword arguments after the > varargs argument as well, but that's harder to implement, and I > don't really see the value in it.) I think keeping the positional arguments before the varargs argument is the right thing to do. > The kwargs argument would remain unchanged - there's no real > reason to allow it to be moved around in the argument list. This also sounds right. > So the argument list syntax would be (in psuedo-grammar): > > (keyword | positional )* [*varargs] keyword* [**kwargs] This seems unnecessarily flexible to me. I'd prefer: positional* keyword* ["*"varargs] keyword* ["**"kwargs] That is, the only difference from the status quo is the ability to put the keyword arguments after the varargs argument instead of before it. > (This is of course omitting all of the complexities of generator > expressions for the moment.) Genexp's pretty much result in a giant "or" with this whole section of the grammar anyway. > Representing the argument list. The current _arguments struct > looks like this: > > struct _arguments { > asdl_seq *args; > identifier vararg; > identifier kwarg; > asdl_seq *defaults; > }; Calling this the argument list is a mistake in the current AST definition - this structure is only used for the parameter list in a function definition. It is *NOT* used for call arguments. (I'd personally be happy with a patch that fixed this discrepancy by changing the name to "parameters"). The actual call definition wouldn't require any changes to handle this: Call(expr func, expr* args, keyword* keywords, expr? starargs, expr? kwargs) Changes to the 'arguments' structure would only be needed if Guido extended this idea to cover declaration of keyword-only arguments. > The approach that I would take would be to remove the 'vararg', > and instead put the vararg argument into args. A special sentinel > value in the *defaults array (say, (void*)-1) would be used to > indicate that this argument was a varargs argument. > > Thus, for each argument (other than kwargs), you would have > both an argument name and a default, where a NULL value indicated > a positional argument, a non-NULL value indicated a keyword argument, > and a sentinel value indicated a varargs argument. Since this is only needed if Guido decides to permit keyword-only arguments in function definitions, I'd suggest simply adding a second sequence for keyword arguments after the varargs argument: struct _arguments { asdl_seq *args; identifier vararg; asdl_seq *kwargs; identifier kwarg; asdl_seq *defaults; }; > Executing a call: This is the tricky part. Specifically, we want to insure > that the mapping of values to formal parameters remains the same > for existing code. In the simple case, this shouldn't have to change at all, since the AST for it won't change. In the case where keyword-only arguments are permitted, it's simply a matter of checking the called function's "kwargs" list in addition to its normal "args" list before dumping an unregognised keyword parameter in the double-star dictionary. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ianb at colorstudy.com Mon Apr 17 06:37:51 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 16 Apr 2006 23:37:51 -0500 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <20060417032930.GB25577@panix.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> Message-ID: <44431B9F.4030005@colorstudy.com> Aahz wrote: > On Sun, Apr 16, 2006, Tim Peters wrote: >> Now that I think of it, though, I've been burned perhaps twice in my >> total Python life by recursing on a string when I didn't intend to. >> Apart from flatten()-ish functions, I'm not sure I've written anything >> vulnerable to that. > > I've been burned a few times, but only because I was expecting a list > input and got a string. I think that improving code analysis tools and > making better use of unit tests is the proper cure for that. The problem with testing is that there's no exception, just bad data, and so you have to test much more aggressively to find the bug. Also, when you encounter the bad it's often not clear where exactly it came from. Unit testing works best when things fail early and fail with an exception, so I don't think this is a case where we should lean on unit testing. Also, this leads to integration bugs -- when someone calls your code with the wrong kind of argument. For any public API that takes a list of strings, I always try to do "assert not isinstance(arg, basestring)", because a runtime assertion is the only kind of test you can do -- it's outside the bounds of unit testing. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From guido at python.org Mon Apr 17 07:26:47 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Apr 2006 07:26:47 +0200 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: <44430CB5.5020907@gmail.com> References: <44430CB5.5020907@gmail.com> Message-ID: > >> On 4/15/06, Talin acm.org> wrote: > >>> 1) I don't know if this is already on the table, but I sure would like > >>> to be able to have more options as far as mixing positional and > >>> keyword arguments. > > Guido van Rossum python.org> writes: > >> I agree. Please produce a patch that implements what you want. On 4/17/06, Nick Coghlan wrote: > A question for Guido: Is this flexibility for argument passing only, or does > it apply to parameter declarations as well? [Nick, could you trim your posts a bit? I'm on a trip with very short intervals of net access, and long posts tend to be put in the "later" bin forever.] I think that's a reasonable idea. I didn't read the rest of your mail yet. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 17 07:35:40 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Apr 2006 07:35:40 +0200 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: On 4/17/06, Neal Norwitz wrote: > Awwww, I was planning to rope you in as mentor. :-) I plan to mentor, > I'm not sure how many projects I can handle though. Don't plan on any for yourself; you're too busy already with the 2.5 release and with managing the mentors. > I hope all the mentors from last year will mentor again. I'm still > not clear on exactly how mentors request access. I know I can accept > them once they are put into the system though. Isn't that covered in one of the SoC FAQs? It does mention you need a Google account. (I have to admit I couldn't find a login link but perhaps that's because I am always logged in to gmail.) The PSF (you) needs to create a page (in the wiki is fine) with instructions for mentors, and make sure that is linked to from the code.google.com/soc/ page as "(ideas)" after our name, just like the Perl foundation. The PSF (you; or I'll do it if you write some text for me) should also start advertising for mentors on python-dev, python-3000, and perhaps even on c.l.py. There is a lot of competition on that page now -- the list of mentoring orgs has at least doubled since Friday! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 17 07:42:07 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Apr 2006 07:42:07 +0200 Subject: [Python-3000] Test and Assign [was: More wishful thinking] In-Reply-To: References: <008701c660dc$4d141070$2452fea9@bagio> <17473.31731.38680.400480@montanaro.dyndns.org> <014501c6616a$d169b030$2452fea9@bagio> <17474.32338.297184.343720@montanaro.dyndns.org> <02d201c66184$b4a1ae70$2452fea9@bagio> <17474.47696.600874.855528@montanaro.dyndns.org> Message-ID: On 4/16/06, Talin wrote: > I'm really more interested in the general case where you have a test, and > then you have the results of the test. It would be nice to put the conditional > statement first, and then once you know that the test succeeded, start to > look at the specific details. I think Skip's response applies equally here. This is an old chestnut; many folks have barked upon this tree in the last 15 years, without being able to crack this nut. Unless you have a concrete solution (short of allowing '=' in expressions :-) there's no solution and you're wasting everybody's time for explaining how useful it would be. This is different from e.g. unpacking syntax because that's a concrete solution, it just needs to be shown that the extra complexity added to the language is made up for by more readable code, and not offset by more bugs; or something like that. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From talin at acm.org Mon Apr 17 07:47:56 2006 From: talin at acm.org (Talin) Date: Mon, 17 Apr 2006 05:47:56 +0000 (UTC) Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) References: <44430CB5.5020907@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > A question for Guido: Is this flexibility for argument passing only, or does > it apply to parameter declarations as well? Actually, parameter declarations are what I am talking about. Now go back and re-read the whole message with that in mind. :) (Its probably my fault for using inaccurate terminology.) However, we really ought to address both, which tells me that the prior posting is only half the job. OK then... As far as "too flexible", let's talk about that. On the definition side, you don't want to see definitions that look like this: def func( first, second=default, third ): ... In this case, I think your argument makes sense - what's the point of supplying a default for the second parameter, but not the third? And on the calling side: def func2( adama, valeri, thrace ): ... func2( valeri='cylon', 'human', 'human' ) In this case, however, I don't see why we can't have the keyword arguments out of order. In the above example, the 'valeri' parameter gets filled in first, and then the two remaining slots get filled in with the positional argumens. The important thing is that the behavior be clear and unambiguous, which I think this is. -- Talin From nnorwitz at gmail.com Mon Apr 17 09:43:33 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 17 Apr 2006 00:43:33 -0700 Subject: [Python-3000] Summer of Code preparation Message-ID: We've only got a short time to get setup for Google's Summer of Code. We need to start identifying mentors and collecting ideas for students to implement. We have the SimpleTodo list (http://wiki.python.org/moin/SimpleTodo), but nothing on the SoC page yet (http://wiki.python.org/moin/SummerOfCode). I can help manage the process from inside Google, but I need help gathering mentors and ideas. I'm not certain of the process, but if you are interested in being a mentor, send me an email. I will try to find all the necessary info and post here again tomorrow. Pass the word! I hope all mentors from last year will return again this year. Can someone take ownership of drumming up mentors and ideas? We also need to spread the word to c.l.p and beyond. Thanks, n From ncoghlan at gmail.com Mon Apr 17 10:41:40 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Apr 2006 18:41:40 +1000 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> Message-ID: <444354C4.1010701@gmail.com> Guido van Rossum wrote: > On 4/17/06, Nick Coghlan wrote: >> A question for Guido: Is this flexibility for argument passing only, or does >> it apply to parameter declarations as well? > > [Nick, could you trim your posts a bit? I'm on a trip with very short > intervals of net access, and long posts tend to be put in the "later" > bin forever.] Will do. > I think that's a reasonable idea. I didn't read the rest of your mail yet. The rest of it was just suggesting that permitting keyword parameters/arguments after *args should be the *only* change we make in this area. I don't see any point in permitting positional arguments to come after a keyword argument, even at the call site (this ability was a side effect of Talin's suggested implementation strategy). Keeping the positional-before-keyword limitation also permits the implementation to be simpler. On the calling side, it's just a parser tweak that doesn't require touching the AST definition. The function definition side will be a bit trickier. In addition to rippling all the way down through the AST to the bytecode compiler, function objects will need some way of reporting "keyword only" arguments as being part of their signature (probably via an extra attribute). This will affect the evaluation of function calls (another place to check for keyword arguments) and may also ripple all the way up to the API of the inspect module. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Mon Apr 17 10:45:14 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 17 Apr 2006 18:45:14 +1000 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> Message-ID: <4443559A.2090502@gmail.com> Talin wrote: > And on the calling side: > > def func2( adama, valeri, thrace ): > ... > > func2( valeri='cylon', 'human', 'human' ) > > In this case, however, I don't see why we can't have the keyword > arguments out of order. In the above example, the 'valeri' parameter > gets filled in first, and then the two remaining slots get filled in with > the positional argumens. The question to ask in this case is "Why change it?", rather than "Why not?". In the case of allowing additional keyword arguments after *args, there's a clear improvement in expressiveness. For function definitions, it permits keyword-only arguments, for function calls it permits all positional arguments to be supplied before any keyword arguments. There isn't any comparable benefit in changing the positional-before-keyword rule. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From kendall at monkeyfist.com Mon Apr 17 15:29:08 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Mon, 17 Apr 2006 09:29:08 -0400 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: <4230BDAA-B613-43C2-A800-A4FAC39C1B83@monkeyfist.com> On Apr 16, 2006, at 4:40 PM, Guido van Rossum wrote: > On 4/16/06, Kendall Clark wrote: >> A perfectly reasonable suggestion. I like this better than writing a >> PEP. Alas, that doesn't mean I have time to do it, which sucks. I >> guess I shouldn't have proposed a feature for Py3K w/out the time to >> back it up with code. > > Perhaps you have the time to mentor someone else doing it for the > Google Summer of Code program? The PSF is looking for projects and > mentors. (Even though there's nothing on the soc page yet.) Yes, I think I can probably do that. Cheers, Kendall From gustavo at niemeyer.net Mon Apr 17 16:10:33 2006 From: gustavo at niemeyer.net (Gustavo Niemeyer) Date: Mon, 17 Apr 2006 11:10:33 -0300 Subject: [Python-3000] More wishful thinking In-Reply-To: References: Message-ID: <20060417141033.GA10368@localhost.localdomain> > What I would like to be able to write is: > > def function( *args, option=False ): > ... (...) For the same reasons, being able to do the equivalent when calling any given function would be nice as well. e.g. function(first, second, *args, something=True) If something like this is implemented and accepted, it would be a good candidate for 2.X. -- Gustavo Niemeyer http://niemeyer.net From steven.bethard at gmail.com Mon Apr 17 18:36:21 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 17 Apr 2006 10:36:21 -0600 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: <44430CB5.5020907@gmail.com> References: <44430CB5.5020907@gmail.com> Message-ID: [Talin] > 1) I don't know if this is already on the table, but I sure would like > to be able to have more options as far as mixing positional and > keyword arguments. I thought it might be useful to look at a few places where we currently have to do some hackery to get the function signature we want. I believe the first of these can be solved by the proposed change. I'm not sure if anything can (or should) be done about the latter two. ------------------------------------------------------------------------ min/max (http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/389659) ------------------------------------------------------------------------ :: def __call__(self, *seq, **kwargs): key = kwargs.pop('key', None) if kwargs: raise TypeError("only 'key' accepted as a " "keyword argument") ... I believe this is straightforwardly cleaned up by the current proposal by making key a keyword only argument:: def __call__(self, *seq, key=None): ... ------------------------- UserDict.DictMixin.update ------------------------- :: def update(self, other=None, **kwargs): ... this is actually wrong, since you can't create a dict like ``dict(self='foo', other='bar')``. It needs to be written as:: def __init__(*args, **kwargs): if len(args) == 1: self, = args other = None else: self, other = args ... To clean this up, we'd need the ability to identify self and dict as positional only arguments. AFAICT, the current proposal doesn't solve this problem. Off the top of my head, I don't see an easy way of supporting this either... ---------------------- UserDict.DictMixin.pop ---------------------- :: def pop(self, key, *args): if len(args) > 1: raise TypeError, "pop expected at most 2 arguments, got "\ + repr(1 + len(args)) try: value = self[key] except KeyError: if args: return args[0] raise del self[key] return value Here, you *have* to know how many arguments were supplied because an exception must be raised if there were 2 and not if there were three. And even if you could define "default" as a keyword-only argument, e.g.:: def pop(self, key, *args, default=None): you still wouldn't know if "default" was None from the default in the signature or because the value None was supplied by the caller. STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From guido at python.org Mon Apr 17 20:40:41 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Apr 2006 19:40:41 +0100 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: <444354C4.1010701@gmail.com> References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> Message-ID: On 4/17/06, Nick Coghlan wrote: > The rest of it was just suggesting that permitting keyword > parameters/arguments after *args should be the *only* change we make in this > area. I don't see any point in permitting positional arguments to come after a > keyword argument, even at the call site (this ability was a side effect of > Talin's suggested implementation strategy). Right. In the call foo(1, 2, abc=42, 3), what's the sequential parameter position for the 3? I think it's ambiguous in the sense that reasonable users could disagree what would be the best choice. But in foo(1, 2, *args, 3, 4) there's no harm; nor is there harm in foo(1, 2, *args, abc=42, **kwds). I'm on the fence about foo(*args, **kwds, abc=42). Here's a related but more complicated wish: define a function in such a way that certain parameters *must* be passed as keywords, *without* using *args or **kwds. This may require a new syntactic crutch. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tjreedy at udel.edu Mon Apr 17 20:44:10 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Apr 2006 14:44:10 -0400 Subject: [Python-3000] Cleaning up argument list parsing (was Re: Morewishful thinking) References: <44430CB5.5020907@gmail.com> Message-ID: "Talin" wrote in message news:loom.20060417T072709-664 at post.gmane.org... > The important thing is that the behavior be clear and unambiguous, > which I think this is. Many would also prefer that functions calls not become noticeable slower than they already are. tjr From tjreedy at udel.edu Mon Apr 17 21:20:44 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Apr 2006 15:20:44 -0400 Subject: [Python-3000] Python 3000 and the Google Summer of Code References: Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604162235w7b37ac26v34b5a5647688a27a at mail.gmail.com... > The PSF (you) needs to create a page (in the wiki is fine) with > instructions for mentors, and make sure that is linked to from the > code.google.com/soc/ page as "(ideas)" after our name, just like the > Perl foundation. The PSF (you; or I'll do it if you write some text > for me) should also start advertising for mentors on python-dev, > python-3000, and perhaps even on c.l.py. The PyPy group also has a few prospective mentors, projects, and mentees. On the presumption that PSF coordinating sponsership is not limited to CPython only projects, I forwarded Neal's SoC py-dev post to their list. Gmane.org mirrors at least 50 python-related mailing lists. If, when the wiki pages are done and mentor/project registration in place, you want an announcement sent to such lists, sent me something and I will forward it to any that seem appropriate. (The only signup needed is to hit reply to 'Are you a person and did you post?' messages.) Terry Jan Reedy From nnorwitz at gmail.com Mon Apr 17 21:27:36 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 17 Apr 2006 12:27:36 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: On 4/17/06, Terry Reedy wrote: > > The PyPy group also has a few prospective mentors, projects, and mentees. > On the presumption that PSF coordinating sponsership is not limited to > CPython only projects, I forwarded Neal's SoC py-dev post to their list. Excellent. I was thinking about specifically mailing PyPy, IronPython, and Jython. I think we should support them in addition to CPython. > Gmane.org mirrors at least 50 python-related mailing lists. If, when the > wiki pages are done and mentor/project registration in place, you want an > announcement sent to such lists, sent me something and I will forward it to > any that seem appropriate. (The only signup needed is to hit reply to 'Are > you a person and did you post?' messages.) I know what it takes to sign a mentor up. It's just a link. I haven't publicised it yet. But I plan to soon. If you would like to craft a nice message I can give you all the info so you can get the word out. n From nnorwitz at gmail.com Mon Apr 17 21:31:55 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 17 Apr 2006 12:31:55 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: One thing I forgot to mention. There will likely be people we don't know who offer to be mentors. It's great to get new people involved. Though we need to ensure they can handle the job. Guido suggested and I think it's a good idea to have any unknown person (*) to get 2 references that are known on python-dev to vouch for them. Terry are you interested in being a mentor? n (*) Unknown to me I guess, since I'm currently the only approver of mentors. -- On 4/17/06, Neal Norwitz wrote: > On 4/17/06, Terry Reedy wrote: > > > > The PyPy group also has a few prospective mentors, projects, and mentees. > > On the presumption that PSF coordinating sponsership is not limited to > > CPython only projects, I forwarded Neal's SoC py-dev post to their list. > > Excellent. I was thinking about specifically mailing PyPy, > IronPython, and Jython. > I think we should support them in addition to CPython. > > > Gmane.org mirrors at least 50 python-related mailing lists. If, when the > > wiki pages are done and mentor/project registration in place, you want an > > announcement sent to such lists, sent me something and I will forward it to > > any that seem appropriate. (The only signup needed is to hit reply to 'Are > > you a person and did you post?' messages.) > > I know what it takes to sign a mentor up. It's just a link. I > haven't publicised it yet. But I plan to soon. If you would like to > craft a nice message I can give you all the info so you can get the > word out. > > n > From p.f.moore at gmail.com Mon Apr 17 21:42:59 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 17 Apr 2006 20:42:59 +0100 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) Message-ID: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> On 4/17/06, Terry Reedy wrote: > > "Talin" wrote in message > news:loom.20060417T072709-664 at post.gmane.org... > > The important thing is that the behavior be clear and unambiguous, > > which I think this is. > > Many would also prefer that functions calls not become noticeable slower > than they already are. How slow *are* function calls? There seems to be a common "function calls in Python are slow" meme. It's never been a significant issue for me, but my code is pretty much always IO-bound, so that doesn't surprise me. Are they slow enough to warrant serious effort to speed them up? More specifically (and more on-topic :-)) is there any room for speeding up function calls in Python 3000? Given that pure performance improvements are clearly OK for 2.x, I'm thinking of cases where it would be necessary to impose language-level restrictions to ease the optimisation process. Paul. From walter at livinglogic.de Mon Apr 17 21:47:07 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 17 Apr 2006 21:47:07 +0200 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> Message-ID: <4443F0BB.40001@livinglogic.de> Guido van Rossum wrote: > On 4/17/06, Nick Coghlan wrote: >> The rest of it was just suggesting that permitting keyword >> parameters/arguments after *args should be the *only* change we make in this >> area. I don't see any point in permitting positional arguments to come after a >> keyword argument, even at the call site (this ability was a side effect of >> Talin's suggested implementation strategy). > > Right. In the call foo(1, 2, abc=42, 3), what's the sequential > parameter position for the 3? I think it's ambiguous in the sense that > reasonable users could disagree what would be the best choice. > > But in foo(1, 2, *args, 3, 4) there's no harm; nor is there harm in > foo(1, 2, *args, abc=42, **kwds). I'm on the fence about foo(*args, > **kwds, abc=42). BTW, do we want to allow foo(1, *args1, 2, *args2, 3)? And what about foo(bar=17, **kwargs1, baz=23, **kwargs2)? Servus, Walter From janssen at parc.com Mon Apr 17 23:24:02 2006 From: janssen at parc.com (Bill Janssen) Date: Mon, 17 Apr 2006 14:24:02 PDT Subject: [Python-3000] Test and Assign [was: More wishful thinking] In-Reply-To: Your message of "Sun, 16 Apr 2006 22:42:07 PDT." Message-ID: <06Apr17.142412pdt."58633"@synergy1.parc.xerox.com> > Unless you have a concrete solution (short of allowing '=' in > expressions :-) there's no solution and you're wasting everybody's > time for explaining how useful it would be. If we had symbols, we could write something like if assign('x, some_expression): print x Presumably a reference to a Python symbol would be not just the symbol name string, but also an indicator of the namespace of the symbol. But frankly, I don't think symbols would bring much else to Python, and I'm -1 on adding them. It seems to me that something very like the above can already be done with partial function application. It could even be regularized/automated with a bit of metaclass hacking. Bill From rhettinger at ewtllc.com Mon Apr 17 23:01:04 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Mon, 17 Apr 2006 14:01:04 -0700 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <20060417032930.GB25577@panix.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> Message-ID: <44440210.9010201@ewtllc.com> Aahz wrote: >On Sun, Apr 16, 2006, Tim Peters wrote: > > >>Now that I think of it, though, I've been burned perhaps twice in my >>total Python life by recursing on a string when I didn't intend to. >>Apart from flatten()-ish functions, I'm not sure I've written anything >>vulnerable to that. >> >> > >I've been burned a few times, but only because I was expecting a list >input and got a string. I think that improving code analysis tools and >making better use of unit tests is the proper cure for that. > > -1 on making strings non-iterable. The cure is worse than the disease. In ML, characters and strings are different datatypes and it is a complete PITA. I suspect that making strings non-iterable would make the language more difficult to use rather than simpler. Besides, if you took away iterability, you would spend the rest of your life responding to frequent requests to add it back ;-) I'm also -1 on almost all proposals of this kind. IMHO, the first cut of Py3000 should be directed at subtracting the cruft and consolidating what we have. It should be a better Python, not just a different Python. The surest route to that goal is to build on what has been shown to work and experiment with random alternatives that may or may not ultimately prove their worth. What if this non-iterable string proposal were accepted and after a year or so we found that there was a distaste for it that had reduced the acceptance of the language? Alternatively, what if we found that people just coded through it by reverting to pre-iterator style: for i in range(len(s)): do_something(s[i])? I would hate to start seeing code like that again. Py3000 should try avoid the "second system effect" and not lose its grounding. This includes proposals to put assignments in expressions, new symbol syntaxes, and all the other random oddities being tossed out. my two cents, Raymond From guido at python.org Tue Apr 18 00:32:38 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Apr 2006 23:32:38 +0100 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) In-Reply-To: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> Message-ID: On 4/17/06, Paul Moore wrote: > How slow *are* function calls? There seems to be a common "function > calls in Python are slow" meme. It's never been a significant issue > for me, but my code is pretty much always IO-bound, so that doesn't > surprise me. Are they slow enough to warrant serious effort to speed > them up? Depends on what you are doing whether it matters, but function calls are most surely slow. Almost any time where code is CPU bound (and you've already done all the obvious optimizations), profiling shows that function calls take up a significant amount of time. Inlining code obviously addresses this, but that's often killing code structure. > More specifically (and more on-topic :-)) is there any room for > speeding up function calls in Python 3000? Given that pure performance > improvements are clearly OK for 2.x, I'm thinking of cases where it > would be necessary to impose language-level restrictions to ease the > optimisation process. I definitely don't want to make optimization a goal of Python 3000. At the same time I don't want it to be *slower* than Python 2.x. So if at some point it appears to be getting slower (it's way too soon to tell) we may have to do some optimization work just so that people can't reject it on the basis of the relative speed alone. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Tue Apr 18 00:40:19 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 17 Apr 2006 17:40:19 -0500 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <44440210.9010201@ewtllc.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> Message-ID: <44441953.6090809@colorstudy.com> Raymond Hettinger wrote: > -1 on making strings non-iterable. The cure is worse than the disease. > In ML, characters and strings are different datatypes and it is a > complete PITA. I suspect that making strings non-iterable would make > the language more difficult to use rather than simpler. Besides, if you > took away iterability, you would spend the rest of your life responding > to frequent requests to add it back ;-) The proposal wasn't to remove iterability in general, only to require the use of an explicit method when iterating over strings. It's not a huge change by any means; a speed bump for a small number of people, perhaps, but no real functionality would be removed. Backward compatibility is also certainly a concern -- and what seems to have soured Guido on the idea as much as anything -- but that doesn't seem to be what has you bothered. Also, there was no proposal to add a character type. It was noted that the lack of a character type is part of what makes string iteration problematic, because strings are not character containers, and iteration over a string actually produces a set of slices of the string, it doesn't expose the contents of the string. I think Greg proposed removing __getitem__, but that seems a bit extreme. So I don't think there's any parallel to ML or other languages with character types. > I'm also -1 on almost all proposals of this kind. IMHO, the first cut > of Py3000 should be directed at subtracting the cruft and consolidating > what we have. It should be a better Python, not just a different > Python. The surest route to that goal is to build on what has been > shown to work and experiment with random alternatives that may or may > not ultimately prove their worth. I don't see how this is so dramatically different. I proposed it specifically to remove something that at least I considered a language wart: an area prone to errors that provides an interface inconsistent with other parts of the language. > What if this non-iterable string proposal were accepted and after a year > or so we found that there was a distaste for it that had reduced the > acceptance of the language? Alternatively, what if we found that > people just coded through it by reverting to pre-iterator style: for i > in range(len(s)): do_something(s[i])? I would hate to start seeing > code like that again. I would hope that in any finished version the exception message would point people to .chars(), or perhaps simply signal a warning with the same message to begin with. Py3k is going to be used for a while before any part of it is truly committed to, so it's perfectly fine if things go in and are then retracted. > Py3000 should try avoid the "second system effect" and not lose its > grounding. This includes proposals to put assignments in expressions, > new symbol syntaxes, and all the other random oddities being tossed out. Those are all rather conservative proposals; I don't see how second system comes into it. Type annotations might fall in that realm, 'make', some others... but suggesting some syntactic twiddling, though perhaps not very useful, is not second system effect. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From guido at python.org Tue Apr 18 00:44:49 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Apr 2006 23:44:49 +0100 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: <4443F0BB.40001@livinglogic.de> References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> <4443F0BB.40001@livinglogic.de> Message-ID: On 4/17/06, Walter D?rwald wrote: > BTW, do we want to allow foo(1, *args1, 2, *args2, 3)? > > And what about foo(bar=17, **kwargs1, baz=23, **kwargs2)? There's probably not much use for those but at least the first isn't ambiguous so could be allowed if it doesn't complicate the implementation. The second begs the questions about what works if kwargs1 and kwargs2 contain overlapping keys with different values; it's probably best not to allow this and perhaps **kwds should only be allowed as the very last parameter. Since keyword args aren't position-dependent, there's no reason to need a different position. If we did end up allowing multiple **kwds, the code implementing calls should check that they don't overlap with each other or with other keyword args or with named positional args, to be consistent with the current state of affairs, where f(a=1, **{'a':2}) is rejected with a TypeError. (The complexity of all those cross-checks is hopefully enough to dissuade anyone from proposing this. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 18 00:57:52 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 17 Apr 2006 23:57:52 +0100 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <44441953.6090809@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> Message-ID: On 4/17/06, Ian Bicking wrote: > The proposal wasn't to remove iterability in general, only to require > the use of an explicit method when iterating over strings. It's not a > huge change by any means; a speed bump for a small number of people, > perhaps, but no real functionality would be removed. Backward > compatibility is also certainly a concern -- and what seems to have > soured Guido on the idea as much as anything -- but that doesn't seem to > be what has you bothered. "A speed bump for a small number of people"? "No real functionality remove"? You gotta be kidding. It drops polymorphism between strings and other sequences. This is not just of theoretical value -- difflib relies on this, for example. Your (earlier) assertion that the stdlib is atypical is wishful thinking; the stdlib contains lots of code just like code written by regular people. Perhaps you've had Perl on your mind too much recently? Ian, please admit defeat when it stares you in the eyes. This change just isn't going to happen. It's pushing up the daisies. It's joined the choir invisible. THIS IS AN EX-PROPOSAL! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ferringb at gmail.com Tue Apr 18 01:08:05 2006 From: ferringb at gmail.com (Brian Harring) Date: Mon, 17 Apr 2006 16:08:05 -0700 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <44441953.6090809@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> Message-ID: <20060417230805.GA23791@nightcrawler> On Mon, Apr 17, 2006 at 05:40:19PM -0500, Ian Bicking wrote: > Raymond Hettinger wrote: > > -1 on making strings non-iterable. The cure is worse than the disease. > > I'm also -1 on almost all proposals of this kind. IMHO, the first cut > > of Py3000 should be directed at subtracting the cruft and consolidating > > what we have. It should be a better Python, not just a different > > Python. The surest route to that goal is to build on what has been > > shown to work and experiment with random alternatives that may or may > > not ultimately prove their worth. > > I don't see how this is so dramatically different. I proposed it > specifically to remove something that at least I considered a language > wart: an area prone to errors that provides an interface inconsistent > with other parts of the language. The issue I'm seeing is that the wart you're pointing at is a general issue not limited to strings- everyone sooner or later has flattening code that hits the "recursively iterate over this container, except for instances of these classes". General algo problem for nested sequences. I'd rather see a builtin that makes that case easier then to go and remove iteration on strings- something akin to def iter_flatten(lists, dont_iterate=[str]): stack=[iter(lists)] while stack: for x in stack[-1]: if hasattr(x, "__iter__") and isinstance(x, dont_iterate): stack.append(iter(x)) break yield x else: stack.pop(-1) That (assuming I didn't typo it) addresses the issue in a general way, without removing (useful) __iter__ from instances that occasionally we don't want to fully expand. An easier way to invoke a func of the sort from above gets the best of both worlds- still have iteration over strings, and general solution to common problem. Downside is that it forces an extra func call for the following if it's unknown if the instance is a string or a normal sequence- for x in iter_flatten(unknown_type_instance): do_something_with_it(x) ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060417/ad89a65e/attachment.pgp From fredrik at pythonware.com Tue Apr 18 01:04:33 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Tue, 18 Apr 2006 01:04:33 +0200 Subject: [Python-3000] Function call speed (Was: Cleaning up argumentlist parsing) References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> Message-ID: Guido van Rossum wrote: > Inlining code obviously addresses this, but that's often killing code > structure. unless you can get the compiler to do it for you. From RHettinger at ewtllc.com Tue Apr 18 01:47:18 2006 From: RHettinger at ewtllc.com (Raymond Hettinger) Date: Mon, 17 Apr 2006 16:47:18 -0700 Subject: [Python-3000] iter_flatten was: Making strings non-iterable Message-ID: [Brian Harring] > The issue I'm seeing is that the wart you're pointing at is a > general issue not limited to strings- everyone sooner or later > has flattening code that hits the "recursively iterate over > this container, except for instances of these classes". Good recap. For more info, search comp.lang.python for previous discussions about a proposed iter_flatten() function. One of the conclusions was that a general algorithm would need to accept a user specified test for atomicity so it would know where to recurse and a where not to (the answer to that is application dependent). Another conclusion was that an iterator version of flattening was a just one of many possible algorithms for traversing a hierarchical structure (i.e. a pre-order tree traversal). But some use cases needed more control points than provided by overly specific solutions (os.walk() and depth-first puzzle solvers are prime examples). The discussion itself was an interesting case-study because the flattening algorithm variants out-numbered the use cases (IOW, it's more fun to write flatten() than to use it). It was also interesting because the problem produced a tail-wagging-the-dog effect as discussions went from "how can I write a more general flatten" and turned into "how can I can modify the language to make my flatten variant apply more generally". Raymond From jack at performancedrivers.com Tue Apr 18 01:58:55 2006 From: jack at performancedrivers.com (Jack Diederich) Date: Mon, 17 Apr 2006 19:58:55 -0400 Subject: [Python-3000] Specializing the dicts in __dict__ Message-ID: <20060417235854.GF5762@performancedrivers.com> In an old thread ("dictionary tuning", 2003[1]) Raymond H laid out the typical use cases of dicts and found there were many subclasses of "typical" and speeding one up hurt the others. The results are in dist/Object/dictnotes.txt A couple of the particular use cases have since been satisfied by sets and the new defaultdict. The dicts used in symbol table lookup (global, local, class) have their own pattern. I mentioned then and would like to resurrect now the idea of making a dict-alike that has two extra properties: 1: the keys can only be str or unicode (py3k: just unicode) 2: the dict-alike is ordered The str/unicode restriction allows for some creative implementation details (maximizing cache hits on small dicts). I don't think this would break much (if any) code. The ordering would eliminate a frequent re-implementation in ORMs, frameworks, and DSLs that use hackery to know in what order the elements of a class/class-as-namespace were defined: class Thing: a = IntField() # increments a counter to 1 b = StringField() # counter == 2 c = StringField() # counter == 3 Thing.b = munge_field(Thing.b) The suggested order would be by key creation so that reassigning a key after manipulation would maintain the original order as in the Thing.b case above. There have been a number of suggested patches over the years to speed up name access but they all impact general dict performance. Isolating symbol table dicts would allow for easy experimentation. To mimic the defaultdict class I would suggest the name symdict, but (Big But) I don't much care what the name is and don't care at all how big your thesaurus is. Easy and arbitrary decisions should be made by proclamation; there may be benefits to driving on one side of the road or the other but the most important thing is that one side is chosen. Guido has asked for working code recently, I'll have a chance to attempt a patch at the "Need for Speed" sprint. One of the suggested goals is an ordered dict, so that works out nicely. afk-to-write-some-checks-to-Uncle-Sam-ly, -Jack [1] http://thread.gmane.org/gmane.comp.python.devel/12160/focus=12195 From greg.ewing at canterbury.ac.nz Tue Apr 18 02:01:00 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Apr 2006 12:01:00 +1200 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <20060417020415.GE28891@tigger.digitaltorque.ca> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> Message-ID: <44442C3C.2090908@canterbury.ac.nz> Michael P. Soulier wrote: > Although you do have to put the current class name in the method calls, as > super() requires it as the first argument. I never understood that. Why would > I wish to use super(Bar) if I'm in class Foo? Cannot Foo be implied here? The problem is that there's no way for Python to know which class the method is "in", in the sense required here. That could be fixed by giving functions defined inside a class statement an attribute referring back to the class. It would create a circular reference, but that's not so much of a problem these days. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 18 02:03:46 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Apr 2006 12:03:46 +1200 Subject: [Python-3000] auto-super() In-Reply-To: <20060417031858.GA354@panix.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> Message-ID: <44442CE2.6020801@canterbury.ac.nz> Aahz wrote: > Remember that in its current form super() is a regular function; it > cannot be implied without stack hackery. I'm not sure it's all that easy even with stack hackery. Remember that you want the class that the def statement of the method was written inside, not the class of self. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 18 02:08:08 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Apr 2006 12:08:08 +1200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> Message-ID: <44442DE8.4040206@canterbury.ac.nz> Tim Peters wrote: > I expect it depends on whether someone writes text-processing > algorithms. If you do, it's quite natural to iterate over strings. Personally I never find it natural to iterate over strings, because it makes me think "this is going to be horrendously inefficient", and I go looking for some way of doing what I want en mass, using the existing string methods, or regular expressions, or something. That's why I didn't think it would have all that much effect on real-life code. But maybe others aren't so averse to string-iterating as I am. -- Greg From ianb at colorstudy.com Tue Apr 18 02:34:53 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 17 Apr 2006 19:34:53 -0500 Subject: [Python-3000] Making strings non-iterable In-Reply-To: References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> Message-ID: <4444342D.2010007@colorstudy.com> Guido van Rossum wrote: > On 4/17/06, Ian Bicking wrote: > >>The proposal wasn't to remove iterability in general, only to require >>the use of an explicit method when iterating over strings. It's not a >>huge change by any means; a speed bump for a small number of people, >>perhaps, but no real functionality would be removed. Backward >>compatibility is also certainly a concern -- and what seems to have >>soured Guido on the idea as much as anything -- but that doesn't seem to >>be what has you bothered. > > > "A speed bump for a small number of people"? "No real functionality > remove"? You gotta be kidding. It drops polymorphism between strings > and other sequences. This is not just of theoretical value -- difflib > relies on this, for example. Your (earlier) assertion that the stdlib > is atypical is wishful thinking; the stdlib contains lots of code just > like code written by regular people. Perhaps you've had Perl on your > mind too much recently? Polymorphism between things that aren't similar is bad; in another context it is called implicit coersion, like '1'+2=='12'. That's what string iteration looks like to me -- strings aren't sequences of strings. It can be convenient to treat them like sequences, but lots of things can be convenient, including '1'+2=='12'. But I can accept and understand that the change introduces too many backward compatibility problems, especially subtle ones like removing type equivalencies, which can be very hard to detect or resolve. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Tue Apr 18 02:37:46 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 17 Apr 2006 19:37:46 -0500 Subject: [Python-3000] auto-super() In-Reply-To: <44442CE2.6020801@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <44442CE2.6020801@canterbury.ac.nz> Message-ID: <444434DA.3080500@colorstudy.com> Greg Ewing wrote: > Aahz wrote: >>Remember that in its current form super() is a regular function; it >>cannot be implied without stack hackery. > > > I'm not sure it's all that easy even with stack hackery. > Remember that you want the class that the def statement > of the method was written inside, not the class of > self. Whatever resolves __private variables could be used to the same effect to bind super, couldn't it? Still a language-level change, but not one without precedence. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Tue Apr 18 02:53:03 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 17 Apr 2006 19:53:03 -0500 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> Message-ID: <4444386F.2010603@colorstudy.com> Guido van Rossum wrote: > Here's a related but more complicated wish: define a function in such > a way that certain parameters *must* be passed as keywords, *without* > using *args or **kwds. This may require a new syntactic crutch. Perhaps: def func(x, *args=(), foo=1): ... Well, that doesn't really relate at all to what = means in that position. "def func(x, *None, foo=1)"? "def func(x, **, foo=1)"... oh, well why not just "def func(x, &optional foo=1)". Maybe related is a way of telling if the keyword was passed at all. In other words, a replacement for: sentinal = [] def func(x, y=sentinal): if y is sentinal: # calculate from x Or, without any danger of a misused sentinal: def func(x, *args, **kw): if args: assert not kw assert len(args) == 1 y = args[0] elif kw: y = kw.pop('y') assert not kw else: # calculate from x But the second gives a ruined signature, and a proper implementation has error messages that aren't assertion errors, and so would take much longer to write. But I can't even think of stupid syntax for the latter; or I guess I can't think of useful semantics. The best I can think of (ignoring syntax): def func(x, [y=sentinal]): try: y except NameError: # calculate from x ... which is annoying, but maybe better than the alternatives. Well, I'm not sure if it is actually better than just using a sentinal, which generally works just fine. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From rasky at develer.com Tue Apr 18 02:57:44 2006 From: rasky at develer.com (Giovanni Bajo) Date: Tue, 18 Apr 2006 02:57:44 +0200 Subject: [Python-3000] super(): issues [Was: Removing 'self' from method definitions] References: <443E7F47.9060308@colorstudy.com><20060413171301.GA9869@localhost.localdomain><443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> Message-ID: <064401c66283$1a392bb0$2452fea9@bagio> Greg Ewing wrote: > The use case for super() is where you have a chain of > methods that are sufficiently compatible that you don't > need to know or care which one you're calling next. > That's not usually the case with __init__ methods. It worked fine for me when I forced my clients (for a specific hierarchy) to always pass keyword arguments to __init__. This allowed me to use super() succesfully. Of course, Python non-existing support for mandatory keyword arguments makes things a little harder than they could be. I'll also point people to this page, which contains a very accurate description of the issues with super: http://fuhm.net/super-harmful/ Giovanni Bajo From greg.ewing at canterbury.ac.nz Tue Apr 18 03:13:38 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Apr 2006 13:13:38 +1200 Subject: [Python-3000] Test and Assign [was: More wishful thinking] In-Reply-To: <06Apr17.142412pdt.58633@synergy1.parc.xerox.com> References: <06Apr17.142412pdt.58633@synergy1.parc.xerox.com> Message-ID: <44443D42.606@canterbury.ac.nz> Bill Janssen wrote: > Presumably a reference to a Python symbol would be not just the symbol > name string, but also an indicator of the namespace of the symbol. That would be something very new -- nothing like that was implied by the original suggestion, and no other language I know of that has symbols gives them any such powers. What you seem to be suggesting here is lvalues as first-class objects, which is an interesting idea, but probably rather too radical even for py3k... -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 18 03:23:33 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Apr 2006 13:23:33 +1200 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) In-Reply-To: References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> Message-ID: <44443F95.7070707@canterbury.ac.nz> Guido van Rossum wrote: > Inlining > code obviously addresses this, but that's often killing code > structure. Would automated inlining be a legitimate use for ast-hacking? -- Greg From aahz at pythoncraft.com Tue Apr 18 03:26:45 2006 From: aahz at pythoncraft.com (Aahz) Date: Mon, 17 Apr 2006 18:26:45 -0700 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <4444342D.2010007@colorstudy.com> References: <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> <4444342D.2010007@colorstudy.com> Message-ID: <20060418012645.GA27998@panix.com> On Mon, Apr 17, 2006, Ian Bicking wrote: > > Polymorphism between things that aren't similar is bad; in another > context it is called implicit coersion, like '1'+2=='12'. That's > what string iteration looks like to me -- strings aren't sequences of > strings. It can be convenient to treat them like sequences, but lots > of things can be convenient, including '1'+2=='12'. But string iteration isn't about treating strings as sequences of strings, it's about treating strings as sequences of characters. The fact that characters are also strings is the reason we have problems, but characters are strings for other good reasons. There's just no way around that conundrum, and changing the current situation will only exchange our current set of problems for another set. Let's stick with the devil we know. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From greg.ewing at canterbury.ac.nz Tue Apr 18 03:29:32 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Apr 2006 13:29:32 +1200 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <44441953.6090809@colorstudy.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> Message-ID: <444440FC.3040901@canterbury.ac.nz> Ian Bicking wrote: > So I don't think > there's any parallel to ML or other languages with character types. If you're after parallels, there's BASIC, which also didn't have a character type, and didn't try to pretend that strings were arrays of anything. The equivalent of s[i] would be mid$(s, i, 1), which is more akin to a slice than an indexing operation. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 18 03:32:01 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Apr 2006 13:32:01 +1200 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> <4443F0BB.40001@livinglogic.de> Message-ID: <44444191.9020406@canterbury.ac.nz> Guido van Rossum wrote: > If we did end up allowing multiple **kwds, the code implementing calls > should check that they don't overlap with each other or with other > keyword args or with named positional args, An alternative might be to add a + operator to dicts which disallowed overlaps. That could be useful in other contexts as well. -- Greg From greg.ewing at canterbury.ac.nz Tue Apr 18 03:50:26 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Tue, 18 Apr 2006 13:50:26 +1200 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> Message-ID: <444445E2.9030506@canterbury.ac.nz> Talin wrote: > In this case, however, I don't see why we can't have the keyword > arguments out of order. But what's the *benefit* of being allowed to write that? > The important thing is that the behavior be clear and unambiguous, > which I think this is. It might be unambiguous, but I disagree completely about the 'clear'. Upon encountering such a call, I would have to stop and think quite hard, and probably go and consult the definition, to figure out what it's doing. To me, the only reason to pass non-optional positional arguments by keyword is when the call signature is not short and well-known enough for positional arguments to be clear -- and in that case, it's much better to pass *all* of them by keyword. -- Greg From krstic at fas.harvard.edu Tue Apr 18 04:26:20 2006 From: krstic at fas.harvard.edu (Ivan Krstic) Date: Mon, 17 Apr 2006 22:26:20 -0400 Subject: [Python-3000] Logix Message-ID: <44444E4C.3040607@fas.harvard.edu> I imagine most of the people on this list have seen it already, but if not, I wanted to point out Logix: http://www.livelogix.net/logix/ It's an interesting (working) experiment in building a Python with some very serious twists. In the context of this list, it might serve as a springboard for Guido to spectacularly shoot down some of the crazier potential Python 3k ideas ;) 60 sec intro: http://www.livelogix.net/logix/intro.html Presentation: http://solarsail.hcs.harvard.edu/~krstic/livelogix.pdf (I have no connection to Logix, I just converted the Powerpoint to PDF for your convenience.) certainly-wouldn't-mind-defop-macros-in-Python-though-it'll-never-happen-ly y'rs, -- Ivan Krstic | GPG: 0x147C722D From tjreedy at udel.edu Tue Apr 18 04:44:42 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 17 Apr 2006 22:44:42 -0400 Subject: [Python-3000] Python 3000 and the Google Summer of Code References: Message-ID: "Neal Norwitz" wrote in message news:ee2a432c0604171227y24d94102te2e77a18c0343146 at mail.gmail.com... > On 4/17/06, Terry Reedy wrote: >> >> The PyPy group also has a few prospective mentors, projects, and >> mentees. >> On the presumption that PSF coordinating sponsership is not limited to >> CPython only projects, I forwarded Neal's SoC py-dev post to their list. > > Excellent. I was thinking about specifically mailing PyPy, > IronPython, and Jython. > I think we should support them in addition to CPython. Your next post answered my question as to mentor eligibility: known of by you or two references therefrom. But back to project eligibility: how far beyond direct implementation-related projects? How about progammer support like pylint/pychecker? Or library extensions like numpy/scipy? Or webframeworks? Is there a Google page I should read for their views, as well as getting yours (and Guido's) as to what PSF wants to sponser? I don't want to mislead people. > I know what it takes to sign a mentor up. It's just a link. I > haven't publicised it yet. But I plan to soon. If you would like to > craft a nice message I can give you all the info so you can get the > word out. I will try to draft something tomorrow (Tues.) with blanks left for missing info. I am curious about last year's results for PSF projects: how many of how many actually finished enough to collect the full stipend? Is there a report that I never saw? Terry Jan Reedy From barry at python.org Tue Apr 18 05:54:57 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 17 Apr 2006 23:54:57 -0400 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <44440210.9010201@ewtllc.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> Message-ID: <1145332497.21740.52.camel@geddy.wooz.org> On Mon, 2006-04-17 at 14:01 -0700, Raymond Hettinger wrote: > -1 on making strings non-iterable. The cure is worse than the disease. > In ML, characters and strings are different datatypes and it is a > complete PITA. I suspect that making strings non-iterable would make > the language more difficult to use rather than simpler. Besides, if you > took away iterability, you would spend the rest of your life responding > to frequent requests to add it back ;-) Thing is, strings more so than any other collection want to be treated sometimes atomically and sometimes as a collection. You don't read the word "string" as ess, tee, are, eye, en, gee, you read it as the atomic unit "string". This is true in other languages than Python too, but in Python by default historically the sequential nature of strings takes precedence over the atomic nature of strings. I think there's value in exploring making strings atomic by default, and having to be explicit when you want to treat strings like a collection. I think the majority of programming uses of strings is as an atomic unit, not as a collection of characters. Sure, you might want to treat a string as a sequence of characters to iterate over them, but I believe that is a minority use case and so should not be implicitly supported. I haven't followed the views topic too closely, but ISTM little penalty to have to say "for c in mystring.characters()" for the very rare times I want to iterate over all the characters in a string (as opposed to taking a slice or checking startswith() or endswith()). It also seems like a price worth paying to not have to say "if not isinstance(s, basestring)" when flattening. It may not be possible to make this change for practical reasons, but I think that treating strings as atoms by default makes sense. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060417/2a9a3a1e/attachment.pgp From barry at python.org Tue Apr 18 06:00:23 2006 From: barry at python.org (Barry Warsaw) Date: Tue, 18 Apr 2006 00:00:23 -0400 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <20060417230805.GA23791@nightcrawler> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> <20060417230805.GA23791@nightcrawler> Message-ID: <1145332823.21740.56.camel@geddy.wooz.org> On Mon, 2006-04-17 at 16:08 -0700, Brian Harring wrote: > The issue I'm seeing is that the wart you're pointing at is a general > issue not limited to strings- everyone sooner or later has flattening > code that hits the "recursively iterate over this container, except > for instances of these classes". I wouldn't want to generalize this, but it is not infrequent that people mistakenly iterate over strings when they want to treat them atomically. difflib not withstanding, and keeping Guido's pronouncement in mind, I do think people want to treat strings atomically much more often then they want to treat them as a sequence of characters. nuff-said-moving-on-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060418/cf391b0d/attachment-0001.pgp From tdelaney at avaya.com Tue Apr 18 06:09:34 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Tue, 18 Apr 2006 14:09:34 +1000 Subject: [Python-3000] Making strings non-iterable Message-ID: <2773CAC687FD5F4689F526998C7E4E5FF1E675@au3010avexu1.global.avaya.com> Barry Warsaw wrote: > I wouldn't want to generalize this, but it is not infrequent that > people mistakenly iterate over strings when they want to treat them > atomically. difflib not withstanding, and keeping Guido's > pronouncement in mind, I do think people want to treat strings > atomically much more often then they want to treat them as a sequence > of characters. Perhaps we need a "no really - I really want to iterate over this thing" built-in and magic method. Then things that should generally be treated atomically would not implement __iter__, but could implement __deep_iter__ (which would return a standard iterator). This would allow both use cases, but atomicity being the default (i.e. a for loop continues to use __iter__). for e in deep_iter(obj): print e Tim Delaney From nnorwitz at gmail.com Tue Apr 18 07:22:22 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Mon, 17 Apr 2006 22:22:22 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: On 4/17/06, Terry Reedy wrote: > > Your next post answered my question as to mentor eligibility: known of by > you or two references therefrom. But back to project eligibility: how far > beyond direct implementation-related projects? How about progammer support > like pylint/pychecker? Or library extensions like numpy/scipy? Or > webframeworks? Is there a Google page I should read for their views, as > well as getting yours (and Guido's) as to what PSF wants to sponser? I > don't want to mislead people. Speaking for myself (not PSF and not Google), I think all of those above are fine. If it generally helps python, it's a good thing. A new module for python, that might never go in the stdlib is fine. For example, the PSF funded a grant to enhance PySNMP. This module is unlikely to ever be included in the stdlib, but it's still beneficial to have. I think it was appropriate to fund and would have been fine as an SoC project too. Your questions are best answered from Google's site (I don't know the answers): http://code.google.com/soc/ > I am curious about last year's results for PSF projects: how many of how > many actually finished enough to collect the full stipend? Is there a > report that I never saw? Here's a write up about last year's projects: http://wiki.python.org/moin/SummerOfCode/2005 Payment was all or nothing. I thought we had something like 18 projects, there are 14 up on the page above. I don't really know anything about last year though. Perhaps David Ascher knows, I think he may have been in charge of SoC last year from the PSFs side. I don't know if David is on this list, but I copied him. n From talin at acm.org Tue Apr 18 08:08:28 2006 From: talin at acm.org (Talin) Date: Tue, 18 Apr 2006 06:08:28 +0000 (UTC) Subject: [Python-3000] Function call speed (Was: Cleaning up argument listparsing) References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> Message-ID: Paul Moore gmail.com> writes: > > On 4/17/06, Terry Reedy udel.edu> wrote: > > > > "Talin" acm.org> wrote in message > > news:loom.20060417T072709-664 post.gmane.org... > > > The important thing is that the behavior be clear and unambiguous, > > > which I think this is. > > > > Many would also prefer that functions calls not become noticeable slower > > than they already are. > > How slow *are* function calls? There seems to be a common "function > calls in Python are slow" meme. It's never been a significant issue > for me, but my code is pretty much always IO-bound, so that doesn't > surprise me. Are they slow enough to warrant serious effort to speed > them up? > > More specifically (and more on-topic ) is there any room for > speeding up function calls in Python 3000? Given that pure performance > improvements are clearly OK for 2.x, I'm thinking of cases where it > would be necessary to impose language-level restrictions to ease the > optimisation process. > > Paul. Well, I spent a little time looking at the function calling code. Admittedly, there are quite a few things that I don't understand yet, however I did notice a couple of things. First, it isn't the Python language that's the problem, its the Python C API. For various good reasons, the Python-to-Python function calling code goes through a number of C API functions, such as PyObject_Call and friends. This makes it very easy to call C code, and for C code to call Python. However, it also means that the interface between caller and callee is fairly abstract and heavy - except in a few specialized cases, arguments have to be pulled off the stack, converted into tuples and dicts (with the associated memory allocation hits), and then pushed back onto the stack so that the caller can read them. This is especially significant considering that most functions with keyword arguments generally only have one or two of them, yet we are paying the price to create a whole new dictionary object each time. (In fact, I would wager that if you got rid of the dict and simply passed them in as an array of key/value pairs, it would be a speed win - the cost of linearly searching the names would be low compared to the cost of building the dict - for one thing, the array would have a smaller cache-miss footprint. The few cases where you have a lot of arguments would be rare enough that they wouldn't dominate performance.) It might be possible to write an algorithm that arranges the calling arguments on the stack directly in the form that the caller needs. However, in order for this to work, either the calling code would need to know a lot more information about the callee's method signature, or vice versa, the callee would need to know more about the arrangement of the caller's arguments on the stack. Making radical performance improvements in this area would be a fairly hefty challenge; My approach to the problem would be to start by limiting the number of design constraints, in particular stating from the outset that we're going to break the C API. Then, what I'd do is design an optimal performance pathway for internal calls, and then create a new C API which sits on top of that. Code using the external API would probably require a conversion step for arguments being passed to the outside world, but I don't think such a conversion step need be much more onerous or slow than what's being done already. Again, I could be completely full of it here - so caveat lector :) -- Talin From talin at acm.org Tue Apr 18 08:19:12 2006 From: talin at acm.org (Talin) Date: Tue, 18 Apr 2006 06:19:12 +0000 (UTC) Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) References: <44430CB5.5020907@gmail.com> <444445E2.9030506@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > Talin wrote: > > > In this case, however, I don't see why we can't have the keyword > > arguments out of order. > > But what's the *benefit* of being allowed to write that? > > > The important thing is that the behavior be clear and unambiguous, > > which I think this is. > > It might be unambiguous, but I disagree completely about the > 'clear'. Upon encountering such a call, I would have to stop and > think quite hard, and probably go and consult the definition, > to figure out what it's doing. No worries. I'm only pressing the issue because I want to make sure it gets examined before being dismissed; My primary concern is more about the "keywords after varargs" aspect of it, and less about the other stuff. (Although, all other things being equal, when arguing over whether a programming language should / should not allow programmers to do certain things, I think the onus is on the side advocating restriction to show why the programmer should so be restricted; which is not to say that there aren't good reasons, or that all other things *are* equal in this case. Especially now that I've looked at the function calling code, and I see why it would be difficult to implement.) -- Talin From david.ascher at gmail.com Tue Apr 18 08:37:39 2006 From: david.ascher at gmail.com (David Ascher) Date: Mon, 17 Apr 2006 23:37:39 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: > > I am curious about last year's results for PSF projects: how many of how > > many actually finished enough to collect the full stipend? Is there a > > report that I never saw? > > Here's a write up about last year's projects: > http://wiki.python.org/moin/SummerOfCode/2005 > > Payment was all or nothing. I thought we had something like 18 > projects, there are 14 up on the page above. I don't really know > anything about last year though. Perhaps David Ascher knows, I think > he may have been in charge of SoC last year from the PSFs side. I > don't know if David is on this list, but I copied him. No report was written up that i know of. Some people just flaked out and didn't do anything. I think that was the major source of dropouts. --david -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060417/aa1472f7/attachment.htm From baptiste13 at altern.org Tue Apr 18 10:12:16 2006 From: baptiste13 at altern.org (Baptiste Carvello) Date: Tue, 18 Apr 2006 10:12:16 +0200 Subject: [Python-3000] iter_flatten was: Making strings non-iterable In-Reply-To: References: Message-ID: Raymond Hettinger a ?crit : > [Brian Harring] > >>The issue I'm seeing is that the wart you're pointing at is a >> general issue not limited to strings- everyone sooner or later >> has flattening code that hits the "recursively iterate over >> this container, except for instances of these classes". > > > Good recap. For more info, search comp.lang.python for previous > discussions about a proposed iter_flatten() function. One of the > conclusions was that a general algorithm would need to accept a user > specified test for atomicity so it would know where to recurse and a > where not to (the answer to that is application dependent). > one useful test for that case could be: >>> item in item This returns True for string-like objects, False for other sane (non infinite) iterables. Anyhow, when it returns True, you most definitely want to stop recursing! Baptiste From guido at python.org Tue Apr 18 10:36:21 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 09:36:21 +0100 Subject: [Python-3000] Specializing the dicts in __dict__ In-Reply-To: <20060417235854.GF5762@performancedrivers.com> References: <20060417235854.GF5762@performancedrivers.com> Message-ID: On 4/18/06, Jack Diederich wrote: > I mentioned then and would like to resurrect now the idea of making > a dict-alike that has two extra properties: > > 1: the keys can only be str or unicode (py3k: just unicode) This seems useful -- although the current dict type is already optimized for keys consisting solely of strings (8-bit only) so I'm skeptical about the obtainable speed-up. > 2: the dict-alike is ordered I've recently figured out what people mean by an "ordered dict" -- most people seem to agree that it's a dict that behaves like you describe: > The suggested order would be by key creation so that reassigning > a key after manipulation would maintain the original order as in > the Thing.b case above. Unfortunately I don't think you can implement this without significantly slowing things down. This belief is in part based on my own prototype and the ordereddict implementations I found through a bit of searching on Google; in part because if such an algorithm existed (i.e. an ordered dict faster than hashing) it would be very popular -- but I've never heard of one. So I'm against making the class dict an ordereddict by default (but I'd be for making it a symdict if we can implement one that's faster than a plain dict). Instead, I'd focus on changing the way the class statement works so that the metaclass can provide the dict object to be used for the local namespace in the class body. The API might be something like a class method on 'type' that takes some metadata like the proposed class name and the tuple of base classes, and returns a dict or a dict-ish object. The default implementation on 'type' could ignore the arguments and return a fresh symdict instance (or a dict instance if we find we can't implement a faster symdict). In order to be useful for Py3k, the symdict should focus exclusively on Unicode strings. However I'm not sure that we'll keep the current Unicode implementation exactly. So perhaps it's too early to attempt to produce one for Py3k -- but you could work the 'type' API details and changes to the class statement easily. PS: if this change to the class statement were to happen, the crowd discussing the 'make' PEP (which I don't like but don't have the power to stop being discussed) should take notice. Or perhaps they could run with the idea and make it the focal point of their proposal. Or whatever. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 18 10:49:28 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 09:49:28 +0100 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: <44442C3C.2090908@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <44442C3C.2090908@canterbury.ac.nz> Message-ID: On 4/18/06, Greg Ewing wrote: > The problem is that there's no way for Python to know > which class the method is "in", in the sense required > here. > > That could be fixed by giving functions defined inside > a class statement an attribute referring back to the > class. It would create a circular reference, but that's > not so much of a problem these days. This begs two questions (at least): - How would the function be given the attribute? At the time the function is being defined the class doesn't exist yet; and by the time the class exists, the function may be wrapped in any number of layers of decorators which may or may not pass on attribute assignment (most implementations of decorators I've seen don't -- at best they copy the attributes that already exist on the function into the wrapper). - How would the code representing the function body access the attribute? By the time the code is executing all that is known are the argument values and the globals. There's no reference to the function object itself (nor to the bound method). (Nor to the class, for that matter.) You may be able to come up with suitable answers -- but IMO these questions (again) point out that the coupling between methods and the class to which they belong is much looser in Python than in most other languages. This started as a hack when classic classes were invented as an afterthought; but this has now become a fundamental and desirable feature of the language. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 18 10:52:44 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 09:52:44 +0100 Subject: [Python-3000] auto-super() In-Reply-To: <444434DA.3080500@colorstudy.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <44442CE2.6020801@canterbury.ac.nz> <444434DA.3080500@colorstudy.com> Message-ID: On 4/18/06, Ian Bicking wrote: > Whatever resolves __private variables could be used to the same effect > to bind super, couldn't it? Still a language-level change, but not one > without precedence. Hardly the same mechanism; __private is done by low-level name mangling in the early stages of the compiler. super() would need a class object to be reliable; the class *name* (which is all that's available at compile time) might resolve to something else at runtime. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ronaldoussoren at mac.com Tue Apr 18 10:56:28 2006 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Tue, 18 Apr 2006 10:56:28 +0200 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <44442C3C.2090908@canterbury.ac.nz> Message-ID: On 18-apr-2006, at 10:49, Guido van Rossum wrote: > On 4/18/06, Greg Ewing wrote: >> The problem is that there's no way for Python to know >> which class the method is "in", in the sense required >> here. >> >> That could be fixed by giving functions defined inside >> a class statement an attribute referring back to the >> class. It would create a circular reference, but that's >> not so much of a problem these days. > > This begs two questions (at least): > > - How would the function be given the attribute? At the time the > function is being defined the class doesn't exist yet; and by the time > the class exists, the function may be wrapped in any number of layers > of decorators which may or may not pass on attribute assignment (most > implementations of decorators I've seen don't -- at best they copy the > attributes that already exist on the function into the wrapper). And related to this: a function may be used as a method on several classes, storing the attribute on the function is therefore not possible without significantly changing the language. class A (object): pass class B (object): pass def myMethod(self): print "hello from", self.__class__ A.myMethod = myMethod B.myMethod = myMethod A().myMethod() B().myMethod() Ronald From nick at craig-wood.com Tue Apr 18 11:02:15 2006 From: nick at craig-wood.com (Nick Craig-Wood) Date: Tue, 18 Apr 2006 10:02:15 +0100 Subject: [Python-3000] A minor cleanup: instances from bound methods Message-ID: <20060418090215.GB26873@craig-wood.com> I noticed this the other day. Perhaps it is suitable for a python 3000 cleanup? It certainly seems illogical, but probably too intrusive to change in python 2.x. I needed to find the instance from a bound method, with obj.im_self. Eg >>> class C(object): ... def fn(self): print "hello" ... >>> c=C() >>> fn=c.fn >>> fn > >>> fn.im_self <__main__.C object at 0xb7dd2acc> >>> fn.__self__ Traceback (most recent call last): File "", line 1, in ? AttributeError: 'function' object has no attribute '__self__' >>> But I discovered that builtin objects work in a completely different way with obj.__self__, Eg >>> fd=open("myfile","w") >>> fn=fd.write >>> fn >>> fn.im_self Traceback (most recent call last): File "", line 1, in ? AttributeError: 'builtin_function_or_method' object has no attribute 'im_self' >>> fn.__self__ >>> I suggest that either im_self or __self__ is renamed! -- Nick Craig-Wood -- http://www.craig-wood.com/nick From birchb at anz.com Tue Apr 18 11:19:37 2006 From: birchb at anz.com (Birch, Bill) Date: Tue, 18 Apr 2006 19:19:37 +1000 Subject: [Python-3000] Type Expressions Message-ID: Hi, There is a massive amount of discussion about adding type checking to Python going back a long way, however most discussions gloss over exactly what type expressions are exactly. I'm working on this question with the goal of putting up a proposal of some kind to this list. So far I have some conclusions/opinions which I'd like to share and get feedback and help on. Here's some points which I want to get out there: (a) A lot of work on types does not consider composite types. A composite type is something like: [int, int, str], ie a type consisting of atomic types held together by collections. As a programmer I want to be able to formally define composite types. I am parsing a CSV file I'd like to be able to declare the file format as a type. These types are very important in the following contexts: (1) reading input and validating it. (XML anyone?) (2) unit testing - did my code create the right output? (3) documenting the types my code uses for human readers in a standard way (b) Much of the academic papers I have read on types are focussed on compile-time static type checking. This is understandable, but Python is a dynamic language and an object can change it's structure at run-time so programs will always outwit the compiler. So IMHO Python needs to have a _dynamic_ type expression system. This also gives Python programmers access to type expressions, they should not be the sole preserve of the compiler writers. (b) I work with some heavy-hitting java developers who don't write a line of code without unit tests. Lots of unit tests. Even developers using a statically typed language rely on unit tests for program correctness. I trust Python programmers are the same. So any discussion of type systems needs to consider what benefits to unit test writing. IMHO dynamic type expressions with dynamic type checking predicates are very useful in unit tests. They can help answer the question "is my program creating the right structures?" Even when those structures are untyped collections. (c) Simple composite type expressions with dynamic predicates are in my view more useful and universal than complex Interface or Adaption schemes. I would prefer to have a flexible and expressive type expression facility than Interfaces. (e) Unlike a compiled type system, a dynamic type system does have access to values. So we can define types where values are given. e.g. the type ('href', str) is the set of all 2-tuples where the first element is 'href'. Compiled static type systems don't normally allow this. (d) Working code. So far I have been coding some simple examples of type expressions with some predicates. I am (ab)using the existing Python syntax to get type expressions. Here are some examples: (int, int) # a tuple with two ints [int, etc] # a list of ints # (etc = zero or more of the previous element) {'b':[12, 34], 'c':list, 'a':long} # dict Union(int, float) function(a=int, b=float, returns=float) pyobject(xcoord=int, ycoord=int) # an instance These are all true: isSubType(list, [42]) isSubType((int, etc), (int, int, int)) isSubType(Union(int, float, str), Union(float, int)) isSubType({'a':Number, 'b':object}, {'b':[12, 34], 'c':list, 'a':long}) (e) Python needs a succinct type expression syntax! Really. (f) Predicates on types: "isInstance(T1, x)". Given a Python object, can x be used where type T1 is expected? And isSubType(T1, T2) - can T2 be used where a T1 is expected? This is a _structural_ subtyping predicate. Given point (e), isSubType() and isInstance() look like the same thing. If a value is also a valid type expression there's no distinction. For now my implementation of isSubType() accepts instances too. (f) A dynamic type expression system also allows programmers to _generate_ objects from type expressions. I have a function generateRandom(T) which returns a random instance of type T. In a loop this is a fun way to break code. (g) Parametric types are simple function calls: lambda T: {'stack': [T], 'pop': function(returns=T) , 'push': function(newtop=T)}. Type names are variable bindings, just like values. (h) Interfaces are just rather complex type expressions. In summary a type system should be _dynamic_ and not static. It's reflection API should be trivial to use. It should help programmers to answer these questions: Is this data I just received in the right structure? Did my program just output something in the right structure? What structure am I expected to pass to this library? My view is that a dynamic type system used in unit testing will find more defects than a static type system. And it will be fun. From guido at python.org Tue Apr 18 11:50:08 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 10:50:08 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: References: Message-ID: You seem to have missed some of my blog entries on Artima early last year. I proposed a syntax that should address most of your issues, e.g. (int, str, float) as a tuple of thos three types, T1|T2 for unions, list[int] for a list of ints. --Guido On 4/18/06, Birch, Bill wrote: > Hi, > > There is a massive amount of discussion about adding type checking to Python going back a long way, however most discussions gloss over exactly what type expressions are exactly. I'm working on this question with the goal of putting up a proposal of some kind to this list. > > So far I have some conclusions/opinions which I'd like to share and get feedback and help on. Here's some points which I want to get out there: > > (a) A lot of work on types does not consider composite types. A composite type is something like: [int, int, str], ie a type consisting of atomic types held together by collections. As a programmer I want to be able to formally define composite types. I am parsing a CSV file I'd like to be able to declare the file format as a type. These types are very important in the following contexts: > (1) reading input and validating it. (XML anyone?) > (2) unit testing - did my code create the right output? > (3) documenting the types my code uses for human readers in a standard way > > (b) Much of the academic papers I have read on types are focussed on compile-time static type checking. This is understandable, but Python is a dynamic language and an object can change it's structure at run-time so programs will always outwit the compiler. So IMHO Python needs to have a _dynamic_ type expression system. This also gives Python programmers access to type expressions, they should not be the sole preserve of the compiler writers. > > (b) I work with some heavy-hitting java developers who don't write a line of code without unit tests. Lots of unit tests. Even developers using a statically typed language rely on unit tests for program correctness. I trust Python programmers are the same. So any discussion of type systems needs to consider what benefits to unit test writing. IMHO dynamic type expressions with dynamic type checking predicates are very useful in unit tests. They can help answer the question "is my program creating the right structures?" Even when those structures are untyped collections. > > (c) Simple composite type expressions with dynamic predicates are in my view more useful and universal than complex Interface or Adaption schemes. I would prefer to have a flexible and expressive type expression facility than Interfaces. > > (e) Unlike a compiled type system, a dynamic type system does have access to values. So we can define types where values are given. e.g. the type ('href', str) is the set of all 2-tuples where the first element is 'href'. Compiled static type systems don't normally allow this. > > (d) Working code. So far I have been coding some simple examples of type expressions with some predicates. I am (ab)using the existing Python syntax to get type expressions. Here are some examples: > > (int, int) # a tuple with two ints > > [int, etc] # a list of ints > # (etc = zero or more of the previous element) > > {'b':[12, 34], 'c':list, 'a':long} # dict > > Union(int, float) > > function(a=int, b=float, returns=float) > > pyobject(xcoord=int, ycoord=int) # an instance > > These are all true: > > isSubType(list, [42]) > isSubType((int, etc), (int, int, int)) > isSubType(Union(int, float, str), Union(float, int)) > isSubType({'a':Number, 'b':object}, {'b':[12, 34], 'c':list, 'a':long}) > > (e) Python needs a succinct type expression syntax! Really. > > (f) Predicates on types: "isInstance(T1, x)". Given a Python object, can x be used where type T1 is expected? And isSubType(T1, T2) - can T2 be used where a T1 is expected? This is a _structural_ subtyping predicate. Given point (e), isSubType() and isInstance() look like the same thing. If a value is also a valid type expression there's no distinction. For now my implementation of isSubType() accepts instances too. > > (f) A dynamic type expression system also allows programmers to _generate_ objects from type expressions. I have a function generateRandom(T) which returns a random instance of type T. In a loop this is a fun way to break code. > > (g) Parametric types are simple function calls: lambda T: {'stack': [T], 'pop': function(returns=T) , 'push': function(newtop=T)}. Type names are variable bindings, just like values. > > (h) Interfaces are just rather complex type expressions. > > In summary a type system should be _dynamic_ and not static. It's reflection API should be trivial to use. It should help programmers to answer these questions: Is this data I just received in the right structure? Did my program just output something in the right structure? What structure am I expected to pass to this library? My view is that a dynamic type system used in unit testing will find more defects than a static type system. And it will be fun. > > > > > > > > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 18 12:43:58 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 11:43:58 +0100 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) In-Reply-To: <44443F95.7070707@canterbury.ac.nz> References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> <44443F95.7070707@canterbury.ac.nz> Message-ID: On 4/18/06, Greg Ewing wrote: > Guido van Rossum wrote: > > > Inlining > > code obviously addresses this, but that's often killing code > > structure. > > Would automated inlining be a legitimate use for ast-hacking? Perhaps, if you can do it in a way that it also works for IronPython and Jython... Though anything for which you can't come up with a better word than "hacking" probably oughtn't be in the language spec. :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From boris at batiment71.net Tue Apr 18 14:06:17 2006 From: boris at batiment71.net (Boris Borcic) Date: Tue, 18 Apr 2006 14:06:17 +0200 Subject: [Python-3000] genexp syntax for reduce (already in 2.5a1) Message-ID: <4444D639.6030108@batiment71.net> Hello, Since the notion of a "generator comprehension" reduce syntax for py3k was discussed on this list - it may have escaped attention that py2.5a1 already allows the following, admittedly not very efficient (nor too robust in this proof-of-concept implementation). hth, Boris Borcic def ireduce(gen) : """ Generator expression syntax for reduce and generalizations Usage examples: - (yield) nothing as below if there is no seed value *and* the expression consists of a single arithmetic op >>> ireduce(x+(yield) for x in range(101)) # famous Gauss example 5050 >>> ireduce(x*(yield) for x in range(1,6)) 120 - if there is a seed, yield it (unless it's None) >>> ireduce(x*(yield 2) for x in range(1,6)) # 2*1*2*3*4*5 240 >>> ireduce(x*(yield 2) for x in range(2,6)) # 2*2*3*4*5 240 >>> ireduce(x*(yield 2) for x in range(6)) # 2*0*1*2*3*4*5 0 - if the seed is None, (yield INone) instead >>> ireduce((x,(yield INone)) for x in "for all good men".split()) ('men', ('good', ('all', ('for', None)))) >>> ireduce([(yield INone),x] for x in "for all good men".split()) [[[[None, 'for'], 'all'], 'good'], 'men'] - do as below if you want no special seed and the operation to reduce isn't a simple arithmetic operation >>> ireduce((x,(yield Seed(x))) for x in "for all good men".split()) ('men', ('good', ('all', 'for'))) >>> ireduce(x+' '+(yield Seed(x)) for x in "for all good men".split()) 'men good all for' >>> ireduce({ x : (yield Seed(x))} for x in "for all good men".split()) {'men': {'good': {'all': 'for'}}} - and notice these for a comparison >>> ireduce({ x : (yield x)} for x in "for all good men".split()) {'men': {'good': {'all': {'for': 'for'}}}} >>> ireduce({ x : (yield Seed(None))} for x in "for all good men".split()) {'men': {'good': {'all': None}}} """ cumulate = gen.next() if cumulate is None : cumulate = Neutral() elif cumulate is INone : cumulate = None elif isinstance(cumulate,Seed) : cumulate = cumulate.seed gen.send(cumulate) gen.next() try : while True : cumulate = gen.send(cumulate) gen.next() except StopIteration : return cumulate class Neutral : # this would need reworking def __coerce__(self,other) : self.other = other return (self,self) def __getattr__(self,attr) : return lambda *x,**y : self.__dict__['other'] class INone : pass class Seed : def __init__(self,seed) : self.seed = seed From lists at janc.be Tue Apr 18 14:41:55 2006 From: lists at janc.be (Jan Claeys) Date: Tue, 18 Apr 2006 14:41:55 +0200 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: <1145364116.12589.24.camel@localhost.localdomain> Op ma, 17-04-2006 te 22:44 -0400, schreef Terry Reedy: > Your next post answered my question as to mentor eligibility: known of > by you or two references therefrom. But back to project eligibility: > how far beyond direct implementation-related projects? [...] Or > webframeworks? Well, I know the Django (web framework) people are maybe interested in someone doing a SoC project for them: -- Jan Claeys From murman at gmail.com Tue Apr 18 15:29:49 2006 From: murman at gmail.com (Michael Urman) Date: Tue, 18 Apr 2006 08:29:49 -0500 Subject: [Python-3000] auto-super() In-Reply-To: References: <443E7F47.9060308@colorstudy.com> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <44442CE2.6020801@canterbury.ac.nz> <444434DA.3080500@colorstudy.com> Message-ID: On 4/18/06, Guido van Rossum wrote: > Hardly the same mechanism; __private is done by low-level name > mangling in the early stages of the compiler. super() would need a > class object to be reliable; the class *name* (which is all that's > available at compile time) might resolve to something else at runtime. That's all the current super() gets, right? super(Base, self).meth() compiles with a LOAD_GLOBAL (Base)... Any mechanisms I can think of to solidify that into the actual class reference either rely on __mangled_attrs being assigned at or after class construction, or potentially break in inherited classes. Michael -- Michael Urman http://www.tortall.net/mu/blog From thomas at python.org Tue Apr 18 16:34:56 2006 From: thomas at python.org (Thomas Wouters) Date: Tue, 18 Apr 2006 16:34:56 +0200 Subject: [Python-3000] auto-super() In-Reply-To: <20060417031858.GA354@panix.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> Message-ID: <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> On 4/17/06, Aahz wrote: > > On Sun, Apr 16, 2006, Michael P. Soulier wrote: > > On 17/04/06 Greg Ewing said: > >> > >> The other possible reason for using super() is so you don't have > >> to write the name of the base class into all your inherited method > >> calls. But that's a separate issue that would be better addressed by > >> a different mechanism, rather than conflating the two in super(). > > > > Although you do have to put the current class name in the method > > calls, as super() requires it as the first argument. I never > > understood that. Why would I wish to use super(Bar) if I'm in class > > Foo? Cannot Foo be implied here? > > Remember that in its current form super() is a regular function; it > cannot be implied without stack hackery. The autosuper metaclass should > probably become part of the regular type metaclass in 3.0, but I think > that requires a PEP -- and in keeping with Guido's recent admonishment, > someone should probably try implementing it first and see what happens > with the test suite. The autosuper class (and any other class that tries to do the same thing the same way) has a serious problem (which it inherits from name-mangled attrs): nameclashes. I usually use an autosuper-style metaclass (which also does auto-class/staticmethoding and autopropping) that explicitly prohibits subclasses with the same name, but I can only do that because I tend to control the entire class hierarchy. I wouldn't want to use it in public code, certainly not for common baseclasses. I'm in favour of super(). The arguments against super() are understandable -- if you only think about your own class, or completely disregard MI. I think that's irresponsible: super() always calls the right baseclass method, even if it changes its signature. If your class ends up being used in MI *and* changing baseclass signatures (in unexpected order), it will break with super() just as hard as without super(), but not vice versa ;) I'd really like an easier way to spell super() in 3.0 (or 2.6), as well as have super() support old-style classes. The ideas I've so far come up with: - Compiler hackery involving a magical variable name, say '__class__' or '__CLASS__'. The compiler would treat this specially, probably stored in the class dict, and type() (which is, after all, called to actually create the class) would stuff the actual class object in there. It causes a reference cycle, but all newstyle classes already do that ;P The main issue is that __CLASS__ would be new magic. It wouldn't exist when the class body is executed, and it would be a special form of enclosed variable afterwards (it should be extracted from the class namespace, using a similar mechanism as closures.) There's also the conceptually new idea of a variable having a static, compile-time meaning, inviting suggestions for __FILE__ and __LINE__ (which are actually easy to implement ;P) - Fixing __-name-mangling so nameclashes don't occur, by using the 'fully qualified' classname in the mangling. The attribute wouldn't be accessible using normal attribute retrieval, but getattr() can get at it (and so can the C code, of course.) This has obvious repercussions, although I'm not sure if they're bad. I've also considered stackframe hackery to get at the desired class from super(), or retroactively patching all methods of a class to have a 'my class' attribute or variable, but those won't work (as Guido exlained.) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060418/b63c1994/attachment.htm From aahz at pythoncraft.com Tue Apr 18 17:30:28 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 18 Apr 2006 08:30:28 -0700 Subject: [Python-3000] auto-super() In-Reply-To: <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> Message-ID: <20060418153028.GB21539@panix.com> On Tue, Apr 18, 2006, Thomas Wouters wrote: > > - Compiler hackery involving a magical variable name, say '__class__' > or '__CLASS__'. The compiler would treat this specially, probably > stored in the class dict, and type() (which is, after all, called to > actually create the class) would stuff the actual class object in > there. It causes a reference cycle, but all newstyle classes already > do that ;P The main issue is that __CLASS__ would be new magic. It > wouldn't exist when the class body is executed, and it would be a > special form of enclosed variable afterwards (it should be extracted > from the class namespace, using a similar mechanism as closures.) It's not clear to me that this requires compiler hackery, just metaclass hackery. Am I missing something? Also, my idea is that self.super is a bound method/closure that already contains a reference to the class. This makes dynamic classes more difficult in some ways, but anyone who wants to play those sorts of games should expect to do some magic mangling. If it's reasonably well-documented, it wouldn't even be that difficult. > - Fixing __-name-mangling so nameclashes don't occur, by using the > 'fully qualified' classname in the mangling. The attribute wouldn't be > accessible using normal attribute retrieval, but getattr() can get at > it (and so can the C code, of course.) This has obvious repercussions, > although I'm not sure if they're bad. +1 Of course, we will still have name problems, but they won't be clashes. I don't know that this is needed for the super() issue, but there are other problems with the current mangling system. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From janssen at parc.com Tue Apr 18 18:24:35 2006 From: janssen at parc.com (Bill Janssen) Date: Tue, 18 Apr 2006 09:24:35 PDT Subject: [Python-3000] Test and Assign [was: More wishful thinking] In-Reply-To: Your message of "Mon, 17 Apr 2006 18:13:38 PDT." <44443D42.606@canterbury.ac.nz> Message-ID: <06Apr18.092438pdt."58633"@synergy1.parc.xerox.com> > > Presumably a reference to a Python symbol would be not just the symbol > > name string, but also an indicator of the namespace of the symbol. > > That would be something very new -- nothing like that was > implied by the original suggestion, and no other language > I know of that has symbols gives them any such powers. Really? Common Lisp symbols have the form "[:]", where defaults to the local namespace (well, actually the default is a bit more complicated). The original poster was suggesting ":", which I suppose I read in its Lisp interpretation as a symbol in the KEYWORD package. (See the "symbol-package" function in Common Lisp.) Bill From steven.bethard at gmail.com Tue Apr 18 18:54:33 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Tue, 18 Apr 2006 10:54:33 -0600 Subject: [Python-3000] Specializing the dicts in __dict__ In-Reply-To: References: <20060417235854.GF5762@performancedrivers.com> Message-ID: On 4/18/06, Guido van Rossum wrote: > PS: if this change to the class statement were to happen, the crowd > discussing the 'make' PEP (which I don't like but don't have the power > to stop being discussed) should take notice. If you pronounce it dead, I can make sure the discussion stops. :-) > Or perhaps they could run with the idea and make it the focal > point of their proposal. There has been discussion of something like this: http://mail.python.org/pipermail/python-dev/2006-April/063706.html http://mail.python.org/pipermail/python-list/2006-April/336586.html But I think the consensus was that there weren't a lot of good use-cases and adding such support would just make the statement more complex. However, if type objects started to support something like this, I guess the make statement should reconsider it. Could someone put out some use-cases where it would be really helpful to be able to specify a different dict type for the class dict? Steve -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From ianb at colorstudy.com Tue Apr 18 18:58:08 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 18 Apr 2006 11:58:08 -0500 Subject: [Python-3000] auto-super() In-Reply-To: <20060418153028.GB21539@panix.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <20060418153028.GB21539@panix.com> Message-ID: <44451AA0.6090009@colorstudy.com> Aahz wrote: > On Tue, Apr 18, 2006, Thomas Wouters wrote: > >> - Compiler hackery involving a magical variable name, say '__class__' >>or '__CLASS__'. The compiler would treat this specially, probably >>stored in the class dict, and type() (which is, after all, called to >>actually create the class) would stuff the actual class object in >>there. It causes a reference cycle, but all newstyle classes already >>do that ;P The main issue is that __CLASS__ would be new magic. It >>wouldn't exist when the class body is executed, and it would be a >>special form of enclosed variable afterwards (it should be extracted >>from the class namespace, using a similar mechanism as closures.) > > > It's not clear to me that this requires compiler hackery, just metaclass > hackery. Am I missing something? > > Also, my idea is that self.super is a bound method/closure that already > contains a reference to the class. This makes dynamic classes more > difficult in some ways, but anyone who wants to play those sorts of games > should expect to do some magic mangling. If it's reasonably > well-documented, it wouldn't even be that difficult. Well: class A(object): def whoami(self): return 'A' def dostuff(self): print 'I (%s) am doing A stuff...' % (self.whoami()) class B(A): def whoami(self): return 'B-ish %s' % (self.super.whoami()) def dostuff(self): print 'Some B stuff...' self.super.dostuff() class C(B): def whoami(self): return 'C' def dostuff(self): print 'Some C stuff...' self.super.dostuff() Reading this statically, I think it's clear how this should work. So, when C().dostuff() is called, we want to see: Some C stuff... Some B stuff... I (C) am doing A stuff... When B().dostuff() is called, we want to see: Some B stuff... I (B-ish A) am doing A stuff... But how can we do that without the function being bound to a class? self.whoami() in A.dostuff needs to access the 'real' self. self.super in B.dostuff should be super(B, self), even when self.__class__ is C. Because Python *isn't* reading the functions statically to determine class and method layout, actually making this work seems very hard. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Tue Apr 18 18:59:43 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 18 Apr 2006 11:59:43 -0500 Subject: [Python-3000] Specializing the dicts in __dict__ In-Reply-To: References: <20060417235854.GF5762@performancedrivers.com> Message-ID: <44451AFF.6050801@colorstudy.com> Guido van Rossum wrote: >>The suggested order would be by key creation so that reassigning >>a key after manipulation would maintain the original order as in >>the Thing.b case above. > > > Unfortunately I don't think you can implement this without > significantly slowing things down. This belief is in part based on my > own prototype and the ordereddict implementations I found through a > bit of searching on Google; in part because if such an algorithm > existed (i.e. an ordered dict faster than hashing) it would be very > popular -- but I've never heard of one. > > So I'm against making the class dict an ordereddict by default (but > I'd be for making it a symdict if we can implement one that's faster > than a plain dict). Instead, I'd focus on changing the way the class > statement works so that the metaclass can provide the dict object to > be used for the local namespace in the class body. The API might be > something like a class method on 'type' that takes some metadata like > the proposed class name and the tuple of base classes, and returns a > dict or a dict-ish object. The default implementation on 'type' could > ignore the arguments and return a fresh symdict instance (or a dict > instance if we find we can't implement a faster symdict). I assume that means the metaclass wouldn't just be defined by __dict__['__metaclass__'], since that's only determined after the body of the class has been evaluated, and any custom dict for the namespace has to be determined earlier. If type(name, bases, dict) coerced the dict object into a Real Dict Object, that would still give metaclasses the power to determine order, without having to effect the actual performance of the class. It would be up to metaclasses to save information they were interested in. Some ordered dict classes just do that for you, I think -- they have an underlying dict (with normal dict performance), plus they keep track of the order of the keys. There's more memory used, but __getitem__ performance needn't be substantially effected. (Any setting of keys has to be tracked, so there's a minor performance penalty.) The potential for this performance characteristic is perhaps what distinguishes an ordered dict from an ordered multidict, which can't be implemented with just an underlying dict. > PS: if this change to the class statement were to happen, the crowd > discussing the 'make' PEP (which I don't like but don't have the power > to stop being discussed) should take notice. Or perhaps they could run > with the idea and make it the focal point of their proposal. Or > whatever. :-) A custom dict (working like how you describe) was proposed for 'make', though I think it was kind of passed over because it was too big a departure from the class statement. If the class statement allowed for custom dicts -- or I suppose even if the idea got an approving nod -- then certainly the idea makes sense there. In the context of the 'make' statement, dicts beyond just ordered dicts have some potential utility (though also a strong clever hack potential); in the context of classes I don't really see use cases beyond ordering. Unless some more powerful class statement was used in lieu of the make statement, and all sorts of non-class-definitions were kludged into that syntax. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Tue Apr 18 18:59:41 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 18 Apr 2006 11:59:41 -0500 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) In-Reply-To: References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> <44443F95.7070707@canterbury.ac.nz> Message-ID: <44451AFD.2030205@colorstudy.com> Guido van Rossum wrote: >>>Inlining >>>code obviously addresses this, but that's often killing code >>>structure. >> >>Would automated inlining be a legitimate use for ast-hacking? > > > Perhaps, if you can do it in a way that it also works for IronPython > and Jython... Is it expected that IronPython and Jython will have similar performance characteristics to CPython? That seems hard to achieve. I would assume that the AST hacking would only effect performance (and maybe remove some potential dynamicism), so un-AST-hacked code would work identically. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Tue Apr 18 19:10:15 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 18 Apr 2006 12:10:15 -0500 Subject: [Python-3000] Specializing the dicts in __dict__ In-Reply-To: References: <20060417235854.GF5762@performancedrivers.com> Message-ID: <44451D77.4070603@colorstudy.com> Steven Bethard wrote: > Could someone put out some use-cases where it would be really helpful > to be able to specify a different dict type for the class dict? In SQLObject and FormEncode, you can do something like: class Registration(formencode.Schema): fname = String(not_empty=True) lname = String(not_empty=True) I'd like to know the order of fname and lname internally; in SQLObject I'd like to know the order of the columns. You can get away without knowing the order, but the result feels distinctly sloppy. Or else you get stuff like: class Registration(SomeClearlyInferiorSchemaClass): fields = [String('fname', not_empty=True), String('lname', not_empty=True)] In this case the classes and objects are all cooperative, and the problem can be resolved by tracking the order in which these String instances were created, though it only allows for the ordering of like objects. Another case is document generation through introspection; usually methods and instances were ordered in the source for some reason, but this ordering is lost. You can order the methods by looking at their line numbers (which is tracked), but you can't order attributes, you can't order descriptors, and you can't interleave attributes with methods. In the case of documentation, you can't expect the classes and objects to be cooperative, and you can't expect metaclasses to provide ordered dicts for their implementations (unless you were able to monkeypatch the entire system to use ordered dicts, which would be acceptable for documentation generation even if it slowed the system down). In the case of documentation, AST inspection might be a better way to go, but it does limit the accuracy of the documentation in other ways as nothing dynamic done at import time can really be understood without actually importing the code. So the better systems typically have to do a bit of both source and object inspection. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From aahz at pythoncraft.com Tue Apr 18 19:27:03 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 18 Apr 2006 10:27:03 -0700 Subject: [Python-3000] auto-super() In-Reply-To: <44451AA0.6090009@colorstudy.com> References: <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <20060418153028.GB21539@panix.com> <44451AA0.6090009@colorstudy.com> Message-ID: <20060418172703.GA8830@panix.com> On Tue, Apr 18, 2006, Ian Bicking wrote: > Aahz wrote: >>On Tue, Apr 18, 2006, Thomas Wouters wrote: >> >>>- Compiler hackery involving a magical variable name, say '__class__' >>>or '__CLASS__'. The compiler would treat this specially, probably >>>stored in the class dict, and type() (which is, after all, called to >>>actually create the class) would stuff the actual class object in >>>there. It causes a reference cycle, but all newstyle classes already >>>do that ;P The main issue is that __CLASS__ would be new magic. It >>>wouldn't exist when the class body is executed, and it would be a >>>special form of enclosed variable afterwards (it should be extracted >>>from the class namespace, using a similar mechanism as closures.) >> >>It's not clear to me that this requires compiler hackery, just metaclass >>hackery. Am I missing something? >> >>Also, my idea is that self.super is a bound method/closure that already >>contains a reference to the class. This makes dynamic classes more >>difficult in some ways, but anyone who wants to play those sorts of games >>should expect to do some magic mangling. If it's reasonably >>well-documented, it wouldn't even be that difficult. > > Well: > > class A(object): > def whoami(self): > return 'A' > def dostuff(self): > print 'I (%s) am doing A stuff...' % (self.whoami()) > > class B(A): > def whoami(self): > return 'B-ish %s' % (self.super.whoami()) > def dostuff(self): > print 'Some B stuff...' > self.super.dostuff() > > class C(B): > def whoami(self): > return 'C' > def dostuff(self): > print 'Some C stuff...' > self.super.dostuff() > > Reading this statically, I think it's clear how this should work. So, > when C().dostuff() is called, we want to see: > > Some C stuff... > Some B stuff... > I (C) am doing A stuff... > > When B().dostuff() is called, we want to see: > > Some B stuff... > I (B-ish A) am doing A stuff... > > But how can we do that without the function being bound to a class? > self.whoami() in A.dostuff needs to access the 'real' self. self.super > in B.dostuff should be super(B, self), even when self.__class__ is C. Hrm. I think I'm forgetting that Python doesn't have any dynamic scopes; that would make this reasonably straightforward (the original self.super() call would put something into the call stack that following calls to self.super() would retrieve and update). -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From jimjjewett at gmail.com Tue Apr 18 19:27:18 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 18 Apr 2006 13:27:18 -0400 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) In-Reply-To: <44443F95.7070707@canterbury.ac.nz> References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> <44443F95.7070707@canterbury.ac.nz> Message-ID: On 4/17/06, Greg Ewing wrote: > Guido van Rossum wrote: > > Inlining code obviously addresses this, but that's often > > killing code structure. > Would automated inlining be a legitimate use for > ast-hacking? To keep the same semantics, you need some notification that the (real version of the) inlined code just got modified. Which again calls for namespaces as slightly different from dictionaries. (Perahps with warnings even now if a module is modified after the initialization is "completed".) Making this change at strictly the python level is a bit of a pain, because modules and classes are evaluated in a standard dictionary (from PyDictNew), before calling the metaclass. I don't see any reason it would need to wait for python 3. -jJ From jcarlson at uci.edu Tue Apr 18 19:40:47 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 18 Apr 2006 10:40:47 -0700 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) In-Reply-To: References: <44443F95.7070707@canterbury.ac.nz> Message-ID: <20060418103614.A807.JCARLSON@uci.edu> "Jim Jewett" wrote: > > On 4/17/06, Greg Ewing wrote: > > Guido van Rossum wrote: > > > > Inlining code obviously addresses this, but that's often > > > killing code structure. > > > Would automated inlining be a legitimate use for > > ast-hacking? > > To keep the same semantics, you need some notification that the (real > version of the) inlined code just got modified. Not necessarily. I am of the opinion that the function inlining should be done explicitly via decorator (see http://bytecodehacks.sourceforge.net/bch-docs/bch/module-bytecodehacks.inline.html for a sample implementation). - Josiah From ianb at colorstudy.com Tue Apr 18 19:41:15 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Tue, 18 Apr 2006 12:41:15 -0500 Subject: [Python-3000] auto-super() In-Reply-To: <20060418172703.GA8830@panix.com> References: <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <20060418153028.GB21539@panix.com> <44451AA0.6090009@colorstudy.com> <20060418172703.GA8830@panix.com> Message-ID: <444524BB.5040502@colorstudy.com> Aahz wrote: >>But how can we do that without the function being bound to a class? >>self.whoami() in A.dostuff needs to access the 'real' self. self.super >>in B.dostuff should be super(B, self), even when self.__class__ is C. > > > Hrm. I think I'm forgetting that Python doesn't have any dynamic > scopes; that would make this reasonably straightforward (the original > self.super() call would put something into the call stack that following > calls to self.super() would retrieve and update). I thought about that too -- since you can inspect stacks to get something like dynamic scope, after all -- but super calls could also be interleaved, or even done recursively. The class context is really something like an argument to the function which is passed in selectively. Recursive calls seem the trickiest in this case (after trying a couple other things): class A(object): def sum(self, seq): if not seq: return 0 return 1 + seq[0] + self.sum(seq[1:]) class B(A): def sum(self, seq): return 2 * self.super.sum(seq) b = B() b.sum([1, 2]) That should call: B.sum(b, [1, 2]) A.sum(b, [1, 2]) B.sum(b, [2]) A.sum(b, [2]) B.sum(b, []) A.sum(b, []) Achieving that through stack inspection doesn't seem possible. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From kendall at monkeyfist.com Tue Apr 18 19:58:57 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 18 Apr 2006 13:58:57 -0400 Subject: [Python-3000] Test and Assign [was: More wishful thinking] In-Reply-To: <06Apr18.092438pdt."58633"@synergy1.parc.xerox.com> References: <06Apr18.092438pdt."58633"@synergy1.parc.xerox.com> Message-ID: <92C4722A-4E98-4DC5-8AA7-2372A8AE1C4B@monkeyfist.com> On Apr 18, 2006, at 12:24 PM, Bill Janssen wrote: >>> Presumably a reference to a Python symbol would be not just the >>> symbol >>> name string, but also an indicator of the namespace of the symbol. >> >> That would be something very new -- nothing like that was >> implied by the original suggestion, and no other language >> I know of that has symbols gives them any such powers. > > Really? Common Lisp symbols have the form "[:]", where > defaults to the local namespace (well, actually the default > is a bit more complicated). Exactly. I tried to explicitly distinguish between symbols in Python (as I suggested them) and symbols in Common Lisp, where they are significantly different. I see you (Bill) are -1 on them anyway, which is fine, but I wasn't suggesting CL-like symbols for Py3K, FWIW. > The original poster was suggesting > ":", which I suppose I read in its Lisp interpretation as a > symbol in the KEYWORD package. Yes, I think that's exactly right. (Ruby symbols are essentially CL keywords, as I understand things, and I was really suggesting Ruby- like symbols for Py3K.) Cheers, Kendall From guido at python.org Tue Apr 18 20:38:21 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 19:38:21 +0100 Subject: [Python-3000] genexp syntax for reduce (already in 2.5a1) In-Reply-To: <4444D639.6030108@batiment71.net> References: <4444D639.6030108@batiment71.net> Message-ID: I wouldn't recommend this approach to anyone. On 4/18/06, Boris Borcic wrote: > Hello, > > Since the notion of a "generator comprehension" reduce syntax for py3k was > discussed on this list - it may have escaped attention that py2.5a1 already > allows the following, admittedly not very efficient (nor too robust in this > proof-of-concept implementation). > > hth, Boris Borcic > > > def ireduce(gen) : > """ > Generator expression syntax for reduce and generalizations > > Usage examples: > > - (yield) nothing as below if there is no seed value > *and* the expression consists of a single arithmetic op > > >>> ireduce(x+(yield) for x in range(101)) # famous Gauss example > 5050 > >>> ireduce(x*(yield) for x in range(1,6)) > 120 > > - if there is a seed, yield it (unless it's None) > > >>> ireduce(x*(yield 2) for x in range(1,6)) # 2*1*2*3*4*5 > 240 > >>> ireduce(x*(yield 2) for x in range(2,6)) # 2*2*3*4*5 > 240 > >>> ireduce(x*(yield 2) for x in range(6)) # 2*0*1*2*3*4*5 > 0 > > - if the seed is None, (yield INone) instead > > >>> ireduce((x,(yield INone)) for x in "for all good men".split()) > ('men', ('good', ('all', ('for', None)))) > > >>> ireduce([(yield INone),x] for x in "for all good men".split()) > [[[[None, 'for'], 'all'], 'good'], 'men'] > > - do as below if you want no special seed and the operation to reduce > isn't a simple arithmetic operation > > >>> ireduce((x,(yield Seed(x))) for x in "for all good men".split()) > ('men', ('good', ('all', 'for'))) > > >>> ireduce(x+' '+(yield Seed(x)) for x in "for all good men".split()) > 'men good all for' > > >>> ireduce({ x : (yield Seed(x))} for x in "for all good men".split()) > {'men': {'good': {'all': 'for'}}} > > - and notice these for a comparison > > >>> ireduce({ x : (yield x)} for x in "for all good men".split()) > {'men': {'good': {'all': {'for': 'for'}}}} > > >>> ireduce({ x : (yield Seed(None))} for x in "for all good men".split()) > {'men': {'good': {'all': None}}} > """ > > cumulate = gen.next() > if cumulate is None : > cumulate = Neutral() > elif cumulate is INone : > cumulate = None > elif isinstance(cumulate,Seed) : > cumulate = cumulate.seed > gen.send(cumulate) > gen.next() > try : > while True : > cumulate = gen.send(cumulate) > gen.next() > except StopIteration : > return cumulate > > class Neutral : # this would need reworking > def __coerce__(self,other) : > self.other = other > return (self,self) > > def __getattr__(self,attr) : > return lambda *x,**y : self.__dict__['other'] > > class INone : pass > > class Seed : > def __init__(self,seed) : > self.seed = seed > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 18 20:43:53 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 19:43:53 +0100 Subject: [Python-3000] Specializing the dicts in __dict__ In-Reply-To: References: <20060417235854.GF5762@performancedrivers.com> Message-ID: On 4/18/06, Steven Bethard wrote: > On 4/18/06, Guido van Rossum wrote: > > PS: if this change to the class statement were to happen, the crowd > > discussing the 'make' PEP (which I don't like but don't have the power > > to stop being discussed) should take notice. > > If you pronounce it dead, I can make sure the discussion stops. :-) Let's kill it. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 18 20:50:15 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 19:50:15 +0100 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) In-Reply-To: <44451AFD.2030205@colorstudy.com> References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> <44443F95.7070707@canterbury.ac.nz> <44451AFD.2030205@colorstudy.com> Message-ID: On 4/18/06, Ian Bicking wrote: > Guido van Rossum wrote: > > Perhaps, if you can do it in a way that it also works for IronPython > > and Jython... > > Is it expected that IronPython and Jython will have similar performance > characteristics to CPython? That seems hard to achieve. I would assume > that the AST hacking would only effect performance (and maybe remove > some potential dynamicism), so un-AST-hacked code would work identically. Since it's hard to ensure that AST hacking doesn't change the semantics in any way I don't think you can guarantee that. You could perhaps guarantee that for some kind of inlining, but if you support AST-hacking for inlining that you automatically make it available for other code transformations. Anyway, this is rather wild py-in-the-sky -- I'd rather not spend time discussing it until there's a working implementation and a PEP. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Tue Apr 18 20:53:01 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 19:53:01 +0100 Subject: [Python-3000] auto-super() In-Reply-To: References: <443E7F47.9060308@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <44442CE2.6020801@canterbury.ac.nz> <444434DA.3080500@colorstudy.com> Message-ID: On 4/18/06, Michael Urman wrote: > On 4/18/06, Guido van Rossum wrote: > > Hardly the same mechanism; __private is done by low-level name > > mangling in the early stages of the compiler. super() would need a > > class object to be reliable; the class *name* (which is all that's > > available at compile time) might resolve to something else at runtime. > > That's all the current super() gets, right? super(Base, self).meth() > compiles with a LOAD_GLOBAL (Base)... Any mechanisms I can think of to > solidify that into the actual class reference either rely on > __mangled_attrs being assigned at or after class construction, or > potentially break in inherited classes. Sure. But that's all for anyone to see in the code. OTOH, if super were implicit, it would be fair to expect it to do the right thing w.r.t. the current class object. IOW an implicit super is held to a higher standard than the current explicit super. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Tue Apr 18 20:58:42 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 18 Apr 2006 14:58:42 -0400 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On 4/14/06, Guido van Rossum wrote: > Then, on 4/14/06, Nick Coghlan wrote: > [...] > class C: > x = property(._get_x, ._set_x, ._del_x, > "This is the x property") > [...] > This is slightly more Pythonic, and unambiguous, but I'd rather > reserve leading dot for a more powerful feature, perhaps .foo as an > abbreviation for self.foo, or some other scope-ish operator. (I also > note that in a proportional font the . is easily missed.) Does that conflict? Wouldn't self._get_x fall through to self.__class__._get_x, which is exactly what you want? (Except that you *might* the ability to override methods on a per-instance basis.) > So this begs the question: would the following assertion pass or fail? > > assert .foo == "foo" > > What about this one? > > assert type(.foo) == str If .foo is a syntactic change, it might not require that an object be created (for the name) at all. That said, if an object is created, then I believe assert .foo is "foo" should be true, because foo is constant string that looks like a name, and should therefore be interned. -jJ From guido at python.org Tue Apr 18 21:13:01 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 20:13:01 +0100 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On 4/18/06, Jim Jewett wrote: > On 4/14/06, Guido van Rossum wrote: > > Then, on 4/14/06, Nick Coghlan wrote: > > [...] > > class C: > > x = property(._get_x, ._set_x, ._del_x, > > "This is the x property") > > [...] > > > This is slightly more Pythonic, and unambiguous, but I'd rather > > reserve leading dot for a more powerful feature, perhaps .foo as an > > abbreviation for self.foo, or some other scope-ish operator. (I also > > note that in a proportional font the . is easily missed.) > > Does that conflict? Wouldn't self._get_x fall through to > self.__class__._get_x, which is exactly what you want? (Except that > you *might* the ability to override methods on a per-instance basis.) Not in this context. When the expression "property(._get_x, ._set_x, ...)" is evaluated, there is no self yet (unless it is an unrelated outer self). > > So this begs the question: would the following assertion pass or fail? > > > > assert .foo == "foo" > > > > What about this one? > > > > assert type(.foo) == str > > If .foo is a syntactic change, it might not require that an object be > created (for the name) at all. We may be misunderstanding each other, but I don't see how, if ".foo" is used as an expression, it can not return some object. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Tue Apr 18 23:52:47 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 18 Apr 2006 17:52:47 -0400 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> Message-ID: On 4/17/06, Talin wrote: > def func( first, second=default, third ): > ... what's the point of supplying a default for the second > parameter, but not the third? You may be wrapping an external library. (Or at least a well-known API.) I'll grant that the second argument probably ought to have been third, but with an existing library, it often isn't. The real choices are def func(first, second, third): # Everyone supplies the junk second on every call def func(first, second=default, third=sentinel): if third is sentinel: raise ValueError("Need a real 'third' key") def func_newname(first, third): none of which are very satisfying. That said, I'm not sure the use case is common enough to justify the confusion. -jJ From jimjjewett at gmail.com Wed Apr 19 00:16:44 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 18 Apr 2006 18:16:44 -0400 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On 4/18/06, Guido van Rossum wrote: > On 4/18/06, Jim Jewett wrote: > > On 4/14/06, Guido van Rossum wrote: > > > Then, on 4/14/06, Nick Coghlan wrote: > > > [...] > > > class C: > > > x = property(._get_x, ._set_x, ._del_x, > > > "This is the x property") > > > [...] > > > > > This is slightly more Pythonic, and unambiguous, but I'd rather > > > reserve leading dot for a more powerful feature, perhaps .foo as an > > > abbreviation for self.foo, or some other scope-ish operator. (I also > > > note that in a proportional font the . is easily missed.) > > Does that conflict? Wouldn't self._get_x fall through to > > self.__class__._get_x, which is exactly what you want? (Except that > > you *might* the ability to override methods on a per-instance basis.) > Not in this context. When the expression "property(._get_x, ._set_x, > ...)" is evaluated, there is no self yet (unless it is an unrelated > outer self). Ah ... you were thinking of ._get_x as an expression which would be evaluated at compile-time to produce an actual method. I was still thinking of symbols, and reading ".name" as syntactic sugar for "self.name", with the caveat that the translation was made at runtime rather that compile-time. Therefore property(._get_x, ._set_x) would be syntactic sugar for def __getx(self): return self._get_x() def __setx(self): return self._set_x() property(__getx, __setx) except that the __getx and __setx methods wouldn't be added to the visible namespace, even in mangled form. -jJ From tdelaney at avaya.com Wed Apr 19 00:39:56 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Wed, 19 Apr 2006 08:39:56 +1000 Subject: [Python-3000] auto-super() Message-ID: <2773CAC687FD5F4689F526998C7E4E5F07436D@au3010avexu1.global.avaya.com> Ian Bicking wrote: > But how can we do that without the function being bound to a class? > self.whoami() in A.dostuff needs to access the 'real' self. > self.super in B.dostuff should be super(B, self), even when > self.__class__ is C. > > Because Python *isn't* reading the functions statically to determine > class and method layout, actually making this work seems very hard. Believe me, it is. Have a look at my `self.super` recipe at: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/286195 and the considerably faster Pyrex version at: http://members.optusnet.com.au/tcdelaney/python.html#autosuper I'm sure there are some edge cases which are not covered by the implementation, but it gets the vast majority of them. And I haven't tested it with Jython or IronPython. It works using sys._getframe() (to get the code object and method name), then walks the MRO to find which class has a method with the same name and code object (there's a bit more to it to prevent recursion). `self.super` is also callable - if you call `self.super(params)` it invokes the method on the superclass with the same name as the current method. Tim Delaney From guido at python.org Wed Apr 19 00:51:00 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 23:51:00 +0100 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On 4/18/06, Jim Jewett wrote: > I was still thinking of symbols, and reading ".name" as syntactic > sugar for "self.name", with the caveat that the translation was made > at runtime rather that compile-time. > > Therefore > > property(._get_x, ._set_x) > > would be syntactic sugar for > > def __getx(self): return self._get_x() > def __setx(self): return self._set_x() > property(__getx, __setx) > > except that the __getx and __setx methods wouldn't be added to the > visible namespace, even in mangled form. Please stop. This is getting ridiculous. Do you realize that you are describing that the expression .foo is syntactic sugar for lambda self: self._foo() ? That's not what I would call a "symbol type" by any stretch of the imagination. (And if you're claiming that .foo is not "an expression" then how can it appear syntactically in the position of a parameter?) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 00:52:53 2006 From: guido at python.org (Guido van Rossum) Date: Tue, 18 Apr 2006 23:52:53 +0100 Subject: [Python-3000] auto-super() In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5F07436D@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5F07436D@au3010avexu1.global.avaya.com> Message-ID: Let's leave this topic behind for a while. Hacks that can be implemented only using sys._getframe() are not generally suitable for including in the standard library or making part of the language spec. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Wed Apr 19 01:03:56 2006 From: tdelaney at avaya.com (Delaney, Timothy (Tim)) Date: Wed, 19 Apr 2006 09:03:56 +1000 Subject: [Python-3000] auto-super() Message-ID: <2773CAC687FD5F4689F526998C7E4E5F07436E@au3010avexu1.global.avaya.com> Guido van Rossum wrote: > Let's leave this topic behind for a while. Hacks that can be > implemented only using sys._getframe() are not generally suitable for > including in the standard library or making part of the language spec. Yep - there's no way I'd consider such a hack for the python distribution. I was just interested that the conversation appeared to be converging on the recipe (more proof-of-concept) I created quite some time ago :) I'd really like to work out a way to do it without sys._getframe() hacks. Apart from anything else, it really needs to perform at similar speed to super(cls, self). Perhaps when a code object is constructed it could include a reference to the class where it was constructed (if any). Then the class to use in `super` would be obtained directly. I originally wanted to use im_class rather than walk the MRO, but im_class didn't give the correct result, and in any case I'd still need sys._getframe(). This approach would of course not work properly if the same method was then added to another class (a flaw shared by my recipe IIRC). One way to deal with that would be to check that the class was actually in the MRO of the current instance, and throw a TypeError if it's not. Tim Delaney From guido at python.org Wed Apr 19 01:50:51 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 00:50:51 +0100 Subject: [Python-3000] auto-super() In-Reply-To: <2773CAC687FD5F4689F526998C7E4E5F07436E@au3010avexu1.global.avaya.com> References: <2773CAC687FD5F4689F526998C7E4E5F07436E@au3010avexu1.global.avaya.com> Message-ID: On 4/19/06, Delaney, Timothy (Tim) wrote: > Perhaps when a code object is constructed it could include a reference > to the class where it was constructed (if any). No, code objects shouldn't contain references to run-time artifacts (other than other code objects). Code objects must be marshallable. It's the function objects created from code objects that reference other run-time state (like globals, default values, and closure cells). > Then the class to use in > `super` would be obtained directly. I originally wanted to use im_class > rather than walk the MRO, but im_class didn't give the correct result, > and in any case I'd still need sys._getframe(). Perhaps the best way to implement super might be to create it as a "cell". Functions already have a mechanism to access cells that are passed in by the environment when the function objects is created. Perhaps 'super' could use a cell with a magic name to access the class. > This approach would of course not work properly if the same method was > then added to another class (a flaw shared by my recipe IIRC). One way > to deal with that would be to check that the class was actually in the > MRO of the current instance, and throw a TypeError if it's not. Methods that are shared between classes shouldn't reference super in either form. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Wed Apr 19 02:15:02 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 18 Apr 2006 17:15:02 -0700 Subject: [Python-3000] auto-super() In-Reply-To: References: <2773CAC687FD5F4689F526998C7E4E5F07436D@au3010avexu1.global.avaya.com> Message-ID: <20060418170742.A814.JCARLSON@uci.edu> "Guido van Rossum" wrote: > Let's leave this topic behind for a while. Hacks that can be > implemented only using sys._getframe() are not generally suitable for > including in the standard library or making part of the language spec. Assuming name uniqueness among all classes in the mro, assuming the mro is available in other implementations, assuming name mangling for __attributes are consistant, and assuming that one checks for attributes on supers before calling, one can do it without frame hacking [1], though I don't know how it compares (in terms of speed) to super(cls, self) . - Josiah [1] >>> class Super(object): ... def __getattr__(self, key): ... if key.endswith('__superclass'): ... name = key[1:-12] ... for cls in self.__class__.__mro__: ... if cls.__name__ == name: ... return cls ... raise AttributeError, "%s instance has no attribute %s"%( ... self.__class__.__name__, key) ... >>> class foo(Super): ... def go(self): ... print "in foo" ... super(self.__superclass, self).go() ... >>> class bar(foo): ... def go(self): ... print "in bar" ... super(self.__superclass, self).go() ... >>> bar().go() in bar in foo Traceback (most recent call last): File "", line 1, in ? File "", line 4, in go File "", line 4, in go AttributeError: 'super' object has no attribute 'go'>>> class Super(object): ... def __getattr__(self, key): ... if key.endswith('__superclass'): ... name = key[1:-12] ... for cls in self.__class__.__mro__: ... if cls.__name__ == name: ... return cls ... raise AttributeError, "%s instance has no attribute %s"%( ... self.__class__.__name__, key) ... >>> class foo(Super): ... def go(self): ... print "in foo" ... super(self.__superclass, self).go() ... >>> class bar(foo): ... def go(self): ... print "in bar" ... super(self.__superclass, self).go() ... >>> bar().go() in bar in foo Traceback (most recent call last): File "", line 1, in ? File "", line 4, in go File "", line 4, in go AttributeError: 'super' object has no attribute 'go' >>> From jcarlson at uci.edu Wed Apr 19 02:32:18 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 18 Apr 2006 17:32:18 -0700 Subject: [Python-3000] auto-super() In-Reply-To: <20060418170742.A814.JCARLSON@uci.edu> References: <20060418170742.A814.JCARLSON@uci.edu> Message-ID: <20060418173142.A817.JCARLSON@uci.edu> [snip] Sorry about the double-pasting in the last post. - Josiah From greg.ewing at canterbury.ac.nz Wed Apr 19 02:43:35 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 12:43:35 +1200 Subject: [Python-3000] Removing 'self' from method definitions In-Reply-To: References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <44442C3C.2090908@canterbury.ac.nz> Message-ID: <444587B7.10400@canterbury.ac.nz> Guido van Rossum wrote: > - How would the function be given the attribute? > At the time the > function is being defined the class doesn't exist yet; Good point. I'll have to think about that some more. BTW, I'm not particularly advocating the idea -- I was really just pointing out that there are difficulties with auto-superclass-determination the way things currently work. I'm happy to continue explicitly naming my superclasses for the time being. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 02:59:38 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 12:59:38 +1200 Subject: [Python-3000] Type Expressions In-Reply-To: References: Message-ID: <44458B7A.9060208@canterbury.ac.nz> Birch, Bill wrote: > [int, etc] # a list of ints > # (etc = zero or more of the previous element) The obvious way to express that would be [int, ...] or at least it would be if ... weren't restricted to slices! Py3k suggestion: make ... a generally legal expression. -- Greg From talin at acm.org Wed Apr 19 03:09:58 2006 From: talin at acm.org (Talin) Date: Wed, 19 Apr 2006 01:09:58 +0000 (UTC) Subject: [Python-3000] =?utf-8?q?Function_call_speed_=28Was=3A_Cleaning_up?= =?utf-8?q?_argument=09listparsing=29?= References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> Message-ID: I wanted to follow up a bit on the issue of speeding up function calls with some more specifics. Currently, the function call mechanism has to do a lot of conversion operations on the function arguments. Specifically, it takes the various positional arguments, keyword arguments, varargs argument, and kwargs, and combines them into a single tuple and a single dict. It then calls the target function via PyObject_Call. On the recieving side, a different conversion takes place, depending on the type of the callable. For user-defined functions, it maps the various arguments in the tuple and dict into the function's parameter slots, along with considerations of default values and such. One way to speed this up would be to determine cases where the conversion steps can be skipped. There is already a method for this in the case where there are only positional arguments, no varargs or keywords. Ideally, we would like to be able to pass the caller's argument stack pretty much as-is to the reciever. At the moment, that consists of an array of positional arguments, followed by an array of key/value pairs for the keyword arguments, followed by an optional vararg or kwarg. Assume for a moment, then, that we had a variation of PyObject_Call which had a function signature that was exactly that: PyObject_CallFast( func, PyObject **args, int numArgs, PyObject **kargs, int numKwArgs, PyObject *starargs, PyObject *kwarg ); The reciever would have to do extra work, because now it has to check two places for each argument. For a keyword argument, for example, it has to check both the key/value array and the kwargs dict argument. However, this is far less work than building a new dictionary. In addition, we know that the keywords are interned strings, not arbitrary objects, so we can speed up the comparison operation by avoiding type tests. Also, because the keyword args are in a linear list, they will exhibit better cache behavior, and the lookup will be very fast - a simple loop and test. (If you want to write optimal code for today's processors, you pretty much have to throw out everything you learned about unrolling loops and such, and learn to think in terms of cache lines, not bytes and words.) Now, it may be that in some cases, the reciever really does require a tuple and a dict. However, in such a case, the onus should be on the reciever to convert the argument list into the preferred form. So in other words, it would be the reciever's job to convert the arguments into whatever data structure the reciever required. For recievers with relatively simple calling signatures, it may mean no conversion is done at all, and the caller's argument list can be used directly. The most complex part would be validating the arguments - making sure that a keyword argument didn't get assigned more than once, and so on. What about older code? Well, I suppose you could add an additional field to the recieving object, indicating whether it accepted the traditional PyObject_Call-style args, or the newer format. Essentially you would have two function pointers in the object, one for newstyle calling, and one for oldstyle. If the newstyle was present, then the interpreter would use it, otherwise it would fall back to the older means of transforming the arguments into a tuple and a dict. In any case, the reason why this is "Py3K" material is that it involves busting the existing C API to some extent. -- Talin From edcjones at comcast.net Wed Apr 19 03:17:27 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Tue, 18 Apr 2006 21:17:27 -0400 Subject: [Python-3000] Is reference counting still needed? Message-ID: <44458FA7.10605@comcast.net> Wild Idea: From Section 1.10 of "Extending and Embedding": Maybe some day a sufficiently portable automatic garbage collector will be available for C. Until then, we'll have to live with reference counts. Any hope such a GC will be available before Python 3.0? From edcjones at comcast.net Wed Apr 19 03:20:44 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Tue, 18 Apr 2006 21:20:44 -0400 Subject: [Python-3000] Use Py_CLEAR only Message-ID: <4445906C.7060006@comcast.net> Wild idea: Can Py_CLEAR completely replace Py_DECREF and Py_XDECREF? From greg.ewing at canterbury.ac.nz Wed Apr 19 03:26:31 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 13:26:31 +1200 Subject: [Python-3000] Function call speed (Was: Cleaning up argument list parsing) In-Reply-To: References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> <44443F95.7070707@canterbury.ac.nz> Message-ID: <444591C7.8080802@canterbury.ac.nz> Guido van Rossum wrote: > On 4/18/06, Greg Ewing wrote: >>Would automated inlining be a legitimate use for ast-hacking? > > Perhaps, if you can do it in a way that it also works for IronPython > and Jython... Would automatically falling back on non-inlining be acceptable if suitable ast support was not available? > Though anything for which you can't come up with a better word than > "hacking" probably oughtn't be in the language spec. :-) Hmmm... all right, "automatic transparent compile-time code rewriting", then. Or "throatwarbling" for short. -- Greg From jcarlson at uci.edu Wed Apr 19 03:36:53 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 18 Apr 2006 18:36:53 -0700 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <44458FA7.10605@comcast.net> References: <44458FA7.10605@comcast.net> Message-ID: <20060418183236.A81A.JCARLSON@uci.edu> "Edward C. Jones" wrote: > > Wild Idea: > > From Section 1.10 of "Extending and Embedding": > > Maybe some day a sufficiently portable automatic garbage collector will > be available for C. Until then, we'll have to live with reference counts. > > Any hope such a GC will be available before Python 3.0? Short answer: not likely. Longer answer: Even if py3k is really Python 3.0, there is the standard 1.5-2 years between 2.x releases, and 3.0 comes 1.5-2 years after 2.9, I would bet modest sums of money against Python 3.0 having non-reference counting GC. Why? Reference counting has worked reasonably well so far (quite well if you ask me). There have been numerous discussions over the years about getting a "real" garbage collector into Python, google for it. - Josiah From edcjones at comcast.net Wed Apr 19 03:41:20 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Tue, 18 Apr 2006 21:41:20 -0400 Subject: [Python-3000] Special object to return from functions that return only None Message-ID: <44459540.1040709@comcast.net> Wild idea. When I was a newbie, I repeatedly make the mistake of writing alist = alist.sort() I suggest a singleton object "UseForbidden" ("Py_UseForbidden" in C). "UseForbidden" can be used only to return from a function. Any other attempt to use "UseForbidden" raises an exception. The incorrect code above would give the error message "cannot assign to result of function". From edcjones at comcast.net Wed Apr 19 03:50:38 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Tue, 18 Apr 2006 21:50:38 -0400 Subject: [Python-3000] After 2.6 should be 3.0 Message-ID: <4445976E.1070806@comcast.net> Wild idea. I would like to see Python 3.0 before the turn of the next millennium. I suggest that Python 2.6 be devoted to implementing some of the backward compatible changes for Python 3.0. In particular, do the long-overdue reorganization of the libraries. After 2.6 should be 3.0. From edcjones at comcast.net Wed Apr 19 04:25:26 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Tue, 18 Apr 2006 22:25:26 -0400 Subject: [Python-3000] A wish: build C, C++, FORTRAN library wrapping into Python. Message-ID: <44459F96.6040507@comcast.net> Wild wish. Many C, C++ and FORTRAN libraries have been wrapped in Python. These wrapped libraries are used a lot. A number of programs have been written that automate this process, the standard example being SWIG. These programs have become fairly mature. How much of the wrapping process can be built into Python? What changes in Python or added capabilities would make the wrapping process easier? What would Beasley, Abrahams, Ewing, et al want to see done? From andy at neotitans.com Tue Apr 18 22:21:19 2006 From: andy at neotitans.com (Andy Sy) Date: Wed, 19 Apr 2006 04:21:19 +0800 Subject: [Python-3000] mechanism for handling asynchronous concurrency Message-ID: <44454A3F.7070705@neotitans.com> The Io language has what (on paper) seems to be a very neat, Pythonic mechanism for doing asynchronous concurrency. The mechanism is called futures and is described at: http://iolanguage.com/darcs/Io/_docs/guide.html#TOC17133 Perhaps Python 3000 can incorporate some similar ideas in its syntax or standard library? From greg.ewing at canterbury.ac.nz Wed Apr 19 04:22:45 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 14:22:45 +1200 Subject: [Python-3000] auto-super() In-Reply-To: <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> Message-ID: <44459EF5.4070604@canterbury.ac.nz> Thomas Wouters wrote: > The arguments against super() are > understandable -- if you only think about your own class, or completely > disregard MI. I think that's irresponsible: super() always calls the > right baseclass method, On the contrary, it's precisely when you *don't* know what classes are going be mixed in with you that super() is likely to go haywire. Your super() call ends up calling a method that you know nothing about, and the method being called isn't expecting to be called in that context either. Whether that will do the "right" thing, or whether there even is a right thing to be done, is anybody's guess. What causes trouble isn't multiple inheritance, it's diamond inheritance. Without diamond inheritance, explicit calls to inherited methods usually work fine, and continue to work fine when your classes get inherited, multiply or otherwise, by other people's classes, as long as they do something sensible with their inherited calls. Personally I think that the use of diamond inheritance should be severely discouraged, if not banned completely. There's a fundamental problem with it: different subclasses can have different ideas on how the diamond- inherited class should be initialised, and otherwise used. That seems to me an inherent flaw in the notion of diamond inheritance, and not something that super() or anything else can fix. Whenever I've tried to use diamond inheritance, I've always ended up regretting it, and re-designing my class hierarchy to eliminate it. Once I've done that, any reason I had to use super() invariably disappears. To put it another way, it seems to me that super() is only there to try to work around problems introduced by diamond inheritance, which IMO you shouldn't be trying to use in the first place. That's why I regard super() as a misfeature. -- Greg From jcarlson at uci.edu Wed Apr 19 05:11:19 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 18 Apr 2006 20:11:19 -0700 Subject: [Python-3000] After 2.6 should be 3.0 In-Reply-To: <4445976E.1070806@comcast.net> References: <4445976E.1070806@comcast.net> Message-ID: <20060418200406.A81D.JCARLSON@uci.edu> "Edward C. Jones" wrote: > Wild idea. > > I would like to see Python 3.0 before the turn of the next millennium. I > suggest that Python 2.6 be devoted to implementing some of the backward > compatible changes for Python 3.0. In particular, do the long-overdue > reorganization of the libraries. After 2.6 should be 3.0. -1. Python 3k (non-alpha, beta, etc.) should come out when it's ready. No sooner. Personally, I see Py3k as a vetting mechanism for all those hair-brained ideas that really shouldn't make it into any Python version (3k or otherwise), with the possible inclusion of things that *should* make life better for Python users. With that said, aside from the stdlib reorganization (which should happen in the 2.x series anyways), so far I haven't seen anything that necessitates backwards incompatible changes to Python 3k, and I predict that many of the changes to py3k will be included into mainline 2.x during 3k's development. - Josiah From tjreedy at udel.edu Wed Apr 19 05:08:45 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Apr 2006 23:08:45 -0400 Subject: [Python-3000] Python 3000 and the Google Summer of Code References: <1145364116.12589.24.camel@localhost.localdomain> Message-ID: "Jan Claeys" wrote in message news:1145364116.12589.24.camel at localhost.localdomain... > Well, I know the Django (web framework) people are maybe interested in > someone doing a SoC project for them: > So interested that they have signed up as a sponsoring org themselves ;-) tjr From jcarlson at uci.edu Wed Apr 19 05:15:42 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 18 Apr 2006 20:15:42 -0700 Subject: [Python-3000] mechanism for handling asynchronous concurrency In-Reply-To: <44454A3F.7070705@neotitans.com> References: <44454A3F.7070705@neotitans.com> Message-ID: <20060418201344.A820.JCARLSON@uci.edu> Andy Sy wrote: > > The Io language has what (on paper) seems to be a very > neat, Pythonic mechanism for doing asynchronous concurrency. > > > The mechanism is called futures and is described at: > > http://iolanguage.com/darcs/Io/_docs/guide.html#TOC17133 > > > Perhaps Python 3000 can incorporate some similar ideas > in its syntax or standard library? This has been the case since Python 2.2 (if I remember correctly), see generators. Python 2.5 adds a mechanism to pass information back into generators (gen.send()). - Josiah From aahz at pythoncraft.com Wed Apr 19 05:24:41 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 18 Apr 2006 20:24:41 -0700 Subject: [Python-3000] auto-super() In-Reply-To: <44459EF5.4070604@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> Message-ID: <20060419032441.GA27988@panix.com> On Wed, Apr 19, 2006, Greg Ewing wrote: > > What causes trouble isn't multiple inheritance, it's diamond > inheritance. Without diamond inheritance, explicit calls to inherited > methods usually work fine, and continue to work fine when your classes > get inherited, multiply or otherwise, by other people's classes, as > long as they do something sensible with their inherited calls. > > Personally I think that the use of diamond inheritance should be > severely discouraged, if not banned completely. But multiple inheritance with new-style classes is inherently diamond inheritance -- how do you propose to deal with that? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From tjreedy at udel.edu Wed Apr 19 05:27:01 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Apr 2006 23:27:01 -0400 Subject: [Python-3000] Python 3000 and the Google Summer of Code References: Message-ID: "Neal Norwitz" wrote in message news:ee2a432c0604172222t4522f1f6te3f2b48362d6fde3 at mail.gmail.com... > Here's a write up about last year's projects: > http://wiki.python.org/moin/SummerOfCode/2005 > > Payment was all or nothing. I thought we had something like 18 > projects, there are 14 up on the page above. I now count 18 or 19 after recent additions. At least 13 seem to have been successful. A few don't specify. tjr From tjreedy at udel.edu Wed Apr 19 05:32:45 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 18 Apr 2006 23:32:45 -0400 Subject: [Python-3000] Python 3000 and the Google Summer of Code References: Message-ID: [This discussion could just as well go to py-dev, but it is here, this is here;-] Draft of announcement to send to various Python related lists. -------- Google 2006 Summer of Code and Python Software Foundation This spring and summer, Google corporation will again provide several hundred stipends for students (18+, undergraduate thru PhD programs) to write new open-source code. The main page is http://code.google.com/summerofcode.html. At the bottom are links to StudentFAQ, MentorFAQ, and TermsOfService. The first two have the timeline. Note that student applications are due between May 1, 17:00 PST through May 8, same time. The Python Software Foundation (PSF) will again act as a sponsoring organization matching mentors and projects benefiting Python and Python programmers. Projects can include work on Python implementations, programmer utilitilies, libraries, packages, or frameworks. Project ideas are welcome and can be added to http://wiki.python.org/moin/SummerOfCode, possibly as links to fuller explanations elsewhere. The projects from last summer are described at http://wiki.python.org/moin/SummerOfCode/2005 People interested in mentoring a student though PSF are encouraged to contact Neal Norwitz at __________________. People unknown to Neal or Guido should try to find a couple of people who are to vouch for them. --------- Terry Jan Reedy From tjreedy at udel.edu Wed Apr 19 06:17:57 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Apr 2006 00:17:57 -0400 Subject: [Python-3000] After 2.6 should be 3.0 References: <4445976E.1070806@comcast.net> <20060418200406.A81D.JCARLSON@uci.edu> Message-ID: "Josiah Carlson" wrote in message news:20060418200406.A81D.JCARLSON at uci.edu... > Personally, I see Py3k as a vetting mechanism for all those hair-brained > ideas that really shouldn't make it into any Python version (3k or > otherwise), with the possible inclusion of things that *should* make > life better for Python users. With that said, aside from the stdlib > reorganization (which should happen in the 2.x series anyways), so far I > haven't seen anything that necessitates backwards incompatible changes > to Python 3k, and I predict that many of the changes to py3k will be > included into mainline 2.x during 3k's development. The somewhat backwards-incompatible integer division change discussed and approved some years ago has been suspended pending 3.0, so since I would like to not have to (indefinitely) import 'division' from __future__ to get the change, I too, on that score, would like to see 3.0 sooner than later. Terry Jan Reedy From nnorwitz at gmail.com Wed Apr 19 06:29:23 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Tue, 18 Apr 2006 21:29:23 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: Guido, are you willing to send this? Would you rather me, Terry, or someone else send it? My changes are inline. They aren't extensive. On 4/18/06, Terry Reedy wrote: > between May 1, 17:00 PST through May 8, 17:00 PST. Might as well be explicit. :-) > The Python Software Foundation (PSF) will again act as a sponsoring If you send html, you should include a link to the PSF, http://www.python.org/psf/ > organization matching mentors and projects benefiting Python and Python > programmers. Projects can include work on Python implementations, > programmer utilitilies, libraries, packages, or frameworks. Project ideas ... or frameworks related to Python > People interested in mentoring a student though PSF are encouraged to > contact Neal Norwitz at nnorwitz at gmail.com. People unknown to Neal or > Guido should find a couple of people known within the Python community that are willing to act as references. Complete version with edits: Google 2006 Summer of Code and Python Software Foundation This spring and summer, Google corporation will again provide several hundred stipends for students (18+, undergraduate thru PhD programs) to write new open-source code. The main page is http://code.google.com/summerofcode.html. At the bottom are links to StudentFAQ, MentorFAQ, and TermsOfService. The first two have the timeline. Note that student applications are due between May 1, 17:00 PST through May 8, 17:00 PST. The Python Software Foundation (PSF) will again act as a sponsoring organization matching mentors and projects benefiting Python and Python programmers. Projects can include work on Python implementations, programmer utilitilies, libraries, packages, or frameworks. Project ideas are welcome and can be added to http://wiki.python.org/moin/SummerOfCode, possibly as links to fuller explanations elsewhere. The projects from last summer are described at http://wiki.python.org/moin/SummerOfCode/2005 People interested in mentoring a student though PSF are encouraged to contact Neal Norwitz at nnorwitz at gmail.com. People unknown to Neal or Guido should find a couple of people known within the Python community that are willing to act as references. From andy at neotitans.com Wed Apr 19 07:41:40 2006 From: andy at neotitans.com (Andy Sy) Date: Wed, 19 Apr 2006 13:41:40 +0800 Subject: [Python-3000] Futures in Python 3000 (was Re: mechanism for handling asynchronous concurrency) In-Reply-To: <20060418201344.A820.JCARLSON@uci.edu> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> Message-ID: <4445CD94.1000409@neotitans.com> Josiah Carlson wrote: > Andy Sy wrote: >> The Io language has what (on paper) seems to be a very >> neat, Pythonic mechanism for doing asynchronous concurrency. >> >> >> The mechanism is called futures and is described at: >> >> http://iolanguage.com/darcs/Io/_docs/guide.html#TOC17133 >> >> >> Perhaps Python 3000 can incorporate some similar ideas >> in its syntax or standard library? > > This has been the case since Python 2.2 (if I remember correctly), see > generators. Python 2.5 adds a mechanism to pass information back into > generators (gen.send()). > > - Josiah I am quite aware of what generators are, thank you. Io's 'Futures' syntax seems to be a far simpler and more general way to express general asynchronous concepts (like waiting for I/O to complete - try dealing with that using generators...) whereas generator syntax is more suited to the specific case of lazy iteration. You will still need to do an implementation of coroutines in terms of generators and not have a natural syntax for it. I don't know about you, but with the addition of send() to deal with the problem outlined in PEP 342, generators are starting to look more and more like a Rube Goldberg contraption. Could anything be more Pythonic than Io's: f := url @fetch // f is a future that will eventually hold the // URL's contents, but you don't block on the fetch. A futures abtraction and corresponding Python syntax might offer a clean start. It sounds like futures are very closely related to what Stackless is trying to achieve but does so in a far more lucid way. I found some additional info regarding Futures below: http://www.ps.uni-sb.de/alice/manual/futures.html http://kasparov.skife.org/blog-live/src/futures.writeback http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/Future.html Apparently, this abstraction is not limited to Io alone. From greg.ewing at canterbury.ac.nz Wed Apr 19 07:21:00 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 17:21:00 +1200 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <44458FA7.10605@comcast.net> References: <44458FA7.10605@comcast.net> Message-ID: <4445C8BC.5010202@canterbury.ac.nz> Edward C. Jones wrote: > Maybe some day a sufficiently portable automatic garbage collector will > be available for C. Until then, we'll have to live with reference counts. Even if it did become available, we might not want to use it. In recent times I've come round to the view that, on modern architectures where cacheing is all-important, refcounting + cyclic garbage collection may well be *better* than mark-and-sweep or some variation thereof. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 07:24:59 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 17:24:59 +1200 Subject: [Python-3000] Special object to return from functions that return only None In-Reply-To: <44459540.1040709@comcast.net> References: <44459540.1040709@comcast.net> Message-ID: <4445C9AB.5090004@canterbury.ac.nz> Edward C. Jones wrote: > I suggest a singleton object "UseForbidden" ("Py_UseForbidden" in C). It should be called "Mu". :-) Although such an object could cause a lot of pain to debuggers and other such introspective code. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 07:59:48 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 17:59:48 +1200 Subject: [Python-3000] Use Py_CLEAR only In-Reply-To: <4445906C.7060006@comcast.net> References: <4445906C.7060006@comcast.net> Message-ID: <4445D1D4.8030503@canterbury.ac.nz> Edward C. Jones wrote: > Can Py_CLEAR completely replace Py_DECREF and Py_XDECREF? It would be less efficient in the cases where you don't need to check for NULL and/or clear the reference afterwards. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 08:11:04 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 18:11:04 +1200 Subject: [Python-3000] A wish: build C, C++, FORTRAN library wrapping into Python. In-Reply-To: <44459F96.6040507@comcast.net> References: <44459F96.6040507@comcast.net> Message-ID: <4445D478.1020204@canterbury.ac.nz> Edward C. Jones wrote: > How much of the wrapping process can > be built into Python? What would Beasley, Abrahams, Ewing, > et al want to see done? Well, my idea of what a Pythonic Python-wrapping system should be is Pyrex, and I don't think I'd want too much of that incorporated into Python itself. Better to leave it as a separate tool. Maybe one day Pyrex could make use of some of Python's parsing and AST machinery -- or maybe not. Some of the tricks Pyrex does with its recursive descent parser would be difficult to reproduce with the likes of pgen. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 08:18:37 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 18:18:37 +1200 Subject: [Python-3000] auto-super() In-Reply-To: <20060419032441.GA27988@panix.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> <20060419032441.GA27988@panix.com> Message-ID: <4445D63D.3030800@canterbury.ac.nz> Aahz wrote: > But multiple inheritance with new-style classes is inherently diamond > inheritance -- how do you propose to deal with that? Yes, that's a bit of a problem. Or rather, it's not really a problem, because object doesn't provide any methods of its own, so having it at the top of every class hierarchy doesn't cause any problems in practice. But any mechanism that tried to enforce a ban on diamond inheritance would need to treat it as a special case. I'm not really advocating an enforced ban, because there are cases where it can be made to work with sufficient care. But I do think it should be regarded as a do-it- at-your-own-risk kind of thing, and not to be generally encouraged. -- Greg From pje at telecommunity.com Wed Apr 19 08:34:51 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 19 Apr 2006 02:34:51 -0400 Subject: [Python-3000] A super() idea - no _getframe() involved Message-ID: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> Here's how you solve the "how does super() get the current class" problem, using existing compiler and VM constructs, and without relying on class names, or on functions not being decorated, or anything like that. And it's so simple you'll slap your forehead for not thinking of it first. :) Closures in Python use "cell" objects that store a reference to an object, and the VM has operations to efficiently load and store to these cells. So what we do is that when the compiler sees a super operation, it should treat the reference to the containing class as a *local read of a closure variable*. That is, it defines the function with MAKE_CLOSURE instead of MAKE_FUNCTION. It also puts a reference to the cell object in the class's dictionary, let's say under __cell__. And so the function code just uses LOAD_DEREF to get at the contents of that cell. At this point, of course, the cell is empty because there's no class yet. But the MAKE_CLASS opcode, after creating the class, checks to see if the resulting object has a __cell__ attribute, and if so, stores the class's self-pointer into it. Or perhaps the 'type' constructor takes care of this, but if we can have class decorators it might be better to have MAKE_CLASS do it so it always points to the object that *will* be bound to the class name in the enclosing scope. This is just a sketch, of course, and there might be something I'm missing, but it doesn't seem to require anything particularly outlandish; all the pieces are there and it seems like any Python implementation that does closures ought to be able to handle it. And it doesn't depend on name mangling, the name not being reassigned, or anything like that. The only bit that might get tricky is that whether the code object generated for a class body can have cell variables or not. Since classes don't normally partake in the lexical scope system, it's possible that the implementation in some way precludes you from executing LOAD_CLOSURE in a class body code object, in which case there would need to be some other way to create the initial cell object. (Or maybe the way that the class code object is executed precludes this.) Apart from that potential issue, however, it actually seems simple enough to implement in the 2.x line, as long as the chosen syntax for expressing these automatic super calls is compatible with the 2.x grammar. Regarding the grammar, I have no strong opinions, although using the 'class' keyword, as in "super(class,self)" has a nice ring to it. But at least now you can talk about the *idea* of an automatic super, without arguing about sys._getframe() or other denigrated hacks. You're welcome. :) From greg.ewing at canterbury.ac.nz Wed Apr 19 08:35:34 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 18:35:34 +1200 Subject: [Python-3000] Futures in Python 3000 (was Re: mechanism for handling asynchronous concurrency) In-Reply-To: <4445CD94.1000409@neotitans.com> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> Message-ID: <4445DA36.9080509@canterbury.ac.nz> Andy Sy wrote: > I don't know about you, but with the addition of send() to deal with > the problem outlined in PEP 342, generators are starting to look more > and more like a Rube Goldberg contraption. Could anything be more > Pythonic than Io's: > > f := url @fetch // f is a future that will eventually hold the > // URL's contents, but you don't block on the fetch. There's a lot more to this than syntax. The oddities surrounding Python generators are mostly due to their "one-level-deep" nature, i.e. they're not full coroutines. And there are deep implementation reasons for that. If syntax is all you're concerned about, you could translate that into Python as something like f = url(future(fetch)) Now, how is that future() function going to be implemented, again? :-) -- Greg From walter at livinglogic.de Wed Apr 19 08:47:30 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 19 Apr 2006 08:47:30 +0200 Subject: [Python-3000] Use Py_CLEAR only In-Reply-To: <4445D1D4.8030503@canterbury.ac.nz> References: <4445906C.7060006@comcast.net> <4445D1D4.8030503@canterbury.ac.nz> Message-ID: <4445DD02.9060805@livinglogic.de> Greg Ewing wrote: > Edward C. Jones wrote: > >> Can Py_CLEAR completely replace Py_DECREF and Py_XDECREF? > > It would be less efficient in the cases where you don't > need to check for NULL and/or clear the reference > afterwards. Both should be optimized away by the compiler. But what we gain is one obvious way to do it, which would be more robust. This sounds like somebody should simply try it out (but I'm not sure, if blindly replacing Py_DECREF and Py_XDECREF with Py_CLEAR would be enough). Servus, Walter From greg.ewing at canterbury.ac.nz Wed Apr 19 09:08:05 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 19:08:05 +1200 Subject: [Python-3000] Use Py_CLEAR only In-Reply-To: <4445DD02.9060805@livinglogic.de> References: <4445906C.7060006@comcast.net> <4445D1D4.8030503@canterbury.ac.nz> <4445DD02.9060805@livinglogic.de> Message-ID: <4445E1D5.5020702@canterbury.ac.nz> Walter D?rwald wrote: > Greg Ewing wrote: >> It would be less efficient in the cases where you don't >> need to check for NULL and/or clear the reference >> afterwards. > > Both should be optimized away by the compiler. How? I don't see how the compiler can know either of those things. -- Greg From jcarlson at uci.edu Wed Apr 19 09:18:06 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 19 Apr 2006 00:18:06 -0700 Subject: [Python-3000] After 2.6 should be 3.0 In-Reply-To: References: <20060418200406.A81D.JCARLSON@uci.edu> Message-ID: <20060418234215.A823.JCARLSON@uci.edu> "Terry Reedy" wrote: > "Josiah Carlson" wrote in message > news:20060418200406.A81D.JCARLSON at uci.edu... > > Personally, I see Py3k as a vetting mechanism for all those hair-brained > > ideas that really shouldn't make it into any Python version (3k or > > otherwise), with the possible inclusion of things that *should* make > > life better for Python users. With that said, aside from the stdlib > > reorganization (which should happen in the 2.x series anyways), so far I > > haven't seen anything that necessitates backwards incompatible changes > > to Python 3k, and I predict that many of the changes to py3k will be > > included into mainline 2.x during 3k's development. > > The somewhat backwards-incompatible integer division change discussed and > approved some years ago has been suspended pending 3.0, so since I would > like to not have to (indefinitely) import 'division' from __future__ to get > the change, I too, on that score, would like to see 3.0 sooner than later. Again, in my opinion, features should necessitate the Python 3.x release. Is the integer division change sufficient to necessitate 3.0 after 2.6? Goodness, I hope not. I understand the dislike of repeated __future__ imports (I was using 'from __future__ import generators' for longer than I would have liked to), but this particular feature doesn't seem to imply to me that 3.0 should come out sooner, rather it says that people should be made aware of the integer division operation change, and it should make it into the 2.x series (how long have we been talking about integer division changes?) - Josiah From walter at livinglogic.de Wed Apr 19 09:30:51 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 19 Apr 2006 09:30:51 +0200 Subject: [Python-3000] Use Py_CLEAR only In-Reply-To: <4445E1D5.5020702@canterbury.ac.nz> References: <4445906C.7060006@comcast.net> <4445D1D4.8030503@canterbury.ac.nz> <4445DD02.9060805@livinglogic.de> <4445E1D5.5020702@canterbury.ac.nz> Message-ID: <4445E72B.3010906@livinglogic.de> Greg Ewing wrote: > Walter D?rwald wrote: >> Greg Ewing wrote: >>> It would be less efficient in the cases where you don't >>> need to check for NULL and/or clear the reference >>> afterwards. >> >> Both should be optimized away by the compiler. > > How? I don't see how the compiler can know either of > those things. The "checking for NULL" scenario looks somewhat like this: PyObject *foo = NULL; foo = get_foo(); if (foo) do something useful else return; if (foo) foo = NULL; The compiler should be able to detect that the second foo check is redundant, if "do something useful" doesn't use foo. And the "clearing the reference" scenario should look ike this: { PyObject *foo = get_foo(); do something useful foo = NULL; } The compiler should be able to detect that foo isn't used any more after the last assignment, so the assignment can be optimized away. Servus, Walter From pje at telecommunity.com Wed Apr 19 09:48:22 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 19 Apr 2006 03:48:22 -0400 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: <4445E4DA.1030905@canterbury.ac.nz> References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060419034338.0418ba08@mail.telecommunity.com> At 07:20 PM 4/19/2006 +1200, Greg Ewing wrote: >Phillip J. Eby wrote: > >>That is, it defines the function with MAKE_CLOSURE instead of >>MAKE_FUNCTION. It also puts a reference to the cell object in the >>class's dictionary, let's say under __cell__. > >Presumably it would first check if there was already a __cell__ >put there by another function and use that. No, the compiler does it, it's not an execution time thing. That is, the code object is set for a certain number of "cell vars", and that's what allocates the cell object when the code is executed and the frame is set up. >What happens if the function gets wrapped by a decorator? I >suppose it still works, because the cell remains attached to >the innermost function where it's needed, right? Right, it's part of the function object's func_closure tuple. >>Or perhaps the 'type' constructor takes care of this, but if we can have >>class decorators it might be better to have MAKE_CLASS do it so it always >>points to the object that *will* be bound to the class name in the >>enclosing scope. > >Isn't that backwards? MAKE_CLASS is going to bind it to the >class created by the basic class statement, before the >decorators get to work on it. The object returned by the >decorators might be something different. Sorry, I was thinking of PyProtocols/zope.interface style class decorators, which work by metaclass munging. That probably means that the scope enclosing the class should define the cell, and have a stanza like: MAKE_CLASS ... decorators, if any ... DUP_TOP STORE_DEREF __cell_for_this_class__ STORE_NAME "classname" And this way, it doesn't need to be an attribute of the class object. From guido at python.org Wed Apr 19 10:06:41 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 09:06:41 +0100 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> Message-ID: On 4/19/06, Phillip J. Eby wrote: > Here's how you solve the "how does super() get the current class" problem, > using existing compiler and VM constructs, and without relying on class > names, or on functions not being decorated, or anything like that. And > it's so simple you'll slap your forehead for not thinking of it first. :) Actually, I *did* think of it first. :-) http://mail.python.org/pipermail/python-3000/2006-April/000947.html And no, I didn't use the time machine to plant that. :-) Regarding the syntax, IMO *if* we use this we should do it so that you can write super.foobar(args) where currently you'd write super(ClassName, self).foobar(args) 'super' could be a new keyword. In 3.0 we could just make the keyword expand to a magic built-in function with the appropriate arguments (the magic cell and the first function argument). We could probably introduce this in 2.6 if the super object, when called, would return itself; then IMO no __future__ statement would be required since existing code using super would continue to work. Oh, I believe super() also supports static and/or class methods. I'm not sure how to handle this but I'm sure you can think of something. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 10:10:30 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 09:10:30 +0100 Subject: [Python-3000] auto-super() In-Reply-To: <44459EF5.4070604@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> Message-ID: On 4/19/06, Greg Ewing wrote: > On the contrary, it's precisely when you *don't* know > what classes are going be mixed in with you that super() > is likely to go haywire. Your super() call ends up > calling a method that you know nothing about, and > the method being called isn't expecting to be called > in that context either. Whether that will do the > "right" thing, or whether there even is a right thing > to be done, is anybody's guess. Um, you don't seem to be familiar with the theory of cooperative method calls I believe I cited the book from which I got this in some original writings regarding new-style classes (no time to look it up right now). It may not work too well in Python due to lack of enforcement, but in theory it is sound. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From pje at telecommunity.com Wed Apr 19 10:40:17 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed, 19 Apr 2006 04:40:17 -0400 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060419041759.04223da0@mail.telecommunity.com> At 09:06 AM 4/19/2006 +0100, Guido van Rossum wrote: >On 4/19/06, Phillip J. Eby wrote: > > Here's how you solve the "how does super() get the current class" problem, > > using existing compiler and VM constructs, and without relying on class > > names, or on functions not being decorated, or anything like that. And > > it's so simple you'll slap your forehead for not thinking of it first. :) > >Actually, I *did* think of it first. :-) > > http://mail.python.org/pipermail/python-3000/2006-April/000947.html > >And no, I didn't use the time machine to plant that. :-) I read this list via digest, so I hadn't seen it yet. >We could probably >introduce this in 2.6 if the super object, when called, would return >itself; then IMO no __future__ statement would be required since >existing code using super would continue to work. I assume you mean that the compiler would expand super.attr to the magic form, but super() would continue to be old-style. If so, then that still *would* require __future__, since you could legitimately have a variable named super. >Oh, I believe super() also supports static and/or class methods. I'm >not sure how to handle this but I'm sure you can think of something. If super.foobar(args) -> super(ThisClass,firstarg).foobar(args), then it will Just Work(TM) for class methods, since their first argument is the class, and super(someclass,myclass).aClassMethod(...) is the normal way of invoking a superclass classmethod. From ncoghlan at gmail.com Wed Apr 19 11:08:31 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Apr 2006 19:08:31 +1000 Subject: [Python-3000] super(): issues [Was: Removing 'self' from method definitions] In-Reply-To: <064401c66283$1a392bb0$2452fea9@bagio> References: <443E7F47.9060308@colorstudy.com><20060413171301.GA9869@localhost.localdomain><443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <064401c66283$1a392bb0$2452fea9@bagio> Message-ID: <4445FE0F.9040406@gmail.com> Giovanni Bajo wrote: > Greg Ewing wrote: > >> The use case for super() is where you have a chain of >> methods that are sufficiently compatible that you don't >> need to know or care which one you're calling next. >> That's not usually the case with __init__ methods. > > It worked fine for me when I forced my clients (for a specific hierarchy) to > always pass keyword arguments to __init__. This allowed me to use super() > succesfully. Of course, Python non-existing support for mandatory keyword > arguments makes things a little harder than they could be. > > I'll also point people to this page, which contains a very accurate description > of the issues with super: > http://fuhm.net/super-harmful/ Hmm, the reference to Dylan's next method (along with a few other comments in this thread) makes me believe a magic autosuper() equivalent would really need to be based on a thread-local context, rather than any kind of static code analysis. An interesting point is that being able to call the "next method" for generic functions has exactly the same problem. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Wed Apr 19 09:20:58 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 19 Apr 2006 19:20:58 +1200 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> Message-ID: <4445E4DA.1030905@canterbury.ac.nz> Phillip J. Eby wrote: > That is, it defines the function with MAKE_CLOSURE instead of > MAKE_FUNCTION. It also puts a reference to the cell object in the class's > dictionary, let's say under __cell__. Presumably it would first check if there was already a __cell__ put there by another function and use that. What happens if the function gets wrapped by a decorator? I suppose it still works, because the cell remains attached to the innermost function where it's needed, right? > Or perhaps the 'type' constructor takes care > of this, but if we can have class decorators it might be better to have > MAKE_CLASS do it so it always points to the object that *will* be bound to > the class name in the enclosing scope. Isn't that backwards? MAKE_CLASS is going to bind it to the class created by the basic class statement, before the decorators get to work on it. The object returned by the decorators might be something different. > Regarding the grammar, I have no strong opinions, although using the > 'class' keyword, as in "super(class,self)" has a nice ring to it. I'd like something decoupled from the super() function, so you can use it for normal inherited calls as well. Not sure what, though. -- Greg From thomas at python.org Wed Apr 19 11:52:27 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 19 Apr 2006 11:52:27 +0200 Subject: [Python-3000] auto-super() In-Reply-To: <44459EF5.4070604@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> Message-ID: <9e804ac0604190252q4c4e4140vcd43f8b03a7a56e8@mail.gmail.com> On 4/19/06, Greg Ewing wrote: > > Thomas Wouters wrote: > > > The arguments against super() are > > understandable -- if you only think about your own class, or completely > > disregard MI. I think that's irresponsible: super() always calls the > > right baseclass method, > > On the contrary, it's precisely when you *don't* know > what classes are going be mixed in with you that super() > is likely to go haywire. No, really, on the contrary, it's precisely when you don't know what classes are going to be mixed in with you that super() is what you need to use. If you do not use super(), you *will* be calling the wrong method. If you do use super(), you will be calling the right method, but you *may* pass the wrong arguments. The set of failure cases is very simply just smaller for super()-using classes. Your super() call ends up > calling a method that you know nothing about, This is violence inherent in the system. This is *why* you're calling the baseclass method. You don't want to figure out the work yourself, you want to ask your baseclass "do your thing as if I weren't there and you got these arguments". The only assumption you make is that the method signature is compatible, and that is indeed unfortunate (but unavoidable, in the current state of affairs. Exclusively using keyword arguments helps a bit, though.) and the method being called isn't expecting to be called > in that context either. A method that isn't expecting to be called by subclasses sounds like a broken method. Whether that will do the > "right" thing, or whether there even is a right thing > to be done, is anybody's guess. If two methods of the same name on different classes do wildly different things, your class hierarchy is broken, and it's completely unrelated to using super() or not. If you want the 'left' part of the diamond to take care of method 'spam' even though the 'right' part of the diamond also defines 'spam', not using super() will do what you want -- except it's horribly fragile. As soon as the right part of the diamond calls spam (and expects the right-handed spam to be called, which it can't help but do) things blow up; it will get the left-handed spam instead. That situations in unfixably broken. Using super() just makes the break more apparent (and, apparently, gives you an easy target to blame :-) If you have two methods of the same name that do the same thing, and expect to be called in the same situations, but have incompatible function signatures, it is also unfixably broken. When you use super(), the left side ends up calling the right-handed method as if it was the baseclass method, possibly passing too few arguments or the wrong ones, or not passing a bit of extra information the right-side could have used. When you don't use super(), you end up ignoring the right hand side entirely, quite likely breaking the entire functionality of that part of the inheritance tree. And if you have two methods that don't have a signature incompatible from the baseclass signature (meaning any extra arguments are optional), super() does the right thing but explicit baseclass calls do not. It's as simple as that. > What causes trouble isn't multiple inheritance, it's > diamond inheritance. Sorry, that's wishful thinking. Multiply-inheriting classes in Py3K are always involved in diamond inheritance, because they are always what we now call "new-style". And it's not an insignificant diamond, 'object' provides methods such as __getattribute__ that are really necessary, and really used. Without diamond inheritance, > explicit calls to inherited methods usually work fine, > and continue to work fine when your classes get > inherited, multiply or otherwise, by other people's > classes, as long as they do something sensible with > their inherited calls. No, as long as there is *no* conflict between the methods of the two sides. Assuming that is, again, wishful thinking, since you have no control over what subclasses do. And if they 'do something sensible', super() *also* does the right thing -- it just does the right thing in more cases to boot. > Personally I think that the use of diamond inheritance > should be severely discouraged, if not banned completely. > There's a fundamental problem with it: different > subclasses can have different ideas on how the diamond- > inherited class should be initialised, and otherwise > used. That seems to me an inherent flaw in the notion > of diamond inheritance, and not something that super() > or anything else can fix. As Guido pointed out, diamond inheritance isn't fundamentally flawed, it just requires a level of co-operation we aren't explicitly requiring, in Python. Fortunately, Python is quite flexible, and we can add that requirement in a custom metaclass. It would walk the namespace of any newly created class, recording method signatures, and comparing them against signatures of baseclasses. Or we could use decorators to indicate willing co-operation and warn against methods of the same name that don't say they play well with others. I've thought about using a metaclass like that, but I end up not needing it enough. Likewise, if you insist on not using MI (or super()), a metaclass could easily be made to warn against uses of MI -- and I did make that one for someone, once: import warnings class DontMIType(type): def __init__(self, name, bases, dict): if bases and len(bases) > 1: # Find the first baseclass to be of our type firstbase = self for base in bases: if not isinstance(base, DontMIType): break firstbase = base warnings.warn("Class %s multiply inherits, but baseclass %s " "disallows multiply inheriting" % (self, firstbase)) return super(DontMIType, self).__init__(name, bases, dict) It's only avisory, because you can easily circumvent this metaclass in subclasses by subclassing the metaclass and not calling the base-metaclass __init__, then using the sub-metaclass as metaclass for subclasses of whatever class doesn't like participating in MI. Besides, mandating such things isn't very Pythonic. It's a clear enough signal that the gun is now pointed at your wriggly little toes. Whenever I've tried to use diamond inheritance, I've > always ended up regretting it, and re-designing my class > hierarchy to eliminate it. Once I've done that, any > reason I had to use super() invariably disappears. If you have that luxury, that's fine. I'd encourage anyone to avoid MI (and use the above metaclass to avoid stepping into unexpected puddles later in life, of course.) I tend to use metaclasses instead of MI myself -- except *in* metaclass hierarchies, but that has a practical reason: Python *does* require metaclass compatibility. Metaclasses make for a good example on how (and why) to use super(), because the methods you use on metaclasses (by and large) are the __init__, __new__, __getattribute__ and __setattribute__ methods, and changing their signature is folly. But there are actual use-cases for MI in 'normal' classes, too, and Python still has to try and cater to them as best it (and we) can. And super() is part of that. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060419/71b7fc9d/attachment-0001.htm From thomas at python.org Wed Apr 19 12:15:17 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 19 Apr 2006 12:15:17 +0200 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> Message-ID: <9e804ac0604190315w49ecd802y832653ab0401d219@mail.gmail.com> On 4/19/06, Guido van Rossum wrote: > > On 4/19/06, Phillip J. Eby wrote: > > Here's how you solve the "how does super() get the current class" > problem, > > using existing compiler and VM constructs, and without relying on class > > names, or on functions not being decorated, or anything like that. And > > it's so simple you'll slap your forehead for not thinking of it > first. :) > > Actually, I *did* think of it first. :-) > > http://mail.python.org/pipermail/python-3000/2006-April/000947.html And yet before that: http://mail.python.org/pipermail/python-3000/2006-April/000922.html Needless to say, I'm in favour, if it can be swung ;-) Oh, I believe super() also supports static and/or class methods. I'm > not sure how to handle this but I'm sure you can think of something. It should just use the descriptor magic to retrieve the right method object from the stored 'current class', passing the class and instance (if any) as usual. That does mean, though, that super (however you want to spell it) needs to be informed what the instance or class is. The current super() call is explicitly passed 'self' (or 'cls' for classmethods), but it has no way of operating in staticmethods unless they are really classmethods in disguise (like __new__, a staticmethod that gets the class as first argument.) since, with neither 'self' nor 'cls', there is no way for super to figure out the right MRO. So, super would have to be used like so: def meth(self, arg, kwarg=kwval): return super.meth(self, arg, kwarg=kwval) @classmethod def cmeth(cls, arg, kwarg=kwval): return super.cmeth(cls, arg, kwarg=kwval) @staticmethod def smeth(cls, arg, kwarg=kwval): # cls is not necessarily the right class, but it's the best guess return super.smeth(cls, arg, kwarg=kwval) Which looks quite a bit like the old BaseClass.meth(self, ...) calls. I think it's a pity we can't make it look more like normal methodcalls, maybe by passing the class or instance explicitly: super(self).meth(arg, kwargv=kwval) (and super(cls) for classmethods.) Unfortunately, that is entirely and silently and somewhat surprisingly incompatible with Python 2.x, where super(cls) creates an unbound super proxy. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060419/b421d6ab/attachment.html From ncoghlan at gmail.com Wed Apr 19 12:17:50 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Apr 2006 20:17:50 +1000 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <1145332823.21740.56.camel@geddy.wooz.org> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> <20060417230805.GA23791@nightcrawler> <1145332823.21740.56.camel@geddy.wooz.org> Message-ID: <44460E4E.2010202@gmail.com> Barry Warsaw wrote: > On Mon, 2006-04-17 at 16:08 -0700, Brian Harring wrote: > >> The issue I'm seeing is that the wart you're pointing at is a general >> issue not limited to strings- everyone sooner or later has flattening >> code that hits the "recursively iterate over this container, except >> for instances of these classes". > > I wouldn't want to generalize this, but it is not infrequent that people > mistakenly iterate over strings when they want to treat them atomically. > difflib not withstanding, and keeping Guido's pronouncement in mind, I > do think people want to treat strings atomically much more often then > they want to treat them as a sequence of characters. Raymond brought something like this up on python-dev over a year ago. The last message in the thread was from me [1], with a proposed itertools.walk function that provided: - the choice of depth-first or breadth-first iteration - a simple stop-list of types not to be iterated over - block infinite recursion due to simple cycles (such as length 1 strings) - permitted use of an iterator factory other than the builtin iter The last feature allows iterables to be excluded from iteration based on factors other than an isinstance check or whether or not they're self-recursive. I'd be inclined to call YAGNI on it, except that it works well with unbound methods. For example, a tree walking function could be written: def walk_tree(root, depth_first=True): # Walk the tree, yielding all leaf elements return itertools.walk(root, depth_first, iter_factory=TreeNode.__iter__) Unfortunately, the thread fizzled without generating any additional interest. I don't recall the topic really coming up since then. Example usage: Py> seq [['123', '456'], 'abc', 'abc', 'abc', 'abc', ['xyz']] Py> list(walk(seq)) ['123', '456', 'abc', 'abc', 'abc', 'abc', 'xyz'] Py> list(walk(seq, depth_first=False)) ['abc', 'abc', 'abc', 'abc', '123', '456', 'xyz'] Py> list(walk(seq, atomic_types=())) ['1', '2', '3', '4', '5', '6', 'a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'x', 'y', 'z'] Py> list(walk(seq, depth_first=False, atomic_types=())) ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', '1', '2', '3', '4', '5', '6', 'x', 'y', 'z'] Cheers, Nick. [1] http://mail.python.org/pipermail/python-dev/2005-March/052245.html -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Wed Apr 19 12:20:41 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 11:20:41 +0100 Subject: [Python-3000] super(): issues [Was: Removing 'self' from method definitions] In-Reply-To: <4445FE0F.9040406@gmail.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <064401c66283$1a392bb0$2452fea9@bagio> <4445FE0F.9040406@gmail.com> Message-ID: On 4/19/06, Nick Coghlan wrote: > Hmm, the reference to Dylan's next method (along with a few other comments in > this thread) makes me believe a magic autosuper() equivalent would really need > to be based on a thread-local context, rather than any kind of static code > analysis. No, no, no! The current thread doesn't enter into it. It *must* be done through static code analysis. The inputs are the class and the instance. There's nothing that the thread can add. > An interesting point is that being able to call the "next method" for generic > functions has exactly the same problem. But thread-dependency doesn't enter into it. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 12:28:47 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 11:28:47 +0100 Subject: [Python-3000] Adaptation and type declarations In-Reply-To: References: Message-ID: On 4/11/06, Ron Adam wrote: > If the __signature__ is parsed from the first line, it puts it right > below the function definition. > > def pattern_search(seq, pattern, start=0): > """ fn(str, str, int) -> (str, int) > > Get a sub-string and index using a pattern. > """ > Please don't. Parsing stuff out of docstrings is a poor substitute for explicit syntax, either using decorators or the proposed argument annotations. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Wed Apr 19 12:32:54 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Apr 2006 20:32:54 +1000 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> Message-ID: <444611D6.1080500@gmail.com> Guido van Rossum wrote: > Here's a related but more complicated wish: define a function in such > a way that certain parameters *must* be passed as keywords, *without* > using *args or **kwds. This may require a new syntactic crutch. A single '*' could indicate that no additional positional arguments were permitted. def f(*, paramA, paramB=10): print paramA, paramB Here paramA and paramB can only be passed as keywords, and paramA *has* to be passed as it has no default. Utterly magical to anyone not already familiar with the use of *args, though. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Wed Apr 19 12:35:03 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 11:35:03 +0100 Subject: [Python-3000] Function call speed (Was: Cleaning up argument listparsing) In-Reply-To: References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> Message-ID: [I don't want to lose this message, but I don't have the time to read it and do the complementary source research right now. So I'm recommending you do some more research yourself.] You're right that the API is very complex. However have you tried stepping through some typical calls? There's a "fast path" that avoids most of the overhead at least if no keywords or */** notation are used, and I believe there's even a semi-fast path for when there are keyword args or defaults but no */** in either the call or the def. I recommend profiling. I seem to recall that last time we did this, we found that creating the frame object is a significant cost factor. --Guido On 4/19/06, Talin wrote: > I wanted to follow up a bit on the issue of speeding up function calls with > some more specifics. > > Currently, the function call mechanism has to do a lot of conversion > operations on the function arguments. Specifically, it takes the various > positional arguments, keyword arguments, varargs argument, and kwargs, and > combines them into a single tuple and a single dict. It then calls the target > function via PyObject_Call. > > On the recieving side, a different conversion takes place, depending on the > type of the callable. For user-defined functions, it maps the various > arguments in the tuple and dict into the function's parameter slots, > along with considerations of default values and such. > > One way to speed this up would be to determine cases where the conversion > steps can be skipped. There is already a method for this in the case where > there are only positional arguments, no varargs or keywords. > > Ideally, we would like to be able to pass the caller's argument stack pretty > much as-is to the reciever. At the moment, that consists of an array of > positional arguments, followed by an array of key/value pairs for the keyword > arguments, followed by an optional vararg or kwarg. > > Assume for a moment, then, that we had a variation of PyObject_Call which had > a function signature that was exactly that: > > PyObject_CallFast( > func, > PyObject **args, int numArgs, > PyObject **kargs, int numKwArgs, > PyObject *starargs, > PyObject *kwarg ); > > The reciever would have to do extra work, because now it has to check two > places for each argument. For a keyword argument, for example, it has to check > both the key/value array and the kwargs dict argument. > > However, this is far less work than building a new dictionary. In addition, we > know that the keywords are interned strings, not arbitrary objects, so we can > speed up the comparison operation by avoiding type tests. > > Also, because the keyword args are in a linear list, they will exhibit better > cache behavior, and the lookup will be very fast - a simple loop and test. (If > you want to write optimal code for today's processors, you pretty much have to > throw out everything you learned about unrolling loops and such, and learn to > think in terms of cache lines, not bytes and words.) > > Now, it may be that in some cases, the reciever really does require a tuple > and a dict. However, in such a case, the onus should be on the reciever to > convert the argument list into the preferred form. > > So in other words, it would be the reciever's job to convert the arguments > into whatever data structure the reciever required. For recievers with > relatively simple calling signatures, it may mean no conversion is done at > all, and the caller's argument list can be used directly. The most complex > part would be validating the arguments - making sure that a keyword argument > didn't get assigned more than once, and so on. > > What about older code? Well, I suppose you could add an additional field to > the recieving object, indicating whether it accepted the traditional > PyObject_Call-style args, or the newer format. Essentially you would have two > function pointers in the object, one for newstyle calling, and one for > oldstyle. If the newstyle was present, then the interpreter would use it, > otherwise it would fall back to the older means of transforming the arguments > into a tuple and a dict. > > In any case, the reason why this is "Py3K" material is that it involves > busting the existing C API to some extent. > > -- Talin > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 12:41:20 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 11:41:20 +0100 Subject: [Python-3000] Separating out CPython and core Python tests In-Reply-To: References: <20060411120235.kktmlrlfexpcgk8w@login.werra.lunarpages.com> <20060411192142.GO476@tigger.digitaltorque.ca> Message-ID: On 4/12/06, Terry Reedy wrote: > > "Guido van Rossum" wrote in message > > I agree that such a test suite would be helpful. I don't think it > > should wait for Python 3000. > > Possible plan: > > Add 'spec' or 'specification' or 'language', 'implementation', and possibly > 'support' subdirs to ...Lib/test. Put most test files in 'spec', possibly > support files in 'support' and any implementation-only tests in > 'implementation'. Test_peepholer.py at least would go in the latter. > Should tests for C-coded modules go here too? Perhaps. I suggest taking this to python-dev; I don't think we need to wait until 3.0 to do this. > Unless done immediately for 2.5, do this for trunk (future 2.6) only after > 2.5 maintenance branch established. It would probably slow down the 2.5 release; let's do it for 2.6. > Adjust 'from test.something import ...' statements to 'from > test.subdir.something import ...' with Python script. > > Split a few files known to have implementation tests, such as the test > somewhere that small ints up to a certain size are preallocated. This > could break even for a customized CPython build. Would having tests like > this segregated in an implementation test directory help even CPython > customizers? I doubt that that particular customization (or any other, really) is ever invoked -- these are customizable so we can tweak it in future implementations. > As Michael suggested, let Jython, IronPython, and PyPy people suggest > additional file splits or movements. More reason to invoke python-dev. (I also like Brett's idea of using decorators to provide fine-grained annotations rather than having to split everything across to multiple files according to scope. It's quite possible that some test are moved between categories in the future. Not that I'm against splitting across files if there's a clear separation.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 12:47:33 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 11:47:33 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: <44458B7A.9060208@canterbury.ac.nz> References: <44458B7A.9060208@canterbury.ac.nz> Message-ID: On 4/19/06, Greg Ewing wrote: > Py3k suggestion: make ... a generally legal expression. Yup. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Wed Apr 19 12:59:44 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Apr 2006 20:59:44 +1000 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> Message-ID: <44461820.6010005@gmail.com> Steven Bethard wrote: > ------------------------- > UserDict.DictMixin.update > ------------------------- ... > To clean this up, we'd need the ability to identify self and dict as > positional only arguments. AFAICT, the current proposal doesn't solve > this problem. Off the top of my head, I don't see an easy way of > supporting this either... One way would be to expand assignment to support positional argument to parameter matching when the sequence on the left is parenthesised: def __init__(*args, **kwargs): (self, other=None) = args ... The signature info would still be lacking details on self & other, but the kwarg conflict would be addressed. Taking this idea one step further, the info could be moved back into the signature by doing: def __init__(*(self, other=None), **kwargs): ... Which actually suggests an alternate approach for keyword-only arguments: permitting argument expansion in a tuple after the **. Then you'd have: def f(a, *(b, c=1, *args), **(d, e=2, **kwds)): # Silly function 'a' would be a normal positional-or-keyword argument 'b' would be a required positional-only argument 'c' would be an optional 3rd positional-only argument 'args' would contain any positional arguments after the 3rd 'd' would be a required keyword-only argument 'e' would be an optional keyword-only argument 'kwds' would contain any keyword arguments other than 'a', 'd' or 'e' > ---------------------- > UserDict.DictMixin.pop > ---------------------- ... > you still wouldn't know if "default" was None from the default in the > signature or because the value None was supplied by the caller. Making this distinction is where sentinel objects and identity tests are handy: _pop_sentinel = object() def pop(self, key, default=_pop_sentinel): try: value = self[key] except KeyError: if default is not self._pop_sentinel: return default raise del self[key] return value (IOW, this can be made simpler without touching Python's syntax) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Wed Apr 19 13:01:28 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Apr 2006 21:01:28 +1000 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: <444611D6.1080500@gmail.com> References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> <444611D6.1080500@gmail.com> Message-ID: <44461888.7080903@gmail.com> Nick Coghlan wrote: > Guido van Rossum wrote: >> Here's a related but more complicated wish: define a function in such >> a way that certain parameters *must* be passed as keywords, *without* >> using *args or **kwds. This may require a new syntactic crutch. > > A single '*' could indicate that no additional positional arguments were > permitted. > > def f(*, paramA, paramB=10): > print paramA, paramB > > Here paramA and paramB can only be passed as keywords, and paramA *has* to be > passed as it has no default. > > Utterly magical to anyone not already familiar with the use of *args, though. A different idea from another part of the thread would be to allow a parameter list rather than an identifier after **: def f(**(paramA, paramB=10)): print paramA, paramB Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Wed Apr 19 13:18:33 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 12:18:33 +0100 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: If you and/or Terry would post this that would be great -- I do have plenty of net access here but my time is very fragmented. Once it's posted I'll blog a link. One comment: I would add explicitly to the subject that the PSF is currently requesting mentors, not students. Perhaps Subject: Python Software Foundation looking for mentors for the Google Summer of Code would work? How widely are you sending this? I'd say c.l.py.a, c.l.py, python-dev, IronPython, Jython, Zope lists. --Guido On 4/19/06, Neal Norwitz wrote: > Guido, are you willing to send this? Would you rather me, Terry, or > someone else send it? > > My changes are inline. They aren't extensive. > > On 4/18/06, Terry Reedy wrote: > > > between May 1, 17:00 PST through May 8, 17:00 PST. > > Might as well be explicit. :-) > > > The Python Software Foundation (PSF) will again act as a sponsoring > > If you send html, you should include a link to the PSF, > http://www.python.org/psf/ > > > organization matching mentors and projects benefiting Python and Python > > programmers. Projects can include work on Python implementations, > > programmer utilitilies, libraries, packages, or frameworks. Project ideas > > ... or frameworks related to Python > > > People interested in mentoring a student though PSF are encouraged to > > contact Neal Norwitz at nnorwitz at gmail.com. People unknown to Neal or > > Guido should find a couple of people known within the Python community > that are willing to act as references. > > Complete version with edits: > > Google 2006 Summer of Code and Python Software Foundation > > This spring and summer, Google corporation will again provide several > hundred stipends for students (18+, undergraduate thru PhD programs) > to write new open-source code. The main page is > http://code.google.com/summerofcode.html. At the bottom are links to > StudentFAQ, MentorFAQ, and TermsOfService. The first two have the > timeline. Note that student applications are due between May 1, 17:00 > PST through May 8, 17:00 PST. > > The Python Software Foundation (PSF) will again act as a sponsoring > organization matching mentors and projects benefiting Python and > Python programmers. Projects can include work on Python > implementations, programmer utilitilies, libraries, packages, or > frameworks. Project ideas > are welcome and can be added to > http://wiki.python.org/moin/SummerOfCode, possibly as links to fuller > explanations elsewhere. > > The projects from last summer are described at > http://wiki.python.org/moin/SummerOfCode/2005 > > People interested in mentoring a student though PSF are encouraged to > contact Neal Norwitz at nnorwitz at gmail.com. People unknown to Neal or > Guido should find a couple of people known within the Python community > that are willing to act as references. > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 13:20:37 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 12:20:37 +0100 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <4445C8BC.5010202@canterbury.ac.nz> References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: On 4/19/06, Greg Ewing wrote: > Even if it did become available, we might not want to > use it. In recent times I've come round to the view that, > on modern architectures where cacheing is all-important, > refcounting + cyclic garbage collection may well be > *better* than mark-and-sweep or some variation thereof. Not that I disagree -- do you have specific data or reasoning to back this up? I'd love to hear that we were right all the time! :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 13:24:16 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 12:24:16 +0100 Subject: [Python-3000] Special object to return from functions that return only None In-Reply-To: <44459540.1040709@comcast.net> References: <44459540.1040709@comcast.net> Message-ID: On 4/19/06, Edward C. Jones wrote: > Wild idea. > > When I was a newbie, I repeatedly make the mistake of writing > > alist = alist.sort() > > I suggest a singleton object "UseForbidden" ("Py_UseForbidden" in C). > "UseForbidden" can be used only to return from a function. Any other > attempt to use "UseForbidden" raises an exception. The incorrect code > above would give the error message "cannot assign to result of function". Apart from assignment, None already has very few methods, so it already has this purpose. Trapping this on assignment would require *every* assignment (and argument passing, and who knows what else) to pay for the overhead for an additional specific check. I don't think that's feasible. I also don't think it's all that important as a feature, so I wouldn't want to go to length to implement this. Unless you can show a working patch that doesn't affect performance I consider this idea rejected (but thanks for throwing it out anyway -- this is the time to generate and quickly reject lots of ideas!). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 13:27:37 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 12:27:37 +0100 Subject: [Python-3000] Futures in Python 3000 (was Re: mechanism for handling asynchronous concurrency) In-Reply-To: <4445DA36.9080509@canterbury.ac.nz> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> Message-ID: On 4/19/06, Greg Ewing wrote: > Andy Sy wrote: > > > I don't know about you, but with the addition of send() to deal with > > the problem outlined in PEP 342, generators are starting to look more > > and more like a Rube Goldberg contraption. Could anything be more > > Pythonic than Io's: > > > > f := url @fetch // f is a future that will eventually hold the > > // URL's contents, but you don't block on the fetch. > > There's a lot more to this than syntax. The oddities > surrounding Python generators are mostly due to their > "one-level-deep" nature, i.e. they're not full coroutines. > And there are deep implementation reasons for that. > > If syntax is all you're concerned about, you could translate > that into Python as something like > > f = url(future(fetch)) > > Now, how is that future() function going to be implemented, > again? :-) Amen. If you want this, please spend time doing a prototype implementation so you can tell us how it should work, in all details. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Wed Apr 19 13:47:01 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 19 Apr 2006 21:47:01 +1000 Subject: [Python-3000] super(): issues [Was: Removing 'self' from method definitions] In-Reply-To: References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <064401c66283$1a392bb0$2452fea9@bagio> <4445FE0F.9040406@gmail.com> Message-ID: <44462335.5080406@gmail.com> Guido van Rossum wrote: > On 4/19/06, Nick Coghlan wrote: >> Hmm, the reference to Dylan's next method (along with a few other comments in >> this thread) makes me believe a magic autosuper() equivalent would really need >> to be based on a thread-local context, rather than any kind of static code >> analysis. > > No, no, no! The current thread doesn't enter into it. It *must* be > done through static code analysis. The inputs are the class and the > instance. There's nothing that the thread can add. Involving the thread was a really roundabout way of getting at the class. Using closures is far more sane, but the historic disconnect between classes and closures meant my brain failed to make that connection :( Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Wed Apr 19 14:01:25 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 00:01:25 +1200 Subject: [Python-3000] Use Py_CLEAR only In-Reply-To: <4445E72B.3010906@livinglogic.de> References: <4445906C.7060006@comcast.net> <4445D1D4.8030503@canterbury.ac.nz> <4445DD02.9060805@livinglogic.de> <4445E1D5.5020702@canterbury.ac.nz> <4445E72B.3010906@livinglogic.de> Message-ID: <44462695.3040503@canterbury.ac.nz> Walter D?rwald wrote: > The "checking for NULL" scenario looks somewhat like this: > > And the "clearing the reference" scenario should look ike this: But only if the reference is a local to the function, and it's used in a disciplined enough way, and the compiler is smart enough. None of those things are guaranteed true for all uses of Py_DECREF/Py_XDECREF. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 14:07:28 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 00:07:28 +1200 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: <5.1.1.6.0.20060419034338.0418ba08@mail.telecommunity.com> References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> <5.1.1.6.0.20060419034338.0418ba08@mail.telecommunity.com> Message-ID: <44462800.4090008@canterbury.ac.nz> Phillip J. Eby wrote: > No, the compiler does it, it's not an execution time thing. That is, > the code object is set for a certain number of "cell vars", and that's > what allocates the cell object when the code is executed and the frame > is set up. What I mean is, however it's done, all the functions defined in a particular class need to end up sharing the same cell. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 14:14:26 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 00:14:26 +1200 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> Message-ID: <444629A2.3090904@canterbury.ac.nz> Guido van Rossum wrote: > Regarding the syntax, IMO *if* we use this we should do it so that you can write > > super.foobar(args) I was thinking of something like __super__.foobar(self, args) i.e. __super__ would just do the work of finding the appropriate unbound method. This would make using __super__ more like an ordinary inherited call, and in the case where you only have one base class, __super__ is a drop-in replacement for the name of the base class. And __super__ could just be super if you're happy to have a new keyword: super.foobar(self, args) -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 14:17:21 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 00:17:21 +1200 Subject: [Python-3000] auto-super() In-Reply-To: References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> Message-ID: <44462A51.6070709@canterbury.ac.nz> Guido van Rossum wrote: > Um, you don't seem to be familiar with the theory of cooperative > method calls I believe I cited the book from which I got this in some > original writings regarding new-style classes (no time to look it up > right now). I'd be interested to see that when you do get time. I wasn't aware that there was a formal theory behind this. -- Greg From greg.ewing at canterbury.ac.nz Wed Apr 19 14:22:34 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 00:22:34 +1200 Subject: [Python-3000] super(): issues [Was: Removing 'self' from method definitions] In-Reply-To: <4445FE0F.9040406@gmail.com> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <064401c66283$1a392bb0$2452fea9@bagio> <4445FE0F.9040406@gmail.com> Message-ID: <44462B8A.9000002@canterbury.ac.nz> Nick Coghlan wrote: > Hmm, the reference to Dylan's next method (along with a few other > comments in this thread) makes me believe a magic autosuper() equivalent > would really need to be based on a thread-local context, rather than any > kind of static code analysis. Why? I don't see what could differ from one thread to another that would make the "next method" be different. Unless somehow there were thread-local inheritance hierarchies or something bizarre like that. Even if Dylan has them, I'm quite sure Python doesn't! -- Greg From walter at livinglogic.de Wed Apr 19 14:23:40 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Wed, 19 Apr 2006 14:23:40 +0200 Subject: [Python-3000] Use Py_CLEAR only In-Reply-To: <44462695.3040503@canterbury.ac.nz> References: <4445906C.7060006@comcast.net> <4445D1D4.8030503@canterbury.ac.nz> <4445DD02.9060805@livinglogic.de> <4445E1D5.5020702@canterbury.ac.nz> <4445E72B.3010906@livinglogic.de> <44462695.3040503@canterbury.ac.nz> Message-ID: <44462BCC.9010202@livinglogic.de> Greg Ewing wrote: > Walter D?rwald wrote: > >> The "checking for NULL" scenario looks somewhat like this: >> >> And the "clearing the reference" scenario should look ike this: > > But only if the reference is a local to the function, True Py_CLEAR(foo->bar->baz) probably won't be optimized. > and it's used in a disciplined enough way, and the > compiler is smart enough. None of those things are > guaranteed true for all uses of Py_DECREF/Py_XDECREF. I tried simply replacing all Py_DECREF()/Py_XDECREF() calls with Py_CLEAR with the result that nothing works any more. ;) Modules/posixmodule.c uses Py_DECREF(Py_None) which can't work with Py_CLEAR, because Py_None is not an lvalue. Python/import.c::PyImport_AddModule() returns m after doing a Py_DECREF, so this would have to continue to use Py_DECREF() (or do a dance with a second local variable). Python/compile.c::makecode() must use Py_DECREF, because the variable is still checked afterwards. Python/symtable.c::symtable_new_tmpname() and Python/compile.c::compiler_listcomp() (and probable many others) use the same var name as the Py_CLEAR macro which doesn't work. Objects/dictobject.c::insertdict() does a Py_DECREF(dummy), but that probably shouldn't clear dummy. And then I gave up. If we would do this, IMHO it would only make sense to do it the other way round: Replace Py_(X)DECREF with Py_CLEAR in selected spots. But that's not much different from what has happened up to now. Servus, Walter From ferringb at gmail.com Wed Apr 19 14:55:42 2006 From: ferringb at gmail.com (Brian Harring) Date: Wed, 19 Apr 2006 05:55:42 -0700 Subject: [Python-3000] Special object to return from functions that return only None In-Reply-To: <44459540.1040709@comcast.net> References: <44459540.1040709@comcast.net> Message-ID: <20060419125542.GA27455@nightcrawler> On Tue, Apr 18, 2006 at 09:41:20PM -0400, Edward C. Jones wrote: > Wild idea. > > When I was a newbie, I repeatedly make the mistake of writing > > alist = alist.sort() > > I suggest a singleton object "UseForbidden" ("Py_UseForbidden" in C). > "UseForbidden" can be used only to return from a function. Any other > attempt to use "UseForbidden" raises an exception. The incorrect code > above would give the error message "cannot assign to result of function". Not to beat the horse a bit further, but... This would also have the unfortunate side affect of forbidding snippets like- groupped_lists = generate_list_of_lists() [l.sort() for l in groupped_lists] Yes, it's lazy (could use a for loop doesn't generate a list), but the short hand there is common enough, as is the map equiv. Rough guess, to properly handle above (ie, support it while disallowing assignment to vars) requires tracing back the assignment- no longer is it just "am I assigning to a var", it's now "am I being handed to a var that is going to go away immediately" which is far uglier/nastier. My interpretation at least, either way -1 (especially if it kills off the short hand) :) ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060419/3b86b695/attachment.pgp From thomas at python.org Wed Apr 19 15:07:05 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 19 Apr 2006 15:07:05 +0200 Subject: [Python-3000] auto-super() In-Reply-To: <44462A51.6070709@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> <44462A51.6070709@canterbury.ac.nz> Message-ID: <9e804ac0604190607g1395d3dbi87c004145a6d1bd4@mail.gmail.com> On 4/19/06, Greg Ewing wrote: > > Guido van Rossum wrote: > > > Um, you don't seem to be familiar with the theory of cooperative > > method calls I believe I cited the book from which I got this in some > > original writings regarding new-style classes (no time to look it up > > right now). > > I'd be interested to see that when you do get time. > I wasn't aware that there was a formal theory behind > this. I believe Guido is referring to http://www.awl.com/cseng/titles/0-201-43305-2 (see the references in http://www.python.org/2.2.3/descrintro.html ) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060419/3b2c9d3e/attachment.html From guido at python.org Wed Apr 19 15:10:35 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 14:10:35 +0100 Subject: [Python-3000] auto-super() In-Reply-To: <9e804ac0604190607g1395d3dbi87c004145a6d1bd4@mail.gmail.com> References: <443E7F47.9060308@colorstudy.com> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> <44462A51.6070709@canterbury.ac.nz> <9e804ac0604190607g1395d3dbi87c004145a6d1bd4@mail.gmail.com> Message-ID: > > Guido van Rossum wrote: > > > Um, you don't seem to be familiar with the theory of cooperative > > > method calls I believe I cited the book from which I got this in some > > > original writings regarding new-style classes (no time to look it up > > > right now). > On 4/19/06, Greg Ewing wrote: > > I'd be interested to see that when you do get time. > > I wasn't aware that there was a formal theory behind > > this. On 4/19/06, Thomas Wouters wrote: > I believe Guido is referring to > http://www.awl.com/cseng/titles/0-201-43305-2 (see the > references in http://www.python.org/2.2.3/descrintro.html ) Right, that's exactly what I meant. Thanks, Thomas! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Wed Apr 19 15:14:53 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 01:14:53 +1200 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: <444637CD.8020508@canterbury.ac.nz> Guido van Rossum wrote: > Not that I disagree -- do you have specific data or reasoning to back > this up? I'd love to hear that we were right all the time! :-) No, I don't, sorry, it's just a hypothesis. But I do like the way most Python garbage goes away almost immediately, instead of hanging around polluting the heap until the next time someone decides to do a GC. It just feels tidier, somehow. -- Greg From aahz at pythoncraft.com Wed Apr 19 15:19:14 2006 From: aahz at pythoncraft.com (Aahz) Date: Wed, 19 Apr 2006 06:19:14 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: <20060419131914.GA15589@panix.com> [Neal asked me to post this nitpick in case someone else posts the announcement] On 4/19/06, Neal Norwitz wrote: > > This spring and summer, Google corporation will again provide several > hundred stipends for students (18+, undergraduate thru PhD programs) > to write new open-source code. The main page is > http://code.google.com/summerofcode.html. At the bottom are links to > StudentFAQ, MentorFAQ, and TermsOfService. The first two have the > timeline. Note that student applications are due between May 1, 17:00 > PST through May 8, 17:00 PST. Either s/between/from/ or s/through/and/ Python: because we care about language nitpicks ;-) Actually, looking at that, I have one more nitpick. It's a Bad Idea IMO to put a period immediately following an URL. I would put the URL on its own line with no punctuation. Most e-mail clients can understand that, and people using text e-mail clients can more easily copy/paste the URL. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From aahz at pythoncraft.com Wed Apr 19 15:44:36 2006 From: aahz at pythoncraft.com (Aahz) Date: Wed, 19 Apr 2006 06:44:36 -0700 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> Message-ID: <20060419134435.GB15589@panix.com> On Wed, Apr 19, 2006, Guido van Rossum wrote: > > Regarding the syntax, IMO *if* we use this we should do it so that you > can write > > super.foobar(args) > > where currently you'd write > > super(ClassName, self).foobar(args) > > 'super' could be a new keyword. In 3.0 we could just make the keyword > expand to a magic built-in function with the appropriate arguments > (the magic cell and the first function argument). We could probably > introduce this in 2.6 if the super object, when called, would return > itself; then IMO no __future__ statement would be required since > existing code using super would continue to work. -1 on keyword in 2.x -- my company already has our own super() function that does call-stack crawling and works with classic classes (since we don't dare convert to new-style classes). We can't be the only ones. I shan't argue too forcefully, though; it wouldn't be that hard to rename it. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From guido at python.org Wed Apr 19 15:46:22 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 14:46:22 +0100 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: <5.1.1.6.0.20060419041759.04223da0@mail.telecommunity.com> References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> <5.1.1.6.0.20060419041759.04223da0@mail.telecommunity.com> Message-ID: On 4/19/06, Phillip J. Eby wrote: [Guido] > >We could probably > >introduce this in 2.6 if the super object, when called, would return > >itself; then IMO no __future__ statement would be required since > >existing code using super would continue to work. > > I assume you mean that the compiler would expand super.attr to the magic > form, but super() would continue to be old-style. No, I was suggesting that (perhaps) the keyword 'super', regardless of context, would expand to an object that behaves like the object returned by the current super(ClassName, self) call, but that in addition has a __call__ method that ignores its arguments and returns itself. Then super(C, self).foo(args) would be equivalent to today's super(C, self)(C, self).foo(args) and the second call would just be a no-op. I believe this is possible because the only meaningful operation on the return value of super(...) is __getattr__. But I'm not dead set on this approach. I do like the idea of supporting both the old and the new syntax for a while. > If so, then that still > *would* require __future__, since you could legitimately have a variable > named super. Unfortunately, yes. I'd been hoping that since it's been a built-in since 2.2, code that uses the name super in any other way would be rare enough not to care about breaking it. But that's not a very safe assumption. I think that if we require from __future__ import super_keyword we might as well not bother with the b/w compatibility hack described above. > > >Oh, I believe super() also supports static and/or class methods. I'm > >not sure how to handle this but I'm sure you can think of something. > > If super.foobar(args) -> super(ThisClass,firstarg).foobar(args), then it > will Just Work(TM) for class methods, since their first argument is the > class, and super(someclass,myclass).aClassMethod(...) is the normal way of > invoking a superclass classmethod. But how about static methods? Inside a static method, I believe you're allowed to write super(ThisClass).foobar(args) which would call ThisClass.__base__.foobar(args) (assuming single inheritance for a moment) i.e. nothing is added to the argument list. That's a bit tricky to do with a super keyword since the compiler isn't supposed to know what the decorators mean, so it can't switch to this interpretation depending on whether @staticmethod is present (there could be a different decorator that happens to have a similar effect). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 15:50:07 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 14:50:07 +0100 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: <444611D6.1080500@gmail.com> References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> <444611D6.1080500@gmail.com> Message-ID: On 4/19/06, Nick Coghlan wrote: > Guido van Rossum wrote: > > Here's a related but more complicated wish: define a function in such > > a way that certain parameters *must* be passed as keywords, *without* > > using *args or **kwds. This may require a new syntactic crutch. > > A single '*' could indicate that no additional positional arguments were > permitted. > > def f(*, paramA, paramB=10): > print paramA, paramB > > Here paramA and paramB can only be passed as keywords, and paramA *has* to be > passed as it has no default. I once considered and rejected this syntax since another logical interpretation would be that any positional arguments are accepted but *ignored*. > Utterly magical to anyone not already familiar with the use of *args, though. But so would any other newly invented syntax, probably, so that doesn't count heavily. Would *None be too bizarre? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 15:52:14 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 14:52:14 +0100 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: <44461888.7080903@gmail.com> References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> <444611D6.1080500@gmail.com> <44461888.7080903@gmail.com> Message-ID: On 4/19/06, Nick Coghlan wrote: > A different idea from another part of the thread would be to allow a parameter > list rather than an identifier after **: > > def f(**(paramA, paramB=10)): > print paramA, paramB Hm, -1. Without the explanation I would have *no* idea what that could mean. Even with explanation I'm not sure... (Is paramA a positional parameter? If not, why not, and what's its default?) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 16:00:42 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 15:00:42 +0100 Subject: [Python-3000] After 2.6 should be 3.0 In-Reply-To: <4445976E.1070806@comcast.net> References: <4445976E.1070806@comcast.net> Message-ID: On 4/19/06, Edward C. Jones wrote: > I would like to see Python 3.0 before the turn of the next millennium. I > suggest that Python 2.6 be devoted to implementing some of the backward > compatible changes for Python 3.0. In particular, do the long-overdue > reorganization of the libraries. After 2.6 should be 3.0. The proposed schedule (see PEP 3000) puts 3.0 somewhere in 2008. With 2.5 planned for July/August this year, and releases about 18 months apart, 2.6 should be out in early 2008, so you may get the relative timing you're asking for. BUT... I am proposing to do at least 2.7 and possibly 2.8 and 2.9 after that (but no more). The transition to 3.0 will be difficult for people with large code bases and strict reliability requirements -- some enterprise users that I know will need to do at least 6 months of testing before they are comfortable with a transition. (Google is slower, in case you thought that's a cloaked reference to Google. :-) And probably 2.6 will be too early to "backport" 3.0 features except for some that mature very early in the 3.0 cycle. 2.7 seems a more likely target. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 16:03:17 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 15:03:17 +0100 Subject: [Python-3000] After 2.6 should be 3.0 In-Reply-To: <20060418234215.A823.JCARLSON@uci.edu> References: <20060418200406.A81D.JCARLSON@uci.edu> <20060418234215.A823.JCARLSON@uci.edu> Message-ID: On 4/19/06, Josiah Carlson wrote: > Again, in my opinion, features should necessitate the Python 3.x release. > Is the integer division change sufficient to necessitate 3.0 after 2.6? > Goodness, I hope not. It's the other way around. int/int alone isn't enough to trigger 3.0; but only 3.0 can change the semantics of int/int. (This has been promised quite a long time ago and I don't think it ought to be revised.) > I understand the dislike of repeated __future__ imports (I was using > 'from __future__ import generators' for longer than I would have liked > to), but this particular feature doesn't seem to imply to me that 3.0 > should come out sooner, rather it says that people should be made aware > of the integer division operation change, and it should make it into the > 2.x series (how long have we been talking about integer division changes?) No. The timeline was fixed in the PEP. We can't change this without seriously affecting Python's image to those users who care about backwards compatibility. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed Apr 19 16:05:49 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 15:05:49 +0100 Subject: [Python-3000] After 2.6 should be 3.0 In-Reply-To: <20060418200406.A81D.JCARLSON@uci.edu> References: <4445976E.1070806@comcast.net> <20060418200406.A81D.JCARLSON@uci.edu> Message-ID: On 4/19/06, Josiah Carlson wrote: > -1. Python 3k (non-alpha, beta, etc.) should come out when it's ready. > No sooner. Right. > Personally, I see Py3k as a vetting mechanism for all those hair-brained > ideas that really shouldn't make it into any Python version (3k or > otherwise), Huh? Why should Py3k be relevant to ideas that shouldn't go in either way? > with the possible inclusion of things that *should* make > life better for Python users. With that said, aside from the stdlib > reorganization (which should happen in the 2.x series anyways), so far I > haven't seen anything that necessitates backwards incompatible changes > to Python 3k, and I predict that many of the changes to py3k will be > included into mainline 2.x during 3k's development. Then you haven't been paying attention. keys() returning a set view is incompatible. A new I/O stack is incompatible. All-Unicode strings are incompatible. Etc. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas at python.org Wed Apr 19 16:17:49 2006 From: thomas at python.org (Thomas Wouters) Date: Wed, 19 Apr 2006 16:17:49 +0200 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> <5.1.1.6.0.20060419041759.04223da0@mail.telecommunity.com> Message-ID: <9e804ac0604190717l5c1b35c1te2d51f5b7b08dfe2@mail.gmail.com> On 4/19/06, Guido van Rossum wrote: > > > > >Oh, I believe super() also supports static and/or class methods. I'm > > >not sure how to handle this but I'm sure you can think of something. > > > > If super.foobar(args) -> super(ThisClass,firstarg).foobar(args), then it > > will Just Work(TM) for class methods, since their first argument is the > > class, and super(someclass,myclass).aClassMethod(...) is the normal way > of > > invoking a superclass classmethod. > > But how about static methods? Inside a static method, I believe you're > allowed to write > > super(ThisClass).foobar(args) > > which would call > > ThisClass.__base__.foobar(args) > > (assuming single inheritance for a moment) i.e. nothing is added to > the argument list. That's not how it works (and that's good; refusing the temptation to guess, and what not :) super() cannot make any assumptions about the next class in the MRO, and since you don't *have* a class in a staticmethod, it has no way to find out. staticmethods just can't sensibly call 'their baseclass method' (unless they're really just "explicit classmethods", like __new__, but those best be real classmethods (even __new__ :)) Currently, super(ThisClass) returns an 'unbound super object', which is not a proxy for 'the next class' but rather an descriptor that would get the right proxy object upon retrieval: >>> class X(object): ... @staticmethod ... def static(*args): ... print "X.static%r" % (args,) ... >>> class Y(X): ... @staticmethod ... def static(*args): ... print "Y.static%r" % (args,) ... super(Y).static(*args) ... >>> Y.static() Y.static() Traceback (most recent call last): File "", line 1, in ? File "", line 5, in static AttributeError: 'super' object has no attribute 'static' >>> y = Y() >>> y.static() Y.static() Traceback (most recent call last): File "", line 1, in ? File "", line 5, in static AttributeError: 'super' object has no attribute 'static' And to prove it's a descriptor: >>> class Z(X): ... @staticmethod ... def static(*args): ... print "Z.static%r" % (args,) ... super(Z).__get__(Z, None).static(*args) ... >>> Z.static() Z.static() X.static() >>> Z().static() Z.static() X.static() Doing super(Z).__get__(Z, None).static is just as static as calling X.static, though (with the only difference being that you name Z multiple times, instead of naming X multiple times.) It doesn't take MI into account at all. And yes, staticmethods are damned near useless. Let's not worry about them calling their baseclass methods -- they honestly shouldn't. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060419/950a71b3/attachment.html From guido at python.org Wed Apr 19 17:39:55 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 16:39:55 +0100 Subject: [Python-3000] A super() idea - no _getframe() involved In-Reply-To: <9e804ac0604190717l5c1b35c1te2d51f5b7b08dfe2@mail.gmail.com> References: <5.1.1.6.0.20060419021123.01e46060@mail.telecommunity.com> <5.1.1.6.0.20060419041759.04223da0@mail.telecommunity.com> <9e804ac0604190717l5c1b35c1te2d51f5b7b08dfe2@mail.gmail.com> Message-ID: On 4/19/06, Thomas Wouters wrote: [responding to some confusing text from me about super in a static method] > That's not how it works (and that's good; refusing the temptation to guess, > and what not :) super() cannot make any assumptions about the next class in > the MRO, and since you don't *have* a class in a staticmethod, it has no way > to find out. staticmethods just can't sensibly call 'their baseclass method' > (unless they're really just "explicit classmethods", like __new__, but those > best be real classmethods (even __new__ :)) Of course. I forgot that -- super is useless with a static method. Whether static methods are useless is a matter of taste (I happen to concur :-). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Wed Apr 19 17:56:14 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 19 Apr 2006 11:56:14 -0400 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <44460E4E.2010202@gmail.com> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> <20060417230805.GA23791@nightcrawler> <1145332823.21740.56.camel@geddy.wooz.org> <44460E4E.2010202@gmail.com> Message-ID: <1145462174.21198.147.camel@resist.wooz.org> On Wed, 2006-04-19 at 20:17 +1000, Nick Coghlan wrote: > Unfortunately, the thread fizzled without generating any additional interest. > I don't recall the topic really coming up since then. Maybe because the proposal got too complex while trying to be so general? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060419/5e4ceb99/attachment.pgp From jimjjewett at gmail.com Wed Apr 19 18:18:53 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 19 Apr 2006 12:18:53 -0400 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: <44461820.6010005@gmail.com> References: <44430CB5.5020907@gmail.com> <44461820.6010005@gmail.com> Message-ID: On 4/19/06, Nick Coghlan wrote: > Then you'd have: > > def f(a, *(b, c=1, *args), **(d, e=2, **kwds)): > # Silly function > > 'a' would be a normal positional-or-keyword argument > 'b' would be a required positional-only argument Am I reading that correctly? Looking only at a and b the possible calling signatures are exactly: f(1, 2) f(a=1, 2) because b can't be named but must appear second, and nothing except a can appear before it because of the rules on positional arguments. -jJ From bborcic at gmail.com Wed Apr 19 18:30:35 2006 From: bborcic at gmail.com (Boris Borcic) Date: Wed, 19 Apr 2006 18:30:35 +0200 Subject: [Python-3000] genexp syntax for reduce In-Reply-To: References: <4444D639.6030108@batiment71.net> Message-ID: <444665AB.9040604@gmail.com> Guido van Rossum wrote: > I wouldn't recommend this approach to anyone. What about a variation ? It's arguably a 2.5a1 bug that yield expressions are permitted in generator expressions, but this suggests a slight syntax extension. Suppose wrapper( for after ) would mean what can in 2.5a1 be clumsily written as wrapper( for _sd in (,) for in (yield _sd) ) then, provided adequate wrapper definitions - I'll provide below - one could write >>> fixpoint(sqrt(x+1) for x after 2) # golden ratio is attractive fixed point 1.6180339887727981 >>> fibo = seedbackloop(x+y for x,y after (1,1)) >>> first10 = lambda gen : tuple(itertools.islice(gen,10)) >>> first10(fibo) (1, 1, 2, 3, 5, 8, 13, 21, 34, 55) >>> superfibo = seedbackloop(sum(trio) for trio after (1,1,1)) >>> first10(superfibo) (1, 1, 1, 3, 5, 9, 17, 31, 57, 105) >>> data = "put something intelligent here".split() >>> seedfold(pair for pair after ("Now please",data)) (((('Now please','put'),'something'),'intelligent'),'here') >>> seedfold([y,x] for x,y after ("Now please",data)) ['here',['intelligent',['something',['put','Now please']]]] Here for proof-of-concept definitions of the wrappers (without attention to corner cases/error messages, but tested with the above examples through the 2.5a1 equivalent syntax) def seedfold(gen) : from itertools import repeat feedback,source = gen.next() source = iter(source) next = source.next() feedback = gen.send((feedback,next) for _ in repeat(None)) for next in source : feedback = gen.next() return feedback def seedbackloop(gen) : from collections import deque from itertools import repeat feedback = deque(gen.next()) for item in feedback : yield item feedback.append(gen.send(tuple(feedback) for _ in repeat(None))) while True : feedback.popleft() yield feedback[-1] feedback.append(gen.next()) def fixpoint(gen,maxiter=1000,eps=1e-10) : from itertools import repeat cnt = maxiter anterior = gen.next() posterior = gen.send(anterior for _ in repeat(None)) while abs(posterior-anterior)>eps : cnt -= 1 if cnt == 0 : raise ValueError("Expression did not converge after " + str(maxiter) + " iterations") anterior = posterior posterior = gen.next() return posterior Regards, Boris Borcic -- assert "304" in "340343", "P4R4D15E M15M47C8" From nnorwitz at gmail.com Wed Apr 19 19:11:36 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 19 Apr 2006 10:11:36 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: Terry, if you can post this during the day, please go ahead and do so and incorporate Guido's and Aahz's comments. If you don't have a chance today, I will post it tonight when I get home which will almost definitely be after 8pm PT/11pm ET. Thanks, n -- On 4/19/06, Guido van Rossum wrote: > If you and/or Terry would post this that would be great -- I do have > plenty of net access here but my time is very fragmented. Once it's > posted I'll blog a link. > > One comment: I would add explicitly to the subject that the PSF is > currently requesting mentors, not students. Perhaps > > Subject: Python Software Foundation looking for mentors for the > Google Summer of Code > > would work? > > How widely are you sending this? I'd say c.l.py.a, c.l.py, python-dev, > IronPython, Jython, Zope lists. > > --Guido > > On 4/19/06, Neal Norwitz wrote: > > Guido, are you willing to send this? Would you rather me, Terry, or > > someone else send it? > > > > My changes are inline. They aren't extensive. > > > > On 4/18/06, Terry Reedy wrote: > > > > > between May 1, 17:00 PST through May 8, 17:00 PST. > > > > Might as well be explicit. :-) > > > > > The Python Software Foundation (PSF) will again act as a sponsoring > > > > If you send html, you should include a link to the PSF, > > http://www.python.org/psf/ > > > > > organization matching mentors and projects benefiting Python and Python > > > programmers. Projects can include work on Python implementations, > > > programmer utilitilies, libraries, packages, or frameworks. Project ideas > > > > ... or frameworks related to Python > > > > > People interested in mentoring a student though PSF are encouraged to > > > contact Neal Norwitz at nnorwitz at gmail.com. People unknown to Neal or > > > Guido should find a couple of people known within the Python community > > that are willing to act as references. > > > > Complete version with edits: > > > > Google 2006 Summer of Code and Python Software Foundation > > > > This spring and summer, Google corporation will again provide several > > hundred stipends for students (18+, undergraduate thru PhD programs) > > to write new open-source code. The main page is > > http://code.google.com/summerofcode.html. At the bottom are links to > > StudentFAQ, MentorFAQ, and TermsOfService. The first two have the > > timeline. Note that student applications are due between May 1, 17:00 > > PST through May 8, 17:00 PST. > > > > The Python Software Foundation (PSF) will again act as a sponsoring > > organization matching mentors and projects benefiting Python and > > Python programmers. Projects can include work on Python > > implementations, programmer utilitilies, libraries, packages, or > > frameworks. Project ideas > > are welcome and can be added to > > http://wiki.python.org/moin/SummerOfCode, possibly as links to fuller > > explanations elsewhere. > > > > The projects from last summer are described at > > http://wiki.python.org/moin/SummerOfCode/2005 > > > > People interested in mentoring a student though PSF are encouraged to > > contact Neal Norwitz at nnorwitz at gmail.com. People unknown to Neal or > > Guido should find a couple of people known within the Python community > > that are willing to act as references. > > > > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) > From janssen at parc.com Wed Apr 19 19:12:33 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 19 Apr 2006 10:12:33 PDT Subject: [Python-3000] auto-super() In-Reply-To: Your message of "Tue, 18 Apr 2006 20:24:41 PDT." <20060419032441.GA27988@panix.com> Message-ID: <06Apr19.101239pdt."58633"@synergy1.parc.xerox.com> > Greg Ewing writes: > > Personally I think that the use of diamond inheritance should be > > severely discouraged, if not banned completely. and Aahz replies: > But multiple inheritance with new-style classes is inherently diamond > inheritance -- how do you propose to deal with that? Multiple inheritance, even with diamond inheritance (which really can't be avoided in most use cases), is so useful in a language where it's actually supported (like Python), that I can't believe more folks don't switch to Python just to use it. Perhaps if Python 3K has real support for building UIs, there would be more recognition of it -- I continually encounter cases in re-factoring Java Swing UIs where it would be very nice to have. Better "super()" support would also help. Bill From jcarlson at uci.edu Wed Apr 19 19:22:14 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 19 Apr 2006 10:22:14 -0700 Subject: [Python-3000] After 2.6 should be 3.0 In-Reply-To: References: <20060418200406.A81D.JCARLSON@uci.edu> Message-ID: <20060419091913.A82C.JCARLSON@uci.edu> "Guido van Rossum" wrote: > On 4/19/06, Josiah Carlson wrote: > > Personally, I see Py3k as a vetting mechanism for all those hair-brained > > ideas that really shouldn't make it into any Python version (3k or > > otherwise), > > Huh? Why should Py3k be relevant to ideas that shouldn't go in either way? I was thinking in terms of the Py3k development process. This list, early test versions of Py3k (presumably there will be a release of py3k prior to 2008), etc. > > with the possible inclusion of things that *should* make > > life better for Python users. With that said, aside from the stdlib > > reorganization (which should happen in the 2.x series anyways), so far I > > haven't seen anything that necessitates backwards incompatible changes > > to Python 3k, and I predict that many of the changes to py3k will be > > included into mainline 2.x during 3k's development. > > Then you haven't been paying attention. keys() returning a set view is > incompatible. A new I/O stack is incompatible. All-Unicode strings are > incompatible. Etc. It seems you are right (I was paying attention, but apparently to the wrong stuff); after going through most of the py3k list history, those changes that would have been backwards compatible were basically rejected. However, that doesn't necessarily mean that all proposed changes will necessarily be so. - Josiah From guido at python.org Wed Apr 19 19:22:24 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 18:22:24 +0100 Subject: [Python-3000] genexp syntax for reduce In-Reply-To: <444665AB.9040604@gmail.com> References: <4444D639.6030108@batiment71.net> <444665AB.9040604@gmail.com> Message-ID: On 4/19/06, Boris Borcic wrote: > Guido van Rossum wrote: > > > I wouldn't recommend this approach to anyone. > > What about a variation ? I haven't seen any convincing argument showing that this feature is useful or that the code becomes more readable when this feature is used. Without such arguments it doesn't matter whether you can come up with unambiguous syntax or anything else. Personally, I believe that all code involving reduce() can be made more readable by removing the reduce() call and writing an explicit loop. Your attempts seem to confirm that alternatives aren't any better. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From andy at neotitans.com Wed Apr 19 19:41:48 2006 From: andy at neotitans.com (Andy Sy) Date: Thu, 20 Apr 2006 01:41:48 +0800 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4445DA36.9080509@canterbury.ac.nz> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> Message-ID: <4446765C.2030306@neotitans.com> Greg Ewing wrote: > There's a lot more to this than syntax. The oddities > surrounding Python generators are mostly due to their > "one-level-deep" nature, i.e. they're not full coroutines. > And there are deep implementation reasons for that. Does this mean that Py3K intends to reuse major portions of Python 2.x's implementation? If Py3K intends to break backwards compatibility, doesn't this mean that if a simpler and more inclusive, more general asynchronous mechanism than generators is possible, then there should be no qualms about dropping or supplementing the latter? Or would this be enough of a difference that it would make for 'a different language'? I had the impression that replacing old abstractions that have acquired cruft with better new ones was one of the main goals of Py3K. Not that I don't find generators cool, but there seem to be some pretty basic asynchronous stuff (blocking on I/O) that they do not address as elegantly as one would like. From jimjjewett at gmail.com Wed Apr 19 19:41:46 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 19 Apr 2006 13:41:46 -0400 Subject: [Python-3000] Futures in Python 3000 (was Re: mechanism for handling asynchronous concurrency) In-Reply-To: <4445DA36.9080509@canterbury.ac.nz> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> Message-ID: On 4/19/06, Greg Ewing wrote: > If syntax is all you're concerned about, you could translate > that into Python as something like > f = url(future(fetch)) Are you sure it wouldn't be f = Future(fetch, url) a bit like result = ActiveObject(function, args) > Now, how is that future() function going to be implemented, > again? :-) where the ActiveObject could be a thread, or a twisted Deferred, or ... I'm not sure that was quite ever resolved. -jJ From ianb at colorstudy.com Wed Apr 19 19:40:34 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 19 Apr 2006 12:40:34 -0500 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> <444611D6.1080500@gmail.com> Message-ID: <44467612.1080106@colorstudy.com> Guido van Rossum wrote: > On 4/19/06, Nick Coghlan wrote: > >>Guido van Rossum wrote: >> >>>Here's a related but more complicated wish: define a function in such >>>a way that certain parameters *must* be passed as keywords, *without* >>>using *args or **kwds. This may require a new syntactic crutch. >> >>A single '*' could indicate that no additional positional arguments were >>permitted. >> >>def f(*, paramA, paramB=10): >> print paramA, paramB >> >>Here paramA and paramB can only be passed as keywords, and paramA *has* to be >>passed as it has no default. > > > I once considered and rejected this syntax since another logical > interpretation would be that any positional arguments are accepted but > *ignored*. While I guess I can understand why that might be interpreted that way, such an interpretation also assume the feature is pretty worthless; why would you ignore the arguments, instead of just ignore the variable the arguments were put into? >>Utterly magical to anyone not already familiar with the use of *args, though. > > > But so would any other newly invented syntax, probably, so that > doesn't count heavily. > > Would *None be too bizarre? Hmm... def f(*(), paramA, paramB=10): *() would kind of imply you have to unpack the positional parameters into (), and of course the only thing that can be unpacked to () is (). Kind of building on the idea that f(*(a, b)) and f(a, b) are equivalent calls, even though in a signature *(a, b) isn't actually allowed (being pointless anyway). But while it has some logical meaning, I don't know if it reads particularly well; too many ()'s, and most people don't encounter empty tuples anyway, and probably don't think of signatures as packing and unpacking. * alone and *None read better to me. * by itself feels most like it is splitting the positional arguments from the keyword arguments: def write_text(s, *, font=None, size=None, color=None, ...): ... The primary use case I see is with a method that has lots of keyword arguments (4+), at which point no one is going to remember the order of the arguments, including people implementing functions with the same signature. With a lot of keyword arguments, **(font=None, size=None, ...) (noted in another message) doesn't feel as nice... though actually as I think about it, maybe that's not true, maybe it does look just as nice. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From aahz at pythoncraft.com Wed Apr 19 19:59:42 2006 From: aahz at pythoncraft.com (Aahz) Date: Wed, 19 Apr 2006 10:59:42 -0700 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4446765C.2030306@neotitans.com> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> <4446765C.2030306@neotitans.com> Message-ID: <20060419175942.GA4784@panix.com> On Thu, Apr 20, 2006, Andy Sy wrote: > Greg Ewing wrote: >> >> There's a lot more to this than syntax. The oddities >> surrounding Python generators are mostly due to their >> "one-level-deep" nature, i.e. they're not full coroutines. >> And there are deep implementation reasons for that. > > Does this mean that Py3K intends to reuse major portions of > Python 2.x's implementation? Absolutely! In fact, the major push in Py3K is to remove portions of code (such as classic classes). -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From jcarlson at uci.edu Wed Apr 19 20:25:10 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 19 Apr 2006 11:25:10 -0700 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4446765C.2030306@neotitans.com> References: <4445DA36.9080509@canterbury.ac.nz> <4446765C.2030306@neotitans.com> Message-ID: <20060419111247.A832.JCARLSON@uci.edu> Andy Sy wrote: > Greg Ewing wrote: > > There's a lot more to this than syntax. The oddities > > surrounding Python generators are mostly due to their > > "one-level-deep" nature, i.e. they're not full coroutines. > > And there are deep implementation reasons for that. > > Does this mean that Py3K intends to reuse major portions of > Python 2.x's implementation? Aahz just answered this. > If Py3K intends to break backwards compatibility, doesn't this > mean that if a simpler and more inclusive, more general > asynchronous mechanism than generators is possible, then there > should be no qualms about dropping or supplementing the latter? > Or would this be enough of a difference that it would make for > 'a different language'? From the discussion I've been paying attention to over the last few years, *some* parts of Py3k will be backwards incompatible. The question really is whether or not Io or stackless style continuations are better than what currently exists. In my experience, I find generators to be quite easy to write, use, modify, and read. Without a sample syntax, execution semantic, and/or implementation (as Guido has already asked for), it is quite difficult for us to measure the positive or negative change (in readability, writability, understandability, etc.) over what we already have. - Josiah From mccollum at fas.harvard.edu Wed Apr 19 20:42:21 2006 From: mccollum at fas.harvard.edu (Andrew McCollum) Date: Wed, 19 Apr 2006 14:42:21 -0400 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: Message-ID: <200604191842.k3JIgKXo031194@us25.unix.fas.harvard.edu> On 4/17/06, Guido van Rossum wrote: > Here's a related but more complicated wish: define a function in such > a way that certain parameters *must* be passed as keywords, *without* > using *args or **kwds. This may require a new syntactic crutch. I don't know if this counts as "not using *args or **kwds", but why can't this simply be done with a decorator? Here's a (tested) example: """ class KeywordError(TypeError): pass def require_keywords(*keywords): def check_keywords(f): def new_f(*args, **kwds): for k in keywords: if k not in kwds: raise KeywordError("%s() requires the keyword " "argument '%s'" % (f.func_name, k)) return f(*args, **kwds) new_f.func_name = f.func_name new_f.func_doc = f.func_doc new_f.func_defaults = f.func_defaults return new_f return check_keywords @require_keywords('font', 'size', 'color') def write_text(s, font=None, size=None, color=None): pass @require_keywords('paramA') def example(paramA, paramB=10): print paramA, paramB """ Here's what happens when you call these functions: """ >>> example(paramA=5) 5 10 >>> example(paramB=5) KeywordError: example() requires the keyword argument 'paramA' >>> example(5, 10) KeywordError: example() requires the keyword argument 'paramA' >>> example(5, paramA=5) TypeError: example() got multiple values for keyword argument 'paramA' """ As you can see, this method still prevents double-passing the arguments. The major disadvantage that I can see is that the function signature of the decorated function becomes (*args, **kwds). By the way, this is my first post to this list, so hello everybody. -Andrew McCollum From tim.peters at gmail.com Wed Apr 19 21:07:50 2006 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 19 Apr 2006 15:07:50 -0400 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: <1f7befae0604191207o2f14551g3c2977be7d8cd88d@mail.gmail.com> [Greg Ewing] >> Even if it [mark-sweepish gc] did become available, we might not >> want to> use it. In recent times I've come round to the view that, >> on modern architectures where cacheing is all-important, >> refcounting + cyclic garbage collection may well be >> *better* than mark-and-sweep or some variation thereof. [Guido] > Not that I disagree -- do you have specific data or reasoning to back > this up? I'd love to hear that we were right all the time! :-) It's always :-) been very easy to see why in simple examples, like: s = 0 for i in xrange(1000000): s += i Current CPython (re)uses just a few int objects, which are almost certain to remain in L1 cache for the duration. Pure mark-sweep waits "until RAM is exhausted", and then crawls over that entire giant blob of address space at least once more to release it. CPython puts enormous pressure on dynamic-memory throughput because _everything_ comes from dynamic memory, even microscopically short-lived integers and stack frames. Many other languages don't, and it matters. I noted this in public in 1999, after Neal Schemenauer tried replacing all of Python's gc with BDW mark-sweep, and reported: I started with Sam Rushing's patch and modified it for Python 1.5.2c. To my surprise, removing the reference counting (Py_INCREF, Py_DECREF) actually slowed down Python a lot (over two times for pystone). That thread should still be required reading for anyone thinking of changing CPython's gc strategy: http://mail.python.org/pipermail/python-list/1999-July/0073.html Note that he got much better performance by leaving refcounting in but using BDW for cycle collection (and also note the pain required to get BDW not to erroneously free memory allocated in Tkinter.c). OTOH, I haven't had bandwidth in years to pay any attention to what other language implementations are doing for gc, and working from ignorance of non-Python practice isn't quite as crushingly convincing as I'd like :-) From ianb at colorstudy.com Wed Apr 19 21:23:29 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 19 Apr 2006 14:23:29 -0500 Subject: [Python-3000] Special object to return from functions that return only None In-Reply-To: References: <44459540.1040709@comcast.net> Message-ID: <44468E31.1080304@colorstudy.com> Guido van Rossum wrote: > On 4/19/06, Edward C. Jones wrote: > >>Wild idea. >> >>When I was a newbie, I repeatedly make the mistake of writing >> >>alist = alist.sort() >> >>I suggest a singleton object "UseForbidden" ("Py_UseForbidden" in C). >>"UseForbidden" can be used only to return from a function. Any other >>attempt to use "UseForbidden" raises an exception. The incorrect code >>above would give the error message "cannot assign to result of function". > > > Apart from assignment, None already has very few methods, so it > already has this purpose. > > Trapping this on assignment would require *every* assignment (and > argument passing, and who knows what else) to pay for the overhead for > an additional specific check. I don't think that's feasible. It also makes pass-through decorators and delegates harder. At least if you make it any stronger than what None already is. None doesn't do anything, but you can pass it around. You could disallow the passing around of a UseForbidden object, but then you need special ways of saying things like "run this function, give me the return value, and really this time it's okay if it is a UseForbidden object". -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ferringb at gmail.com Wed Apr 19 21:29:10 2006 From: ferringb at gmail.com (Brian Harring) Date: Wed, 19 Apr 2006 12:29:10 -0700 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <1f7befae0604191207o2f14551g3c2977be7d8cd88d@mail.gmail.com> References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> <1f7befae0604191207o2f14551g3c2977be7d8cd88d@mail.gmail.com> Message-ID: <20060419192909.GA29351@nightcrawler> On Wed, Apr 19, 2006 at 03:07:50PM -0400, Tim Peters wrote: > and it matters. I noted this in public in 1999, after Neal > Schemenauer tried replacing all of Python's gc with BDW mark-sweep, > and reported: > > I started with Sam Rushing's patch and modified it for Python > 1.5.2c. To my surprise, removing the reference counting > (Py_INCREF, Py_DECREF) actually slowed down Python a lot (over > two times for pystone). > > That thread should still be required reading for anyone thinking of > changing CPython's gc strategy: > > http://mail.python.org/pipermail/python-list/1999-July/0073.html Seem to have missed a few digits in the url... http://mail.python.org/pipermail/python-list/1999-July/007623.html ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060419/5aab42f0/attachment.pgp From skip at pobox.com Wed Apr 19 21:45:05 2006 From: skip at pobox.com (skip at pobox.com) Date: Wed, 19 Apr 2006 14:45:05 -0500 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <1f7befae0604191207o2f14551g3c2977be7d8cd88d@mail.gmail.com> References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> <1f7befae0604191207o2f14551g3c2977be7d8cd88d@mail.gmail.com> Message-ID: <17478.37697.379719.987480@montanaro.dyndns.org> Tim> That thread should still be required reading for anyone thinking of Tim> changing CPython's gc strategy: Tim> http://mail.python.org/pipermail/python-list/1999-July/0073.html I think its article number may have changed in the last few years. I found what I think you referred to here: http://mail.python.org/pipermail/python-list/1999-July/007623.html FYI... Skip From tim.peters at gmail.com Wed Apr 19 21:46:56 2006 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 19 Apr 2006 15:46:56 -0400 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <20060419192909.GA29351@nightcrawler> References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> <1f7befae0604191207o2f14551g3c2977be7d8cd88d@mail.gmail.com> <20060419192909.GA29351@nightcrawler> Message-ID: <1f7befae0604191246o7219c088gebb7c861572d930@mail.gmail.com> [Tim Peters] ... >>That thread should still be required reading for anyone thinking of >> changing CPython's gc strategy: >> >> http://mail.python.org/pipermail/python-list/1999-July/0073.html [Brian Harring] > Seem to have missed a few digits in the url... > > http://mail.python.org/pipermail/python-list/1999-July/007623.html Indeed! Since it was just copy+paste from a Firefox address bar, dropping two digits "in the middle" may seem inexplicable. But I'm not fooled, and now that Guido works at Google perhaps they'll fix this egregious gmail bug ;-) From bborcic at gmail.com Wed Apr 19 22:06:50 2006 From: bborcic at gmail.com (Boris Borcic) Date: Wed, 19 Apr 2006 22:06:50 +0200 Subject: [Python-3000] genexp syntax / lambda Message-ID: <77f4ed2b0604191306i3b9ebf56s5fd4964c2b326cee@mail.gmail.com> On 4/19/06, Guido van Rossum wrote: > ... I'll be perfectly happy to leave the matter (and the mailing list) at that - once said that your complete wording left an unfortunate doubt : it maybe reflects my failure to adjust the subject of the thread to a proposal and examples that hadn't as main/only target, equivalents to reduce(). > I haven't seen any convincing argument showing that this feature is > useful or that the code becomes more readable when this feature is > used [...] BTW, what about replacing the geeky and arguably lengthy keyword "lambda" with either : given - much clearer and slightly more concise; or ? - slightly clearer and much more concise ? Boris Borcic -- assert "304" in "340343", "P424D15E M15M47CH" From steven.bethard at gmail.com Wed Apr 19 23:01:26 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Wed, 19 Apr 2006 15:01:26 -0600 Subject: [Python-3000] genexp syntax / lambda In-Reply-To: <77f4ed2b0604191306i3b9ebf56s5fd4964c2b326cee@mail.gmail.com> References: <77f4ed2b0604191306i3b9ebf56s5fd4964c2b326cee@mail.gmail.com> Message-ID: On 4/19/06, Boris Borcic wrote: > BTW, what about replacing the geeky and arguably lengthy keyword > "lambda" with either : > > given - much clearer and slightly more concise; or > ? - slightly clearer and much more concise ? Guido's already said that lambda will stay as it is. We don't need to start this thread again. If you really like "given", you can just apply Talin's patch: http://bugs.python.org/1434008 Steve -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From tjreedy at udel.edu Wed Apr 19 23:42:01 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Apr 2006 17:42:01 -0400 Subject: [Python-3000] Cleaning up argument list parsing (was Re: Morewishful thinking) References: <44430CB5.5020907@gmail.com><444354C4.1010701@gmail.com><444611D6.1080500@gmail.com> Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604190650x39efc500p503807aa757c25a3 at mail.gmail.com... > I once considered and rejected this syntax since another logical > interpretation would be that any positional arguments are accepted but > *ignored*. I was about to suggest that perhaps this is how it should be interpreted ;-). Perhaps it is too much to use syntax to indicate whether positional args should be silently ignored or raise an exception. Given def f(*ignored, a, b='yes'): positional args will be ignored if 'ignored' if never referenced again. To raise an exception, just add if ignored: raise TypeErrror("Positional args not allowed") Given that keyword only functions should be fairly rare (given that we have survived thus long without), is a syntactic abbreviation needed? Terry Jan Reedy From guido at python.org Wed Apr 19 23:48:52 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 19 Apr 2006 22:48:52 +0100 Subject: [Python-3000] Cleaning up argument list parsing (was Re: Morewishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> <444354C4.1010701@gmail.com> <444611D6.1080500@gmail.com> Message-ID: On 4/19/06, Terry Reedy wrote: > > "Guido van Rossum" wrote in message > news:ca471dc20604190650x39efc500p503807aa757c25a3 at mail.gmail.com... > > I once considered and rejected this syntax since another logical > > interpretation would be that any positional arguments are accepted but > > *ignored*. > > I was about to suggest that perhaps this is how it should be interpreted > ;-). > > Perhaps it is too much to use syntax to indicate whether positional args > should be silently ignored or raise an exception. Given > > def f(*ignored, a, b='yes'): > > positional args will be ignored if 'ignored' if never referenced again. To > raise an exception, just add > > if ignored: raise TypeErrror("Positional args not allowed") > > Given that keyword only functions should be fairly rare (given that we have > survived thus long without), is a syntactic abbreviation needed? You're right. This currently doesn't work because *args must be last (except for **kwds). but we're already considering an addition that allows kwd=default following *args; an explicit check for args==() is easy enough then. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tjreedy at udel.edu Wed Apr 19 23:50:08 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Apr 2006 17:50:08 -0400 Subject: [Python-3000] Type Expressions References: <44458B7A.9060208@canterbury.ac.nz> Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604190347k473b96e8s2ee266b3098423f at mail.gmail.com... > On 4/19/06, Greg Ewing wrote: >> Py3k suggestion: make ... a generally legal expression. > > Yup. An addition for PEP 3100 under core language. From tjreedy at udel.edu Wed Apr 19 23:54:21 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Apr 2006 17:54:21 -0400 Subject: [Python-3000] After 2.6 should be 3.0 References: <20060418200406.A81D.JCARLSON@uci.edu> <20060419091913.A82C.JCARLSON@uci.edu> Message-ID: "Josiah Carlson" wrote in message news:20060419091913.A82C.JCARLSON at uci.edu... > It seems you are right (I was paying attention, but apparently to the > wrong stuff); after going through most of the py3k list history, those > changes that would have been backwards compatible were basically > rejected. That is because numerous more-or-less non-controversial code-breaking changes have already been approved and listed in PEP 3100 http://www.python.org/dev/peps/pep-3100/ many before this list started. Now we are more thoroughly discussing and getting pronouncements on things not approved before. Terry Jan Reedy From ianb at colorstudy.com Wed Apr 19 23:54:26 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 19 Apr 2006 16:54:26 -0500 Subject: [Python-3000] Cleaning up argument list parsing (was Re: Morewishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com><444354C4.1010701@gmail.com><444611D6.1080500@gmail.com> Message-ID: <4446B192.8030405@colorstudy.com> Terry Reedy wrote: > Given that keyword only functions should be fairly rare (given that we have > survived thus long without), is a syntactic abbreviation needed? It's quite common to see documentation warn API users only to call a function with keyword arguments, so there's definitely a desire. But certainly with the already-agreed-upon keywords-after-*args it would be easy to implement this with one assert, where now it is quite tedious. I don't know if there's an advantage to presenting a signature that accurately describes the function; it can never *really* be accurate in all cases, as there can be all sorts of constraints that are impossible to represent in the signature (like mutually exclusive keyword arguments). But then by putting in *args to constrain arguments to keywords, you formally represent the keyword constraint while adding a very inaccurate representation of the function signature for positional arguments. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From lcaamano at gmail.com Thu Apr 20 00:13:46 2006 From: lcaamano at gmail.com (Luis P Caamano) Date: Wed, 19 Apr 2006 18:13:46 -0400 Subject: [Python-3000] GIL Message-ID: It seems that python-3000 is more about language issues than CPython issues. Still, I'll ask the question just to nip it in the bud early: Any possibility of working on the CPython GIL and MP problems in python-3000? The reason I ask is because most solutions might involve incompatibilities with C extensions that rely on the GIL and since python 3000 is not promising any backward compatibility, then it seems this might be a good opportunity to tackle that issue too. I'm not proposing any solution, just trying to figure out whether it's a valid issue or not. -- Luis P Caamano Atlanta, GA USA From brett at python.org Thu Apr 20 00:39:38 2006 From: brett at python.org (Brett Cannon) Date: Wed, 19 Apr 2006 15:39:38 -0700 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> Message-ID: [forgot to cc: python-3000; sorry for the dup, Terry] On 4/19/06, Terry Reedy wrote: > > "Guido van Rossum" wrote in message > news:ca471dc20604190347k473b96e8s2ee266b3098423f at mail.gmail.com... > > On 4/19/06, Greg Ewing wrote: > >> Py3k suggestion: make ... a generally legal expression. > > > > Yup. > > An addition for PEP 3100 under core language. Done in rev. 45572. -Brett From brett at python.org Thu Apr 20 00:44:22 2006 From: brett at python.org (Brett Cannon) Date: Wed, 19 Apr 2006 15:44:22 -0700 Subject: [Python-3000] GIL In-Reply-To: References: Message-ID: On 4/19/06, Luis P Caamano wrote: > It seems that python-3000 is more about language issues than CPython > issues. Still, I'll ask the question just to nip it in the bud early: > > Any possibility of working on the CPython GIL and MP problems in python-3000? > > The reason I ask is because most solutions might involve > incompatibilities with C extensions that rely on the GIL and since > python 3000 is not promising any backward compatibility, then it seems > this might be a good opportunity to tackle that issue too. > > I'm not proposing any solution, just trying to figure out whether it's > a valid issue or not. I just know this is going to fall under Guido's "code first, then we consider" rule. The GIL is not currently going to go anywhere without a thorough rearchitecture of the interpreter and that is not planned for Python 3000. Plus most people here, based on the last time this came up, did not feel like the GIL was that big of a problem. A good IPC/RPC mechanism would be much more appreciated. -Brett From tjreedy at udel.edu Thu Apr 20 01:01:36 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Apr 2006 19:01:36 -0400 Subject: [Python-3000] Python 3000 and the Google Summer of Code References: <20060419131914.GA15589@panix.com> Message-ID: "Aahz" wrote in message news:20060419131914.GA15589 at panix.com... > [Neal asked me to post this nitpick in case someone else posts the > announcement] Thank you. Really. > Either s/between/from/ or s/through/and/ Yes. Late night continuity disconnect. > Python: because we care about language nitpicks ;-) > > Actually, looking at that, I have one more nitpick. It's a Bad Idea IMO > to put a period immediately following an URL. Agreed. Not every reader tolerates puntuation in detecting links like Outlook Express does. > I would put the URL on its own line with no punctuation. Agreed. One newline disappeared between post and your reader (but not mine), so I put a space before and after each to make sure. See revision in my soon-to-be-sent response to Neal. Terry Jan Reedy From ianb at colorstudy.com Thu Apr 20 01:10:09 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 19 Apr 2006 18:10:09 -0500 Subject: [Python-3000] GIL In-Reply-To: References: Message-ID: <4446C351.7040902@colorstudy.com> Brett Cannon wrote: > I just know this is going to fall under Guido's "code first, then we > consider" rule. The GIL is not currently going to go anywhere without > a thorough rearchitecture of the interpreter and that is not planned > for Python 3000. Plus most people here, based on the last time this > came up, did not feel like the GIL was that big of a problem. A good > IPC/RPC mechanism would be much more appreciated. IPC/RPC would be an great Summer of Code project, if there was a mentor interested. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From tjreedy at udel.edu Thu Apr 20 01:28:59 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Apr 2006 19:28:59 -0400 Subject: [Python-3000] Python 3000 and the Google Summer of Code References: Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604190418x4548c45fp410d5df3e9ec4ced at mail.gmail.com... > One comment: I would add explicitly to the subject that the PSF is > currently requesting mentors, not students. Perhaps > > Subject: Python Software Foundation looking for mentors for the > Google Summer of Code > > would work? Given that students' proposals are due in a one week window beginning in 11 days, and that they therefore need to start preparing *soon*, do you really want to make that restriction? Or are you thinking of a second announcement (soon)? > How widely are you sending this? I'd say c.l.py.a, c.l.py, python-dev, > IronPython, Jython, Zope lists. That is for you or Neal to decide, not me. My involvement with this began with an offer to forward an announcement to gmane.comp.python.*.devel newsgroup lists *if* you or Neal wished it. But I forget that there is c.l.p.a for people that don't read c.l.p but want occasional important announcements. In any case, unless you want it only sent to the above lists, I am now thinking it would be better anyway to just add a line to the bottom inviting recipients to share on other forums where they have involvement. If that were done, then I, for instance, would forward to the pygame list which I read and occasionally post to. My latest revision is attached to my response to Neal. Terry Jan Reedy From tjreedy at udel.edu Thu Apr 20 01:30:41 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 19 Apr 2006 19:30:41 -0400 Subject: [Python-3000] Python 3000 and the Google Summer of Code References: Message-ID: "Neal Norwitz" wrote in message news:ee2a432c0604191011h313ad271h1a317794ef7c98f3 at mail.gmail.com... > Terry, if you can post this during the day, please go ahead and do so > and incorporate Guido's and Aahz's comments. My revision is both pasted below and attached as a text file. I am not posting because I prefer you send the original after one more check and because I see two open questions: to intent and title; distribution and a possible added line. See my response to Guido. So decide, revise if necessary, sign, and send. Also, I do not have access to Jython or IronPython lists. Terry Jan Reedy Google 2006 Summer of Code and Python Software Foundation This spring and summer, Google corporation will again provide several hundred stipends for students (18+, undergraduate thru PhD programs) to write new open-source code. The main page is http://code.google.com/summerofcode.html At the bottom are links to StudentFAQ, MentorFAQ, and TermsOfService. The first two have the timeline. Note that student applications are due between May 1, 17:00 PST and May 8, 17:00 PST. The Python Software Foundation (PSF) http://www.python.org/psf/ will again act as a sponsoring organization matching mentors and projects benefiting Python and Python programmers. Besides Python implementationns, projects can include work on programmer utilitilies, libraries, packages, or frameworks related to Python. Project ideas are welcome and can be added to http://wiki.python.org/moin/SummerOfCode possibly as links to fuller explanations elsewhere. The projects from last summer are described at http://wiki.python.org/moin/SummerOfCode/2005 People interested in mentoring a student though PSF are encouraged to >> contact Neal Norwitz at nnorwitz at gmail.com. People unknown to Neal or Guido should find a couple of people known within the Python community that are willing to act as references begin 666 soc.txt M1V]O9VQE(#(P,#8 at 4W5M;65R(&]F($-O9&4 at 86YD(%!Y=&AO;B!3;V9T=V%R M92!&;W5N9&%T:6]N#0H-"E1H:7,@71H;VX at 4V]F='=A71H;VX@<')O9W)A;6UE M71H;VXN("!0B!A="!N M;F]R=VET>D!G;6%I;"YC;VTN("!096]P;&4@=6YK;F]W;B!T;R!.96%L(&]R M($=U:61O('-H;W5L9"!F:6YD(&$@8V]U<&QE(&]F('!E;W!L92!K;F]W;B!W M:71H:6X@=&AE(%!Y=&AO;B!C;VUM=6YI='D@=&AA="!A <44430CB5.5020907@gmail.com><444354C4.1010701@gmail.com><444611D6.1080500@gmail.com> <4446B192.8030405@colorstudy.com> Message-ID: "Ian Bicking" wrote in message news:4446B192.8030405 at colorstudy.com... > I don't know if there's an advantage to presenting a signature that > accurately describes the function; it can never *really* be accurate in > all cases, as there can be all sorts of constraints that are impossible > to represent in the signature (like mutually exclusive keyword > arguments). On the contrary ;-), we could add a parameter operator !^!, used as def f(*prohibited, a=None !^! b=None): blah to make the keywords exclusive and automatically raise an exception if both are given values, or perhaps if both are given values other than, in this case, None. But do we really want to substitute signature syntax for body code to do this? > But then by putting in *args to constrain arguments to > keywords, you formally represent the keyword constraint while adding a > very inaccurate representation of the function signature for positional > arguments. But I intentionally did not use '*args' but instead used '*ignored' in my example and also suggest '*prohibited' as a convention. Either should be clear enough to a code reader if used as consistently as 'self'. Doc strings should just say 'positional args prohibited'. Introspection code that generates alternate signature representations from func and code attributes could be made convention aware. Terry Jan Reedy From janssen at parc.com Thu Apr 20 03:58:29 2006 From: janssen at parc.com (Bill Janssen) Date: Wed, 19 Apr 2006 18:58:29 PDT Subject: [Python-3000] GIL In-Reply-To: Your message of "Wed, 19 Apr 2006 16:10:09 PDT." <4446C351.7040902@colorstudy.com> Message-ID: <06Apr19.185833pdt."58633"@synergy1.parc.xerox.com> It would be interesting to bring ILU up-to-date. Get the ILU C library compiling nicely on POSIX (should be trivial), then bring the Python bindings up-to-date. That would, once again, give us ONC RPC and CORBA compatibility, both IPC and RPC. Bill > Brett Cannon wrote: > > I just know this is going to fall under Guido's "code first, then we > > consider" rule. The GIL is not currently going to go anywhere without > > a thorough rearchitecture of the interpreter and that is not planned > > for Python 3000. Plus most people here, based on the last time this > > came up, did not feel like the GIL was that big of a problem. A good > > IPC/RPC mechanism would be much more appreciated. > > IPC/RPC would be an great Summer of Code project, if there was a mentor > interested. > > > -- > Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/janssen%40parc.com From nnorwitz at gmail.com Thu Apr 20 04:21:17 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Wed, 19 Apr 2006 19:21:17 -0700 Subject: [Python-3000] Function call speed (Was: Cleaning up argument listparsing) In-Reply-To: References: <79990c6b0604171242w7447443eic6d80990ee29e46f@mail.gmail.com> Message-ID: On 4/19/06, Guido van Rossum wrote: > > I recommend profiling. I seem to recall that last time we did this, we > found that creating the frame object is a significant cost factor. That's part of it. Plus there's a tuple allocation (and copy). n From talin at acm.org Thu Apr 20 05:22:20 2006 From: talin at acm.org (Talin) Date: Thu, 20 Apr 2006 03:22:20 +0000 (UTC) Subject: [Python-3000] PEP - string.format Message-ID: I decided to take some of the ideas discussed in the string formatting thread, add a few touches of my own, and write up a PEP. http://viridia.org/python/doc/PEP_AdvancedStringFormatting.txt (I've also submitted the PEP via the normal channels.) I don't have a prototype, however the PEP is explicit enough about the details of operation that a prototype should be fairly straightforward. -- Talin From aahz at pythoncraft.com Thu Apr 20 05:54:10 2006 From: aahz at pythoncraft.com (Aahz) Date: Wed, 19 Apr 2006 20:54:10 -0700 Subject: [Python-3000] [Python-Dev] Raising objections In-Reply-To: <200604201039.29138.anthony@interlink.com.au> References: <4445D80E.8050008@canterbury.ac.nz> <4446776E.5090702@v.loewis.de> <200604201039.29138.anthony@interlink.com.au> Message-ID: <20060420035410.GA1127@panix.com> [major contextectomy of a python-dev post -- despite the header, this is now the python-3000 list, not python-dev] On Thu, Apr 20, 2006, Anthony Baxter wrote: > > urllib->urllib2 That reminds me. Anyone want to write a PEP for merging urllib2 into urllib for Python 3.0? More precisely, we should probably just rename urllib2 to urllib and copy the utility functions from urllib. Any other modules that need similar treatment? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "LL YR VWL R BLNG T S" From andy at neotitans.com Thu Apr 20 08:41:27 2006 From: andy at neotitans.com (Andy Sy) Date: Thu, 20 Apr 2006 14:41:27 +0800 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <20060419111247.A832.JCARLSON@uci.edu> References: <4445DA36.9080509@canterbury.ac.nz> <4446765C.2030306@neotitans.com> <20060419111247.A832.JCARLSON@uci.edu> Message-ID: <44472D17.9010305@neotitans.com> Josiah Carlson wrote: > Andy Sy wrote: >> Does this mean that Py3K intends to reuse major portions of >> Python 2.x's implementation? > > Aahz just answered this. > From the discussion I've been paying attention to over the last few > years, *some* parts of Py3k will be backwards incompatible. > > In my experience, I find generators to be quite easy to write, use, > modify, and read. Without a sample syntax, execution semantic, and/or > implementation (as Guido has already asked for), it is quite difficult > for us to measure the positive or negative change (in readability, > writability, understandability, etc.) over what we already have. Err... thank you once again for being redundant, reiterating what is already pretty much dead clear to everyone and saying nothing new in general. * Yes, Aahz DID answer my question regarding reuse of Python 2.x implementation in Py3K * Yes, Guido DID ask for a prototype implementation for anyone who would lobby for a new async mechanism * Yes, generators are quite easy to write, use, modify and read (once you get the concept), except I don't see how you can easily use them to implement non-blocking I/O in their current state (I believe I've already mentioned that twice)... Let's see... anything else I've missed? Oh yeah... * _Some_ parts of Py3K will be backwards incompatible Could anyone be possibly more condescending as to have to repeat that... ??? You must think other people pay even less attention to things than you do. > The question really is whether or not Io or stackless style > continuations are better than what currently exists. Actually... it's whether or not a futures-inspired abstraction would make for a more pythonic asynchronous mechanism compared to an enhanced generator as described in PEP 342. Nothing to do with Io or Stackless per se. I guess you weren't really paying attention that time too. Feh. From erno at iki.fi Thu Apr 20 09:15:08 2006 From: erno at iki.fi (Erno Kuusela) Date: 20 Apr 2006 10:15:08 +0300 Subject: [Python-3000] Is reference counting still needed? References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: Greg Ewing writes: > Edward C. Jones wrote: > > > Maybe some day a sufficiently portable automatic garbage collector will > > be available for C. Until then, we'll have to live with reference counts. > > Even if it did become available, we might not want to > use it. In recent times I've come round to the view that, > on modern architectures where cacheing is all-important, > refcounting + cyclic garbage collection may well be > *better* than mark-and-sweep or some variation thereof. The refcounting vs generational GC reasoning I've heard argues that refcounting is less cache-friendly: The cache line containing the refcount field of the pointed-to objects is dirtied (or at least loaded) every time something is done with the reference, even when the object itself is not accessed in any way. Generational GC is said to make the most frequent gc-related memory references somewhat localised. -- erno From nnorwitz at gmail.com Thu Apr 20 09:34:22 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 20 Apr 2006 00:34:22 -0700 Subject: [Python-3000] Python 3000 and the Google Summer of Code In-Reply-To: References: Message-ID: On 4/19/06, Terry Reedy wrote: > > My revision is both pasted below and attached as a text file. I am not > posting because I prefer you send the original after one more check and Thanks Terry. I sent it. Everyone feel free to spread the word. n From greg.ewing at canterbury.ac.nz Thu Apr 20 09:59:43 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 19:59:43 +1200 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4446765C.2030306@neotitans.com> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> <4446765C.2030306@neotitans.com> Message-ID: <44473F6F.9020809@canterbury.ac.nz> Andy Sy wrote: > Does this mean that Py3K intends to reuse major portions of > Python 2.x's implementation? I expect that almost all of it will be reused. The difficulty with improving on the current generator model is that the only way to do any better is to use full coroutines. That means either using threads (which you can do now if you want), or turning the whole implementation inside-out in the manner of the original Stackless, which would break all existing C extensions and force writers of future extensions to use a very convoluted API. If there were any easy way to improve generators, it would already have been done by now. -- Greg From greg.ewing at canterbury.ac.nz Thu Apr 20 10:08:46 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 20:08:46 +1200 Subject: [Python-3000] Theory of cooperative super calls (Re: auto-super()) In-Reply-To: <9e804ac0604190607g1395d3dbi87c004145a6d1bd4@mail.gmail.com> References: <443E7F47.9060308@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> <44462A51.6070709@canterbury.ac.nz> <9e804ac0604190607g1395d3dbi87c004145a6d1bd4@mail.gmail.com> Message-ID: <4447418E.1080400@canterbury.ac.nz> Thomas Wouters wrote: > I believe Guido is referring to > http://www.awl.com/cseng/titles/0-201-43305-2 Are there any published papers about this, or is that book the only source? -- Greg From greg.ewing at canterbury.ac.nz Thu Apr 20 10:19:16 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 20:19:16 +1200 Subject: [Python-3000] genexp syntax / lambda In-Reply-To: <77f4ed2b0604191306i3b9ebf56s5fd4964c2b326cee@mail.gmail.com> References: <77f4ed2b0604191306i3b9ebf56s5fd4964c2b326cee@mail.gmail.com> Message-ID: <44474404.9070600@canterbury.ac.nz> Boris Borcic wrote: > given - much clearer and slightly more concise; or > ? - slightly clearer and much more concise ? That would be replacing something that's familiar to at least some people (even if they are geeks) with something that's familiar to nobody. If you really want to replace it with something, 'proc' or 'func' would be better. Most geeks would be able to guess what they meant, I think. -- Greg From greg.ewing at canterbury.ac.nz Thu Apr 20 10:34:02 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 20:34:02 +1200 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: <4447477A.3080306@canterbury.ac.nz> Erno Kuusela wrote: > The refcounting vs generational GC reasoning I've heard argues that > refcounting is less cache-friendly: The cache line containing the > refcount field of the pointed-to objects is dirtied (or at least > loaded) every time something is done with the reference, Has anyone actually measured this effect in a real system, or is it just theorising? If it's a real effect, would this be helped at all if the refcounts weren't stored with the objects, but kept all together in one block of memory? Or would that just make things worse? With dynamic languages becoming increasingly important these days, I wonder whether anyone has thought about what sort of cache or other memory architecture modifications might make things like refcounting more efficient. Ideally one would like a memory architecture that is wired in such a way that it just "knows" which objects are in use, without needing any explicit refcounting or garbage collection algorithms. Haven't figured out how to build such a thing at reasonable cost, though... -- Greg From guido at python.org Thu Apr 20 10:44:23 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 09:44:23 +0100 Subject: [Python-3000] Changing the import machinery Message-ID: I'm changing the list and the subject, pulling this quote out of python-dev: On 4/20/06, in python-dev, Fredrik Lundh wrote: > I was hoping that for Python 3.0, we could get around to unkludge the > sys.path/meta_path/path_hooks/path_importer_cache big ball of hacks, > possibly by replacing sys.path with something a bit more intelligent than > a plain list. That's an excellent idea. Are there any volunteers here to help out? Even just listing specific use cases / scenarios that are currently difficult to solve right would be tremendously helpful. (I think that Phillip's and others' experience with setuptools might be very useful.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 20 10:49:07 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 09:49:07 +0100 Subject: [Python-3000] Theory of cooperative super calls (Re: auto-super()) In-Reply-To: <4447418E.1080400@canterbury.ac.nz> References: <443E7F47.9060308@colorstudy.com> <4442F264.7030307@canterbury.ac.nz> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> <44462A51.6070709@canterbury.ac.nz> <9e804ac0604190607g1395d3dbi87c004145a6d1bd4@mail.gmail.com> <4447418E.1080400@canterbury.ac.nz> Message-ID: On 4/20/06, Greg Ewing wrote: > Thomas Wouters wrote: > > > I believe Guido is referring to > > http://www.awl.com/cseng/titles/0-201-43305-2 > > Are there any published papers about this, or is that book > the only source? What can be more published than a book? :-) I'm sure the book has a long list of references; if someone has a copy handy perhaps they could help out here; my copy is at home and I'm not. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From aurelien.campeas at logilab.fr Thu Apr 20 10:57:50 2006 From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Thu, 20 Apr 2006 10:57:50 +0200 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <44473F6F.9020809@canterbury.ac.nz> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> <4446765C.2030306@neotitans.com> <44473F6F.9020809@canterbury.ac.nz> Message-ID: <20060420085750.GD7995@crater.logilab.fr> On Thu, Apr 20, 2006 at 07:59:43PM +1200, Greg Ewing wrote: > Andy Sy wrote: > > > Does this mean that Py3K intends to reuse major portions of > > Python 2.x's implementation? > > I expect that almost all of it will be reused. > Couldn't PyPy be considered an interesting contender ? > The difficulty with improving on the current generator > model is that the only way to do any better is to use > full coroutines. That means either using threads (which > you can do now if you want), or turning the whole > implementation inside-out in the manner of the original > Stackless, which would break all existing C extensions > and force writers of future extensions to use a very > convoluted API. Sorry to jump in like this, but what about the *current* stackless ? Does it also break all existing C extensions ... ? (I guess I should do my homework and ask the stackless people, but ...) > > If there were any easy way to improve generators, it > would already have been done by now. > > -- > Greg > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 From guido at python.org Thu Apr 20 11:06:47 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 10:06:47 +0100 Subject: [Python-3000] genexp syntax / lambda In-Reply-To: <44474404.9070600@canterbury.ac.nz> References: <77f4ed2b0604191306i3b9ebf56s5fd4964c2b326cee@mail.gmail.com> <44474404.9070600@canterbury.ac.nz> Message-ID: Renaming lambda is off the table. It would be change for change's sake. On 4/20/06, Greg Ewing wrote: > Boris Borcic wrote: > > > given - much clearer and slightly more concise; or > > ? - slightly clearer and much more concise ? > > That would be replacing something that's familiar to > at least some people (even if they are geeks) with > something that's familiar to nobody. > > If you really want to replace it with something, > 'proc' or 'func' would be better. Most geeks would > be able to guess what they meant, I think. > > -- > Greg > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 20 11:16:01 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 10:16:01 +0100 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: On 20 Apr 2006 10:15:08 +0300, Erno Kuusela wrote: > The refcounting vs generational GC reasoning I've heard argues that > refcounting is less cache-friendly: The cache line containing the > refcount field of the pointed-to objects is dirtied (or at least > loaded) every time something is done with the reference, even when the > object itself is not accessed in any way. Generational GC is said to > make the most frequent gc-related memory references somewhat > localised. I guess it depends on how refcounting is used. Python has many places where references are passed without manipulating the refcount. Also, the argument about dirtying cache lines seem to be relevant to SMP architectures and probably doesn't apply much to Python as long as we have the GIL (which is not going away). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Thu Apr 20 11:19:59 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 21:19:59 +1200 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> Message-ID: <4447523F.4030504@canterbury.ac.nz> Some wilder ideas for keyword-only arguments: def spam(a, b, c, {d, e, f=x}): # d, e are mandatory keyword-only # f is optional keyword-only Writing them inside {} is meant to make them look like a set, therefore unordered. Problem is it looks a little ugly when all you have is keyword-only args: def spam({d, e, f=x}): ... So maybe you should be able to write that as def spam{d, e, f=x}: ... Then if you have some positional args as well, def spam(a, b, c){d, e, f=x}: ... And incorporating all the possible features, def spam(a, b, c=y, *d){e, f=x, **g}: ... Although now that the ** arg is inside the {}, there's no longer any need for the double *, so it could just be def spam(a, b, c=y, *d){e, f=x, *g}: ... -- Greg From guido at python.org Thu Apr 20 11:20:21 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 10:20:21 +0100 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <44473F6F.9020809@canterbury.ac.nz> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> <4446765C.2030306@neotitans.com> <44473F6F.9020809@canterbury.ac.nz> Message-ID: On 4/20/06, Greg Ewing wrote: > The difficulty with improving on the current generator > model is that the only way to do any better is to use > full coroutines. That means either using threads (which > you can do now if you want), or turning the whole > implementation inside-out in the manner of the original > Stackless, which would break all existing C extensions > and force writers of future extensions to use a very > convoluted API. Perhaps more importantly, it would make it impossible (or at least hugely impractical or inefficient) to implement the feature in Jython or IronPython. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 20 11:23:33 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 10:23:33 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: <4447523F.4030504@canterbury.ac.nz> References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: On 4/20/06, Greg Ewing wrote: > Some wilder ideas for keyword-only arguments: > > def spam(a, b, c, {d, e, f=x}): > # d, e are mandatory keyword-only > # f is optional keyword-only IMO anything using any kind of nested brackets inside the argument list is doomed. Such syntax is too close to resembling a single argument with a complex sub-structure, even if there's a prefix operator. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Thu Apr 20 11:25:53 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Apr 2006 19:25:53 +1000 Subject: [Python-3000] Making strings non-iterable In-Reply-To: <1145462174.21198.147.camel@resist.wooz.org> References: <443E91E0.5020304@colorstudy.com> <4441D404.6030802@canterbury.ac.nz> <20060416052258.GA12059@panix.com> <4442E71A.2090501@canterbury.ac.nz> <1f7befae0604162025v3cfe1941qc4860f6e6797b27f@mail.gmail.com> <20060417032930.GB25577@panix.com> <44440210.9010201@ewtllc.com> <44441953.6090809@colorstudy.com> <20060417230805.GA23791@nightcrawler> <1145332823.21740.56.camel@geddy.wooz.org> <44460E4E.2010202@gmail.com> <1145462174.21198.147.camel@resist.wooz.org> Message-ID: <444753A1.5090907@gmail.com> Barry Warsaw wrote: > On Wed, 2006-04-19 at 20:17 +1000, Nick Coghlan wrote: > >> Unfortunately, the thread fizzled without generating any additional interest. >> I don't recall the topic really coming up since then. > > Maybe because the proposal got too complex while trying to be so > general? One way to simplify it would be to special case strings rather than provide a general stop list (this would be significantly simpler to do in C, too): from itertools import chain def walk(iterable, depth_first=True, iter_strings=False): itr = iter(iterable) while True: for item in itr: if isinstance(item, basestring): # We have a string # Don't iterate if flag is not set # Never iterate if length is only 1 if not iter_strings or len(item) == 1: yield item continue try: subitr = iter(item) except TypeError: yield item continue if depth_first: itr = chain(subitr, itr) else: itr = chain(itr, subitr) break # Restart the inner for loop else: break # Finish the outer while loop Py> seq = [['123', '456'], 'abc', 'abc', 'abc', 'abc', ['xyz']] Py> seq [['123', '456'], 'abc', 'abc', 'abc', 'abc', ['xyz']] Py> list(walk(seq)) ['123', '456', 'abc', 'abc', 'abc', 'abc', 'xyz'] Py> list(walk(seq, iter_strings=True)) ['1', '2', '3', '4', '5', '6', 'a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'x', 'y', 'z'] Py> list(walk(seq, depth_first=False)) ['abc', 'abc', 'abc', 'abc', '123', '456', 'xyz'] Py> list(walk(seq, depth_first=False, iter_strings=True)) ['a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', 'a', 'b', 'c', '1', '2', '3', '4', '5', '6', 'x', 'y', 'z'] -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From erno at iki.fi Thu Apr 20 11:39:08 2006 From: erno at iki.fi (Erno Kuusela) Date: 20 Apr 2006 12:39:08 +0300 Subject: [Python-3000] Is reference counting still needed? References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: "Guido van Rossum" writes: > Also, > the argument about dirtying cache lines seem to be relevant to SMP > architectures and probably doesn't apply much to Python as long as we > have the GIL (which is not going away). I guess the primary cost would probably be incurring the cache miss and resulting stall, but it does also pollute the cache a bit to have lots of 128 or 64-byte lines around for the single refcount field contained therein... But this is just armchair talk, some numbers would be nice. Maybe one could measure it by making the refcounting macros real non-inline functions and seeing how they show up in eg oprofile... -- erno From tim.peters at gmail.com Thu Apr 20 12:41:27 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 20 Apr 2006 06:41:27 -0400 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <4447477A.3080306@canterbury.ac.nz> References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> <4447477A.3080306@canterbury.ac.nz> Message-ID: <1f7befae0604200341i1cfa8dfcl22430e718b0d4b81@mail.gmail.com> [Erno Kuusela] >> The refcounting vs generational GC reasoning I've heard argues that >> refcounting is less cache-friendly: The cache line containing the >> refcount field of the pointed-to objects is dirtied (or at least >> loaded) every time something is done with the reference, [Greg Ewing] > Has anyone actually measured this effect in a real > system, or is it just theorising? Of course people have tried it in real systems. Then they write about it, and everyone gets confused <0.5 wink>. Efficiency of gc strategy is deeply dependent, in highly complex ways, on _many_ aspects of the system in question. It's a tempting but basically idiotic mistake to imagine, e.g., that a strategy that works well for LISP would also work well for CPython (or vice versa). > If it's a real effect, would this be helped at all if the > refcounts weren't stored with the objects, but kept all > together in one block of memory? Or would that just make > things worse? That's also been tried, and "it depends". An obvious thing about CPython is that you can't do anything non-trivial with an object O without reading up O's type pointer first -- but as soon as you do that, chances are high that you have O's refcount in L1 cache too, since the refcount is adjacent to the type pointer in the base PyObject struct, and that's true of _all_ objects in CPython. "Reading up the refcount" is essentially free then, given that you had to read up the type pointer anyway (or "reading up the type pointer" is essentially free, given that you already read up the refcount). > With dynamic languages becoming increasingly important > these days, I wonder whether anyone has thought about > what sort of cache or other memory architecture modifications > might make things like refcounting more efficient. Yes, but I don't think anyone's offering to build a P3K chip for us ;-) From ncoghlan at gmail.com Thu Apr 20 12:50:46 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 20 Apr 2006 20:50:46 +1000 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> <44461820.6010005@gmail.com> Message-ID: <44476786.5060608@gmail.com> Jim Jewett wrote: > On 4/19/06, Nick Coghlan wrote: > >> Then you'd have: >> >> def f(a, *(b, c=1, *args), **(d, e=2, **kwds)): >> # Silly function >> >> 'a' would be a normal positional-or-keyword argument >> 'b' would be a required positional-only argument > > Am I reading that correctly? > > Looking only at a and b the possible calling signatures are exactly: > > f(1, 2) > f(a=1, 2) > > because b can't be named but must appear second, and nothing except a > can appear before it because of the rules on positional arguments. Both would be illegal. The first one misses the required keyword argument 'd', the second one is a SyntaxError because you can't have a keyword argument before a positional one. Some legal combinations: f(1, 2, 3, d=1) # Positional args a, b and c f(1, 2, 3, 4, 5, 6, d=1) # Positional args a, b and c and (4, 5, 6) as args f(2, a=1, d=1) # Provide a as a keyword arg instead You'd never define a function with a signature as insane as the one in the example though - you'd only use it for cases where you accepted arbitray keyword arguments, but wanted to permit some positional arguments (like 'self') without accidental conflicts. This seems to be more of a theoretical problem than a practical one though. . . Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Thu Apr 20 13:24:09 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 20 Apr 2006 23:24:09 +1200 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <1f7befae0604200341i1cfa8dfcl22430e718b0d4b81@mail.gmail.com> References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> <4447477A.3080306@canterbury.ac.nz> <1f7befae0604200341i1cfa8dfcl22430e718b0d4b81@mail.gmail.com> Message-ID: <44476F59.1070303@canterbury.ac.nz> Tim Peters wrote: > Yes, but I don't think anyone's offering to build a P3K chip for us ;-) Since you mention it, one of the things that's been occupying my mind in idle moments recently is trying to imagine what a Python chip *would* be like, and maybe even designing it and prototyping it with an FPGA. Whether the project will ever get that far is highly uncertain yet, but it's an interesting thing to think about. Some of the ideas I've been considering include keeping all the object headers, including refcount, size, type pointer and maybe some flags, in a separate specialised memory with its own cache and maybe its own bus. Another idea is just keeping the low-order byte of the refcount in a special high-speed memory, so that you only need to touch main memory for about 1 in 256 inc/decrefs. This is getting off-topic for Py3k, though. Any hardware buffs interested in discussing these ideas further are welcome to contact me by email. -- Greg From p.f.moore at gmail.com Thu Apr 20 13:53:21 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 20 Apr 2006 12:53:21 +0100 Subject: [Python-3000] Theory of cooperative super calls (Re: auto-super()) In-Reply-To: References: <443E7F47.9060308@colorstudy.com> <20060417020415.GE28891@tigger.digitaltorque.ca> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> <44462A51.6070709@canterbury.ac.nz> <9e804ac0604190607g1395d3dbi87c004145a6d1bd4@mail.gmail.com> <4447418E.1080400@canterbury.ac.nz> Message-ID: <79990c6b0604200453s353fd8cbk338854f041496cbb@mail.gmail.com> On 4/20/06, Guido van Rossum wrote: > On 4/20/06, Greg Ewing wrote: > > Thomas Wouters wrote: > > > > > I believe Guido is referring to > > > http://www.awl.com/cseng/titles/0-201-43305-2 > > > > Are there any published papers about this, or is that book > > the only source? > > What can be more published than a book? :-) Sadly, the book seems to be out of print, and second hand copies seem very expensive (?70+ from Amazon UK, more than I really want to spend...) Paul. From walter at livinglogic.de Thu Apr 20 14:02:17 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 20 Apr 2006 14:02:17 +0200 Subject: [Python-3000] Chaning the import machinery; was: Re: [Python-Dev] setuptools in 2.5. In-Reply-To: References: <200604201456.13148.anthony@interlink.com.au> <200604201652.51448.anthony@interlink.com.au> <44473D71.6060107@v.loewis.de> <200604201801.29943.anthony@interlink.com.au> <444747EA.1010707@livinglogic.de> Message-ID: <44477849.7000806@livinglogic.de> Guido van Rossum wrote: > On 4/20/06, Walter D?rwald wrote: >> I'd like to be able to simply import a file if I have the filename. And >> I'd like to be able to load sourcecode from an Oracle database and have >> useful "filename" and line number information if I get a traceback. > > Great use cases. Could I ask you to elaborate these in the Python-3000 > list? It would be very useful if you attempted to specify what > (approximately) the API for these would look like and how it would > work (e.g. the immediate question with import from a file or URL is > what happens if another module imports the same thing). Maybe import should become a function (maybe even a generic function ;)) so we can do: cStringIO = import(url("file:/usr/local/lib/python2.5/cStringIO")) import cx_Oracle db = cx_Oracle.connect("...") mymodule = import(oraclesource(db=db, query="select source from modules where name='mymodule'")) code = """ cache = 1000 color = True """ options = import(literal(code)) Simple imports could look like this: urllib2 = import("urllib2") (i.e. pass the module name as a string). Unfortunately this means that the module name has to be specified twice. All objects passed to import() should be usable as dictionary keys, so that they can be stored as keys in sys.modules which would continue to be the module cache. In a traceback the repr() of this object should be displayed (maybe with the exception of import("urllib2") which should display the real filename). So far this means that we would have to get rid of relative imports. Another option would be to make a relative import be the responsibility of the module to which the import is relative. Even better would be if import() would do some kind of dependency tracking during import for modules that are recursively imported during import of the original module. Then mymodule = reload(oraclesource(db=db, query="select source from modules where name='mymodule'") could do the right thing (i.e. import the module if it's not in sys.modules or it has changed since the last import or one of the modules it uses has changed; otherwise use the cached module). This would mean that each import resource would have to provide some kind of cookie (a checksum or a timestamp), so that it's possible to detect if the source has changed (and of course it needs a method to return the real source code string). I've implemented something like this once, but abandoned the idea because tracebacks simply have "" as the filename, and that makes debugging a PITA. Anyway the source code for this is here: http://styx.livinglogic.de/~walter/pythonimport/resload.py Servus, Walter From rasky at develer.com Thu Apr 20 14:12:34 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 20 Apr 2006 14:12:34 +0200 Subject: [Python-3000] super(): issues [Was: Removing 'self' from method definitions] References: <443E7F47.9060308@colorstudy.com><20060413171301.GA9869@localhost.localdomain><443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <064401c66283$1a392bb0$2452fea9@bagio> <4445FE0F.9040406@gmail.com> Message-ID: <0d0301c66473$b52d2850$2452fea9@bagio> Nick Coghlan wrote: >> I'll also point people to this page, which contains a very accurate >> description of the issues with super: >> http://fuhm.net/super-harmful/ > > Hmm, the reference to Dylan's next method (along with a few other > comments in this thread) makes me believe a magic autosuper() > equivalent would really need to be based on a thread-local context, > rather than any kind of static code analysis. > > An interesting point is that being able to call the "next method" for > generic functions has exactly the same problem. Right. Part of the problems would be solved if we forced keyword arguments only in super calls (as everything else is really a latent bug), and if we had an implicit, defaul tway of "doing the right thing" with super (that is, forwarding everything). Giovanni Bajo From guido at python.org Thu Apr 20 15:06:34 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 14:06:34 +0100 Subject: [Python-3000] Chaning the import machinery; was: Re: [Python-Dev] setuptools in 2.5. In-Reply-To: <44477849.7000806@livinglogic.de> References: <200604201456.13148.anthony@interlink.com.au> <200604201652.51448.anthony@interlink.com.au> <44473D71.6060107@v.loewis.de> <200604201801.29943.anthony@interlink.com.au> <444747EA.1010707@livinglogic.de> <44477849.7000806@livinglogic.de> Message-ID: Sorry, there's so much here that seems poorly thought out that I don't know where to start. Getting rid of the existing import syntax in favor of the incredibly verbose and ugly foo = import("foo") just isn't acceptable. Importing from remote URLs is a non-starter from a security POV; and using HTTPS would be too slow. For code that's known to reside remotely, a better approach is to use setuptools to install that code once and for all. How would a module know its own name? How do you deal with packages (importing a module from a package implies importing/loading the package's __init__.py). I suggest that instead of answering these questions from the perspective of the solution you're offering here, you tell us a bit more about the use cases that make you think of this solution. What are you trying to do? Why are you importing code from a specific file instead of configuring sys.path so the file will be found naturally? I suspect that your module-from-a-database use case is actually intended as a building block for an import hook. I think we ought to redesign the import machinery in such a way that you could do things like importing from a specific file or from a database, but without changing the import statement syntax -- instead, we should change what import actually *does*. While we're at it, we should also fix the silliness that __import__("foo.bar") imports foo.bar but returns foo. --Guido On 4/20/06, Walter D?rwald wrote: > Guido van Rossum wrote: > > > On 4/20/06, Walter D?rwald wrote: > >> I'd like to be able to simply import a file if I have the filename. And > >> I'd like to be able to load sourcecode from an Oracle database and have > >> useful "filename" and line number information if I get a traceback. > > > > Great use cases. Could I ask you to elaborate these in the Python-3000 > > list? It would be very useful if you attempted to specify what > > (approximately) the API for these would look like and how it would > > work (e.g. the immediate question with import from a file or URL is > > what happens if another module imports the same thing). > > Maybe import should become a function (maybe even a generic function ;)) > so we can do: > > cStringIO = import(url("file:/usr/local/lib/python2.5/cStringIO")) > > import cx_Oracle > db = cx_Oracle.connect("...") > > mymodule = import(oraclesource(db=db, query="select source from modules > where name='mymodule'")) > > code = """ > cache = 1000 > color = True > """ > > options = import(literal(code)) > > Simple imports could look like this: > urllib2 = import("urllib2") > > (i.e. pass the module name as a string). Unfortunately this means that > the module name has to be specified twice. > > All objects passed to import() should be usable as dictionary keys, so > that they can be stored as keys in sys.modules which would continue to > be the module cache. > > In a traceback the repr() of this object should be displayed (maybe with > the exception of import("urllib2") which should display the real filename). > > So far this means that we would have to get rid of relative imports. > Another option would be to make a relative import be the responsibility > of the module to which the import is relative. > > Even better would be if import() would do some kind of dependency > tracking during import for modules that are recursively imported during > import of the original module. Then > > mymodule = reload(oraclesource(db=db, query="select source from modules > where name='mymodule'") > > could do the right thing (i.e. import the module if it's not in > sys.modules or it has changed since the last import or one of the > modules it uses has changed; otherwise use the cached module). This > would mean that each import resource would have to provide some kind of > cookie (a checksum or a timestamp), so that it's possible to detect if > the source has changed (and of course it needs a method to return the > real source code string). > > I've implemented something like this once, but abandoned the idea > because tracebacks simply have "" as the filename, and that > makes debugging a PITA. Anyway the source code for this is here: > > http://styx.livinglogic.de/~walter/pythonimport/resload.py > > Servus, > Walter > > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 20 15:15:37 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 14:15:37 +0100 Subject: [Python-3000] Theory of cooperative super calls (Re: auto-super()) In-Reply-To: <79990c6b0604200453s353fd8cbk338854f041496cbb@mail.gmail.com> References: <443E7F47.9060308@colorstudy.com> <20060417031858.GA354@panix.com> <9e804ac0604180734pb0eec14i4312853c374c895a@mail.gmail.com> <44459EF5.4070604@canterbury.ac.nz> <44462A51.6070709@canterbury.ac.nz> <9e804ac0604190607g1395d3dbi87c004145a6d1bd4@mail.gmail.com> <4447418E.1080400@canterbury.ac.nz> <79990c6b0604200453s353fd8cbk338854f041496cbb@mail.gmail.com> Message-ID: On 4/20/06, Paul Moore wrote: > On 4/20/06, Guido van Rossum wrote: > > On 4/20/06, Greg Ewing wrote: > > > Thomas Wouters wrote: > > > > > > > I believe Guido is referring to > > > > http://www.awl.com/cseng/titles/0-201-43305-2 > > > > > > Are there any published papers about this, or is that book > > > the only source? > > > > What can be more published than a book? :-) > > Sadly, the book seems to be out of print, and second hand copies seem > very expensive (?70+ from Amazon UK, more than I really want to > spend...) So write the authors asking to put the text on-line. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From walter at livinglogic.de Thu Apr 20 16:11:04 2006 From: walter at livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Thu, 20 Apr 2006 16:11:04 +0200 Subject: [Python-3000] Chaning the import machinery; was: Re: [Python-Dev] setuptools in 2.5. In-Reply-To: References: <200604201456.13148.anthony@interlink.com.au> <200604201652.51448.anthony@interlink.com.au> <44473D71.6060107@v.loewis.de> <200604201801.29943.anthony@interlink.com.au> <444747EA.1010707@livinglogic.de> <44477849.7000806@livinglogic.de> Message-ID: <44479678.3070803@livinglogic.de> Guido van Rossum wrote: > Sorry, there's so much here that seems poorly thought out that I don't > know where to start. Consider it a collection of wild ideas. > Getting rid of the existing import syntax in favor of the incredibly > verbose and ugly > > foo = import("foo") > > just isn't acceptable. OK, then how about import foo import foo from url("file:~guido/python-packages/foo") How would this work with multiple imports? import foo, bar from url("file:~guido/python-packages/") How would it recognize whether url() refers to the module itself or to a package from which modules are imported? > Importing from remote URLs is a non-starter from a security POV; and > using HTTPS would be too slow. For code that's known to reside > remotely, a better approach is to use setuptools to install that code > once and for all. I don't see how that changes anything from a security POV. You have to trust the source in both cases. Performancewise you are right, it wouldn't make sense to call such an import in a tight loop. > How would a module know its own name? It probably would only have a real name for a standard "import foo". import foo from url("file:~guido/python-packages/foo") would create a module with foo.__origin__ == url("file:~guido/python-packages/foo") > How do you deal with packages > (importing a module from a package implies importing/loading the > package's __init__.py). I have no idea. This probably won't work (at least not in the sense that importing something imports all its parent modules). But packages seem to be a problem for setuptools too (at least if parts of the package are distributed separately). Maybe for import os.path from url("file:~guido/python-packages/os/path") url("file:~guido/python-packages/os/path").parent() should return url("file:~guido/python-packages/os") which then gets imported and url("file:~guido/python-packages") should return None. But this would mean that the number of dots on the left side has to match the number of times calling parent() on the right side returns somehing not None. Not good. Maybe we should leave the current import syntax alone and add a new one for importing from files etc.. > I suggest that instead of answering these questions from the > perspective of the solution you're offering here, you tell us a bit > more about the use cases that make you think of this solution. What > are you trying to do? Why are you importing code from a specific file > instead of configuring sys.path so the file will be found naturally? The Python files I'm importing define "layout classes" that are used for generating a bunch of static HTML files (actually JSP files). I.e. something like foo.py: def link(text, href): return "%s" % (href, text) bar.py: foo = specialimport("foo.py") def linklist(links): return "
    %s
" % "".join("
  • %s
    • " % bar.link(text, href) for (text, href) in links) This function linklist() is used for generating the HTML files. The interactive Python shell is used as a "make shell", so when I change foo.py in an editor and do >>> project.build("install") in the shell all the HTML files that depend on bar.py (which in turn depends on foo.py) must be rebuilt. > I > suspect that your module-from-a-database use case is actually intended > as a building block for an import hook. Maybe, but ACAIK import hooks can't solve the dependency and reload problem. > I think we ought to redesign the import machinery in such a way that > you could do things like importing from a specific file or from a > database, but without changing the import statement syntax -- instead, > we should change what import actually *does*. But there has to be a spot where I can actually specify from *where* I want to import, and IMHO this should be in the import statement, not by putting some hooks somewhere. > While we're at it, we > should also fix the silliness that __import__("foo.bar") imports > foo.bar but returns foo. +1 Servus, Walter From jimjjewett at gmail.com Thu Apr 20 16:20:17 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 20 Apr 2006 10:20:17 -0400 Subject: [Python-3000] parameter lists [was: Type Expressions] Message-ID: On 4/20/06, Greg Ewing wrote: > Some wilder ideas for keyword-only arguments: > def spam(a, b, c, {d, e, f=x}): > # d, e are mandatory keyword-only > # f is optional keyword-only I see several sources of complexity. Long sequences where order matters are bad, but probably can't be eliminated. We have this today. A rest-of-the-arguments collector is a useful special case. We have this today with *. Arguments which can be either positional or keywords are tricky, but also useful. We have this today. A distinction between expected and unexpected keywords is awkward, but sometimes useful. We have this today with **. Today, we do not have any way to say that a parameter is keyword only without also marking it unexpected. This motivates the addition of def(pos1, pos2, *args, key1=XXX): I also see the value of keyword-only arguments without unlimited positional arguments. Whether it deserves syntax and what that syntax should be (such as * or *None) may need a pronouncement eventually, but it doesn't have to be complex. The *new* complexity seems to come almost entirely from trying to support keyword-only arguments without a default. I don't think (mandatory) + (keyword only) + (no default) + (no sentinel) is important enough to justify that complexity. (That said, I would like a better way to tell whether the keyword was passed explicitly; creating a sentinel outside the function always seems awkward.) -jJ From jimjjewett at gmail.com Thu Apr 20 16:41:04 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 20 Apr 2006 10:41:04 -0400 Subject: [Python-3000] super(): issues [Was: Removing 'self' from method definitions] In-Reply-To: <0d0301c66473$b52d2850$2452fea9@bagio> References: <443E7F47.9060308@colorstudy.com> <20060413171301.GA9869@localhost.localdomain> <443E9193.7000908@colorstudy.com> <4441C3E4.4070504@canterbury.ac.nz> <4442F264.7030307@canterbury.ac.nz> <064401c66283$1a392bb0$2452fea9@bagio> <4445FE0F.9040406@gmail.com> <0d0301c66473$b52d2850$2452fea9@bagio> Message-ID: On 4/20/06, Giovanni Bajo wrote: > Right. Part of the problems would be solved if we forced > keyword arguments only in super calls (as everything else > is really a latent bug), As mentioned elsewhere, for a well-known API like __getattr__, it really isn't a bug to use positional arguments. Otherwise, it makes sense to use keywords for any call, even if it has nothing to do with super. > and if we had an implicit, default > way of "doing the right thing" with super (that is, > forwarding everything). Right now, this requires a slower path allowing *args and **kwargs, even if you don't use them. Right now, this fails if there isn't a super method. (e.g., if you inherit directly from object, and the method isn't there, but might exist on something your object gets composed with.) -jJ From fredrik at pythonware.com Thu Apr 20 16:52:26 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 20 Apr 2006 16:52:26 +0200 Subject: [Python-3000] Changing the import machinery References: Message-ID: Guido van Rossum wrote: > On 4/20/06, in python-dev, Fredrik Lundh wrote: > > I was hoping that for Python 3.0, we could get around to unkludge the > > sys.path/meta_path/path_hooks/path_importer_cache big ball of hacks, > > possibly by replacing sys.path with something a bit more intelligent than > > a plain list. > > That's an excellent idea. Are there any volunteers here to help out? > Even just listing specific use cases / scenarios that are currently > difficult to solve right would be tremendously helpful. (I think that > Phillip's and others' experience with setuptools might be very > useful.) how far would a plain the path can contain either directory names or import handlers modification take us ? (where import handlers are duck-typed objects with an as-small-as-possible interface, and the handler for builtins and frozen objects are included in the list). being able to track modifications to sys.path would also be useful, I think. tracking *modifications* are of course not much of a problem, but what about sys.path = list(stuff) time to add support for module properties, perhaps ? From fredrik at pythonware.com Thu Apr 20 17:19:55 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 20 Apr 2006 17:19:55 +0200 Subject: [Python-3000] OT: Get to work, get to work, build the dam, build the dam Message-ID: I'm pretty sure reddit.com had the Py3K project in mind when it brought this to the top of my "recommended" list. http://www.theonion.com/content/node/47469 must-hurry-ly yrs /F From free.condiments at gmail.com Thu Apr 20 17:16:55 2006 From: free.condiments at gmail.com (Sam Pointon) Date: Thu, 20 Apr 2006 16:16:55 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: <4447523F.4030504@canterbury.ac.nz> References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: On 20/04/06, Greg Ewing wrote: > So maybe you should be able to write that as > > def spam{d, e, f=x}: > ... > > Then if you have some positional args as well, > > def spam(a, b, c){d, e, f=x}: > ... > > And incorporating all the possible features, > > def spam(a, b, c=y, *d){e, f=x, **g}: > ... > > Although now that the ** arg is inside the {}, there's > no longer any need for the double *, so it could just be > > def spam(a, b, c=y, *d){e, f=x, *g}: > ... Following on from this, how about using [] for positional-only arguments as well? * and ** arguments will follow the same rules as they do now - {} for keyword-only arguments obliviates the need for keyword arguments after a * argument. ** will not be allowed in []. Then, we get this legal function definition: def foo[a, b, c = 'spam'](d, e = 'ham', *f){g, h = 'eggs', **i): ... which, IMO, looks natural, and I think is reasonably intuitive ([] for lists, so positional only, {} for dicts, so keyword only, and () like it is now, so no surprises there). Also, none of these are nested inside each other (barring pattern matching on tuples) so Guido's concern about nested brackets is assuaged. Also, all three sections will be optional, so long as there is at least one of them. If I had any real experience past minor tinkering with CPython's internals I'd have a go at implementing this per Guido's appetite for code, but this change will affect not just the parser/compiler, but also the bytecode interpreter at the very least. However, I will be looking in my code and the stdlib for use cases for this idea. --Sam From guido at python.org Thu Apr 20 17:27:02 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 16:27:02 +0100 Subject: [Python-3000] Chaning the import machinery; was: Re: [Python-Dev] setuptools in 2.5. In-Reply-To: <44479678.3070803@livinglogic.de> References: <200604201456.13148.anthony@interlink.com.au> <200604201652.51448.anthony@interlink.com.au> <44473D71.6060107@v.loewis.de> <200604201801.29943.anthony@interlink.com.au> <444747EA.1010707@livinglogic.de> <44477849.7000806@livinglogic.de> <44479678.3070803@livinglogic.de> Message-ID: On 4/20/06, Walter D?rwald wrote: > Guido van Rossum wrote: > > > Sorry, there's so much here that seems poorly thought out that I don't > > know where to start. > > Consider it a collection of wild ideas. > > > Getting rid of the existing import syntax in favor of the incredibly > > verbose and ugly > > > > foo = import("foo") > > > > just isn't acceptable. > > OK, then how about > > import foo > import foo from url("file:~guido/python-packages/foo") Sorry. I wasn't proposing that the import statement had to be extended to support the new functionality; only that the existing functionality should still be available by writing import statements. For the above, I'd much rather write foo = import_from_url("foo", "/home/guido/.../foo") I also don't like your proposal to create a "url" object (as you might have predicted from my resistance to the path PEP :-). > How would this work with multiple imports? > > import foo, bar from url("file:~guido/python-packages/") > > How would it recognize whether url() refers to the module itself or to a > package from which modules are imported? You tell me. > > Importing from remote URLs is a non-starter from a security POV; and > > using HTTPS would be too slow. For code that's known to reside > > remotely, a better approach is to use setuptools to install that code > > once and for all. > > I don't see how that changes anything from a security POV. You have to > trust the source in both cases. With http, even if I trusted the source, I still shouldn't trust that the data I get from the URL actually came from the source. With HTTPS, at least man-in-the-middle attacks should be thwarted. > Performancewise you are right, it > wouldn't make sense to call such an import in a tight loop. > > > How would a module know its own name? > > It probably would only have a real name for a standard "import foo". Thats a problem IMO. > import foo from url("file:~guido/python-packages/foo") > > would create a module with foo.__origin__ == > url("file:~guido/python-packages/foo") > > > How do you deal with packages > > (importing a module from a package implies importing/loading the > > package's __init__.py). > > I have no idea. This probably won't work (at least not in the sense that > importing something imports all its parent modules). > > But packages seem to be a problem for setuptools too (at least if parts > of the package are distributed separately). So please do some research and find out what their problems are, how your problems are similar, and what should be done about it. At this point I believe you're running out of quick wild ideas that are actually helpful. > Maybe for > > import os.path from url("file:~guido/python-packages/os/path") > > url("file:~guido/python-packages/os/path").parent() should return > url("file:~guido/python-packages/os") which then gets imported and > url("file:~guido/python-packages") should return None. But this would > mean that the number of dots on the left side has to match the number of > times calling parent() on the right side returns somehing not None. Not > good. Maybe we should leave the current import syntax alone and add a > new one for importing from files etc.. I think we should design a new OO API that captures the essence of the current import machinery (but cleaned up), create a new mapping from all current syntactic variants of the import statements to that API, and design a separate extension (through subclassing or whatever) to do imports from non-traditional sources. > > I suggest that instead of answering these questions from the > > perspective of the solution you're offering here, you tell us a bit > > more about the use cases that make you think of this solution. What > > are you trying to do? Why are you importing code from a specific file > > instead of configuring sys.path so the file will be found naturally? > > The Python files I'm importing define "layout classes" that are used for > generating a bunch of static HTML files (actually JSP files). I.e. > something like > > foo.py: > def link(text, href): > return "%s" % (href, text) > > bar.py: > foo = specialimport("foo.py") > > def linklist(links): > return "
      %s
    " % "".join("
  • %s
    • " % bar.link(text, href) > for (text, href) in links) > > This function linklist() is used for generating the HTML files. The > interactive Python shell is used as a "make shell", so when I change > foo.py in an editor and do > >>> project.build("install") > in the shell all the HTML files that depend on bar.py (which in turn > depends on foo.py) must be rebuilt. You've nearly lost me, but it *seems* to me that what you're really doing is use an alternative import mechanism in order to solve the reload() problem for a set of interdependent modules. That's a good thing to attempt to solve more generally, but hardly a use case for a special import function. What does your specialimport() implementation do when the same module is requested twice? Does it load it twice? > > I > > suspect that your module-from-a-database use case is actually intended > > as a building block for an import hook. > > Maybe, but ACAIK import hooks can't solve the dependency and reload problem. Only because nobody has bothered to use them to solve it; they have all the information available needed to solve it. > > I think we ought to redesign the import machinery in such a way that > > you could do things like importing from a specific file or from a > > database, but without changing the import statement syntax -- instead, > > we should change what import actually *does*. > > But there has to be a spot where I can actually specify from *where* I > want to import, and IMHO this should be in the import statement, not by > putting some hooks somewhere. That may be your use case. Most people would prefer to be able to set PYTHONPATH once and then use regular import statements. On Windows, there's a mechanism to specify that a particular module must be loaded from a specifc location. Maybe this would be helpful? It could be a dict whose keys are fully qualified module names, and whose values are pathnames (or URLs or whatever). > > While we're at it, we > > should also fix the silliness that __import__("foo.bar") imports > > foo.bar but returns foo. > > +1 This is an example of what I mentioned above -- redesign the machinery API and then remap the import statement. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 20 17:33:14 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 16:33:14 +0100 Subject: [Python-3000] Changing the import machinery In-Reply-To: References: Message-ID: On 4/20/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > On 4/20/06, in python-dev, Fredrik Lundh wrote: > > > I was hoping that for Python 3.0, we could get around to unkludge the > > > sys.path/meta_path/path_hooks/path_importer_cache big ball of hacks, > > > possibly by replacing sys.path with something a bit more intelligent than > > > a plain list. > > > > That's an excellent idea. Are there any volunteers here to help out? > > Even just listing specific use cases / scenarios that are currently > > difficult to solve right would be tremendously helpful. (I think that > > Phillip's and others' experience with setuptools might be very > > useful.) > > how far would a plain > > the path can contain either directory names or import handlers > > modification take us ? (where import handlers are duck-typed objects > with an as-small-as-possible interface, and the handler for builtins and > frozen objects are included in the list). Probably all the way. PEP 302's rationale discusses some problems that explain why this wasn't done in 2.x; perhaps those concerns can be swept in 3.0. > being able to track modifications to sys.path would also be useful, I think. > tracking *modifications* are of course not much of a problem, but what > about > > sys.path = list(stuff) > > time to add support for module properties, perhaps ? Or make the module a different class instance; people already do that. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 20 17:40:48 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 16:40:48 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: On 4/20/06, Sam Pointon wrote: > Following on from this, how about using [] for positional-only > arguments as well? Forget it. Too different with not enough benefits. Remember, we don't want to emulate Perl 6! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From theller at python.net Thu Apr 20 14:52:16 2006 From: theller at python.net (Thomas Heller) Date: Thu, 20 Apr 2006 14:52:16 +0200 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: <44476F59.1070303@canterbury.ac.nz> References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> <4447477A.3080306@canterbury.ac.nz> <1f7befae0604200341i1cfa8dfcl22430e718b0d4b81@mail.gmail.com> <44476F59.1070303@canterbury.ac.nz> Message-ID: <44478400.2020007@python.net> Greg Ewing wrote: > Tim Peters wrote: > >> Yes, but I don't think anyone's offering to build a P3K chip for us ;-) > > Since you mention it, one of the things that's been > occupying my mind in idle moments recently is trying to > imagine what a Python chip *would* be like, and maybe > even designing it and prototyping it with an FPGA. Since MyHDL now exists, it may even be possible to write and test the prototype in Python. > Whether the project will ever get that far is highly > uncertain yet, but it's an interesting thing to think > about. Some of the ideas I've been considering include > keeping all the object headers, including refcount, > size, type pointer and maybe some flags, in a separate > specialised memory with its own cache and maybe its own > bus. Or even in the FPGA's own memory. > Another idea is just keeping the low-order byte of the > refcount in a special high-speed memory, so that you > only need to touch main memory for about 1 in 256 > inc/decrefs. > > This is getting off-topic for Py3k, though. Any hardware > buffs interested in discussing these ideas further are > welcome to contact me by email. Yes. I'll shut up now. Thomas From jcarlson at uci.edu Thu Apr 20 18:03:18 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 20 Apr 2006 09:03:18 -0700 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <44472D17.9010305@neotitans.com> References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> Message-ID: <20060420084537.A83B.JCARLSON@uci.edu> Andy Sy wrote: > Err... thank you once again for being redundant, reiterating what is > already pretty much dead clear to everyone and saying nothing new > in general. And yet, here I notice that you've still not described a syntax or its execution semantics. Instead you've chosen to spend your time insulting me. Please continue, I'm sure it will make people want to read your future posts all the more. Ultimately, Guido is the judge of what is or is not Pythonic. Without a description of what you want, no one can be for or against it, and ultimately, you are playing yourself to be very much like every other first-poster in python-dev or python-3000; 'I think Python would be better with feature/syntax/language construct X'. This is not a mortal sin. I think it's about time to use a Tim Peters inspired closing. to-inattentive-to-really-be-insulted-ly y'rs - Josiah From lists at janc.be Thu Apr 20 18:06:55 2006 From: lists at janc.be (Jan Claeys) Date: Thu, 20 Apr 2006 18:06:55 +0200 Subject: [Python-3000] Chaning the import machinery; was: Re: [Python-Dev] setuptools in 2.5. In-Reply-To: References: <200604201456.13148.anthony@interlink.com.au> <200604201652.51448.anthony@interlink.com.au> <44473D71.6060107@v.loewis.de> <200604201801.29943.anthony@interlink.com.au> <444747EA.1010707@livinglogic.de> <44477849.7000806@livinglogic.de> <44479678.3070803@livinglogic.de> Message-ID: <1145549216.12589.178.camel@localhost.localdomain> Op do, 20-04-2006 te 16:27 +0100, schreef Guido van Rossum: > On 4/20/06, Walter D?rwald wrote: > > Guido van Rossum wrote: > > > Importing from remote URLs is a non-starter from a security POV; and > > > using HTTPS would be too slow. For code that's known to reside > > > remotely, a better approach is to use setuptools to install that code > > > once and for all. > > > > I don't see how that changes anything from a security POV. You have to > > trust the source in both cases. > > With http, even if I trusted the source, I still shouldn't trust that > the data I get from the URL actually came from the source. With HTTPS, > at least man-in-the-middle attacks should be thwarted. A man-in-the-middle-attack and other ways to "inject" a different module than intended by the author are also possible with the current default filesystem based imports, so I don't think that's a good argument against http-imports (or other similar extensions to import). In both cases you need a way to check & control the whole chain involved (network, OS, python, ...) to make 100% sure every step is safe, that you really import what you intended to import... -- Jan Claeys From ianb at colorstudy.com Thu Apr 20 18:38:50 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 20 Apr 2006 11:38:50 -0500 Subject: [Python-3000] Changing the import machinery In-Reply-To: References: Message-ID: <4447B91A.8090409@colorstudy.com> Guido van Rossum wrote: > I'm changing the list and the subject, pulling this quote out of python-dev: > > On 4/20/06, in python-dev, Fredrik Lundh wrote: > >>I was hoping that for Python 3.0, we could get around to unkludge the >>sys.path/meta_path/path_hooks/path_importer_cache big ball of hacks, >>possibly by replacing sys.path with something a bit more intelligent than >>a plain list. > > > That's an excellent idea. Are there any volunteers here to help out? > Even just listing specific use cases / scenarios that are currently > difficult to solve right would be tremendously helpful. (I think that > Phillip's and others' experience with setuptools might be very > useful.) Cleaning up import stuff would be excellent. Time spent debugging imports is time wasted, but it happens all too often. I would argue against any list of loaders, or list of anything. That builds ambiguity directly into the system. Without a list, if you want ambiguity, a container loader could search a list of loaders. Or if you want to avoid all ambiguity, you could have a loader that was more picky. Setuptools version-based eager loading can give you some confidence that everything you think you need is installed, but can't provide much confidence that everything you *think you are using* is actually what you are using. That is, it's been fairly common in my own experience for me to realize some other version of a package is being loaded than what I thought, or I spend an inordinant amount of time tweaking requirements to get the right version of a package from one place without affecting another package that needs a different version (or perhaps is run with a different sys.path). But, back to more concrete use cases: * Right now it is pretty hard to set up an environment where changes elsewhere on the system can't leak in. That is, installation of something in a system-wide site-packages can cause problems everywhere on the system, and even if you try to avoid these it is quite hard. One strategy is setting up an entirely different environment (aka, a different prefix); this is heavy-feeling. Another is avoiding site.py or using a custom site.py, but the tool support is iffy for that. There's really just no good way to tell Python to leave well enough alone. * Relatedly, installation and management when you don't have root or the cooperation of root can be hard. I think the answer to this is much like the isolated environment, but the use case is fairly different. * Configuration about where to install things (e.g., distutils.cfg) is separate from information about where to look for things (sys.path). These should form a consistent description of the environment, but currently they are disassociated from each other. * Any kind of automatic installation is difficult, because you can't really count on being able to install even the most inoccuous package in an automated way. There's too many manual overrides, and too many redundant options, and few people actually have their system set up to work without tweaking these options through the command line or other feedback. * Personally I've settled on putting everything I make into a Python package that is distutils-installable. But many people don't. I'm not sure if this is just because the tools seem too hard, or the namespaces feel too deep, or all the documentation starts without using packages, or having '.' (sometimes) on sys.path does it, or what. I'd rather there be consistent practices; but the consistent practices that we have that actually work (setup.py scripts and packages) are too heavy for a lot of people. * People are seriously planning on using relative imports to manage their packages, and so an application will be 'installed' by putting it into another package. Presumably unpacking it directly in some other package's directory. Who knows what the version control plans are, or maintenance, or whatever. I think it's a bad idea. We need to give these people a carrot to keep them from doing this. * Right now namespace packages are hard. That is, a Python package (like 'zope') that is used by several distutils packages. I almost feel like namespace packages should be installed flat, like 'zope-interface' and 'zope-tal', and turned into namespaces dynamically. * The module layout is used both as an API and as an internal factoring of the code. If you want to refactor the code you break the API. Personally I really like the strong connection between imports and code location, and appreciate how easily I can find code as a result. But setting up the scaffolding and warnings necessary when moving a module can be tiresome. * Circular imports should fail more nicely. Everyone suffers this at some time; maybe it can't be fixed, but at least it should be clear what's happening. * You can't really tell if "from foo import bar" can be written as "import foo; bar = foo.bar", because it works if foo contains bar, but not if foo is a package and bar is a module in that package. Well... I think that's maybe half way through the list of issues I have, but this email is already much too long. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From guido at python.org Thu Apr 20 19:10:28 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 18:10:28 +0100 Subject: [Python-3000] Changing the import machinery In-Reply-To: <4447B91A.8090409@colorstudy.com> References: <4447B91A.8090409@colorstudy.com> Message-ID: It must not be my day. I don't understand anything you're saying. On 4/20/06, Ian Bicking wrote: > Cleaning up import stuff would be excellent. Time spent debugging > imports is time wasted, but it happens all too often. > > I would argue against any list of loaders, or list of anything. That > builds ambiguity directly into the system. Without a list, if you want > ambiguity, a container loader could search a list of loaders. Or if you > want to avoid all ambiguity, you could have a loader that was more picky. What ambiguity are you talking of here? > Setuptools version-based eager loading Can you explain this? I don't actually know much of what setuptools does (nor does almost anyone else it seems :-) -- I have no frame of reference to understand this. > can give you some confidence that > everything you think you need is installed, but can't provide much > confidence that everything you *think you are using* is actually what > you are using. That is, it's been fairly common in my own experience > for me to realize some other version of a package is being loaded than > what I thought, or I spend an inordinant amount of time tweaking > requirements to get the right version of a package from one place > without affecting another package that needs a different version (or > perhaps is run with a different sys.path). Is this a setuptools thing or a Python thing? I don't understand what you mean by "tweaking requirements". > But, back to more concrete use cases: > > * Right now it is pretty hard to set up an environment where changes > elsewhere on the system can't leak in. That is, installation of > something in a system-wide site-packages can cause problems everywhere > on the system, and even if you try to avoid these it is quite hard. One > strategy is setting up an entirely different environment (aka, a > different prefix); this is heavy-feeling. Another is avoiding site.py > or using a custom site.py, but the tool support is iffy for that. I have a sense of deja vu -- somehow I feel we've been discussing this some years ago. Are you sure you didn't post this same message around the time we were discussing PEP 302? Seriously, can you provide an example of the sort of thing you're thinking of? Again, is this setuptools specific or not? Is the problem caused by users setting their PYTHONPATH a certain way? (If so, how?) > There's really just no good way to tell Python to leave well enough alone. That's a rather general and vague remark. What do you mean? > * Relatedly, installation and management when you don't have root or the > cooperation of root can be hard. I think the answer to this is much > like the isolated environment, but the use case is fairly different. This I understand. What do you suggest we do to improve the situation? > * Configuration about where to install things (e.g., distutils.cfg) is > separate from information about where to look for things (sys.path). > These should form a consistent description of the environment, but > currently they are disassociated from each other. I didn't even know we have a distutils.cfg. I am beginning to believe that you're talking about issues with installation rather than issues in the import machinery proper. But maybe if you clear up some of my earlier confusion I'll understand better. > * Any kind of automatic installation is difficult, because you can't > really count on being able to install even the most inoccuous package in > an automated way. There's too many manual overrides, and too many > redundant options, and few people actually have their system set up to > work without tweaking these options through the command line or other > feedback. Again that's rather vague and general. Please be specific. "Build a better mousetrap" can't be our requirements spec. > * Personally I've settled on putting everything I make into a Python > package that is distutils-installable. But many people don't. I'm not > sure if this is just because the tools seem too hard, or the namespaces > feel too deep, or all the documentation starts without using packages, > or having '.' (sometimes) on sys.path does it, or what. I'd rather > there be consistent practices; but the consistent practices that we have > that actually work (setup.py scripts and packages) are too heavy for a > lot of people. To the contrary, if your entire code fits in a single .py file, I'm not sure what distutils even buys you, and it sure costs a lot of complexity. FWIW Google's internal build system agrees with you and doesn't like 3rd party code that's not in a package. I often have a hard time finding the source code in a distutils-based package that I have unpacked but am hesitant to install just to read the source code; there seems to be no consistent convention as to where the source is. > * People are seriously planning on using relative imports to manage > their packages, and so an application will be 'installed' by putting it > into another package. Presumably unpacking it directly in some other > package's directory. Who knows what the version control plans are, or > maintenance, or whatever. I think it's a bad idea. We need to give > these people a carrot to keep them from doing this. You seem to have version control on your brain. (Not that that's a bad thing, but most people don't -- it's a specific way of looking at things.) I'm not entirely sure I understand what bad practice you're describing here; and what do you propose to do instead? KSurely not killing relative import? > * Right now namespace packages are hard. That is, a Python package > (like 'zope') that is used by several distutils packages. I almost feel > like namespace packages should be installed flat, like 'zope-interface' > and 'zope-tal', and turned into namespaces dynamically. Fine. That's the most advanced usage there is. I'd be happy if we solved all other problems first. > * The module layout is used both as an API and as an internal factoring > of the code. If you want to refactor the code you break the API. > Personally I really like the strong connection between imports and code > location, and appreciate how easily I can find code as a result. But > setting up the scaffolding and warnings necessary when moving a module > can be tiresome. I don't expect there's a silver bullet here; backwards compatibility hacks are always tiresome. > * Circular imports should fail more nicely. Everyone suffers this at > some time; maybe it can't be fixed, but at least it should be clear > what's happening. It probably can't be fixed (or do you see a fix?). Do you mean it should be easier to debug, or do you mean it should be explained better how things work? > * You can't really tell if "from foo import bar" can be written as > "import foo; bar = foo.bar", because it works if foo contains bar, but > not if foo is a package and bar is a module in that package. And is that bad? "from foo import bar" means what it means. Nobody said the other code is equivalent. If you know what foo is you will know whether the other code is equivalent; isn't that enough? > Well... I think that's maybe half way through the list of issues I have, > but this email is already much too long. And still not clear enough. It's difficult to communicate when one party has spent a long time exploring the issues and another is just starting; this time I seem to be on the receiving end (== haven't spent a lot of time). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From janssen at parc.com Thu Apr 20 19:27:49 2006 From: janssen at parc.com (Bill Janssen) Date: Thu, 20 Apr 2006 10:27:49 PDT Subject: [Python-3000] GIL In-Reply-To: Your message of "Wed, 19 Apr 2006 18:58:29 PDT." <06Apr19.185833pdt."58633"@synergy1.parc.xerox.com> Message-ID: <06Apr20.102759pdt."58633"@synergy1.parc.xerox.com> > It would be interesting to bring ILU up-to-date. Get the ILU C > library compiling nicely on POSIX (should be trivial), then bring the > Python bindings up-to-date. That would, once again, give us ONC RPC > and CORBA compatibility, both IPC and RPC. Actually, I tried this last night on a Fedora platform, and after fiddling with the configure.in script to move it from autoconf 2.13 to 2.59, the core library, parser, stubber, and C bindings compile and run fine. Passes all tests. So the remaining work would be to bring the Python binding up-to-date, perhaps using Pyrex or ctypes. ILU also has server-side SSL support, by the way. Bill From ianb at colorstudy.com Thu Apr 20 19:49:38 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 20 Apr 2006 12:49:38 -0500 Subject: [Python-3000] Changing the import machinery In-Reply-To: References: <4447B91A.8090409@colorstudy.com> Message-ID: <4447C9B2.8050009@colorstudy.com> Guido van Rossum wrote: > It must not be my day. I don't understand anything you're saying. Hmm... well, trying again... > On 4/20/06, Ian Bicking wrote: > >>Cleaning up import stuff would be excellent. Time spent debugging >>imports is time wasted, but it happens all too often. >> >>I would argue against any list of loaders, or list of anything. That >>builds ambiguity directly into the system. Without a list, if you want >>ambiguity, a container loader could search a list of loaders. Or if you >>want to avoid all ambiguity, you could have a loader that was more picky. > > > What ambiguity are you talking of here? If you have a sys.path like ['/A', '/B'], and you have a package in /B/foo, you don't know if importing foo will import that package, because it depends on the contents of A. >>Setuptools version-based eager loading > > > Can you explain this? I don't actually know much of what setuptools > does (nor does almost anyone else it seems :-) -- I have no frame of > reference to understand this. When you explicitly activate an egg, setuptools looks at the requirements listed for that egg and activates all those required eggs as well. (Activating doesn't import anything, but adds the egg to sys.path if necessary, and recursively activates that eggs requirements) So you get an early failure if something is simply missing, you don't have to wait for the import to happen. It's mostly an aside, but the "try:import foo except ImportError: foo = None" pattern doesn't really work in this context, because what fails to import might be importable later when another egg is activated. Generally that pattern is best replaced with setuptools requirements. >>can give you some confidence that >>everything you think you need is installed, but can't provide much >>confidence that everything you *think you are using* is actually what >>you are using. That is, it's been fairly common in my own experience >>for me to realize some other version of a package is being loaded than >>what I thought, or I spend an inordinant amount of time tweaking >>requirements to get the right version of a package from one place >>without affecting another package that needs a different version (or >>perhaps is run with a different sys.path). > > > Is this a setuptools thing or a Python thing? I don't understand what > you mean by "tweaking requirements". This is a setuptools thing... but I guess I was getting ahead of myself. I guess I was alluding to a problem I didn't actually describe, then describing how setuptools tries but doesn't entirely succede in resolving the problem. With the ambiguity of sys.path, it's hard to statically determine if you have all the requirements. Well, with the lack of requirement declarations, you have to scan documentation and imports to actually figure anything out. With the lack of version declarations, you have to scan lots of documentation to figure out if you have the right versions. If you need to resolve a problem in a non-global manner, you can do so with alternate installation locations and sys.path manipulation. How you do the sys.path manipulation is very important, as order is very important in this situation. Side effects from later installations are common depending on how you set it up. Sometimes these side effects are desired (e.g., a security upgrade), sometimes not. Usually not. Also the environment you use is very important -- if your manipulations depend on tweaked scripts or $PYTHONPATH, alternate entry points will often give you a non-working process, or partially-working. In theory, with setuptools you can use a single global installation and let setuptools resolve the requirements and activate the correct packages. So setuptools is doing the sys.path manipulation for you. In practice I have found this to still be too implicit. >>But, back to more concrete use cases: >> >>* Right now it is pretty hard to set up an environment where changes >>elsewhere on the system can't leak in. That is, installation of >>something in a system-wide site-packages can cause problems everywhere >>on the system, and even if you try to avoid these it is quite hard. One >>strategy is setting up an entirely different environment (aka, a >>different prefix); this is heavy-feeling. Another is avoiding site.py >>or using a custom site.py, but the tool support is iffy for that. > > > I have a sense of deja vu -- somehow I feel we've been discussing this > some years ago. Are you sure you didn't post this same message around > the time we were discussing PEP 302? I didn't participate in that discussion, but maybe someone else felt the same way ;) > Seriously, can you provide an example of the sort of thing you're > thinking of? Again, is this setuptools specific or not? Is the problem > caused by users setting their PYTHONPATH a certain way? (If so, how?) No, this is not setuptools specific. Setuptools has some workarounds related site.py (which I assume are themselves going away in Python 2.5, but I don't know)... but I don't think these relate. It's hard to safely remove things from sys.path. So site.py runs extremely early in the process, and adds a bunch of things to sys.path (including .pth processing). If you want to create an isolated environment at that point, it feels like a lost cause -- sys.path has all sorts of spurious entries, and figuring out which ones are important and which ones are not important (and just introduce ambiguity) is hard. PYTHONPATH is a way of adding things to the path, but there's no way of keeping things off the path. >>There's really just no good way to tell Python to leave well enough alone. > > > That's a rather general and vague remark. What do you mean? Mostly what I was talking about up there. Package installations frequently have unintended side effects. And there's lots of complicated machinery in place (often in distribution packaging systems) to handle this complexity, but it'd be nicer if the complexity just didn't exist. >>* Relatedly, installation and management when you don't have root or the >>cooperation of root can be hard. I think the answer to this is much >>like the isolated environment, but the use case is fairly different. > > > This I understand. What do you suggest we do to improve the situation? Mostly the ability to set up a localized environment disassociated from the global installation (except the stdlib). So, what I described up there was mostly my frustrations in managing multiple web applications running on the same server, where the apps require considerably different versions of software. The solution I would prefer -- isolated and localized environments -- would also apply to the hosting situation. >>* Configuration about where to install things (e.g., distutils.cfg) is >>separate from information about where to look for things (sys.path). >>These should form a consistent description of the environment, but >>currently they are disassociated from each other. > > > I didn't even know we have a distutils.cfg. It isn't installed by default, but it's like ~/.pydistutils.cfg for the system. It should exist more often than it does. > I am beginning to believe that you're talking about issues with > installation rather than issues in the import machinery proper. But > maybe if you clear up some of my earlier confusion I'll understand > better. Yeah, I guess I've had installation on the brain for a while, but I think the two issues intersect a lot. Some of the import machinery is complex to handle complex installation situations. I think sys.path and the import machinery ignore installation currently, but I think it's a fake simplicity, since nothing appears on the path unless it is put there by someone. There needs to be consistency between the two. In my mind, there's some concept that encompasses both of these items, and I think if it is identified some of this will be easier to understand. >>* Any kind of automatic installation is difficult, because you can't >>really count on being able to install even the most inoccuous package in >>an automated way. There's too many manual overrides, and too many >>redundant options, and few people actually have their system set up to >>work without tweaking these options through the command line or other >>feedback. > > > Again that's rather vague and general. Please be specific. "Build a > better mousetrap" can't be our requirements spec. Again, it's just a frustration at the lack of integration of installation and importing. >>* Personally I've settled on putting everything I make into a Python >>package that is distutils-installable. But many people don't. I'm not >>sure if this is just because the tools seem too hard, or the namespaces >>feel too deep, or all the documentation starts without using packages, >>or having '.' (sometimes) on sys.path does it, or what. I'd rather >>there be consistent practices; but the consistent practices that we have >>that actually work (setup.py scripts and packages) are too heavy for a >>lot of people. > > > To the contrary, if your entire code fits in a single .py file, I'm > not sure what distutils even buys you, and it sure costs a lot of > complexity. FWIW Google's internal build system agrees with you and > doesn't like 3rd party code that's not in a package. > > I often have a hard time finding the source code in a distutils-based > package that I have unpacked but am hesitant to install just to read > the source code; there seems to be no consistent convention as to > where the source is. For experiments I never distutilify my code to start with. No one does. Certainly it's heavier than it should have to be. I guess, to randomly create a parallel, it's a great feature than in Python every .py file is a module. This gets people writing modular code without even realizing it; good practice is automatic, and doesn't even have any real overhead. There's not much disconnect between the Right Way and the Easy Way. I wish it was the same for packages and distutil code. >>* People are seriously planning on using relative imports to manage >>their packages, and so an application will be 'installed' by putting it >>into another package. Presumably unpacking it directly in some other >>package's directory. Who knows what the version control plans are, or >>maintenance, or whatever. I think it's a bad idea. We need to give >>these people a carrot to keep them from doing this. > > > You seem to have version control on your brain. (Not that that's a bad > thing, but most people don't -- it's a specific way of looking at > things.) I'm not entirely sure I understand what bad practice you're > describing here; and what do you propose to do instead? KSurely not > killing relative import? No, definitely not. I'm just noting that I think relative imports will be used by people who want to avoid the current distutil process, even though relative imports are not a good solution. The stick way of keeping people from doing this would be keeping relative imports out of the language. The carrot way is making the good solution easier than the bad solution. >>* Right now namespace packages are hard. That is, a Python package >>(like 'zope') that is used by several distutils packages. I almost feel >>like namespace packages should be installed flat, like 'zope-interface' >>and 'zope-tal', and turned into namespaces dynamically. > > > Fine. That's the most advanced usage there is. I'd be happy if we > solved all other problems first. Yeah, I'm not even sure how important namespace packages are. They don't actually buy you anything; it's not like zope.interface is magically more modular and elegant than zopeinterface. Maybe it should really just be seen as a case of the stability of imports; zope.interface needs to be supported because people are using it. As distributions are refactored, they don't want to also refactor imports. >>* The module layout is used both as an API and as an internal factoring >>of the code. If you want to refactor the code you break the API. >>Personally I really like the strong connection between imports and code >> location, and appreciate how easily I can find code as a result. But >>setting up the scaffolding and warnings necessary when moving a module >>can be tiresome. > > > I don't expect there's a silver bullet here; backwards compatibility > hacks are always tiresome. No; the most obvious solution of declaring and mapping all external interfaces is not a solution I would care for. Setuptools' entry points also offer some help here, basically by providing a such a declarative frontend, but with the query facilities to actually add extra value. >>* Circular imports should fail more nicely. Everyone suffers this at >>some time; maybe it can't be fixed, but at least it should be clear >>what's happening. > > > It probably can't be fixed (or do you see a fix?). Do you mean it > should be easier to debug, or do you mean it should be explained > better how things work? I don't see any fix. Maybe just something like a giving AttributeError('partially-loaded module foo has no attribute bar') instead of the really unhelpful AttributeError("'module' object has no attribute 'foo'"). Just some error that lets the developer know that a circular import is happening. >>* You can't really tell if "from foo import bar" can be written as >>"import foo; bar = foo.bar", because it works if foo contains bar, but >>not if foo is a package and bar is a module in that package. > > > And is that bad? "from foo import bar" means what it means. Nobody > said the other code is equivalent. If you know what foo is you will > know whether the other code is equivalent; isn't that enough? There's no explicit namespace operator, so you don't necessarily know what foo is, nor do you have to. But it's a minor issue. >>Well... I think that's maybe half way through the list of issues I have, >>but this email is already much too long. > > > And still not clear enough. It's difficult to communicate when one > party has spent a long time exploring the issues and another is just > starting; this time I seem to be on the receiving end (== haven't > spent a lot of time). Quite understandable; maybe a few iterations of this and I'll make myself clear ;) I'm also at a loss in some ways, as I'm only really familiar with how Python works. There's probably some useful mechanisms and terminology from other languages that we could learn from -- I've heard interesting things about .NET, for instance. OTOH, there's a *lot* of really bad examples in other languages too ;) When I see references to package management in other languages (especially dynamic ones), most of the problems are very recognizable. I actually suspect a there's a set of practices here which is language-neutral, but I don't know of any community where they could really be worked on. Linux distributions, perhaps, but many of the people involved in packaging come off as surprisingly stodgy. They've spent so much time working around these problems that they dislike any attempt to fix the problem. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Thu Apr 20 19:55:39 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 20 Apr 2006 12:55:39 -0500 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: <4447CB1B.1060306@colorstudy.com> Guido van Rossum wrote: > On 4/20/06, Greg Ewing wrote: > >>Some wilder ideas for keyword-only arguments: >> >> def spam(a, b, c, {d, e, f=x}): >> # d, e are mandatory keyword-only >> # f is optional keyword-only > > > IMO anything using any kind of nested brackets inside the argument > list is doomed. Such syntax is too close to resembling a single > argument with a complex sub-structure, even if there's a prefix > operator. I thought I remembered some talk about removing sub-structure from functions signatures in py3k...? It's a pretty obscure feature, and when the substructure pattern doesn't match the call the exception is hard to decypher. >>> def p((x, y)): pass >>> p(1) Traceback (most recent call last): File "", line 1, in ? File "", line 1, in p TypeError: unpack non-sequence ...that exception is actually considerably worse than I expected. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From rasky at develer.com Thu Apr 20 20:14:01 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 20 Apr 2006 20:14:01 +0200 Subject: [Python-3000] Changing the import machinery References: <4447B91A.8090409@colorstudy.com> Message-ID: <044501c664a6$3361bd30$bf03030a@trilan> Guido van Rossum wrote: >> * Circular imports should fail more nicely. Everyone suffers this at >> some time; maybe it can't be fixed, but at least it should be clear >> what's happening. > > It probably can't be fixed (or do you see a fix?). Do you mean it > should be easier to debug, or do you mean it should be explained > better how things work? Currently, if you hit a recursive import, it either works (if the name that's been looking for in the module which causes the recursion was already bound) or doesn't work and abort with ImportError/AttributeError/NameError (if the name was not bound yet). I think this behaviour is inconsitent, and the error message (along the lines of "foo is not in module bar") is not meaningful because "foo" *is* defined in module "bar", just not *yet*. I guess there are a couple of things we can do: 1) Generate a better error message. If an import statement does not find name A in module M, and module M is being recursively imported, the error message could hint at the problem. 2) Totally disallow recursive imports (!). If import is called for module A, while module was already *being* imported, an ImportError is generated with message "recursive import of module A". The stack trace should make pretty clear the exact source of the problem. Of course, there should be better ways of doing this... -- Giovanni Bajo From guido at python.org Thu Apr 20 20:27:07 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 19:27:07 +0100 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <20060420084537.A83B.JCARLSON@uci.edu> References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> <20060420084537.A83B.JCARLSON@uci.edu> Message-ID: Let me just add that Andy is wasting his time. Python 3000 won't have continuations. On 4/20/06, Josiah Carlson wrote: > > Andy Sy wrote: > > Err... thank you once again for being redundant, reiterating what is > > already pretty much dead clear to everyone and saying nothing new > > in general. > > And yet, here I notice that you've still not described a syntax or its > execution semantics. Instead you've chosen to spend your time insulting > me. Please continue, I'm sure it will make people want to read your > future posts all the more. > > > Ultimately, Guido is the judge of what is or is not Pythonic. Without a > description of what you want, no one can be for or against it, and > ultimately, you are playing yourself to be very much like every other > first-poster in python-dev or python-3000; 'I think Python would be > better with feature/syntax/language construct X'. This is not a mortal > sin. > > I think it's about time to use a Tim Peters inspired closing. > > to-inattentive-to-really-be-insulted-ly y'rs - Josiah > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Thu Apr 20 21:32:40 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Thu, 20 Apr 2006 15:32:40 -0400 Subject: [Python-3000] Changing the import machinery In-Reply-To: <4447C9B2.8050009@colorstudy.com> References: <4447B91A.8090409@colorstudy.com> <4447C9B2.8050009@colorstudy.com> Message-ID: On 4/20/06, Ian Bicking wrote: > With the ambiguity of sys.path, it's hard to statically determine if you > have all the requirements. [And, as you said elsewhere, whether it is the version you expected, and whether it will actually load or get skipped because of an earlier import] Would you like the ability to specify precisely which file a requested module gets loaded from, and to load your own copy if the already-imported version has the wrong __file__ attribute? > If you need to resolve a problem in a non-global manner, you can do so > with alternate installation locations and sys.path manipulation. If you need to resolve it in a non-global manner, doesn't that indicate that you're working with a specific application, which can safely run in (and modify the sys.path of) its own process? > Mostly the ability to set up a localized environment disassociated from > the global installation (except the stdlib). Are you saying that you would like to lock the global environment, so that you won't see anything except the original stdlib without an explicit change to your own application? If so, is the problem just that so many existing applications put themselves on the global path, and [in a hosting situation] these may be installed by someone else, without your knowledge? In theory, the import path could be controlled directly by each importing module, but in practice, I think that would lead to a bigger mess as soon as there were conflicts (= as soon as it starts to matter). > I'm just noting that I think relative imports will > be used by people who want to avoid the current distutil process, Absolutely! They also protect against version skew, which can be particularly important if you're relying on a patched version. > even though relative imports are not a good solution. Why not? You won't get the benefit of automatic upgrades to components you rely upon, but you won't get the risk either. > it's not like zope.interface is > magically more modular and elegant than zopeinterface. One of the objections to the python documentation is that it can be hard to find things if you don't already know where to look. The module index is an obvious starting point, but is way too long, because it lists every module instead of just packages and independent modules. Having to group on zope* rather than zope.* is a step backwards. Looking at just the stdlib, are mailbox and mailcap related? operator and optparse? I don't need to know anything about Carbon.* to know that they're all related; if I have a summary of the Carbon package ("Mac only system bindings") I get to skip the whole batch. -jJ From ianb at colorstudy.com Thu Apr 20 22:07:19 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Thu, 20 Apr 2006 15:07:19 -0500 Subject: [Python-3000] Changing the import machinery In-Reply-To: References: <4447B91A.8090409@colorstudy.com> <4447C9B2.8050009@colorstudy.com> Message-ID: <4447E9F7.5020300@colorstudy.com> Jim Jewett wrote: > On 4/20/06, Ian Bicking wrote: > > >>With the ambiguity of sys.path, it's hard to statically determine if you >>have all the requirements. > > > [And, as you said elsewhere, whether it is the version you expected, > and whether it will actually load or get skipped because of an earlier > import] > > Would you like the ability to specify precisely which file a requested > module gets loaded from, and to load your own copy if the > already-imported version has the wrong __file__ attribute? Probably not, but I guess it would depend on the specifics. In many cases it is not safe to load two versions of the same module into the same process. Allowing for that would be a fairly major change to what it means to be a module, even if it's also a rather subtle change. Having accidentally imported a single module multiple times under different names, it's made me pretty uncomfortable with the idea. That said, in some cases it is possible, and I do just that. Like, if I want string.Template, I do: try: from string import Template except ImportError: from mypackage.backports.string24 import Template Doing this in a more elegent or formalized fashion might be nice. Versioning the standard library should also probably be a bullet point in here somewhere -- I think it's an important aspect to keeping stdlib modules from being forceably put into an eternal freeze. >>If you need to resolve a problem in a non-global manner, you can do so >>with alternate installation locations and sys.path manipulation. > > > If you need to resolve it in a non-global manner, doesn't that > indicate that you're working with a specific application, which can > safely run in (and modify the sys.path of) its own process? I guess I see it as a deployment task, not an application development task, though it would be possible to address it as an application development task. As I noted elsewhere in one of these emails, it's hard to safely remove items from sys.path, so it's hard for an application to fix up sys.path unless it gets its hands on it very early in the process. When this goes wrong it can be very frustrating to debug. For an application installed as a OS distribution (rpm, etc) I think this is absolutely the strategy they should take, because the distribution created in that situation is a merging of both development and deployment. That's also what makes packaging work hard, so simplifying that would be nice. >>Mostly the ability to set up a localized environment disassociated from >>the global installation (except the stdlib). > > > Are you saying that you would like to lock the global environment, so > that you won't see anything except the original stdlib without an > explicit change to your own application? Yes. "Locking" is pretty simple, really -- just don't put site-packages on the path, which in turn is accomplished by not running site.py. So current convention already gets us most of the way there. It's just too damn hard to get the rest of the way :( > If so, is the problem just that so many existing applications put > themselves on the global path, and [in a hosting situation] these may > be installed by someone else, without your knowledge? Yes, I think this is basically the problem. And that "someone else" may be a past or future self, who didn't know the best way to install things or else forgot the best way. Whenever I figure out a good set of command-line arguments to do what I want, I always forget later. And certainly anyone else who comes along will be entirely lost. > In theory, the import path could be controlled directly by each > importing module, but in practice, I think that would lead to a bigger > mess as soon as there were conflicts (= as soon as it starts to > matter). To a degree this is what Setuptools is doing, but indeed conflicts happen and can be hard to resolve. >>I'm just noting that I think relative imports will >>be used by people who want to avoid the current distutil process, > > > Absolutely! > > They also protect against version skew, which can be particularly > important if you're relying on a patched version. > > >>even though relative imports are not a good solution. > > > Why not? You won't get the benefit of automatic upgrades to > components you rely upon, but you won't get the risk either. For all the same reasons that importing based on __file__ is challenging. It breaks the rules -- modules don't get a canonical name, they may exist multiple times in the same process, and a lot of the distinctions between packages and maintainers are covered up. Sometimes it is fine -- like if you are backporting something, or you want to grab some small module without having to put in another requirement, maybe using svn:externals. As a general means of reuse I don't think it's a good idea. >>it's not like zope.interface is >>magically more modular and elegant than zopeinterface. > > > One of the objections to the python documentation is that it can be > hard to find things if you don't already know where to look. The > module index is an obvious starting point, but is way too long, > because it lists every module instead of just packages and independent > modules. > > Having to group on zope* rather than zope.* is a step backwards. > Looking at just the stdlib, are mailbox and mailcap related? operator > and optparse? I don't need to know anything about Carbon.* to know > that they're all related; if I have a summary of the Carbon package > ("Mac only system bindings") I get to skip the whole batch. I think this is a separate issue. The ToC offers more of a categorization, but happens to be layed out in a way that is hard to scan. The module index is easy to scan given a name, but hard given a target functionality. Also, there's not enough editorial control -- for instance, to categorize deprecated modules differently from other modules. Even the index should have editorial control IMHO. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From andy at neotitans.com Thu Apr 20 22:24:38 2006 From: andy at neotitans.com (Andy Sy) Date: Fri, 21 Apr 2006 04:24:38 +0800 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> <20060420084537.A83B.JCARLSON@uci.edu> Message-ID: <4447EE06.90705@neotitans.com> Guido van Rossum wrote: > Let me just add that Andy is wasting his time. Python 3000 won't have > continuations. Huh? Futures are very different from continuations. I still have a hard time understanding continuations (and am no fan of them), but futures seem to be a rather simple abstraction to comprehend. It just seems like clean syntax to do general asynchronous stuff. The links I posted earlier: http://www.ps.uni-sb.de/alice/manual/futures.html http://kasparov.skife.org/blog-live/src/futures.writeback http://java.sun.com/j2se/1.5.0/docs/api/java/util/concurrent/Future.html discuss the abstraction in greater depth. Note that I'm not even lobbying to see futures in Py3K, just that maybe checking them out may inspire some ideas regarding related mechanisms in Python. It's just something I came across that I thought might be of interest. From pje at telecommunity.com Thu Apr 20 22:28:15 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 20 Apr 2006 16:28:15 -0400 Subject: [Python-3000] Changing the import machinery In-Reply-To: Message-ID: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> At 04:33 PM 4/20/2006 +0100, "Guido van Rossum" wrote: >On 4/20/06, Fredrik Lundh wrote: > > Guido van Rossum wrote: > > > > > On 4/20/06, in python-dev, Fredrik Lundh wrote: > > > > I was hoping that for Python 3.0, we could get around to unkludge the > > > > sys.path/meta_path/path_hooks/path_importer_cache big ball of hacks, > > > > possibly by replacing sys.path with something a bit more > intelligent than > > > > a plain list. > > > > > > That's an excellent idea. Are there any volunteers here to help out? > > > Even just listing specific use cases / scenarios that are currently > > > difficult to solve right would be tremendously helpful. (I think that > > > Phillip's and others' experience with setuptools might be very > > > useful.) > > > > how far would a plain > > > > the path can contain either directory names or import handlers > > > > modification take us ? (where import handlers are duck-typed objects > > with an as-small-as-possible interface, and the handler for builtins and > > frozen objects are included in the list). > >Probably all the way. PEP 302's rationale discusses some problems that >explain why this wasn't done in 2.x; perhaps those concerns can be >swept in 3.0. I'm afraid I disagree. PEP 302 actually has some tremendous advantages over a pure objects-on-sys.path approach: * Strings can be put in any configuration file, and used in .pth files * Strings can be put in environment variables (like PYTHONPATH). * Strings can be printed out, with all their contents showing and nothing hidden In short, strings are better for humans. There are only two things wrong with PEP 302 IMO, and neither is its "fault". The first is that the "classic" import machinery isn't on sys.meta_path, and the 'imp' API isn't defined in terms of PEP 302. Those two things can't change without introducing backward incompatibilities. The second is that PEP 302 only covers "location" importers, not "format" importers. That is, if you want to do something like make Kid or Cheetah templates importable, you have to replace things with new machinery. This is a more important problem to solve, IMO, than coming up with yet another set of semantics for the things that *are* covered by PEP 302. PEP 302 is reasonably clear and concise, although there are occasional loopholes. At least three people have re-implemented parts of it in pure Python from the spec: Just van Rossum, Neal Norwitz, and myself (twice, in my case). It is easy and elegant -- the only hard parts are when you have to wedge all the "classic" imports in, or if you want to implement "format" importers. So, my vote for Py3K is to move the classic import machinery to explicit objects on sys.meta_path -- preferably one for each distinct rule, so that it is possible to *remove* them if you don't want them. (For example, the Chandler folks have made a big deal about not wanting general Windows registry imports to apply; it would be nice if they could just delete an object from sys.meta_path rather than having to build their own Python to elide this.) I would also suggest that we ask Just if he can come up with some elegant extension that would allow PEP 302 to support pluggable import *formats* as well as locations. :) From mcherm at mcherm.com Thu Apr 20 22:46:05 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 20 Apr 2006 13:46:05 -0700 Subject: [Python-3000] Futures in Python 3000 Message-ID: <20060420134605.clnsox3h1544kwsk@login.werra.lunarpages.com> Andy Sy writes: > Does this mean that Py3K intends to reuse major portions of > Python 2.x's implementation? Aahz already answered this, but I'd like to restate more forcefully: It is a FUNDAMENTAL PRECONDITION of Py3K that it will BE the existing CPython codebase, with a few changes and some cruft removed. We are NOT going to re-write from the ground up. If you don't know why, go read "http://www.joelonsoftware.com/articles/fog0000000069.html". -- Michael Chermside From guido at python.org Thu Apr 20 22:54:03 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 21:54:03 +0100 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4447EE06.90705@neotitans.com> References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> <20060420084537.A83B.JCARLSON@uci.edu> <4447EE06.90705@neotitans.com> Message-ID: On 4/20/06, Andy Sy wrote: > Guido van Rossum wrote: > > > Let me just add that Andy is wasting his time. Python 3000 won't have > > continuations. > > Huh? Futures are very different from continuations. I still have a > hard time understanding continuations (and am no fan of them), but > futures seem to be a rather simple abstraction to comprehend. Ok, sorry. I was confused by the mentioning of continuations in an earlier response. > It just seems like clean syntax to do general asynchronous stuff. Right. So go right ahead and implement them using threads. Or, actually, you don't have to do anything -- the Queue module already implements something that's more powerful and just as easy to use. If you want to contribute a pure Python module that implements some useful form of futures, you don't have to wait for Python 3.0. But before you contribute this to Python 2.6 (it's too late for 2.5), you should probably have released it successfully as a separate 3rd party (open source) module for a while, and have many happy users. That's the standard for library contributions these days. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Thu Apr 20 23:07:10 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 20 Apr 2006 17:07:10 -0400 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> <20060420084537.A83B.JCARLSON@uci.edu> <4447EE06.90705@neotitans.com> Message-ID: <1f7befae0604201407hc28f635wbae404d3a062cb99@mail.gmail.com> [Guido, to Andy Sy] > ... > If you want to contribute a pure Python module that implements some > useful form of futures, you don't have to wait for Python 3.0. But > before you contribute this to Python 2.6 (it's too late for 2.5), you > should probably have released it successfully as a separate 3rd party > (open source) module for a while, and have many happy users. That's > the standard for library contributions these days. Note that Twisted has a big head start on this, with its Deferred objects: Asynch programming wasn't at all a goal of generators, and Twisted is well worth looking into for those who want slicker asynch programming tools. From guido at python.org Thu Apr 20 23:15:42 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 22:15:42 +0100 Subject: [Python-3000] Chaning the import machinery; was: Re: [Python-Dev] setuptools in 2.5. In-Reply-To: <1145549216.12589.178.camel@localhost.localdomain> References: <200604201456.13148.anthony@interlink.com.au> <200604201801.29943.anthony@interlink.com.au> <444747EA.1010707@livinglogic.de> <44477849.7000806@livinglogic.de> <44479678.3070803@livinglogic.de> <1145549216.12589.178.camel@localhost.localdomain> Message-ID: On 4/20/06, Jan Claeys wrote: > A man-in-the-middle-attack and other ways to "inject" a different module > than intended by the author are also possible with the current default > filesystem based imports, so I don't think that's a good argument > against http-imports (or other similar extensions to import). Do you know much about security? If it's not safe to go out at night in your neighborhood, do you use that as an argument that you also shouldn't get out of bed at night to go to the bathroom? Trusting my local disk is asking a lot less than trusting the entire internet. If my machine has been compromised ALL bets are off and I might as well worry about whether 1+1 is still 2. There are many ways to reach a sufficient level of trust in a machine you own (although a small amount of paranoia is fine). But regarding the internet, the only way to survive is a healthy dose of paranoia, combined with state of the art encryption and authentication etc. (And yes, I know for a fact there are no monsters under my bed. I asked, and they said "no". :-) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From mcherm at mcherm.com Fri Apr 21 00:12:55 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Thu, 20 Apr 2006 15:12:55 -0700 Subject: [Python-3000] Futures in Python 3000 Message-ID: <20060420151255.o8m8q8xgrnk080g8@login.werra.lunarpages.com> Guido writes > Let me just add that Andy is wasting his time. Python 3000 won't have > continuations. Andy writes: > Huh? Futures are very different from continuations. I still have a > hard time understanding continuations (and am no fan of them), but > futures seem to be a rather simple abstraction to comprehend. Andy: You are right... futures are not continuations, and there is no reason for Guido to fear them. However, here me out on the idea of futures in Python: Python HAS futures. They are implemented using the Deferred object in Twisted. Anyone writing Python code in a callback style uses the Twisted library, it's the standard tool for that job in Python (and it's quite good at what it does). Unfortunately, it does require you to turn your code "inside out", and you probably want to avoid that. Some languages (like alice) manage to implement futures in a way that allows code to be written "right-side out". But this requires the entire language being built around such a concept. I have heard Bruce Eckel put forth the idea of supporting "Active Objects" which would be a related concept with similar benefits, but even then he never suggested supporting it for all objects. I doubt anyone would seriously consider totally redesigning the core of Python to allow futures. So the best hope for futures would be if they can be implemented without core modifications. I can imagine, for example, an object called 'Future', that was used like this: widget = Future( long_running_get_widget_func, arg1, arg2 ) x = other_slow_function(argA, argB) widget.set_color( x ) The Future object would launch the long-running function in a separate thread then return a proxy object. When attributes of the proxy object were accessed, they would block if the function was still running, and otherwise forward to the result of the function. There are design points to be addressed about whether there would be explicit joins and so forth. The good news is that this is completely possible in Python today! So there is no need to bring this up on the Python 3000 discussion group. Instead, go out there and write the "Future" library, then make it available on cheeseshop.python.org. Promote your library on comp.lang.python. Once your library is popular enough, propose adding it to the standard library. And none of this needs to wait for Py3K... it could all happen by 2.6! Seriously... futures are a GREAT tool. The above is a way to get it moving in Python without needing support from the core language team... and that means it gets done faster! -- Michael Chermside From guido at python.org Fri Apr 21 00:26:58 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 20 Apr 2006 23:26:58 +0100 Subject: [Python-3000] Help needed in py3k branch Message-ID: There are still a bunch of tests failing in the py3k (sic) branch: test_class fails because hash() of a new-style class that defines __eq__ or __cmp__ doesn't raise an exception. There's a design issue here: should the default hash be changed to raise an exception in this case (perhaps triggering other test failures) or should the tests be deleted? (This is from a test module that used to test classic classes.) test_exception fails with a SystemError for popping a non-exception off the stack test_importhooks fails test_pickletools fails test_pyclbr fails test_scriptpackages fails a bunch of tests for XML modules all fail (perhaps some of these are related): minidom, sax, xdrlib, xml_etree I recently fixed repr.py and test_repr.py, but I wonder if we shouldn't delete that module... If anyone wants to help out debugging these, please do! If you solve one, just mail me a patch (or check it in if you've got checkin perms and feel confident -- just still drop me a line). I'd really like the p3yk branch to be error-free so that if further changes cause new errors, the failures are easily attributable. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From thomas at python.org Fri Apr 21 00:40:35 2006 From: thomas at python.org (Thomas Wouters) Date: Fri, 21 Apr 2006 00:40:35 +0200 Subject: [Python-3000] Help needed in py3k branch In-Reply-To: References: Message-ID: <9e804ac0604201540w268636fcl76bfaaf3fa4a8ff@mail.gmail.com> (Sorry, Guido, meant to reply to list, not to you privately.) On 4/21/06, Guido van Rossum wrote: > test_exception fails with a SystemError for popping a non-exception > off the stack I already fixed that one (it's not just a test fix, it's an actual bug, so per your request I didn't check it in. It's an SF patch assigned to you.) test_pyclbr fails I looked at this, and thought I'd fixed it... IIRC, it was a class-hierarchy check that didn't take 'object' into account. a bunch of tests for XML modules all fail (perhaps some of these are > related): minidom, sax, xdrlib, xml_etree I'm not sure if they're related, but I do believe they're all caused by classic-vs-new-style class semantics. Oh, no, xdrlib is a truedivision issue. Fixed that, now. (xdrlib isn't an XML module anyway, it's Sun RPC-related.) If anyone wants to help out debugging these, please do! If you solve one, > just mail me a patch (or check it in if you've got checkin perms and feel > confident -- just still drop me a line). Oh, I guess I'll check in the exceptions fix, then ;-) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060421/f6d2fc9d/attachment.htm From brett at python.org Fri Apr 21 01:00:23 2006 From: brett at python.org (Brett Cannon) Date: Thu, 20 Apr 2006 16:00:23 -0700 Subject: [Python-3000] Help needed in py3k branch In-Reply-To: References: Message-ID: On 4/20/06, Guido van Rossum wrote: > There are still a bunch of tests failing in the py3k (sic) branch: > > test_class fails because hash() of a new-style class that defines > __eq__ or __cmp__ doesn't raise an exception. There's a design issue > here: should the default hash be changed to raise an exception in this > case (perhaps triggering other test failures) or should the tests be > deleted? (This is from a test module that used to test classic > classes.) > I say raise an exception. iter() raises a TypeError if you pass it an object it can't get an iterator for. hash() should raise an exception for an object that cannot be used as a key in a dict. -Brett From greg.ewing at canterbury.ac.nz Fri Apr 21 02:41:43 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Apr 2006 12:41:43 +1200 Subject: [Python-3000] Chaning the import machinery; was: Re: [Python-Dev] setuptools in 2.5. In-Reply-To: <44479678.3070803@livinglogic.de> References: <200604201456.13148.anthony@interlink.com.au> <200604201652.51448.anthony@interlink.com.au> <44473D71.6060107@v.loewis.de> <200604201801.29943.anthony@interlink.com.au> <444747EA.1010707@livinglogic.de> <44477849.7000806@livinglogic.de> <44479678.3070803@livinglogic.de> Message-ID: <44482A47.8020001@canterbury.ac.nz> Walter D?rwald wrote: > import foo from url("file:~guido/python-packages/foo") Hard-wiring pathnames, URLs etc. into one's code on a regular basis would be an extremely BAD idea. The way to do that is via configuration mechanisms external to the program. It might be useful to have the ability for use in very specialised circumstances, such as loading plugins. But it would be rarely used, and there shouldn't be any new syntax for it, just a function. -- Greg From greg.ewing at canterbury.ac.nz Fri Apr 21 02:48:33 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Apr 2006 12:48:33 +1200 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: References: Message-ID: <44482BE1.8030109@canterbury.ac.nz> Jim Jewett wrote: > I also see the value of keyword-only arguments without unlimited > positional arguments. Whether it deserves syntax and what that syntax > should be (such as * or *None) may need a pronouncement eventually, > but it doesn't have to be complex. My previous post was just wild brainstorming. I'd be perfectly happy with def f(a, b, *, c, d = x): # c and d are keyword only, c is required, # no extra positional args allowed I don't buy the argument that bare * should mean "extra positional args ignored", since there's already a way of saying that: def f(a, b, *others): # and then don't bother doing anything with others As shown above, I don't see any extra difficulty there. -- Greg From nas at arctrix.com Fri Apr 21 06:49:50 2006 From: nas at arctrix.com (Neil Schemenauer) Date: Fri, 21 Apr 2006 04:49:50 +0000 (UTC) Subject: [Python-3000] Is reference counting still needed? References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: Erno Kuusela wrote: > The refcounting vs generational GC reasoning I've heard argues that > refcounting is less cache-friendly I believe that's correct. A state of the art generational GC would outperform reference counting, even given Python's enormous allocation rate. However, those systems require man years of development effort and are not widely portable. Perhaps even more seriously, any system that could outperform refcounting would require moving objects (at least, AFAIK). If we would want to use such a system, extensions would have to be written in a completely different manner. The portability problem might be solved by have a pluggable GC system: high performance on platforms that support it, simple implementation (e.g. mark and sweep) on platforms that don't. Neil From rrr at ronadam.com Fri Apr 21 07:54:30 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 21 Apr 2006 00:54:30 -0500 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: <44482BE1.8030109@canterbury.ac.nz> References: <44482BE1.8030109@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Jim Jewett wrote: > >> I also see the value of keyword-only arguments without unlimited >> positional arguments. Whether it deserves syntax and what that syntax >> should be (such as * or *None) may need a pronouncement eventually, >> but it doesn't have to be complex. > > My previous post was just wild brainstorming. I'd be > perfectly happy with > > def f(a, b, *, c, d = x): > # c and d are keyword only, c is required, > # no extra positional args allowed Or just ... def f(a, b, c=?, d=x): I think this might be clearer if some suitable replacement for the '?' in the above could be decided on. This *may* relate to None being an object which isn't the same as "not a value". There currently isn't a way (that I know of) to specify a generally null object outside of sequences. def f(a, b, c=Null, d=x): # Using None here wouldn't work. Recently I found a case where I wanted to return something that was more literally *nothing* than a None is. So maybe a null symbol of some sort might be useful in other cases as well? Cheers, Ron From jcarlson at uci.edu Fri Apr 21 08:28:20 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Thu, 20 Apr 2006 23:28:20 -0700 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: References: Message-ID: <20060420230332.F1DF.JCARLSON@uci.edu> Neil Schemenauer wrote: > Erno Kuusela wrote: > > The refcounting vs generational GC reasoning I've heard argues that > > refcounting is less cache-friendly > > I believe that's correct. A state of the art generational GC would > outperform reference counting, even given Python's enormous > allocation rate. However, those systems require man years of > development effort and are not widely portable. > > Perhaps even more seriously, any system that could outperform > refcounting would require moving objects (at least, AFAIK). If we > would want to use such a system, extensions would have to be written > in a completely different manner. > > The portability problem might be solved by have a pluggable GC > system: high performance on platforms that support it, simple > implementation (e.g. mark and sweep) on platforms that don't. In my mind, the real question is whether or not the possible speed improvements on those platforms is worth the "man years of development effort" (smells doubtful). If someone begins such an undertaking, it would also be quite nice if the effort could result in a non-moving GC (for us C extension writers). - Josiah Also; great work on the original set of Bohem GC patches. When Tim posted the link on Wednesday, I almost couldn't believe that it had already been done. I had heard occasional references to someone trying it before in python-dev, and it was good to read through that thread. From guido at python.org Fri Apr 21 08:46:58 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 07:46:58 +0100 Subject: [Python-3000] Help needed in py3k branch In-Reply-To: <9e804ac0604201540w268636fcl76bfaaf3fa4a8ff@mail.gmail.com> References: <9e804ac0604201540w268636fcl76bfaaf3fa4a8ff@mail.gmail.com> Message-ID: On 4/20/06, Thomas Wouters wrote: > On 4/21/06, Guido van Rossum wrote: > > a bunch of tests for XML modules all fail (perhaps some of these are > > related): minidom, sax, xdrlib, xml_etree > > I'm not sure if they're related, but I do believe they're all caused by > classic-vs-new-style class semantics. The minidom problems seem to be mostly due to excessive backwards compatibility hacks (related to classic classes and more). Shall we just kill minicompat.py? It's got hacks for supporting versions all the way back to 2.1... > Oh, I guess I'll check in the exceptions fix, then ;-) Thanks! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From just at letterror.com Fri Apr 21 09:02:51 2006 From: just at letterror.com (Just van Rossum) Date: Fri, 21 Apr 2006 09:02:51 +0200 Subject: [Python-3000] Changing the import machinery In-Reply-To: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: Phillip J. Eby wrote: > There are only two things wrong with PEP 302 IMO, and neither is its > "fault". > > The first is that the "classic" import machinery isn't on > sys.meta_path, and the 'imp' API isn't defined in terms of PEP 302. > Those two things can't change without introducing backward > incompatibilities. The former was planned for implementation in the realease following basic 302 support (and I don't see any potential compatibility issues). I just never got round to actually doing that, probably having to do with the fact that hacking import.c stops being "fun" after a while... I appreciate you defending PEP 302, it's pretty neat. However, I'm not convinced that we need all that much flexibility. Back then it seemed to satisfy most people's needs, while still keeping maximum compatibility. Still it may be somewhat overengineered. The original idea of simply allowing importer-ish objects on sys.path (or just callables) may be feasible for p3yk, although I would really like to see import semantics to be simplified. Eg. after all those years I still don't like pkg.__path__ at all. I proposed to kill back in the forming days of PEP 302, but was unable to convince Guido. Not sure if it's smart to try again four years later :) Whatever we end up with, I think it would be a tremendous win if we could implement most of the import logic in Python. Phillip: I'm not sure what "format importers" are or do. Can you give an example of how you'd use those? Just From greg.ewing at canterbury.ac.nz Fri Apr 21 10:10:07 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Apr 2006 20:10:07 +1200 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <20060420085750.GD7995@crater.logilab.fr> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> <4446765C.2030306@neotitans.com> <44473F6F.9020809@canterbury.ac.nz> <20060420085750.GD7995@crater.logilab.fr> Message-ID: <4448935F.20605@canterbury.ac.nz> Aur?lien Camp?as wrote: > On Thu, Apr 20, 2006 at 07:59:43PM +1200, Greg Ewing wrote: > > Andy Sy wrote: > > > Does this mean that Py3K intends to reuse major portions of > > > Python 2.x's implementation? > > I expect that almost all of it will be reused. > > Couldn't PyPy be considered an interesting contender ? That would be far too big a change. Last time I read anything about PyPy, it was even more head-exploding than metaclasses. > Sorry to jump in like this, but what about the *current* stackless ? > Does it also break all existing C extensions ... ? As far as I know, the current Stackless is some sort of compromise that keeps the current C API but doesn't always work properly (I seem to recall there being some trouble with the way Tkinter uses the C stack, for example). Then there's something around called "greenlets" that was supposed to be some brilliant stroke of genius that solved all these problems, but I haven't heard anything about it since. So currently there seems to be no magic bullet for these problems. -- Greg From greg.ewing at canterbury.ac.nz Fri Apr 21 10:39:48 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Apr 2006 20:39:48 +1200 Subject: [Python-3000] Changing the import machinery In-Reply-To: <044501c664a6$3361bd30$bf03030a@trilan> References: <4447B91A.8090409@colorstudy.com> <044501c664a6$3361bd30$bf03030a@trilan> Message-ID: <44489A54.3060108@canterbury.ac.nz> Giovanni Bajo wrote: > 2) Totally disallow recursive imports (!). That would be extremely undesirable. I've used languages in which mutual imports weren't possible, and it's a massive pain in the posterior. You end up having to modularise things in awkward and unnatural ways to get around the restriction. Possibly it might work to allow mutual imports only of the form "import spam" and not "from spam import grail", but I'm not sure about that. -- Greg From greg.ewing at canterbury.ac.nz Fri Apr 21 11:16:27 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Apr 2006 21:16:27 +1200 Subject: [Python-3000] Changing the import machinery In-Reply-To: <4447E9F7.5020300@colorstudy.com> References: <4447B91A.8090409@colorstudy.com> <4447C9B2.8050009@colorstudy.com> <4447E9F7.5020300@colorstudy.com> Message-ID: <4448A2EB.8020401@canterbury.ac.nz> Ian Bicking wrote: > try: > from string import Template > except ImportError: > from mypackage.backports.string24 import Template > > Doing this in a more elegent or formalized fashion might be nice. Have you seen my proposal for "or" in import statements? Would you consider that elegant enough? > Versioning the standard library My thought on versioning is that it should look something like import spam[3:], movies.grail[2:4] # version 3 or later of spam, version 2, 3 or 4 of movies.grail I would like this much better than some kind of require() function. Integrating the version specification with the import would solve some problems, such as the one mentioned earlier where a tentative import fails when it would have succeeded if some later require() had been done first. The form that the version numbers should take is up to debate, but I'd tend to keep it extremely simple, i.e. just integers. The baroque possibilities offered by the setuptools require() seem like overkill to me. -- Greg From greg.ewing at canterbury.ac.nz Fri Apr 21 11:33:10 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Apr 2006 21:33:10 +1200 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4447EE06.90705@neotitans.com> References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> <20060420084537.A83B.JCARLSON@uci.edu> <4447EE06.90705@neotitans.com> Message-ID: <4448A6D6.30003@canterbury.ac.nz> Andy Sy wrote: > Huh? Futures are very different from continuations. I still have a > hard time understanding continuations (and am no fan of them), but > futures seem to be a rather simple abstraction to comprehend. Isn't a future just a coroutine, or something equivalent? Sometimes there's a bit of confusion when people use the terms "coroutine" and "continuation" interchangeably. They're not quite the same thing. True continuations can be "resumed" more than once from the same state, whereas a coroutine can only be resumed once, after which its state has changed. This makes true continuations a good deal more complicated to implement, *unless* you've planned your entire language implementation from the outset with support for them in mind. CPython was *not* so designed. Coroutines are somewhat easier, but still present considerable difficulties for an implementation that works the way CPython does. Researching the history of Stackless will show why. Attempting to channel Guido a bit here, I'd say he also means that Py3k will not have *coroutines* either, or anything equivalent to coroutines, except insofar as they are provided by the operating system in the form of threads. Generators already go as far as it is possible to go into coroutine-land without unacceptably large changes to the implementation. -- Greg From guido at python.org Fri Apr 21 11:42:37 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 10:42:37 +0100 Subject: [Python-3000] Changing the import machinery In-Reply-To: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: On 4/20/06, Phillip J. Eby wrote: > I'm afraid I disagree. PEP 302 actually has some tremendous advantages > over a pure objects-on-sys.path approach: > > * Strings can be put in any configuration file, and used in .pth files > > * Strings can be put in environment variables (like PYTHONPATH). > > * Strings can be printed out, with all their contents showing and nothing > hidden > > In short, strings are better for humans. I think I like this. I wonder if there's a parallel with my preference for strings as paths instead of path objects... > There are only two things wrong with PEP 302 IMO, and neither is its "fault". > > The first is that the "classic" import machinery isn't on sys.meta_path, > and the 'imp' API isn't defined in terms of PEP 302. Those two things > can't change without introducing backward incompatibilities. So it's a perfect py3k project. > The second is that PEP 302 only covers "location" importers, not "format" > importers. That is, if you want to do something like make Kid or Cheetah > templates importable, you have to replace things with new machinery. This > is a more important problem to solve, IMO, than coming up with yet another > set of semantics for the things that *are* covered by PEP 302. > > PEP 302 is reasonably clear and concise, although there are occasional > loopholes. At least three people have re-implemented parts of it in pure > Python from the spec: Just van Rossum, Neal Norwitz, and myself (twice, in > my case). It is easy and elegant -- the only hard parts are when you have > to wedge all the "classic" imports in, or if you want to implement "format" > importers. > > So, my vote for Py3K is to move the classic import machinery to explicit > objects on sys.meta_path -- preferably one for each distinct rule, so that > it is possible to *remove* them if you don't want them. (For example, the > Chandler folks have made a big deal about not wanting general Windows > registry imports to apply; it would be nice if they could just delete an > object from sys.meta_path rather than having to build their own Python to > elide this.) +1 > I would also suggest that we ask Just if he can come up with some elegant > extension that would allow PEP 302 to support pluggable import *formats* as > well as locations. :) I guess you'll need to explain this particular problem a bit better... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Fri Apr 21 11:54:18 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Apr 2006 21:54:18 +1200 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <1f7befae0604201407hc28f635wbae404d3a062cb99@mail.gmail.com> References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> <20060420084537.A83B.JCARLSON@uci.edu> <4447EE06.90705@neotitans.com> <1f7befae0604201407hc28f635wbae404d3a062cb99@mail.gmail.com> Message-ID: <4448ABCA.30800@canterbury.ac.nz> Tim Peters wrote: > Asynch programming wasn't at all a goal of generators, and Twisted is > well worth looking into for those who want slicker asynch programming > tools. There might possibly be room for an alternative way of presenting the generator machinery that makes it look like less of an abuse when it's used inside-out. A while back I was thinking about having something like result = call some_generator(args) which would be syntactic sugar for yielding an object that gets used by a generator-scheduling framework to simulate a coroutine. -- Greg From mike_traynar at credence.com Fri Apr 21 11:20:22 2006 From: mike_traynar at credence.com (Mike Traynar) Date: Fri, 21 Apr 2006 11:20:22 +0200 Subject: [Python-3000] New built-in function: bin() Message-ID: <4448A3D6.5090602@credence.com> I've always wondered why there isn't a bin() built-in in Python. Built-in functions already exist for converting ints to hexadecimal and octal strings and vice versa i.e. s = hex() and int(s, 16) s = oct() and int(s, 8) but no bin() function for binary strings s = ??? and int(s, 2)? Working with hardware I occasionally need to display an integer in binary format, or manipulate the bits of integer. A simple example would be reversing the order of the bits. Converting an int to binary string, reversing the string, then converting back to an int is a simple way to do this. Right now I use the recipe in http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/219300 and am happy to continue to do so, but it has always struck me as odd that there isn't a bin() built-in. Mike PS. I'll let someone else argue the case for the tri(), quad(), penta()... built-in functions. From ncoghlan at gmail.com Fri Apr 21 12:23:39 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Apr 2006 20:23:39 +1000 Subject: [Python-3000] Cleaning up argument list parsing (was Re: More wishful thinking) In-Reply-To: References: <44430CB5.5020907@gmail.com> <44461820.6010005@gmail.com> <44476786.5060608@gmail.com> Message-ID: <4448B2AB.4090302@gmail.com> Jim Jewett wrote: > On 4/20/06, Nick Coghlan wrote: >> Some legal combinations: >> >> f(1, 2, 3, d=1) # Positional args a, b and c >> f(1, 2, 3, 4, 5, 6, d=1) # Positional args a, b and c and (4, 5, 6) as args >> f(2, a=1, d=1) # Provide a as a keyword arg instead > > So positional would still have to occur before keywords, but only if > they were passed as positional? Today, that third one gets a > TypeError for assigning multiple values to a. My mistake - I thought that was currently legal, and didn't check it. So no, I'd keep the current rule - once you pass one positional argument as a keyword, you have to do the same for all subsequent positional arguments. Which implies that "positional-only" arguments would have to come *before* the current normal arguments that can be passed either way. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From rasky at develer.com Fri Apr 21 12:34:46 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 21 Apr 2006 12:34:46 +0200 Subject: [Python-3000] parameter lists [was: Type Expressions] References: <44482BE1.8030109@canterbury.ac.nz> Message-ID: <0eb101c6652f$363ec730$2452fea9@bagio> Ron Adam wrote: > This *may* relate to None being an object which isn't the same as > "not a value". There currently isn't a way (that I know of) to > specify a generally null object outside of sequences. > > def f(a, b, c=Null, d=x): # Using None here wouldn't work. > > > Recently I found a case where I wanted to return something that was > more literally *nothing* than a None is. So maybe a null symbol of > some sort might be useful in other cases as well? And what's wrong with: class Null: pass def f(a, b, c=Null, d=x): [...] Giovanni Bajo From rasky at develer.com Fri Apr 21 12:48:12 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 21 Apr 2006 12:48:12 +0200 Subject: [Python-3000] Changing the import machinery References: <4447B91A.8090409@colorstudy.com><044501c664a6$3361bd30$bf03030a@trilan> <44489A54.3060108@canterbury.ac.nz> Message-ID: <0eda01c66531$162e3e10$2452fea9@bagio> Greg Ewing wrote: >> 2) Totally disallow recursive imports (!). > > That would be extremely undesirable. I've used languages > in which mutual imports weren't possible, and it's a > massive pain in the posterior. You end up having to > modularise things in awkward and unnatural ways to > get around the restriction. Point taken. > Possibly it might work to allow mutual imports only > of the form "import spam" and not "from spam import > grail", but I'm not sure about that. That would be a first step. What worries me is this fact that you access the contents a module which is not fully imported yet. This makes us play trickes like moving imports up and down to satisfy circular dependencies. I think that if we prohibit the "from x import foo" syntax if x is already being imported, people will end up doing things like: import x # might be circular def func(self): foo = x.foo # fake from-import, but at run-time [...] Another thought. Sometimes, the order of imports does not match the order of (import-time) dependency. It might be that user import module A, which in turns imports module B. B then imports back module A. So the circle is user -> A -> B -> A. Now, it might be that B depends on A's contents at import time, while A does not depend on B's contents at import time. So, if user had imported module B before, things would have worked out. Sometimes, 'user' is just another module we're writing, and we've learnt to reorder our imports so we do so. I guess there's no easy solution for this. But I had in mind something... When you type "from spam import grail", many times you don't really need to bind name "grail" at import-time. Sometimes you do, yes, but many times you just want name "grail" to end up in the global dictionary of that module, but it's not necessary that it's done *right now* when the import is processed. Python could be note this and, if "spam" is being already imported, try to delay the binding of name "grail" after the circular import is finished. In fact, such a feature would really be killer because it would solve all import ordering problems. It would leave ordering problems only if you need name at *import* time, which is an order of magnitude less frequent. Giovanni Bajo From birchb at anz.com Fri Apr 21 05:27:30 2006 From: birchb at anz.com (Birch, Bill) Date: Fri, 21 Apr 2006 13:27:30 +1000 Subject: [Python-3000] Type Comparisons with Godel Numbers Message-ID: With reference to the last Gfdl blog on type checking (http://www.artima.com/weblogs/viewpost.jsp?thread=87182). There is concern that type comparison at runtime using objects will be quite expensive, so this posting is about optimisation. An idea is to compute a single canonical string which summarizes the _structural_ type of something. Think of it as a Godel number for the type. (http://diary.carolyn.org/godel2.html) Well not actually a Godel number, since they are very expensive to calculate and large. But close. Lets call them "type Godel strings" with the following properties: * the godel string for identical types will be equal * the godel strings for common types are small and very fast to compare. * godel strings are stateless and serializable You may have experienced something similar in C++ name mangling. (http://en.wikipedia.org/wiki/Name_mangling) There is an algorithm for creating a string from a type. However we are not proposing to mangle anything :-) Here are C++ name mangling examples: void h(int, char) // is mangled to: _Z1hic void h(int) // is mangled to: _Z1hi For Python types we do not need the names so we would have something like: h(int, str) -> void - '1is' I don't want to guess what the strings would look like. The algorithm design could use some kind of statistical analysis to come up with an optimally short encoding. Huffman maybe? Type comparison operators would only need a deep inspection of the types when the godel strings don't match. If most comparisons will be an exact match (not a subtype) the lookup should be faster. My hunch is that in real systems, exact matches will be very common. Some statistical work on real programs would be a good idea... The subtype operator would do: if object_type.godel() == required_type.godel(): # fast string comparison # exact match so OK proceed else: if object_type <: type : # slow to execute Yes proceed. Godel strings would be cacheable. The type comparison operators could be memoized to store the type-subtype lattice, not as a lattice of type objects, but as a lattice type godel strings. The cache would also be able to remember relationships between types for which the original type expression objects have been deleted. This idea implies that every type object in Python must provide a godel() method as a standard. Individual value objects could (provided they were immutable) cache their type's godel string accessible from a godel() method. This would speed up type comparison, since the type checking may not need to visit every sub- component of the object. Mutable objects would see little benefit, since its type can change with every update. At best this plan optimises away the analysis of the required type, but not the mutable instance to be compared. The idea also requires that type expressions be immutable, which is probably not a bad thing anyway. What possible use could there be for a mutable type expression? From guido at python.org Fri Apr 21 13:02:41 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 12:02:41 +0100 Subject: [Python-3000] New built-in function: bin() In-Reply-To: <4448A3D6.5090602@credence.com> References: <4448A3D6.5090602@credence.com> Message-ID: This has been brought up many times before. The value of bin() is really rather minimal except when you're just learning about binary numbers; and then writing it yourself is a useful exercise. I'm not saying that bin() is useless -- but IMO its (small) value doesn't warrant making, maintaining and documenting a new built-in function. --Guido On 4/21/06, Mike Traynar wrote: > I've always wondered why there isn't a bin() built-in in Python. > Built-in functions already exist for converting ints to hexadecimal and > octal strings and vice versa i.e. > > s = hex() and int(s, 16) > s = oct() and int(s, 8) > > but no bin() function for binary strings > > s = ??? and int(s, 2)? > > Working with hardware I occasionally need to display an integer in > binary format, or manipulate the bits of integer. A simple example would > be reversing the order of the bits. Converting an int to binary string, > reversing the string, then converting back to an int is a simple way to > do this. Right now I use the recipe in > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/219300 and am > happy to continue to do so, but it has always struck me as odd that > there isn't a bin() built-in. > > Mike > > PS. I'll let someone else argue the case for the tri(), quad(), > penta()... built-in functions. > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Fri Apr 21 13:09:56 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 21 Apr 2006 23:09:56 +1200 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <20060420151255.o8m8q8xgrnk080g8@login.werra.lunarpages.com> References: <20060420151255.o8m8q8xgrnk080g8@login.werra.lunarpages.com> Message-ID: <4448BD84.5040307@canterbury.ac.nz> Michael Chermside wrote: > > Andy writes: > > > I still have a hard time understanding continuations BTW, if you *really* want to understand continuations, you need to carry out the following exercise: Write a Scheme interpreter in Scheme, doing it in a continuation-passing style. [1] You'll find that it's possible to do this using a subset of Scheme that doesn't itself have continuations, yet it will be trivially easy to make it so that your interpreter implements a Scheme that does. If you carry out this exercise successfully, and make it actually work, at some point along the way you will have a Zen-like flash of insight, and will suddenly understand exactly what a continuation is and how it works. (The other possibility is that you will still be none the wiser. If this happens, you will never understand continuations. Give up programming language implementation and become a potato farmer. :-) Footnote [1]: Continuation-passing style is where every call passes a function[2] telling the called function what to do with its result. No call ever returns. There is no difficulty with this in Scheme, because of tail recursion. Footnote [2]: These functions are not continuations in the sense we're talking about, they're just ordinary functions. So you can write in a continuation-passing style even if the language you're writing in doesn't have continuations. -- Greg From guido at python.org Fri Apr 21 13:15:36 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 12:15:36 +0100 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: References: <44482BE1.8030109@canterbury.ac.nz> Message-ID: On 4/21/06, Ron Adam wrote: > Recently I found a case where I wanted to return something that was more > literally *nothing* than a None is. So maybe a null symbol of some sort > might be useful in other cases as well? You're not gonna get something that's a valid expression *and* is more "nothing" than None. On a case by case basis there might be syntax indicating the absence of something that is distinguishable from 'None'; for example, using *args you can reliaby whether an argument was passed or not (unlike using None as a default, which can be defeated by passing an explicit None). But a generic NULL won't be part of the language. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 21 13:31:19 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 12:31:19 +0100 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: References: Message-ID: This seems a good idea to remember when we get to that point. But let me point out that the key concern I have about the expense of type checking is what would be done when unchecked code calls a function with type-checked arguments. If I have some utterly dynamic code that comes up with a list of a million ints, and then I pass that as an argument to a function that requests the argument type is list[int], then I don't see how we can prevent the cost of doing a million checks to ensure that each and every item in the list is an int; either we do it all at once at the call site, or we amortize the cost over the lifetime of every list object, whether or not it is ever going to be used in a typechecked call. Neither sounds very attractive, and AFAICT your approach doesn't solve this particular issue. --Guido On 4/21/06, Birch, Bill wrote: > With reference to the last Gfdl blog on type checking (http://www.artima.com/weblogs/viewpost.jsp?thread=87182). There is concern that type comparison at runtime using objects will be quite expensive, so this posting is about optimisation. > An idea is to compute a single canonical string which summarizes the _structural_ type of something. Think of it as a Godel number for the type. (http://diary.carolyn.org/godel2.html) Well not actually a Godel number, since they are very expensive to calculate and large. But close. Lets call them "type Godel strings" with the following properties: > * the godel string for identical types will be equal > * the godel strings for common types are small and very fast to compare. > * godel strings are stateless and serializable > You may have experienced something similar in C++ name mangling. (http://en.wikipedia.org/wiki/Name_mangling) There is an algorithm for creating a string from a type. However we are not proposing to mangle anything :-) > Here are C++ name mangling examples: > void h(int, char) // is mangled to: _Z1hic > void h(int) // is mangled to: _Z1hi > For Python types we do not need the names so we would have something like: > h(int, str) -> void - '1is' > I don't want to guess what the strings would look like. The algorithm design could use some kind of statistical analysis to come up with an optimally short encoding. Huffman maybe? > Type comparison operators would only need a deep inspection of the types when the godel strings don't match. If most comparisons will be an exact match (not a subtype) the lookup should be faster. My hunch is that in real systems, exact matches will be very common. Some statistical work on real programs would be a good idea... The subtype operator would do: > if object_type.godel() == required_type.godel(): > # fast string comparison > # exact match so OK proceed > else: > if object_type <: type : # slow to execute > Yes proceed. > Godel strings would be cacheable. The type comparison operators could be memoized to store the type-subtype lattice, not as a lattice of type objects, but as a lattice type godel strings. The cache would also be able to remember relationships between types for which the original type expression objects have been deleted. > This idea implies that every type object in Python must provide a godel() method as a standard. > Individual value objects could (provided they were immutable) cache their type's godel string accessible from a godel() method. This would speed up type comparison, since the type checking may not need to visit every sub- component of the object. Mutable objects would see little benefit, since its type can change with every update. At best this plan optimises away the analysis of the required type, but not the mutable instance to be compared. > The idea also requires that type expressions be immutable, which is probably not a bad thing anyway. What possible use could there be for a mutable type expression? > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Fri Apr 21 14:05:32 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 21 Apr 2006 22:05:32 +1000 Subject: [Python-3000] Changing the import machinery In-Reply-To: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: <4448CA8C.5050701@gmail.com> Phillip J. Eby wrote: > There are only two things wrong with PEP 302 IMO, and neither is its "fault". > > The first is that the "classic" import machinery isn't on sys.meta_path, > and the 'imp' API isn't defined in terms of PEP 302. Those two things > can't change without introducing backward incompatibilities. > > The second is that PEP 302 only covers "location" importers, not "format" > importers. That is, if you want to do something like make Kid or Cheetah > templates importable, you have to replace things with new machinery. This > is a more important problem to solve, IMO, than coming up with yet another > set of semantics for the things that *are* covered by PEP 302. Do you mean an API so you can associate loader objects with new extensions, allowing find_module and load_module to deal with them? A cleaned up replacement for the current _PyImport_FileTab? What about something like a sys.load_hooks list? The default contents of sys.load_hooks (on Windows) would be: PythonPackageLoader (handles directories with __init__.py files) PythonDLLLoader (handles .pyd files) PythonSourceLoader (handles .py and .pyw files, checks for compiled file) PythonCompiledLoader (handles either .pyc and .pyo files, depending on -O) That would let us get rid of a lot of ifdef hackery from import.c, and also permit things like: sys.load_hooks.append(ZipLoader) where "ZipLoader" handles files ending in .zip or .egg, and treats them as packages. Putting "foo.zip" or "foo.egg" somewhere on sys.path would then allow "import foo" and "from foo import bar" to work without requiring a path hook for every module. (Random aside: why the heck is all this machinery stored in sys, anyway? Having it in imp would make far more sense. Was it just because sys.path was already there?) > PEP 302 is reasonably clear and concise, although there are occasional > loopholes. At least three people have re-implemented parts of it in pure > Python from the spec: Just van Rossum, Neal Norwitz, and myself (twice, in > my case). It is easy and elegant -- the only hard parts are when you have > to wedge all the "classic" imports in, or if you want to implement "format" > importers. Make that 4 - I had to reimplement a chunk of it for PEP 338. And as you say, the only part that made it particularly hard was that the classic machinery doesn't play by the rules. Using PEP 302, the path importer cache ends up becoming the container with the special objects for the benefit of the interpreter, while the original path continues to contain the human-readable strings. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From greg.ewing at canterbury.ac.nz Fri Apr 21 14:06:18 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 00:06:18 +1200 Subject: [Python-3000] Is reference counting still needed? In-Reply-To: References: <44458FA7.10605@comcast.net> <4445C8BC.5010202@canterbury.ac.nz> Message-ID: <4448CABA.5000608@canterbury.ac.nz> Neil Schemenauer wrote: > I believe that's correct. A state of the art generational GC would > outperform reference counting, even given Python's enormous > allocation rate. Another thing to consider is that Python's current scheme tends to be more fail-safe when interacting with other systems that don't know about the GC scheme it's using. Most GC systems work by assuming that something is garbage unless the system can prove that it's not. This relies critically on being able to find all roots that could possibly lead to an object. Miss one anywhere and you get a very nasty crash when something is deallocated prematurely. Python's cyclic GC, on the other hand, assumes that nothing is garbage unless it can prove that it is. So if there's an object around somewhere that it can't reach, it might leak some memory, but it won't lead to a crash. This is a nice property to have in something like Python where interfacing with external code is so important. -- Greg From greg.ewing at canterbury.ac.nz Fri Apr 21 14:13:06 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 00:13:06 +1200 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: References: <44482BE1.8030109@canterbury.ac.nz> Message-ID: <4448CC52.3000004@canterbury.ac.nz> Ron Adam wrote: > Or just ... > > def f(a, b, c=?, d=x): But there's nothing about this that particularly suggests that c *must* be specified with a keyword. It looks like just another positional arg with a rather strange-looking default. Keep in mind that there's really no connection between the = used in the definition and the = used in the call. They mean different things and are independent of each other. -- Greg From greg.ewing at canterbury.ac.nz Fri Apr 21 14:36:34 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 00:36:34 +1200 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: References: Message-ID: <4448D1D2.3010306@canterbury.ac.nz> Guido van Rossum wrote: > If I have some utterly dynamic > code that comes up with a list of a million ints, and then I pass that > as an argument to a function that requests the argument type is > list[int], you wrap it in something that checks elements for intness as you access them. It'll still check a million ints if you access them all, but that would have to be done anyway if you were passing the list to fully dynamic code. And it spreads the cost of doing it out over whatever is being done with the ints, rather than doing it all up front. And it doesn't bother checking any ints that you don't use. -- Greg From guido at python.org Fri Apr 21 15:06:33 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 14:06:33 +0100 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: <4448D1D2.3010306@canterbury.ac.nz> References: <4448D1D2.3010306@canterbury.ac.nz> Message-ID: On 4/21/06, Greg Ewing wrote: > Guido van Rossum wrote: > > If I have some utterly dynamic > > code that comes up with a list of a million ints, and then I pass that > > as an argument to a function that requests the argument type is > > list[int], > > you wrap it in something that checks elements for intness > as you access them. It'll still check a million ints if > you access them all, but that would have to be done anyway > if you were passing the list to fully dynamic code. And > it spreads the cost of doing it out over whatever is > being done with the ints, rather than doing it all up > front. And it doesn't bother checking any ints that you > don't use. That's the conclusion we reached last time too. Unfortunately the wrapper slows down each access. I guess we could disable all checking when -O is used... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From edcjones at comcast.net Fri Apr 21 15:35:38 2006 From: edcjones at comcast.net (Edward C. Jones) Date: Fri, 21 Apr 2006 09:35:38 -0400 Subject: [Python-3000] Cleaning up argument list parsing Message-ID: <4448DFAA.20100@comcast.net> For fancy argument list parsing, perhaps we need a module "argsparse" that is analogous to "optparse". From walter at livinglogic.de Fri Apr 21 15:55:09 2006 From: walter at livinglogic.de (=?iso-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 21 Apr 2006 15:55:09 +0200 (CEST) Subject: [Python-3000] Chaning the import machinery; was: Re: [Python-Dev] setuptools in 2.5. In-Reply-To: References: <200604201456.13148.anthony@interlink.com.au> <200604201652.51448.anthony@interlink.com.au> <44473D71.6060107@v.loewis.de> <200604201801.29943.anthony@interlink.com.au> <444747EA.1010707@livinglogic.de> <44477849.7000806@livinglogic.de> <44479678.3070803@livinglogic.de> Message-ID: <61036.89.54.11.221.1145627709.squirrel@isar.livinglogic.de> Guido van Rossum wrote: > On 4/20/06, Walter D?rwald wrote: >> Guido van Rossum wrote: >> >> > Sorry, there's so much here that seems poorly thought out that I don't know where to start. >> >> Consider it a collection of wild ideas. >> >> > Getting rid of the existing import syntax in favor of the incredibly verbose and ugly >> > >> > foo = import("foo") >> > >> > just isn't acceptable. >> >> OK, then how about >> >> import foo >> import foo from url("file:~guido/python-packages/foo") > > Sorry. I wasn't proposing that the import statement had to be extended to support the new functionality; only that the > existing functionality should still be available by writing import statements. For the above, I'd much rather write > > foo = import_from_url("foo", "/home/guido/.../foo") Then this function would need access to some kind of atomic import functionality. Maybe a version of exec where I can specify a filename would be enough: >>> d = {} >>> exec "x = 1/0" in d as "foobar" Traceback (most recent call last): File "", line 1, in ? File "foobar", line 1, in ? ZeroDivisionError: integer division or modulo by zero It would be nice if the traceback did show source code. > I also don't like your proposal to create a "url" object (as you might have predicted from my resistance to the path PEP :-). OK, then dispatching would have to be done based on something else. >> How would this work with multiple imports? >> >> import foo, bar from url("file:~guido/python-packages/") >> >> How would it recognize whether url() refers to the module itself or to a package from which modules are imported? > > You tell me. Maybe url("file:~guido/python-packages/foo/") could be a package an url("file:~guido/python-packages/foo") could be a module (and if ~guido/python-packages/foo is a directory this would be the package module. But this seems to be to unreliable and magic to me. Probaby this shouldn't be used for mass imports, but only for importing single specials modules outside of the stdlib. >> > Importing from remote URLs is a non-starter from a security POV; and using HTTPS would be too slow. For code that's known >> > to reside remotely, a better approach is to use setuptools to install that code once and for all. >> >> I don't see how that changes anything from a security POV. You have to trust the source in both cases. > > With http, even if I trusted the source, I still shouldn't trust that the data I get from the URL actually came from the > source. With HTTPS, at least man-in-the-middle attacks should be thwarted. True, but the man-at-the-end attack still works (although it's less likely I guess) ;) >> Performancewise you are right, it >> wouldn't make sense to call such an import in a tight loop. >> >> > How would a module know its own name? >> >> It probably would only have a real name for a standard "import foo". > > Thats a problem IMO. Instances from classes in those module wouldn't be pickleable. >> import foo from url("file:~guido/python-packages/foo") >> >> would create a module with foo.__origin__ => url("file:~guido/python-packages/foo") >> >> > How do you deal with packages >> > (importing a module from a package implies importing/loading the package's __init__.py). >> >> I have no idea. This probably won't work (at least not in the sense that importing something imports all its parent >> modules). >> >> But packages seem to be a problem for setuptools too (at least if parts of the package are distributed separately). > > So please do some research and find out what their problems are, If the modules foo.bar and foo.baz are distributed as separate setuptools packages, foo/bar.py as installed as e.g. foo-bar-0.1/foo/bar.py and foo/baz.py is installed as foo-baz-0.3/foo/baz.py. setuptools somehow manages to tie those together into a single "virtual packages" (according to http://peak.telecommunity.com/DevCenter/setuptools#namespace-packages by __import__('pkg_resources').declare_namespace(__name__), which fiddles with __path__) However I must admit, I don't understand what pgk_resources.declare_namespace() and pgk_resource._handle_ns() etc. really do. > how your problems are similar, and what should be done about > it. My "load a module from a file" problem is actually totally unrelated to this, except that it gets complicated if packages come into the picture (which I wouldn't need from the "load a module from a file" scenario). > At this point I believe you're running out of quick wild ideas that are > actually helpful. > >> Maybe for >> >> import os.path from url("file:~guido/python-packages/os/path") >> >> url("file:~guido/python-packages/os/path").parent() should return url("file:~guido/python-packages/os") which then gets >> imported and url("file:~guido/python-packages") should return None. But this would mean that the number of dots on the left >> side has to match the number of times calling parent() on the right side returns somehing not None. Not good. Maybe we >> should leave the current import syntax alone and add a new one for importing from files etc.. > > I think we should design a new OO API that captures the essence of the current import machinery (but cleaned up), create a > new mapping from all current syntactic variants of the import statements to that API, and design a separate extension > (through subclassing or whatever) to do imports from non-traditional sources. > >> > I suggest that instead of answering these questions from the >> > perspective of the solution you're offering here, you tell us a bit more about the use cases that make you think of this >> > solution. What are you trying to do? Why are you importing code from a specific file instead of configuring sys.path so >> > the file will be found naturally? >> >> The Python files I'm importing define "layout classes" that are used for generating a bunch of static HTML files (actually >> JSP files). I.e. something like >> >> foo.py: >> def link(text, href): >> return "%s" % (href, text) >> >> bar.py: >> foo = specialimport("foo.py") >> >> def linklist(links): >> return "
      %s
    " % "".join("
  • %s
    • " % bar.link(text, href) >> for (text, href) in links) >> >> This function linklist() is used for generating the HTML files. The interactive Python shell is used as a "make shell", so >> when I change foo.py in an editor and do >> >>> project.build("install") >> in the shell all the HTML files that depend on bar.py (which in turn depends on foo.py) must be rebuilt. > > You've nearly lost me, but it *seems* to me that what you're really doing is use an alternative import mechanism in order to > solve the reload() problem for a set of interdependent modules. Exactly, it's the same problem that a Python-based webservers have with reloading changed modules (and those modules that depend on the changed one). > That's a good thing to attempt to solve more generally, but > hardly a use case for a special import function. > > What does your specialimport() implementation do when the same module is requested twice? Does it load it twice? Only if the module source code has been changed since the first import (or one of the module used by this module has changd), otherwise it's reused from a cache (see Repository._update() in http://styx.livinglogic.de/~walter/pythonimport/resload.py) >> > I >> > suspect that your module-from-a-database use case is actually intended as a building block for an import hook. >> >> Maybe, but ACAIK import hooks can't solve the dependency and reload problem. > > Only because nobody has bothered to use them to solve it; they have all the information available needed to solve it. > >> > I think we ought to redesign the import machinery in such a way that you could do things like importing from a specific >> > file or from a database, but without changing the import statement syntax -- instead, we should change what import >> > actually *does*. >> >> But there has to be a spot where I can actually specify from *where* I want to import, and IMHO this should be in the import >> statement, not by putting some hooks somewhere. > > That may be your use case. Most people would prefer to be able to set PYTHONPATH once and then use regular import statements. I do that for modules that change more slowely (but installing a project from 6 month ago takes a lot of work, because you have to install al the correct version of the packages. But pkg_resources.require() should be able to resolve that). > On Windows, there's a mechanism to specify that a particular module must be loaded from a specifc location. Maybe this would > be helpful? For Python, or are you talking about the Windows OS? > It could be a dict whose keys are fully qualified module names, and whose values are pathnames (or URLs or > whatever). > >> > While we're at it, we >> > should also fix the silliness that __import__("foo.bar") imports foo.bar but returns foo. >> >> +1 > > This is an example of what I mentioned above -- redesign the machinery API and then remap the import statement. Servus, Walter From aahz at pythoncraft.com Fri Apr 21 16:30:33 2006 From: aahz at pythoncraft.com (Aahz) Date: Fri, 21 Apr 2006 07:30:33 -0700 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: <4448D1D2.3010306@canterbury.ac.nz> References: <4448D1D2.3010306@canterbury.ac.nz> Message-ID: <20060421143033.GA16802@panix.com> On Sat, Apr 22, 2006, Greg Ewing wrote: > Guido van Rossum wrote: >> >> If I have some utterly dynamic code that comes up with a list of a >> million ints, and then I pass that as an argument to a function that >> requests the argument type is list[int], > > you wrap it in something that checks elements for intness as you > access them. It'll still check a million ints if you access them all, > but that would have to be done anyway if you were passing the list > to fully dynamic code. And it spreads the cost of doing it out over > whatever is being done with the ints, rather than doing it all up > front. And it doesn't bother checking any ints that you don't use. Guido alluded to this, but let's make a specific example for clarity: def foo(a -> list[int]): for x in a: blah blah for y in a: blah blah How do we handle the cost of doing the typecheck in the second loop? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From jimjjewett at gmail.com Fri Apr 21 16:41:52 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 21 Apr 2006 10:41:52 -0400 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4448A6D6.30003@canterbury.ac.nz> References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> <20060420084537.A83B.JCARLSON@uci.edu> <4447EE06.90705@neotitans.com> <4448A6D6.30003@canterbury.ac.nz> Message-ID: On 4/21/06, Greg Ewing wrote: > Andy Sy wrote: > > Huh? Futures are very different from continuations. I still have a > > hard time understanding continuations (and am no fan of them), but > > futures seem to be a rather simple abstraction to comprehend. > Isn't a future just a coroutine, or something equivalent? Not in the references he supplied (or, at least, not in the Java one). There, a Future is a proxy object, whose actual characteristics will (or may) be filled in later. If you need it, you wait for it, but if you don't need it yet, it can be created in the background. How to make this cleaner than the existing concrete implementations (such as a twisted Deferred, or spawning a thread which puts a return value in a known location) -- that is the part which hasn't been worked out. -jJ From mcherm at mcherm.com Fri Apr 21 17:44:56 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Fri, 21 Apr 2006 08:44:56 -0700 Subject: [Python-3000] Type Comparisons with Godel Numbers Message-ID: <20060421084456.gmk00hfwik0s4k8k@login.werra.lunarpages.com> Building from Aahz's example, what you really want is a mechanism for typechecking that any accessed elements are ints when dynamic code generates a list of a million elements and passes it to a function declared to take a parameter of type "list[int]". And then make sure the following two examples have the desired performance: # # Accesses only a small (but unknown) number # of elements in the list, 'n'. Performance # of typechecking should be O(n), which is far # better than O(len(a)). # def few_accesses(a -> list[int]): i = 0 while a[i] != 42: i += 1 return i # # Accesses the list a large (but unknown) number # of times, 'b'. Performance of typechecking should # be O(len(a)), which is far better than O(b*len(a)). # def many_accesses(a -> list[int], b int): for i in range(b): for x in a: do_something(b, x) It is easy for me to come up with an approach to optimizing typechecking for each function, but analysizing code to decide which is appropriate is right out, and it is difficult for me to imagine a single approach that satisfies both. Perhaps some super-clever technique that starts out wrapping accesses but switches to pre-checking the entire list and then dropping all wrappers once some threshhold of accesses has been reached? That sounds WAY too clever and implicit... it's sure to cause problems. -- Michael Chermside From mark.russell at zen.co.uk Fri Apr 21 16:58:32 2006 From: mark.russell at zen.co.uk (Mark Russell) Date: Fri, 21 Apr 2006 15:58:32 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: On 20 Apr 2006, at 10:23, Guido van Rossum wrote: > IMO anything using any kind of nested brackets inside the argument > list is doomed. Another wild thought: def foo(a, b, @keyword_only c, d): pass Actually that one could go in 2.X - it's currently a syntax error. Mark Russell From guido at python.org Fri Apr 21 18:02:59 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 17:02:59 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: On 4/21/06, Mark Russell wrote: > On 20 Apr 2006, at 10:23, Guido van Rossum wrote: > > IMO anything using any kind of nested brackets inside the argument > > list is doomed. > > Another wild thought: > > def foo(a, b, @keyword_only c, d): > pass > > Actually that one could go in 2.X - it's currently a syntax error. To prevent more abominations like this, let me pronounce that I now like the single-star syntax: def foo(a, b, *, x=1, y=2): ... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Fri Apr 21 18:10:46 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 21 Apr 2006 11:10:46 -0500 Subject: [Python-3000] Changing the import machinery In-Reply-To: <4448A2EB.8020401@canterbury.ac.nz> References: <4447B91A.8090409@colorstudy.com> <4447C9B2.8050009@colorstudy.com> <4447E9F7.5020300@colorstudy.com> <4448A2EB.8020401@canterbury.ac.nz> Message-ID: <44490406.3040104@colorstudy.com> Greg Ewing wrote: > Ian Bicking wrote: >>try: >> from string import Template >>except ImportError: >> from mypackage.backports.string24 import Template >> >>Doing this in a more elegent or formalized fashion might be nice. > > > Have you seen my proposal for "or" in import statements? > Would you consider that elegant enough? No, I haven't; I assume you mean: from string import Template or from mypackage.backports.string24 import Template ... that (realistic) example doesn't look very pretty :( >>Versioning the standard library > > > My thought on versioning is that it should look something > like > > import spam[3:], movies.grail[2:4] > # version 3 or later of spam, version 2, 3 or 4 of movies.grail > > I would like this much better than some kind of require() > function. Integrating the version specification with the > import would solve some problems, such as the one mentioned > earlier where a tentative import fails when it would have > succeeded if some later require() had been done first. One difference of require() is that it works on distributions (well, distribution names), not on Python packages. So you can have a distribution that provides more than one module, or you can have two differently-named distributions that provide the same module (which is nice for forking). Also, I already find it difficult to maintain version requirements declaratively in my setup.py files. I have to speculate on whether some package will depend or conflict with a version of some other package that hasn't even been released yet. For instance, I might release my code A, then realize that there's a bug in another package B that breaks A; after B is fixed, I'd like A to require that new fixed version, but package A hasn't actually changed in any way... yet I have to give it a new version number. Actually putting the requirements in the import statements in the body of my code seems worse. > The form that the version numbers should take is up to > debate, but I'd tend to keep it extremely simple, i.e. > just integers. The baroque possibilities offered by the > setuptools require() seem like overkill to me. A more formalized version makes it possible to use the version number meaningfully. E.g., require >=1.5,<1.6 of a library, even though 1.6 doesn't exist, because there's a policy of not releasing anything but bug fixes without at least a minor version increase. Version parsing *more* sophisticated than setuptools' could make branching and forking of projects more feasible. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From pje at telecommunity.com Fri Apr 21 18:17:54 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Apr 2006 12:17:54 -0400 Subject: [Python-3000] Changing the import machinery In-Reply-To: References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060421120507.01e17e08@mail.telecommunity.com> At 10:42 AM 4/21/2006 +0100, Guido van Rossum wrote: >I think I like this. I wonder if there's a parallel with my preference >for strings as paths instead of path objects... PEP 302 still has things analagous to path objects, it's just that you're allowed to define the mapping between strings and objects. > > I would also suggest that we ask Just if he can come up with some elegant > > extension that would allow PEP 302 to support pluggable import *formats* as > > well as locations. :) > >I guess you'll need to explain this particular problem a bit better... Some users of templating systems want to be able to put template files in their packages, and then import them as Python objects, especially if it's a templating language that compiles to Python code anyway (e.g. Kid, PTL). However, because you can't use PEP 302 hooks to do this, each package that does is forced to write a full-blown import hook or use one of the other import hook frameworks -- and they generally these extensions don't "play nice" with each other. So basically what they would want is a way to add new file extensions that are recognized as "modules" and can be loaded by the import machinery. Think of it as a way to add things to imp.get_suffixes(), if imp.get_suffixes() also gave you handlers instead of just metadata. So, what I'd suggest is that the expanded import mechanism would provide a registry of extensions and handlers, defining how you get from file extension to module, in a way that would allow PEP 302 importers and loaders to delegate this machinery to a handler. Python would also provide built-in handlers for .py, .pyc, .pyo, and so on, so that everybody's importers wouldn't have to reimplement these features, the way they do now. There are a few details to work out, as some formats simply can't work without a real filesystem file (C extensions, for example), so the handler registry would have to contain information about that. Also, some file formats are generated from others, and so the registry needs to give some info about that, and provide hooks for writing back generated files. (Which would have to be optional, since e.g. zip files wouldn't allow it.) All in all, there's nothing particularly revolutionary there, it's just working out some tedious details with a registry and interface. Then, we change the built-in machinery to just be some entries in this registry, and change zipimport and imp to use that registry. And that should even be able to be done in the 2.x series, as it doesn't create any backward incompatibilities, it just enables zipimport and user importers to call back to pieces of the machinery (e.g. to load/check/write Python modules) rather than reimplementing them. From thomas at python.org Fri Apr 21 18:27:44 2006 From: thomas at python.org (Thomas Wouters) Date: Fri, 21 Apr 2006 18:27:44 +0200 Subject: [Python-3000] Changing the import machinery In-Reply-To: <44490406.3040104@colorstudy.com> References: <4447B91A.8090409@colorstudy.com> <4447C9B2.8050009@colorstudy.com> <4447E9F7.5020300@colorstudy.com> <4448A2EB.8020401@canterbury.ac.nz> <44490406.3040104@colorstudy.com> Message-ID: <9e804ac0604210927m4ad0ee60t8108fc852d35a630@mail.gmail.com> On 4/21/06, Ian Bicking wrote: > > Greg Ewing wrote: > > > Have you seen my proposal for "or" in import statements? > > Would you consider that elegant enough? > > No, I haven't; I assume you mean: > > from string import Template or from mypackage.backports.string24 > import Template > > ... that (realistic) example doesn't look very pretty :( The suggestion was from string or mypackage.backports.string24 import Template which looks a bit better. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060421/859e2a6d/attachment.html From pje at telecommunity.com Fri Apr 21 18:38:23 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Apr 2006 12:38:23 -0400 Subject: [Python-3000] Changing the import machinery In-Reply-To: <4448CA8C.5050701@gmail.com> References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060421123217.04393fb8@mail.telecommunity.com> At 10:05 PM 4/21/2006 +1000, Nick Coghlan wrote: >Phillip J. Eby wrote: >>There are only two things wrong with PEP 302 IMO, and neither is its "fault". >>The first is that the "classic" import machinery isn't on sys.meta_path, >>and the 'imp' API isn't defined in terms of PEP 302. Those two things >>can't change without introducing backward incompatibilities. >>The second is that PEP 302 only covers "location" importers, not "format" >>importers. That is, if you want to do something like make Kid or Cheetah >>templates importable, you have to replace things with new >>machinery. This is a more important problem to solve, IMO, than coming >>up with yet another set of semantics for the things that *are* covered by >>PEP 302. > >Do you mean an API so you can associate loader objects with new >extensions, allowing find_module and load_module to deal with them? More precisely, to allow PEP 302 importers and loaders to deal with them. find_module and load_module should go away, replaced with a PEP 302 importer and loader, respectively. >What about something like a sys.load_hooks list? The default contents of >sys.load_hooks (on Windows) would be: > > PythonPackageLoader (handles directories with __init__.py files) > PythonDLLLoader (handles .pyd files) > PythonSourceLoader (handles .py and .pyw files, checks for compiled file) > PythonCompiledLoader (handles either .pyc and .pyo files, depending on -O) Well, it'd be more like sys.module_extensions or some such, but yes. I also don't think there should be a package loader as such. Instead, the other handlers would be applied to the name __init__ within a package. >That would let us get rid of a lot of ifdef hackery from import.c, and >also permit things like: > > sys.load_hooks.append(ZipLoader) > >where "ZipLoader" handles files ending in .zip or .egg, and treats them as >packages. No, let's keep PEP 302, which works just fine for this part. Zip files aren't packages, anyway, and definitely shouldn't be conflated with them. >(Random aside: why the heck is all this machinery stored in sys, anyway? >Having it in imp would make far more sense. Was it just because sys.path >was already there?) Well, without sys there are no modules yet, so how would you import imp? But you'd have to ask Just for his reasons. >>PEP 302 is reasonably clear and concise, although there are occasional >>loopholes. At least three people have re-implemented parts of it in pure >>Python from the spec: Just van Rossum, Neal Norwitz, and myself (twice, >>in my case). It is easy and elegant -- the only hard parts are when you >>have to wedge all the "classic" imports in, or if you want to implement >>"format" importers. > >Make that 4 - I had to reimplement a chunk of it for PEP 338. Oops, I must've got you and Neal mixed up; I just remembered it was an "N" name, and he has two "N" names. :) Sorry. From ianb at colorstudy.com Fri Apr 21 18:37:27 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 21 Apr 2006 11:37:27 -0500 Subject: [Python-3000] Changing the import machinery In-Reply-To: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: <44490A47.3000102@colorstudy.com> Phillip J. Eby wrote: > The second is that PEP 302 only covers "location" importers, not "format" > importers. That is, if you want to do something like make Kid or Cheetah > templates importable, you have to replace things with new machinery. This > is a more important problem to solve, IMO, than coming up with yet another > set of semantics for the things that *are* covered by PEP 302. I think the specific examples (Kid and Cheetah) are places where import machinery is used when it shouldn't be. Or at least, when it would be better to use more path-based machinery. (Brief explanation: Kid and Cheetah are template languages that compile to importable Python code; they have import hooks so you can import .kid and .tmpl files as though they were Python files) Realistic use cases for templates often involve fairly complicated sets of overlapping hierarchies, searched depth-first. I guess not entirely unlike some complicated import machinery, except that the rules are really limited to templates and I don't see a reason to mix in templates. Maybe you wouldn't have to mix it all together... if sys.path, sys.meta_path, and sys.modules (and maybe some other stuff I don't know of) were really instance variables of some ModuleEnvironment object, and you could instantiate a new ModuleEnvironment that was specific to templates. Probably to be filed under crazy ideas. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Fri Apr 21 18:40:34 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 21 Apr 2006 11:40:34 -0500 Subject: [Python-3000] New built-in function: bin() In-Reply-To: References: <4448A3D6.5090602@credence.com> Message-ID: <44490B02.9080704@colorstudy.com> Guido van Rossum wrote: > This has been brought up many times before. The value of bin() is > really rather minimal except when you're just learning about binary > numbers; and then writing it yourself is a useful exercise. > > I'm not saying that bin() is useless -- but IMO its (small) value > doesn't warrant making, maintaining and documenting a new built-in > function. And for some reason no one wants to propose it for any particular stdlib module... -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From edloper at gradient.cis.upenn.edu Fri Apr 21 18:36:05 2006 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Fri, 21 Apr 2006 12:36:05 -0400 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: <4448D1D2.3010306@canterbury.ac.nz> References: <4448D1D2.3010306@canterbury.ac.nz> Message-ID: <3AE4D4F5-63D5-40F4-8560-ECAF32B09860@gradient.cis.upenn.edu> On Apr 21, 2006, at 8:36 AM, Greg Ewing wrote: > Guido van Rossum wrote: >> If I have some utterly dynamic >> code that comes up with a list of a million ints, and then I pass >> that >> as an argument to a function that requests the argument type is >> list[int], > > you wrap it in something that checks elements for intness > as you access them. [...] I think it's important for code to fail early if type constraints aren't satisifed. The approach you suggest (wrapping the object) would cause a function with a badly typed argument to blow up part-way through executing, possibly at a position far-distant from the place where the type constraint was violated, and possibly after various side-effects have already taken place. -Edward From thomas at python.org Fri Apr 21 18:47:45 2006 From: thomas at python.org (Thomas Wouters) Date: Fri, 21 Apr 2006 18:47:45 +0200 Subject: [Python-3000] p3yk branch and merging (was Re: [Python-3000-checkins] r45619 - python/branches/p3yk/Modules/cjkcodecs/multibytecodec.c) Message-ID: <9e804ac0604210947j32cf3267lfee57c9229347061@mail.gmail.com> On 4/21/06, hyeshik.chang wrote: > > Author: hyeshik.chang > Date: Fri Apr 21 18:21:44 2006 > New Revision: 45619 > > Modified: > python/branches/p3yk/Modules/cjkcodecs/multibytecodec.c > Log: > Add empty __init__ methods for stateful multibytecodec instances. > This resolves a problem found by Thomas Wouters: > http://mail.python.org/pipermail/python-dev/2006-April/064051.html Thanks, but please don't "backport" from the p3yk-branch to the trunk. It makes merging harder. Not an issue for this fix, because the trunk-merge happened to be up-to-date, but it can make future merges 'orribly damned hard. If a fix applies to both the trunk and one or more branches (that are actively merged), only stuff it in the trunk, then merge the trunk changes into the branch. (Or get someone else to do it.) And just so everyone knows: done right, merging isn't hard. It's not as easy as it could be, but 'svn merge' makes it a lot easier than it used to be in CVS. Here's how you do it: The last merged revision is stored in the branch, in a property called 'last_merge'; 'svn propget last_merge .' will show it. Right now, it's in human-readable format, partly because a branch off of p3yk would get the last_merge property branched, too, and then you lose track of which branch was merged when. If anyone wants to automate the merging process (not something I'd bother with, myself, since it usually requires too much manual merging anyway), make sure to support grepping for the right revision :) Also, 'last_merge' is a completely arbitrary propname. In a checked out and pristine p3yk branch, run 'svn merge -r : ../path/to/trunk'. 'path to trunk' can be a URL just as easily as a local path. 'svn merge' applies all changes (diffs, but also renames, deletes, property changes, etc) to the local working copy. Resolve the conflicts, configure, compile, make, make test, fix any test failures. When you're about to commit, 'svn propedit last_merge .', fill in the you used in the merge, and 'svn commit' the whole lot. The you use can be any revision, it doesn't have to be now. Also, svn merge sometimes can't deal with the trunk changes; for instance, the trunk had re.py removed and then sre.py renamed to re.py, and svn merge couldn't handle that in one go. Fortunately, 'svn merge' only applies changes locally. Tossing it all out is a matter of 'svn revert -R' or just deleting the working copy. I think Neal wants to keep the p3yk branch pretty up to date with the trunk, likely merging every week, and I think that's a very good idea. It makes merging much less painful ;P -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060421/32eb0818/attachment.html From talin at acm.org Fri Apr 21 18:54:39 2006 From: talin at acm.org (Talin) Date: Fri, 21 Apr 2006 16:54:39 +0000 (UTC) Subject: [Python-3000] New built-in function: bin() References: <4448A3D6.5090602@credence.com> Message-ID: Mike Traynar credence.com> writes: > I've always wondered why there isn't a bin() built-in in Python. > Built-in functions already exist for converting ints to hexadecimal and > octal strings and vice versa i.e. Perhaps a "format_base" function in the strings library, that takes an int and a number base? (The name is horrid, I know.) -- Talin From pje at telecommunity.com Fri Apr 21 19:05:48 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri, 21 Apr 2006 13:05:48 -0400 Subject: [Python-3000] Changing the import machinery In-Reply-To: <44490A47.3000102@colorstudy.com> References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20060421130330.01e56320@mail.telecommunity.com> At 11:37 AM 4/21/2006 -0500, Ian Bicking wrote: >Phillip J. Eby wrote: >>The second is that PEP 302 only covers "location" importers, not "format" >>importers. That is, if you want to do something like make Kid or Cheetah >>templates importable, you have to replace things with new >>machinery. This is a more important problem to solve, IMO, than coming >>up with yet another set of semantics for the things that *are* covered by >>PEP 302. > >I think the specific examples (Kid and Cheetah) are places where import >machinery is used when it shouldn't be. Or at least, when it would be >better to use more path-based machinery. Actually, *my* use cases for this mechanism would run more to running other languages in the Python VM. For example, Logix or my SCALE experiments, or that project, I think it's called "classpath", that lets you import Java classes in Python. The main reason I mentioned the template use cases is because they currently exist and have users. From jimjjewett at gmail.com Fri Apr 21 19:12:55 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 21 Apr 2006 13:12:55 -0400 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: References: Message-ID: On 4/20/06, Birch, Bill wrote: > Type comparison operators would only need a deep > inspection of the types when the godel strings don't match. > If most comparisons will be an exact match (not a subtype) > the lookup should be faster. If you're assuming that, then just checking whether obj.__class__ is requirement.allowed_class should be pretty hard to beat... -jJ From talin at acm.org Fri Apr 21 19:12:46 2006 From: talin at acm.org (Talin) Date: Fri, 21 Apr 2006 17:12:46 +0000 (UTC) Subject: [Python-3000] Type Expressions References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: Guido van Rossum python.org> writes: > To prevent more abominations like this, let me pronounce that I now > like the single-star syntax: > > def foo(a, b, *, x=1, y=2): ... Thank you :) It was getting pretty strange there. The variation that I was thinking of was a little shorter, but not necessarily better: def foo( a, b; x=1, y=2 ): ... BTW I still haven't given up on working on an implementation for keywords after *args, but it's been going rather slowly due to lack of time to sit down and focus. I know you want implementations rather than PEPs at this point, but if there is a consensus on both the "keywords after *args" and "keyword only" arguments, I can write up a PEP for that, if that would be useful. -- Talin From aleaxit at gmail.com Fri Apr 21 19:15:10 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Fri, 21 Apr 2006 10:15:10 -0700 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: On 4/21/06, Guido van Rossum wrote: ... > To prevent more abominations like this, let me pronounce that I now > like the single-star syntax: > > def foo(a, b, *, x=1, y=2): ... So, what will this syntax signify? If the single-star stands for "arbitrary positional arguments", how will the body of foo access them? Sorry if these have been well covered already, but I read back through this thread and couldn't find them. Alex From talin at acm.org Fri Apr 21 19:17:18 2006 From: talin at acm.org (Talin) Date: Fri, 21 Apr 2006 17:17:18 +0000 (UTC) Subject: [Python-3000] PEP - string.format References: Message-ID: Talin acm.org> writes: > I decided to take some of the ideas discussed in the string formatting > thread, add a few touches of my own, and write up a PEP. > > http://viridia.org/python/doc/PEP_AdvancedStringFormatting.txt > > (I've also submitted the PEP via the normal channels.) No responses? I'm surprised... (I assume the PEP editor is on vacation, since I haven't gotten a response from that channel either.) I realize its long, and people might not have had time to read the whole thing. But I wanted to make sure that every issue had been addressed. Or if you just want me to drop it, I can do that as well...just do so politely and I will take no offense :) (I'm a pretty hard person to offend.) -- Talin From ianb at colorstudy.com Fri Apr 21 19:21:16 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 21 Apr 2006 12:21:16 -0500 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: <4449148C.7000708@colorstudy.com> Talin wrote: > I know you want implementations rather than PEPs at this point, > but if there is a consensus on both the "keywords after *args" and > "keyword only" arguments, I can write up a PEP for that, if that > would be useful. Would a doctest be a good middle ground? Helps you write the implementation, deals with highly specific aspects (e.g., what the error messages are), makes it clear what you intend to write. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From osantana at gmail.com Fri Apr 21 19:20:40 2006 From: osantana at gmail.com (Osvaldo Santana) Date: Fri, 21 Apr 2006 14:20:40 -0300 Subject: [Python-3000] Changing the import machinery In-Reply-To: References: Message-ID: Hi, I've another subject related to import machinery to add in this thread. We've discussed about this at py-dev when I found an inconsistent import/zipimport behavior with .pyc/.pyo compiled modules: http://mail.python.org/pipermail/python-dev/2005-November/057959.html http://mail.python.org/pipermail/python-dev/2005-November/057983.html I think that is a good idea to separate the 'optimization' option (-O) and 'no-docstring' option (-OO). I would like to +1 the Phillip J. Eby suggestion about 'JUMP_IF_NOT_DEBUG' opcode to remove the requirement of a .pyo file generation: http://mail.python.org/pipermail/python-dev/2005-November/057985.html Thanks, Osvaldo On 4/20/06, Guido van Rossum wrote: > I'm changing the list and the subject, pulling this quote out of python-dev: > > On 4/20/06, in python-dev, Fredrik Lundh wrote: > > I was hoping that for Python 3.0, we could get around to unkludge the > > sys.path/meta_path/path_hooks/path_importer_cache big ball of hacks, > > possibly by replacing sys.path with something a bit more intelligent than > > a plain list. > > That's an excellent idea. Are there any volunteers here to help out? > Even just listing specific use cases / scenarios that are currently > difficult to solve right would be tremendously helpful. (I think that > Phillip's and others' experience with setuptools might be very > useful.) > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) -- Osvaldo Santana Neto (aCiDBaSe) icq, url = (11287184, "http://www.pythonbrasil.com.br") From rasky at develer.com Fri Apr 21 19:30:15 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 21 Apr 2006 19:30:15 +0200 Subject: [Python-3000] p3yk branch and merging (was Re:[Python-3000-checkins] r45619 -python/branches/p3yk/Modules/cjkcodecs/multibytecodec.c) References: <9e804ac0604210947j32cf3267lfee57c9229347061@mail.gmail.com> Message-ID: <01b701c66569$40713410$bf03030a@trilan> Thomas Wouters wrote: >> And just so everyone knows: done right, merging isn't hard. It's not >> as easy as it could be, but 'svn merge' makes it a lot easier than >> it used to be in CVS. Here's how you do it: >> [...] It can be much much easier than this: SVN 1.3 ships with a client-side tool called svnmerge.py, which implements merge-tracking. Latest (improved) version is here: http://svn.collab.net/repos/svn/trunk/contrib/client-side/svnmerge.py A quick usage guide can be found here: http://www.ruby-forum.com/topic/61372 and here: http://www.dellroad.org/svnmerge/index Incidentally, it also allows to mark revisions that were manually merged into the trunk from the branch (look into the "svnmerge merge --record-only"). This means that people can merge in either direction without making the merge process harder. -- Giovanni Bajo From ianb at colorstudy.com Fri Apr 21 19:30:30 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 21 Apr 2006 12:30:30 -0500 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: References: Message-ID: <444916B6.4010205@colorstudy.com> Guido van Rossum wrote: > This seems a good idea to remember when we get to that point. > > But let me point out that the key concern I have about the expense of > type checking is what would be done when unchecked code calls a > function with type-checked arguments. If I have some utterly dynamic > code that comes up with a list of a million ints, and then I pass that > as an argument to a function that requests the argument type is > list[int], then I don't see how we can prevent the cost of doing a > million checks to ensure that each and every item in the list is an > int; either we do it all at once at the call site, or we amortize the > cost over the lifetime of every list object, whether or not it is ever > going to be used in a typechecked call. Neither sounds very > attractive, and AFAICT your approach doesn't solve this particular > issue. If you do the check early for intness (or lazily with a wrapper), that would hopefully mean that an optimizer could use that information to avoid later type checks, e.g., by inlining code. Though that implies that you are checking for concrete types, not interfaces. Anyway, you have to do a type test at some point when you use an object; if it is done earlier that shouldn't mean more work ultimately (unless you are ignoring lots of objects). Lazy testing for sequences makes some sense, because in the case of an iterator (and maybe even an iterable, which may not be finite) you can only ever do lazy testing. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From rrr at ronadam.com Fri Apr 21 19:49:43 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 21 Apr 2006 12:49:43 -0500 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: <0eb101c6652f$363ec730$2452fea9@bagio> References: <44482BE1.8030109@canterbury.ac.nz> <0eb101c6652f$363ec730$2452fea9@bagio> Message-ID: Giovanni Bajo wrote: > Ron Adam wrote: > >> This *may* relate to None being an object which isn't the same as >> "not a value". There currently isn't a way (that I know of) to >> specify a generally null object outside of sequences. >> >> def f(a, b, c=Null, d=x): # Using None here wouldn't work. >> >> >> Recently I found a case where I wanted to return something that was >> more literally *nothing* than a None is. So maybe a null symbol of >> some sort might be useful in other cases as well? > > And what's wrong with: > > class Null: pass > def f(a, b, c=Null, d=x): > [...] > > Giovanni Bajo That would only be part way there. The interpreter would need to know about the Null object so it could treat it as a missing value in the case of a function definition. And Guido already said NULL wouldn't be part of the language. Cheers, Ron From aahz at pythoncraft.com Fri Apr 21 19:53:23 2006 From: aahz at pythoncraft.com (Aahz) Date: Fri, 21 Apr 2006 10:53:23 -0700 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: <20060421175323.GA21396@panix.com> On Fri, Apr 21, 2006, Guido van Rossum wrote: > On 4/21/06, Mark Russell wrote: >> >> Another wild thought: >> >> def foo(a, b, @keyword_only c, d): >> pass >> >> Actually that one could go in 2.X - it's currently a syntax error. > > To prevent more abominations like this, let me pronounce that I now > like the single-star syntax: > > def foo(a, b, *, x=1, y=2): ... So the way to get you to pronounce in favor of an abomination is to suggest a worse abomination? ;-) -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From rasky at develer.com Fri Apr 21 20:02:03 2006 From: rasky at develer.com (Giovanni Bajo) Date: Fri, 21 Apr 2006 20:02:03 +0200 Subject: [Python-3000] PEP - string.format References: Message-ID: <020901c6656d$b191f090$bf03030a@trilan> Talin wrote: > http://viridia.org/python/doc/PEP_AdvancedStringFormatting.txt Some comments: - Could this be implemented as a "preview" in Python 2.x? Do we want to? - I like having an operator for string formatting. I'm -0 on dropping it for a .format() method. - string.Template becomes obsolete with this PEP (as my understanding is that it was added only to provide positional arguments). Of course other people will have other templating libraries (outside stdlib), but I can't see why you shouldn't declare string.Template obsoleted by this PEP. It violates TOOWTDI for no good reason. - Provide rationale for defaulting to a positional-argument style. It's a good choice IMO, but you should explain why it's good to say {0} {1} instead of {d} {i} or similar. - You say that {} is unspecified. Why don't it make it mean "previous positional argument + 1"? - I'm confused as why your datetime example uses {0:%x} instead of {0:x}. - I'm not sure what __format__ buys you over __unicode__ / __str__. I know there is the additional "specifiers" argument, but maybe there is a way we can unify the two things in Py3k. - I feel strongly that we ought to have a way to implicitly refer to locals() in string formatting. I like your solution in which the formatting operation is explicit (so that a string is not magically interpolated ala Perl, which I find inconvenient), but the locals() dictionary is assumed by default. IMO, it doesn't need a new method, simply calling .format() with no arguments is clear and elegant. My second favourite is the backtick which is free again in Python 3.0. - The PEP is mixmatching the issue of have a simple way to spell .format(**locals()) with the issue of allowing arbitrary complex expressions within the format specifiers. I see that you think that {parser.line} is useful, but I don't see why "Error in file {0.file}, line {0.line}".format(parser) is less useful (notice that using {0},{1} with format(parser.file, parser.line) could be said to violate the "spell it once" principle). I think the two issues should be separated. - As for supporting expression like {a.b} in format specifiers, I'm -0. If we go that way, we end up with Cheetah or whatever. Specifically, why should we allow {a.b} and not {a[0]}? And {a[0].b}? And {a[k].b}? And {a[k+1].b}? And {a[{x}].b}? Basically, we either allow an eval() within the specified context or we do nothing. I think I prefer if we do not allow anything. -- Giovanni Bajo From rrr at ronadam.com Fri Apr 21 20:04:18 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 21 Apr 2006 13:04:18 -0500 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: References: <44482BE1.8030109@canterbury.ac.nz> Message-ID: Guido van Rossum wrote: > On 4/21/06, Ron Adam wrote: >> Recently I found a case where I wanted to return something that was more >> literally *nothing* than a None is. So maybe a null symbol of some sort >> might be useful in other cases as well? > > You're not gonna get something that's a valid expression *and* is more > "nothing" than None. Yes, it would require special casing either by object methods and/or by the interpreter to work in a general way. Which is probably enough of a reason not to do it. > On a case by case basis there might be syntax indicating the absence > of something that is distinguishable from 'None'; for example, using > *args you can reliaby whether an argument was passed or not (unlike > using None as a default, which can be defeated by passing an explicit > None). Yes, in cases where containers are used, as in *args, the identity "x == x + empty_container" works. What I was looking for was the same identity operations with an append method. x == x.append(foo(y)) # when foo returns a NULL object. Not that it's a big deal, I just thought it might also fit with Gregs suggestion as well where a keyword with a NULL could indicate a defualt isn't present and must be supplied when used as a defualt. > But a generic NULL won't be part of the language. I'm fine with that. I realized it may need a lot more *good* use cases to not have the special isn't special enough guide line apply. Cheers, Ron > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 21 20:12:45 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 19:12:45 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: On 4/21/06, Talin wrote: > Guido van Rossum python.org> writes: > > > To prevent more abominations like this, let me pronounce that I now > > like the single-star syntax: > > > > def foo(a, b, *, x=1, y=2): ... > > Thank you :) It was getting pretty strange there. > > The variation that I was thinking of was a little shorter, but not > necessarily better: > > def foo( a, b; x=1, y=2 ): ... That cropped up in my head too long ago. I think I've seen something like this in another language. (Perl? VB?) But it reverses the priority of ';' and that's not very nice. > BTW I still haven't given up on working on an implementation for > keywords after *args, but it's been going rather slowly due to lack > of time to sit down and focus. Just wait for the weekend. :-) > I know you want implementations rather than PEPs at this point, > but if there is a consensus on both the "keywords after *args" and > "keyword only" arguments, I can write up a PEP for that, if that > would be useful. Consensus is a big word, but I'd be happy to see a PEP that explains the currently best proposals and clarifies any gray areas that the informal proposals might have. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jimjjewett at gmail.com Fri Apr 21 20:14:30 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 21 Apr 2006 14:14:30 -0400 Subject: [Python-3000] Pronouncement on parameter lists Message-ID: On 4/21/06, Alex Martelli wrote: > On 4/21/06, Guido van Rossum wrote: > > To prevent more abominations like this, let me pronounce that I now > > like the single-star syntax: > > def foo(a, b, *, x=1, y=2): ... > So, what will this syntax signify? If the single-star stands for > "arbitrary positional arguments", how will the body of foo access > them? Sorry if these have been well covered already, but I read back > through this thread and couldn't find them. No, *args stands for arbitrary positional arguments. The unadorned * stands for "anything after this must be passed as a keyword". It has not yet been specified what would happen to additional positional arguments that get passed in anyway. (Swallow or raise an Exception?) It has not yet been specified whether the keyword-only arguments must each have a default. Most proposals assume not, but then do ugly things to support that assumption. The above suggests def foo(a, b=2, *, x=3, y=4): # OK def foo(a, b=2, *, x, y=4): # not yet decided -jJ From guido at python.org Fri Apr 21 20:22:19 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 19:22:19 +0100 Subject: [Python-3000] Pronouncement on parameter lists In-Reply-To: References: Message-ID: On 4/21/06, Jim Jewett wrote: > On 4/21/06, Alex Martelli wrote: > > On 4/21/06, Guido van Rossum wrote: > > > > To prevent more abominations like this, let me pronounce that I now > > > like the single-star syntax: > > > > def foo(a, b, *, x=1, y=2): ... > > > So, what will this syntax signify? If the single-star stands for > > "arbitrary positional arguments", how will the body of foo access > > them? Sorry if these have been well covered already, but I read back > > through this thread and couldn't find them. > > No, *args stands for arbitrary positional arguments. > > The unadorned * stands for "anything after this must be passed as a keyword". > > It has not yet been specified what would happen to additional > positional arguments that get passed in anyway. (Swallow or raise an > Exception?) It would be useless to swallow them (*ignored already lets you do that). > It has not yet been specified whether the keyword-only arguments must > each have a default. Most proposals assume not, but then do ugly > things to support that assumption. The above suggests > > def foo(a, b=2, *, x=3, y=4): # OK > def foo(a, b=2, *, x, y=4): # not yet decided I first was gonna say "illegal of course" but on second though the feature of having required keyword args is sometimes useful; I believe I've coded such a signature myself once or twice out of **kwds and lots of checking. FWIW Talin, if you're writing up a PEP for this, could I ask you to also specify a new introspection API? A function foo should have a magic attribute foo.__signature__ which provides access to the argument names and defaults (and types if in the future we add argument declarations). __signature__ objects should be copyable and mutable and the attribute should be assignable so that wrapping decorators can copy and modify the signature of the wrapped function onto the wrapper function (the latter is usually declared as *args, **kwds). (Ideally the __signature__ attribute on a bound method should return the unbound method's signature modified to remove 'self'.) Oh, other standard function attributes probably all ought to be renamed from func_whatever to __whatever__. That applies to 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', 'func_globals', 'func_name'. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From p.f.moore at gmail.com Fri Apr 21 20:25:47 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 21 Apr 2006 19:25:47 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: References: <4447523F.4030504@canterbury.ac.nz> Message-ID: <79990c6b0604211125m76685ad5tc929b9f9fe891e9c@mail.gmail.com> On 4/21/06, Guido van Rossum wrote: > On 4/21/06, Talin wrote: > > The variation that I was thinking of was a little shorter, but not > > necessarily better: > > > > def foo( a, b; x=1, y=2 ): ... > > That cropped up in my head too long ago. I think I've seen something > like this in another language. (Perl? VB?) But it reverses the > priority of ';' and that's not very nice. I'm not sure I follow the priority argument (except in a purely parser-technical sense, I guess). For me, the parentheses ensure that I only think about the parameter list in relation to the semicolon, and the semicolon (meaning roughly "and then") nicely splits the positional arguments from the non-positional ones. If keyword-only arguments *without* a default were involved, def foo(a, b; x, y=1): ... doesn't look good. But I'd be happy with mandating that keyword-only arguments must have a default (it's similar in principle to the rule in option parsing that (keyword) options should be optional, and hence have viable defaults. (See the optparse documentation, section 13.3.1.2 in the Python library reference.) I prefer semicolon to a bare *, but I'm not going to start a crusade over it... Paul. From ianb at colorstudy.com Fri Apr 21 20:25:39 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Fri, 21 Apr 2006 13:25:39 -0500 Subject: [Python-3000] PEP - string.format In-Reply-To: References: Message-ID: <444923A3.2050205@colorstudy.com> Talin wrote: > Talin acm.org> writes: > > >>I decided to take some of the ideas discussed in the string formatting >>thread, add a few touches of my own, and write up a PEP. >> >>http://viridia.org/python/doc/PEP_AdvancedStringFormatting.txt >> >>(I've also submitted the PEP via the normal channels.) > > > No responses? I'm surprised... You should have copied the PEP into the email... it was a whole click away, thus easier to ignore ;) The scope of this PEP will be restricted to proposals of built-in string formatting operations (in other words, methods of the built-in string type.) This does not obviate the need for more sophisticated string-manipulation modules in the standard library such as string.Template. In any case, string.Template will not be discussed here, except to say that the this proposal will most likely have some overlapping functionality with that module. s/module/class/ The '%' operator is primarily limited by the fact that it is a binary operator, and therefore can take at most two arguments. One of those arguments is already dedicated to the format string, leaving all other variables to be squeezed into the remaining argument. The current practice is to use either a dictionary or a list as the second argument, but as many people have commented [1], this lacks flexibility. The "all or nothing" approach (meaning that one must choose between only positional arguments, or only named arguments) is felt to be overly constraining. A dictionary, *tuple*, or a single object. That a tuple is special is sometimes confusing (in most other places lists can be substituted for tuples), and that the single object can be anything but a dictionary or tuple can also be confusing. I've seen nervous people avoid the single object form entirely, often relying on the syntactically unappealing single-item tuple ('' % (x,)). Brace characters ('curly braces') are used to indicate a replacement field within the string: "My name is {0}".format( 'Fred' ) While I've argued in an earlier thread that $var is more conventional, honestly I don't care (except that %(var)s is not very nice). A couple other people also preferred $var, but I don't know if they have particularly strong opinions either. The result of this is the string: "My name is Fred" The element within the braces is called a 'field name' can either be a number, in which case it indicates a positional argument, or a name, in which case it indicates a keyword argument. Braces can be escaped using a backslash: "My name is {0} :-\{\}".format( 'Fred' ) Which would produce: "My name is Fred :-{}" Does } have to be escaped? Or just optionally escaped? I assume this is not a change to string literals, so we're relying on '\{' producing the same thing as '\\{' (which of course it does). Each field can also specify an optional set of 'conversion specifiers'. Conversion specifiers follow the field name, with a colon (':') character separating the two: "My name is {0:8}".format( 'Fred' ) The meaning and syntax of the conversion specifiers depends on the type of object that is being formatted, however many of the built-in types will recognize a standard set of conversion specifiers. The conversion specifier consists of a sequence of zero or more characters, each of which can consist of any printable character except for a non-escaped '}'. The format() method does not attempt to intepret the conversion specifiers in any way; it merely passes all of the characters between the first colon ':' and the matching right brace ('}') to the various underlying formatters (described later.) Thus you can't nest formatters, e.g., {0:pad(23):xmlquote}, unless the underlying object understands that. Which is probably unlikely. Potentially : could be special, but \: would be pass the ':' to the underlying formatter. Then {x:pad(23):xmlquote} would mean format(format(x, 'pad(23)'), 'xmlquote') Also, I note that {} doesn't naturally nest in this specification, you have to quote those as well. E.g.: {0:\{a:b\}}. But I don't really see why you'd be inclined to use {} in a formatter anyway ([] and () seem much more likely). Also, some parsing will be required in these formatters, e.g., pad(23) is not parsed in any way and so it's up to the formatter to handle that (and may use different rules than normal Python syntax). When using the 'fformat' variant, it is possible to omit the field name entirely, and simply include the conversion specifiers: "My name is {:pad(23)}" This syntax is used to send special instructions to the custom formatter object (such as instructing it to insert padding characters up to a given column.) The interpretation of this 'empty' field is entirely up to the custom formatter; no standard interpretation will be defined in this PEP. If a custom formatter is not being used, then it is an error to omit the field name. This sounds similar to (?i) in a regex. I can't think of a good use-case, though, since most commands would be localized to a specific formatter or to the formatting object constructor. {:pad(23)} seems like a bad example. {:upper}? Also, it applies globally (or does it?); that is, the formatter can't detect what markers come after the command, and which come before. So {:upper} seems like a bad example. Standard Conversion Specifiers: For most built-in types, the conversion specifiers will be the same or similar to the existing conversion specifiers used with the '%' operator. Thus, instead of '%02.2x", you will say '{0:2.2x}'. There are a few differences however: - The trailing letter is optional - you don't need to say '2.2d', you can instead just say '2.2'. If the letter is omitted, then the value will be converted into its 'natural' form (that is, the form that it take if str() or unicode() were called on it) subject to the field length and precision specifiers (if supplied.) - Variable field width specifiers use a nested version of the {} syntax, allowing the width specifier to be either a positional or keyword argument: "{0:{1}.{2}d}".format( a, b, c ) (Note: It might be easier to parse if these used a different type of delimiter, such as parens - avoiding the need to create a regex that handles the recursive case.) Ah... that's an interesting way to use nested {}. I like that. A class that wishes to implement a custom interpretation of its conversion specifiers can implement a __format__ method: class AST: def __format__( self, specifiers ): ... The 'specifiers' argument will be either a string object or a unicode object, depending on the type of the original format string. The __format__ method should test the type of the specifiers parameter to determine whether to return a string or unicode object. It is the responsibility of the __format__ method to return an object of the proper type. If nested/piped formatting was allowed (like {0:trun(23):xmlquote}) then it would be good if it could return any object, and str/unicode was called on that object ultimately. I don't know if it would be considered an abuse of formatting, but maybe a_dict.__format__('x') could return a_dict['x']. Probably not a good idea. The string.format() will format each field using the following steps: 1) First, see if there is a custom formatter. If one has been supplied, see if it wishes to override the normal formatting for this field. If so, then use the formatter's format() function to convert the field data. 2) Otherwise, see if the value to be formatted has a __format__ method. If it does, then call it. 3) Otherwise, check the internal formatter within string.format that contains knowledge of certain builtin types. If it is a language change, could all those types have __format__ methods added? Is there any way for the object to accept or decline to do formatting? 4) Otherwise, call str() or unicode() as appropriate. Is there a global repr() formatter, like %r? Potentially {0:repr} could be implemented the same way by convention, including in object.__format__? Custom Formatters: If the fformat function is used, a custom formatter object must be supplied. The only requirement is that it have a format() method with the following signature: def format( self, value, specifier, builder ) This function will be called once for each interpolated value. The parameter values will be: 'value' - the value that to be formatted. 'specifier' - a string or unicode object containing the conversion specifiers from the template string. 'builder' - contains the partially constructed string, in whatever form is most efficient - most likely the builder value will be a mutable array or buffer which can be efficiently appended to, and which will eventually be converted into an immutable string. What's the use case for this argument? The formatter should examine the type of the object and the specifier string, and decide whether or not it wants to handle this field. If it decides not to, then it should return False to indicate that the default formatting for that field should be used; Otherwise, it should call builder.append() (or whatever is the appropriate method) to concatenate the converted value to the end of the string, and return True. Well, I guess this is the use case, but it feels a bit funny to me. A concrete use case would be appreciated. Optional Feature: locals() support This feature is ancilliary to the main proposal. Often when debugging, it may be convenient to simply use locals() as a dictionary argument: print "Error in file {file}, line {line}".format( **locals() ) This particular use case could be even more useful if it were possible to specify attributes directly in the format string: print "Error in file {parser.file}, line {parser.line}" \ .format( **locals() ) It is probably not desirable to support execution of arbitrary expressions within string fields - history has shown far too many security holes that leveraged the ability of scripting languages to do this. A fairly high degree of convenience for relatively small risk can be obtained by supporting the getattr (.) and getitem ([]) operators. While it is certainly possible that these operators can be overloaded in a way that a maliciously written string could exploit their behavior in nasty ways, it is fairly rare that those operators do anything more than retargeting to another container. On other other hand, the ability of a string to execute function calls would be quite dangerous by comparison. It could be a keyword option to enable this. Though all the keywords are kind of taken. This itself wouldn't be an issue if ** wasn't going to be used so often. And/or the custom formatter could do the lookup, and so a formatter may or may not do getattr's. One other thing that could be done to make the debugging case more convenient would be to allow the locals() dict to be omitted entirely. Thus, a format function with no arguments would instead use the current scope as a dictionary argument: print "Error in file {p.file}, line {p.line}".format() An alternative would be to dedicate a special method name, other than 'format' - say, 'interpolate' or 'lformat' - for this behavior. It breaks some conventions to have a method that looks into the parent frame; but the use cases are very strong for this. Also, if attribute access was a keyword argument potentially that could be turned on by default when using the form that pulled from locals(). Unlike a string prefix, you can't tell that the template string itself was directly in the source code, so this could encourage some potential security holes (though it's not necessarily insecure). This would require some stack-frame hacking in order that format be able to get access to the scope of the calling function. Other, more radical proposals include backquoting (`), or a new string prefix character (let's say 'f' for 'format'): print f"Error in file {p.file}, line {p.line}" This prefix character could of course be combined with any of the other existing prefix characters (r, u, etc.) This does address the security issue. The 'f' reads better than the '$' prefix previous suggested, IMHO. Syntax highlighting can also be applied this way. (This also has the benefit of allowing Python programmers to quip that they can use "print f debugging", just like C programmers.) Alternate Syntax Naturally, one of the most contentious issues is the syntax of the format strings, and in particular the markup conventions used to indicate fields. Rather than attempting to exhaustively list all of the various proposals, I will cover the ones that are most widely used already. - Shell variable syntax: $name and $(name) (or in some variants, ${name}). This is probably the oldest convention out there, and is used by Perl and many others. When used without the braces, the length of the variable is determined by lexically scanning until an invalid character is found. This scheme is generally used in cases where interpolation is implicit - that is, in environments where any string can contain interpolation variables, and no special subsitution function need be invoked. In such cases, it is important to prevent the interpolation behavior from occuring accidentally, so the '$' (which is otherwise a relatively uncommonly-used character) is used to signal when the behavior should occur. It is my (Talin's) opinion, however, that in cases where the formatting is explicitly invoked, that less care needs to be taken to prevent accidental interpolation, in which case a lighter and less unwieldy syntax can be used. I don't think accidental problems with $ are that big a deal. They don't occur that often, and it's pretty obvious to the eye when they exist. "$lengthin" is pretty clearly not right compared to "${length}in". However, nervous shell programmers often use ${} everywhere, regardless of need, so this is likely to introduce style differences between programmers (some will always use ${}, some will remove {}'s whenever possible). However, it can be reasonable argued that {} is just as readable and easy to work with as $, and it avoids the need to do any changes as you reformat the string (possibly introducing or removing ambiguity), or add formatting. - Printf and its cousins ('%'), including variations that add a field index, so that fields can be interpolated out of order. - Other bracket-only variations. Various MUDs have used brackets (e.g. [name]) to do string interpolation. The Microsoft .Net libraries uses braces {}, and a syntax which is very similar to the one in this proposal, although the syntax for conversion specifiers is quite different. [2] Many languages use {}, including PHP and Ruby, and even $ uses it on some level. The details differ, but {} exists nearly everywhere in some fashion. - Backquoting. This method has the benefit of minimal syntactical clutter, however it lacks many of the benefits of a function call syntax (such as complex expression arguments, custom formatters, etc.) It doesn't have any natural nesting, nor any way to immediately see the difference between opening and closing an expression. It also implies a relation to shell ``, which evaluates the contents. I don't see any benefit to backquotes. Personally I'm very uncomfortable with using str.format(**args) for all named substitution. It removes the possibility of non-enumerable dictionary-like objects, and requires a dictionary copy whenever an actual dictionary is used. In the case of positional arguments it is currently an error if you don't use all your positional arguments with %. Would it be an error in this case? Should the custom formatter get any opportunity to finalize the formatted string (e.g., "here's the finished string, give me what you want to return")? -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From talin at acm.org Fri Apr 21 20:29:37 2006 From: talin at acm.org (Talin) Date: Fri, 21 Apr 2006 18:29:37 +0000 (UTC) Subject: [Python-3000] Pronouncement on parameter lists References: Message-ID: Jim Jewett gmail.com> writes: > It has not yet been specified what would happen to additional > positional arguments that get passed in anyway. (Swallow or raise an > Exception?) Additional positional arguments would be treated exactly as if you attempted to pass too many positional arguments to a function that took a fixed number of arguments. In other words, raise TypeError. > It has not yet been specified whether the keyword-only arguments must > each have a default. Most proposals assume not, but then do ugly > things to support that assumption. The above suggests > > def foo(a, b=2, *, x=3, y=4): # OK > def foo(a, b=2, *, x, y=4): # not yet decided Defaults and keywords have nothing to do with each other, other than the fact that they both use the '=' sign. So both examples above would be OK, however if you failed to supply a value for 'x', that would be an error in the second example, but not the first. -- Talin From rrr at ronadam.com Fri Apr 21 20:27:29 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 21 Apr 2006 13:27:29 -0500 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: <4448CC52.3000004@canterbury.ac.nz> References: <44482BE1.8030109@canterbury.ac.nz> <4448CC52.3000004@canterbury.ac.nz> Message-ID: Greg Ewing wrote: > Ron Adam wrote: > >> Or just ... >> >> def f(a, b, c=?, d=x): > > But there's nothing about this that particularly > suggests that c *must* be specified with a keyword. > It looks like just another positional arg with a > rather strange-looking default. The strange default would after a while be easily recognized for what it is, "A missing value which needs to be replaced at call time", but Guido said there won't be a generic NULL. So we can cross this one off. This would have indicated required keywords that are also positional, but not indicate required keywords that are non-positional. An actual syntax is needed for that instead of a Value placeholder. Your single '*' syntax which Guido now prefers fills this need better. ;-) Cheers, Ron From guido at python.org Fri Apr 21 20:33:04 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 19:33:04 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: <79990c6b0604211125m76685ad5tc929b9f9fe891e9c@mail.gmail.com> References: <4447523F.4030504@canterbury.ac.nz> <79990c6b0604211125m76685ad5tc929b9f9fe891e9c@mail.gmail.com> Message-ID: On 4/21/06, Paul Moore wrote: > I'm not sure I follow the priority argument (except in a purely > parser-technical sense, I guess). I meant that everywhere else in Python, ';' inside parentheses is illegal; ';' terminates a statement (and an error-correcting parser could use a ';' inside parentheses as a clue that a close paren is probably missing). I like this property because it aids readability by humans. There are a few operators that don't always have the same relative priorities (e.g. ',' and 'in' -- in a for-statement, ',' binds more tightly, but in an expression, 'in' binds more tightly) and those are somewhat bothersome to me. The parser has no problems because it knows in which context it is parsing; but it can make skimming code by humans harder. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 21 20:42:31 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 19:42:31 +0100 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: References: <44482BE1.8030109@canterbury.ac.nz> <4448CC52.3000004@canterbury.ac.nz> Message-ID: [Ron Adam] > def f(a, b, c=?, d=x): Another problem with this is that it uses punctuation in a manner that's neither similar to how it is used in written English (or Dutch, or any other Latin language) nor resembling of other areas in Python. I know this is a very strict constraint, and sometimes we violate it (@decorators come to mind) but still, as a general guideline, if it can be avoided, using punctuation in new and strange ways just isn't Pythonic. That also applies to using ';' but not (at least not so strongly) to '*' (since that can easily be seen as "*args without the args"). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Fri Apr 21 20:43:55 2006 From: barry at python.org (Barry Warsaw) Date: Fri, 21 Apr 2006 14:43:55 -0400 Subject: [Python-3000] PEP - string.format In-Reply-To: <444923A3.2050205@colorstudy.com> References: <444923A3.2050205@colorstudy.com> Message-ID: <1145645035.13310.80.camel@resist.wooz.org> On Fri, 2006-04-21 at 13:25 -0500, Ian Bicking wrote: > While I've argued in an earlier thread that $var is more conventional, > honestly I don't care (except that %(var)s is not very nice). A couple > other people also preferred $var, but I don't know if they have > particularly strong opinions either. For the use case that string.Template was originally designed for, I definitely agree. OTOH, as long as we don't change string.Template, it can still be used for simple string substitutions (hence the title of PEP 292) where all the extra formating is unnecessary. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060421/76dcc633/attachment.pgp From skip at pobox.com Fri Apr 21 21:05:21 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 21 Apr 2006 14:05:21 -0500 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: <17481.11505.620748.296453@montanaro.dyndns.org> Guido> To prevent more abominations like this, let me pronounce that I Guido> now like the single-star syntax: Guido> def foo(a, b, *, x=1, y=2): ... Sorry, I haven't been following this thread. What exactly does the naked star mean? Skip From skip at pobox.com Fri Apr 21 21:10:50 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 21 Apr 2006 14:10:50 -0500 Subject: [Python-3000] Type Expressions In-Reply-To: <17481.11505.620748.296453@montanaro.dyndns.org> References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> <17481.11505.620748.296453@montanaro.dyndns.org> Message-ID: <17481.11834.704977.365450@montanaro.dyndns.org> Guido> def foo(a, b, *, x=1, y=2): ... skip> Sorry, I haven't been following this thread. What exactly does skip> the naked star mean? Sorry, should have finished reading my mail first. S From talin at acm.org Fri Apr 21 21:15:14 2006 From: talin at acm.org (Talin) Date: Fri, 21 Apr 2006 19:15:14 +0000 (UTC) Subject: [Python-3000] PEP - string.format References: <020901c6656d$b191f090$bf03030a@trilan> Message-ID: Giovanni Bajo develer.com> writes: > Talin wrote: > > > http://viridia.org/python/doc/PEP_AdvancedStringFormatting.txt > > Some comments: Excellent feedback. I won't address everything at once - instead I'd like to collect feedback from various people and address the issues all at once. However, there's a few things I want to answer right away: > - Could this be implemented as a "preview" in Python 2.x? Do we want to? That is certainly a possibility. I have no opinion on this, but I'd be interest to know what the community wants. > - string.Template becomes obsolete with this PEP (as my understanding is > that it was added only to provide positional arguments). Of course other > people will have other templating libraries (outside stdlib), but I can't > see why you shouldn't declare string.Template obsoleted by this PEP. It > violates TOOWTDI for no good reason. I'll let other people decide on what's obselete or not. > - I'm confused as why your datetime example uses {0:%x} instead of {0:x}. The idea was that the specifier could be passed unmodified to the strftime() function. "%x" is interpreted by strftime() as meaning "the date in the current locale." > - The PEP is mixmatching the issue of have a simple way to spell > .format(**locals()) with the issue of allowing arbitrary complex expressions > within the format specifiers. I see that you think that {parser.line} is > useful, but I don't see why "Error in file {0.file}, line > {0.line}".format(parser) is less useful (notice that using {0},{1} with > format(parser.file, parser.line) could be said to violate the "spell it > once" principle). I think the two issues should be separated. > - As for supporting expression like {a.b} in format specifiers, I'm -0. If > we go that way, we end up with Cheetah or whatever. Specifically, why should > we allow {a.b} and not {a[0]}? And {a[0].b}? And {a[k].b}? And {a[k+1].b}? > And {a[{x}].b}? Basically, we either allow an eval() within the specified > context or we do nothing. I think I prefer if we do not allow anything. I'm not wedded to the idea, and I'd be happy to drop that whole part of if that's what folks want. (That's why I labeled that section "Optional".) The selection of which expressions to support in string templates was based on an informal, seat-of-the-pants estimate of the security risk of various operations. In other words, the partition between what's allowed and what isn't is not based on a logical, mathematical rule, but based on history and practice, which is much messier. -- Talin From jimjjewett at gmail.com Fri Apr 21 21:59:10 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 21 Apr 2006 15:59:10 -0400 Subject: [Python-3000] Signature API Message-ID: On 4/21/06, Guido van Rossum wrote: On 4/21/06, Guido van Rossum wrote: >... pronounce that I now like the single-star syntax: > def foo(a, b, *, x=1, y=2): ... [Alex asked for clarification] I wrote: > It has not yet been specified whether the keyword-only > arguments must each have a default. Most proposals > assume not, but then do ugly things to support that > assumption. The above suggests > def foo(a, b=2, *, x=3, y=4): # OK > def foo(a, b=2, *, x, y=4): # not yet decided On 4/21/06, Guido van Rossum wrote: > I first was gonna say "illegal of course" but on second > thought the feature of having required keyword args is > sometimes useful; I believe I've coded such a signature > myself once or twice out of **kwds and lots of checking. On the other hand, it would be pretty easy to miss that x needed a value, particularly if x were the third or fourth of a dozen keywords. > FWIW Talin, if you're writing up a PEP for this, could I ask > you to also specify a new introspection API? > A function foo should have a magic attribute > foo.__signature__ > which provides access to the argument names and > defaults (and types if in the future we add argument > declarations). >__signature__ objects should be copyable and > mutable and the attribute should be assignable so Do you just mean that the signature can be replaced, or do you really mean that it can be modified in place? > that wrapping decorators can copy and modify the > signature of the wrapped function onto the wrapper > function (the latter is usually declared as *args, > **kwds). Is the __signature__ used for anything except introspection? def f(a, b=2): pass f.__signature__ = () Should this false assignment raise an error? (If so, should a change that only renamed arguments be OK?) If the assignment is valid, should it actually modify f.func_defaults to match? What about f.func_code.co_varnames (etc)? Should calls to f be (in any way) checked against the new (false) signature? > (Ideally the __signature__ attribute on a bound method > should return the unbound method's signature modified to > remove 'self'.) Note: So far this could (theoretically) happen in 2.5 The renamings below might need a bit longer. > Oh, other standard function attributes probably all > ought to be renamed from func_whatever to > __whatever__. That applies to 'func_closure', > 'func_code', 'func_defaults', 'func_dict', 'func_doc', > 'func_globals', 'func_name'. I think some of this is already done; if not, then there are subtle issues. In 2.4, at least for a sample function quickly defined, f.func_dict is f.__dict__ f.func_doc is f.__doc__ f.func_name is f.__name__ I think func_defaults should probably just go away, and become an attribute of the signature. __doc__ and __name__ should probably also be available on the signature, though removing them from the function might be a mistake. Should bound function attributes im_class, im_func, and im_self be renamed? (Maybe not, because of the conflict over __class__) Should code objects' attributes be renamed without the co_ prefix? Should code objects have a signature, since that is where the signature really starts? (Functions add the defaults, change the doc, and may change the name.) Should signatures indicate return type (if types are added)? Are signatures a reasonable place to add other metadata, such as DBC contracts? Should signatures have a recommended text representation? -jJ From thomas at python.org Fri Apr 21 22:27:11 2006 From: thomas at python.org (Thomas Wouters) Date: Fri, 21 Apr 2006 22:27:11 +0200 Subject: [Python-3000] p3yk branch and merging (was Re:[Python-3000-checkins] r45619 -python/branches/p3yk/Modules/cjkcodecs/multibytecodec.c) In-Reply-To: <01b701c66569$40713410$bf03030a@trilan> References: <9e804ac0604210947j32cf3267lfee57c9229347061@mail.gmail.com> <01b701c66569$40713410$bf03030a@trilan> Message-ID: <9e804ac0604211327l497ef6fakcfc88ac42f0921c9@mail.gmail.com> On 4/21/06, Giovanni Bajo wrote: > > Thomas Wouters wrote: > > >> And just so everyone knows: done right, merging isn't hard. It's not > >> as easy as it could be, but 'svn merge' makes it a lot easier than > >> it used to be in CVS. Here's how you do it: > >> [...] > > It can be much much easier than this: SVN 1.3 ships with a client-side > tool > called svnmerge.py, which implements merge-tracking. Latest (improved) > version is here: > http://svn.collab.net/repos/svn/trunk/contrib/client-side/svnmerge.py Ah. Does it require a particular version of the server or client? I can't seem to find that information anywhere. If it doesn't, and everyone who does (or would do) branch merging can live with a python script for merging branches, let's use that instead. :) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060421/452c4416/attachment.htm From brett at python.org Fri Apr 21 22:41:48 2006 From: brett at python.org (Brett Cannon) Date: Fri, 21 Apr 2006 13:41:48 -0700 Subject: [Python-3000] Pronouncement on parameter lists In-Reply-To: References: Message-ID: On 4/21/06, Guido van Rossum wrote: > On 4/21/06, Jim Jewett wrote: > > On 4/21/06, Alex Martelli wrote: > > > On 4/21/06, Guido van Rossum wrote: > > > > > > To prevent more abominations like this, let me pronounce that I now > > > > like the single-star syntax: > > > > > > def foo(a, b, *, x=1, y=2): ... > > > > > So, what will this syntax signify? If the single-star stands for > > > "arbitrary positional arguments", how will the body of foo access > > > them? Sorry if these have been well covered already, but I read back > > > through this thread and couldn't find them. > > > > No, *args stands for arbitrary positional arguments. > > > > The unadorned * stands for "anything after this must be passed as a keyword". > > > > It has not yet been specified what would happen to additional > > positional arguments that get passed in anyway. (Swallow or raise an > > Exception?) > > It would be useless to swallow them (*ignored already lets you do that). > > > It has not yet been specified whether the keyword-only arguments must > > each have a default. Most proposals assume not, but then do ugly > > things to support that assumption. The above suggests > > > > def foo(a, b=2, *, x=3, y=4): # OK > > def foo(a, b=2, *, x, y=4): # not yet decided > > I first was gonna say "illegal of course" but on second though the > feature of having required keyword args is sometimes useful; I believe > I've coded such a signature myself once or twice out of **kwds and > lots of checking. > > FWIW Talin, if you're writing up a PEP for this, could I ask you to > also specify a new introspection API? A function foo should have a > magic attribute foo.__signature__ which provides access to the > argument names and defaults (and types if in the future we add > argument declarations). __signature__ objects should be copyable and > mutable and the attribute should be assignable so that wrapping > decorators can copy and modify the signature of the wrapped function > onto the wrapper function (the latter is usually declared as *args, > **kwds). > > (Ideally the __signature__ attribute on a bound method should return > the unbound method's signature modified to remove 'self'.) > I already have a rough draft PEP and implementation for a __signature__ object from a couple of months ago (sparked my thinking on __introspects__). I will go ahead and start another email thread with the PEP so that details can get hashed out on that separately. > Oh, other standard function attributes probably all ought to be > renamed from func_whatever to __whatever__. That applies to > 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', > 'func_globals', 'func_name'. > That would be nice. Should that be tossed into PEP 3100 for now so it isn't lost? -Brett From rhettinger at ewtllc.com Fri Apr 21 23:39:54 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Fri, 21 Apr 2006 14:39:54 -0700 Subject: [Python-3000] PEP - string.format In-Reply-To: <020901c6656d$b191f090$bf03030a@trilan> References: <020901c6656d$b191f090$bf03030a@trilan> Message-ID: <4449512A.5010800@ewtllc.com> >- I like having an operator for string formatting. I'm -0 on dropping it for >a .format() method. > > I'm +1 on some sort of change from the current %-operator which has issues distinguishing between scalar and collection arguments. Also, there are precedence issues (the current precedence is a vestige of % being used for modular arithmetic -- it is not the precedence we would have chosen otherwise). Also, the docs for %-formatting are more difficult to find because we use an operator instead of a named method. Lastly, the %-operator is not visually distinct from the %-symbols used in the adjacent format string. IMO, a named method solves all of these issues. Raymond From barry at python.org Fri Apr 21 23:45:33 2006 From: barry at python.org (Barry Warsaw) Date: Fri, 21 Apr 2006 17:45:33 -0400 Subject: [Python-3000] PEP - string.format In-Reply-To: <4449512A.5010800@ewtllc.com> References: <020901c6656d$b191f090$bf03030a@trilan> <4449512A.5010800@ewtllc.com> Message-ID: <1145655933.8218.111.camel@resist.wooz.org> On Fri, 2006-04-21 at 14:39 -0700, Raymond Hettinger wrote: > >- I like having an operator for string formatting. I'm -0 on dropping it for > >a .format() method. > IMO, a named method solves all of these > issues. +1 for a method to replace the operator. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060421/7177b550/attachment.pgp From rasky at develer.com Sat Apr 22 00:21:24 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 22 Apr 2006 00:21:24 +0200 Subject: [Python-3000] Type Comparisons with Godel Numbers References: Message-ID: <00dc01c66591$ed190120$a8b62997@bagio> Guido van Rossum wrote: > But let me point out that the key concern I have about the expense of > type checking is what would be done when unchecked code calls a > function with type-checked arguments. If I have some utterly dynamic > code that comes up with a list of a million ints, and then I pass that > as an argument to a function that requests the argument type is > list[int], then I don't see how we can prevent the cost of doing a > million checks to ensure that each and every item in the list is an > int; either we do it all at once at the call site, or we amortize the > cost over the lifetime of every list object, whether or not it is ever > going to be used in a typechecked call. Neither sounds very > attractive, and AFAICT your approach doesn't solve this particular > issue. One way would be to keep boolean flags like "is this a list of integers". It could be updated after each list modification, so that typechecks come for (almost) free. Another (similar) way would be to let the user pay for the high typechecking price in normal cases *but* provide a list[int] class, which is a list decorated with typechecks on modification operations. We could have list[int|float]() to construct a list which can hold either ints or floats. In both the above cases, you don't pay when you read from the list or pass it to a typechecked function, you pay only when you *modify* it. I imply that all these checks will be disabled at -O. Giovanni Bajo From guido at python.org Sat Apr 22 00:25:53 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 21 Apr 2006 23:25:53 +0100 Subject: [Python-3000] Signature API In-Reply-To: References: Message-ID: Jim, Can you try to produce strawman answers for all those questions? It may be a while before I'll get to answering them... --Guido On 4/21/06, Jim Jewett wrote: > On 4/21/06, Guido van Rossum wrote: > > On 4/21/06, Guido van Rossum wrote: > > >... pronounce that I now like the single-star syntax: > > > def foo(a, b, *, x=1, y=2): ... > > [Alex asked for clarification] > > I wrote: > > > It has not yet been specified whether the keyword-only > > arguments must each have a default. Most proposals > > assume not, but then do ugly things to support that > > assumption. The above suggests > > > def foo(a, b=2, *, x=3, y=4): # OK > > def foo(a, b=2, *, x, y=4): # not yet decided > > > On 4/21/06, Guido van Rossum wrote: > > I first was gonna say "illegal of course" but on second > > thought the feature of having required keyword args is > > sometimes useful; I believe I've coded such a signature > > myself once or twice out of **kwds and lots of checking. > > On the other hand, it would be pretty easy to miss that x needed a > value, particularly if x were the third or fourth of a dozen keywords. > > > FWIW Talin, if you're writing up a PEP for this, could I ask > > you to also specify a new introspection API? > > > A function foo should have a magic attribute > > foo.__signature__ > > which provides access to the argument names and > > defaults (and types if in the future we add argument > > declarations). > > >__signature__ objects should be copyable and > > mutable and the attribute should be assignable so > > Do you just mean that the signature can be replaced, or do you really > mean that it can be modified in place? > > > that wrapping decorators can copy and modify the > > signature of the wrapped function onto the wrapper > > function (the latter is usually declared as *args, > > **kwds). > > Is the __signature__ used for anything except introspection? > > def f(a, b=2): pass > > f.__signature__ = () > > Should this false assignment raise an error? (If so, should a change > that only renamed arguments be OK?) > > If the assignment is valid, should it actually modify f.func_defaults > to match? What about f.func_code.co_varnames (etc)? > > Should calls to f be (in any way) checked against the new (false) signature? > > > (Ideally the __signature__ attribute on a bound method > > should return the unbound method's signature modified to > > remove 'self'.) > > Note: So far this could (theoretically) happen in 2.5 > > The renamings below might need a bit longer. > > > Oh, other standard function attributes probably all > > ought to be renamed from func_whatever to > > __whatever__. That applies to 'func_closure', > > 'func_code', 'func_defaults', 'func_dict', 'func_doc', > > 'func_globals', 'func_name'. > > I think some of this is already done; if not, then there are subtle > issues. In 2.4, at least for a sample function quickly defined, > > f.func_dict is f.__dict__ > f.func_doc is f.__doc__ > f.func_name is f.__name__ > > I think func_defaults should probably just go away, and become an > attribute of the signature. __doc__ and __name__ should probably also > be available on the signature, though removing them from the function > might be a mistake. > > Should bound function attributes im_class, im_func, and im_self be > renamed? (Maybe not, because of the conflict over __class__) > > Should code objects' attributes be renamed without the co_ prefix? > > Should code objects have a signature, since that is where the > signature really starts? (Functions add the defaults, change the doc, > and may change the name.) > > Should signatures indicate return type (if types are added)? > > Are signatures a reasonable place to add other metadata, such as DBC contracts? > > Should signatures have a recommended text representation? > > -jJ > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From greg.ewing at canterbury.ac.nz Sat Apr 22 02:33:29 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 12:33:29 +1200 Subject: [Python-3000] Changing the import machinery In-Reply-To: <44490406.3040104@colorstudy.com> References: <4447B91A.8090409@colorstudy.com> <4447C9B2.8050009@colorstudy.com> <4447E9F7.5020300@colorstudy.com> <4448A2EB.8020401@canterbury.ac.nz> <44490406.3040104@colorstudy.com> Message-ID: <444979D9.90000@canterbury.ac.nz> Ian Bicking wrote: > No, I haven't; I assume you mean: > > from string import Template or from mypackage.backports.string24 > import Template No, I mean from string or mypackage.backports.string24 import Template which seems to be about the least ugly it can get given that you need to explicitly refer to a package name that's already rather long and ugly in itself. -- Greg From greg.ewing at canterbury.ac.nz Sat Apr 22 02:43:45 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 12:43:45 +1200 Subject: [Python-3000] Changing the import machinery In-Reply-To: <5.1.1.6.0.20060421120507.01e17e08@mail.telecommunity.com> References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> <5.1.1.6.0.20060421120507.01e17e08@mail.telecommunity.com> Message-ID: <44497C41.6060904@canterbury.ac.nz> Phillip J. Eby wrote: > So, what I'd suggest is that the expanded import mechanism would provide a > registry of extensions and handlers, Urg, global registries again. I'd like this better if the scope of a particular extension and its handler could be restricted to some chosen part of the package namespace. So the author of the 'grail' package can specify a handler for his special '.nun' files, but it only applies to imports within the 'grail' package, and can't conflict with anyone else's registration of a handler for the same extension in other places. Perhaps the handlers pertaining to a package could be attached to the module object for that package, in an __importers__ attribute or something like that. Then, when looking for a handler for a particular file, a search is made upwards from its containing package looking for a handler that matches it. -- Greg From greg.ewing at canterbury.ac.nz Sat Apr 22 03:14:25 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 13:14:25 +1200 Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: <44498371.20502@canterbury.ac.nz> Alex Martelli wrote: >> def foo(a, b, *, x=1, y=2): ... > > So, what will this syntax signify? This particular example means "a and b are required positional arguments, no other positional arguments are allowed, and x and y are optional keyword-only arguments". > If the single-star stands for > "arbitrary positional arguments", how will the body of foo access > them? To allow extra positional args, you would need a name after the *, same as now. -- Greg From greg.ewing at canterbury.ac.nz Sat Apr 22 03:28:50 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 13:28:50 +1200 Subject: [Python-3000] Pronouncement on parameter lists In-Reply-To: References: Message-ID: <444986D2.5060901@canterbury.ac.nz> Jim Jewett wrote: > It has not yet been specified what would happen to additional > positional arguments that get passed in anyway. (Swallow or raise an > Exception?) I've always intended that there would be an exception. -- Greg From talin at acm.org Sat Apr 22 03:34:57 2006 From: talin at acm.org (Talin) Date: Sat, 22 Apr 2006 01:34:57 +0000 (UTC) Subject: [Python-3000] Pronouncement on parameter lists References: Message-ID: Guido van Rossum python.org> writes: > FWIW Talin, if you're writing up a PEP for this, could I ask you to > also specify a new introspection API? A function foo should have a > magic attribute foo.__signature__ which provides access to the > argument names and defaults (and types if in the future we add > argument declarations). __signature__ objects should be copyable and > mutable and the attribute should be assignable so that wrapping > decorators can copy and modify the signature of the wrapped function > onto the wrapper function (the latter is usually declared as *args, > **kwds). All right, sure. BTW, does anyone have a suggestion for the name of this proposed PEP? I can probably get it done this evening. As far as the __signature__ variable goes: OK, let me think about this a bit. First thing that strikes me is that you can't use None to indicate a lack of a default, since a 'None' is a valid default value. Probably something like this: A sequence containing a tuple per argument, each tuple containing the name, an integer representing the number of stars (0, 1 or 2), and optionally a default value. If there's no default then its a 2- tuple, otherwise its a 3-tuple, so you can distinguish between the default=None case and the 'no default' case by looking at the length of the tuple. The 'naked star' parameter is represented by (None, VARARGS) - an unnamed argument of the varargs type. As far as the type of argument goes: There's no different between positional and keyword arguments from the recieving side, *except* that this new API creates such a distinction. However - all of the behaviors of the new API can be deduced by looking at the ordering of arguments. Thus, "keyword only" arguments do not need to be explicitly identified as such. So the function signature: def method( a, b, *args, c=1, **kwargs ): ... Would have a __signature__ variable of: [('a', 0), ('b', 0), ('args', 1), ('c', 0, 1), ('kwargs',2)] As you can see, only the 'c' argument has a 3-tuple (since it has a default), the others are all 2-tuples. 'c' can also be deduced as a position-only argument, since it follows an argument a '1' in the second field. Is that something like what you had in mind? -- Talin From greg.ewing at canterbury.ac.nz Sat Apr 22 03:37:35 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 13:37:35 +1200 Subject: [Python-3000] parameter lists [was: Type Expressions] In-Reply-To: References: <44482BE1.8030109@canterbury.ac.nz> <4448CC52.3000004@canterbury.ac.nz> Message-ID: <444988DF.3040507@canterbury.ac.nz> Guido van Rossum wrote: > (since that can easily be seen as "*args without the > args"). Waitress: Args, star, star, args, star and args without the args? Eeeuuuggh! -- Greg From paul-python at svensson.org Sat Apr 22 03:48:11 2006 From: paul-python at svensson.org (Paul Svensson) Date: Fri, 21 Apr 2006 21:48:11 -0400 (EDT) Subject: [Python-3000] Type Expressions In-Reply-To: References: <44458B7A.9060208@canterbury.ac.nz> <4447523F.4030504@canterbury.ac.nz> Message-ID: <20060421212834.C56623@familjen.svensson.org> On 4/21/06, Talin wrote: > Guido van Rossum python.org> writes: > >> To prevent more abominations like this, let me pronounce that I now >> like the single-star syntax: >> >> def foo(a, b, *, x=1, y=2): ... > > Thank you :) It was getting pretty strange there. > > The variation that I was thinking of was a little shorter, but not > necessarily better: > > def foo( a, b; x=1, y=2 ): ... At the risk of hypergeneralization... If *arg means "put the rest of the positional arguments in arg", then if we simply allow arg to be a tuple... def foo(*(a, b)): ... # disallow keyword args. def foo(a, b, *(), x, y): ... # disallow positional args after b. def foo(*(a, b), x, y): ... # a, b _must_ be positional; x, y keyword. That () = () is currently a syntax error might be considered a reason to use the lone star spelling instead of *(), but I find the tuple more obvious. /Paul From murman at gmail.com Sat Apr 22 04:41:17 2006 From: murman at gmail.com (Michael Urman) Date: Fri, 21 Apr 2006 21:41:17 -0500 Subject: [Python-3000] Brainstorming: literal construction hooks Message-ID: This idea isn't fully fleshed out, but I wanted to air it to see if it took wind or fell flat. Please forgive inaccuracies between lexing and parsing. It's about being able to override what a given literal is turned into. It would only take effect in a limited scope, either per module, per compile/exec, or something similar. When a literal of a registered token type is parsed, its string would be passed to the provided function and whose return value would be used for the object. The following example, when registered, would be used to turn all float literals into decimal instances. def float_literal(literal): return decimal.Decimal(literal) More simply decimal.Decimal could be registered as the literal constructor if just the literal's string is used. Alternative signatures could allow a single function to handle multiple token types, or could make applying modifications to normal types simpler, but would prevent the obvious use of decimal.Decimal as above. I see token_type and value (as normally parsed) as obvious candidates for this. This could be used for people who want to create auto-substituting string literals (which I dislike for I18N reasons) without having to shoehorn its own syntax into the language. There's a lot of holes to this story, including at least how these functions are registered, and which additional arguments (if any) are necessary. Shall we try to fill these holes in? Michael -- Michael Urman http://www.tortall.net/mu/blog From mahs at telcopartners.com Sat Apr 22 04:45:17 2006 From: mahs at telcopartners.com (Michael Spencer) Date: Fri, 21 Apr 2006 19:45:17 -0700 Subject: [Python-3000] Pronouncement on parameter lists In-Reply-To: References: Message-ID: Talin wrote: > Jim Jewett gmail.com> writes: > >> It has not yet been specified what would happen to additional >> positional arguments that get passed in anyway. (Swallow or raise an >> Exception?) > > Additional positional arguments would be treated exactly as if you attempted > to pass too many positional arguments to a function that took a fixed number > of arguments. In other words, raise TypeError. > Does calling with wrong arguments deserve a more specific Exception, e.g., ArgumentsError(TypeError)? The main motivation would be greater clarity and precision in the exception itself, but it would also enable dirt-simple function overloading. Michael From skip at pobox.com Sat Apr 22 05:40:17 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 21 Apr 2006 22:40:17 -0500 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: <00dc01c66591$ed190120$a8b62997@bagio> References: <00dc01c66591$ed190120$a8b62997@bagio> Message-ID: <17481.42401.755948.704375@montanaro.dyndns.org> Giovanni> One way would be to keep boolean flags like "is this a list of Giovanni> integers". It could be updated after each list modification, Giovanni> so that typechecks come for (almost) free. Where would that be done? If it's in the untyped function in Guido's example, how does it know it's supposed to be a list of ints? If it's in the typed function the untyped function calls, it seems to me you'll have to do the entire check of a million elements before any user code is executed to raising a typecheck error after partial execution someone else brought up. Other (difficult or unworkable I think) alternatives: * Require that typed functions may only be called by other typed functions. That way lies madness I think. It seems to me like it would be similar to the "const" virus in ANSI C. * Allow lazy type checking, but implement some sort of transaction that is only committed if the typed function executes successfully. (More madness. I know! Implement the virtual machine objects in a database! Tell the Jython and IronPython folks to suck rocks!) Then again, perhaps brighter folk will think of something that will actually work. (I'm happy to live without type annotations, but that's just me.) Skip From greg.ewing at canterbury.ac.nz Sat Apr 22 05:49:28 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 15:49:28 +1200 Subject: [Python-3000] Signature API In-Reply-To: References: Message-ID: <4449A7C8.8010705@canterbury.ac.nz> Jim Jewett wrote: > On the other hand, it would be pretty easy to miss that x needed a > value, particularly if x were the third or fourth of a dozen keywords. That could be addressed by requiring that all mandatory keyword-only args are listed before the optional ones in the def. Of course, you would be allowed to supply them in any order in the call. -- Greg From greg.ewing at canterbury.ac.nz Sat Apr 22 06:02:53 2006 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 22 Apr 2006 16:02:53 +1200 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: <00dc01c66591$ed190120$a8b62997@bagio> References: <00dc01c66591$ed190120$a8b62997@bagio> Message-ID: <4449AAED.3000604@canterbury.ac.nz> Giovanni Bajo wrote: > Another (similar) way would be to let the user pay for the high typechecking > price in normal cases *but* provide a list[int] class, which is a list > decorated with typechecks on modification operations. We could have > list[int|float]() to construct a list which can hold either ints or floats. This is worth thinking about. Recently when pondering the question of when it would or would not be appropriate to put in type assertions to help catch bugs, I concluded that it makes the most sense to do so when building a data structure, but not when just passing things around between calls. The most difficult-to-debug type errors are the ones where you've put something of the wrong type into a data structure, and it's sat there for a while not causing any obvious problem until something else tries to use it, by which time most evidence of where it came from is gone. So data structures which automatically check what you put into them could be handy to have. -- Greg From ncoghlan at gmail.com Sat Apr 22 06:11:01 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Apr 2006 14:11:01 +1000 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: References: <20060419111247.A832.JCARLSON@uci.edu> <44472D17.9010305@neotitans.com> <20060420084537.A83B.JCARLSON@uci.edu> <4447EE06.90705@neotitans.com> <4448A6D6.30003@canterbury.ac.nz> Message-ID: <4449ACD5.8070700@gmail.com> Jim Jewett wrote: > There, a Future is a proxy object, whose actual characteristics will > (or may) be filled in later. If you need it, you wait for it, but if > you don't need it yet, it can be created in the background. > > How to make this cleaner than the existing concrete implementations > (such as a twisted Deferred, or spawning a thread which puts a return > value in a known location) -- that is the part which hasn't been > worked out. Rampant API theft from weakref.proxy? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From talin at acm.org Sat Apr 22 06:20:16 2006 From: talin at acm.org (Talin) Date: Sat, 22 Apr 2006 04:20:16 +0000 (UTC) Subject: [Python-3000] Type Comparisons with Godel Numbers References: <00dc01c66591$ed190120$a8b62997@bagio> <4449AAED.3000604@canterbury.ac.nz> Message-ID: Greg Ewing canterbury.ac.nz> writes: > Giovanni Bajo wrote: > > > Another (similar) way would be to let the user pay for the high typechecking > > price in normal cases *but* provide a list[int] class, which is a list > > decorated with typechecks on modification operations. We could have > > list[int|float]() to construct a list which can hold either ints or floats. > > This is worth thinking about. Recently when pondering the > question of when it would or would not be appropriate to > put in type assertions to help catch bugs, I concluded > that it makes the most sense to do so when building a > data structure, but not when just passing things around > between calls. This makes total sense to me. You don't check the type of each entry of the list - you check the type of the list itself. This is pretty much the way statically typed languages work. This means that you can't pass a regular list to a function that expects a "list of ints". So its an extra step to convert the list to the proper type. As long as the syntax isn't too hard to type, that shouldn't be a problem, and this way the user knows that they are paying the cost for the conversion. "Explicit is better than implicit". -- Talin From rasky at develer.com Sat Apr 22 06:42:35 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 22 Apr 2006 06:42:35 +0200 Subject: [Python-3000] p3yk branch and merging (wasRe:[Python-3000-checkins] r45619-python/branches/p3yk/Modules/cjkcodecs/multibytecodec.c) References: <9e804ac0604210947j32cf3267lfee57c9229347061@mail.gmail.com><01b701c66569$40713410$bf03030a@trilan> <9e804ac0604211327l497ef6fakcfc88ac42f0921c9@mail.gmail.com> Message-ID: <014f01c665c7$2cfb45c0$a8b62997@bagio> Thomas Wouters wrote: >> Ah. Does it require a particular version of the server or client? I >> can't seem to find that information anywhere. No, it doesn't. I have been using it with SVN 1.1, for instance. Though you'll get speedups using newer clients as they implement more introspection features that svnmerge uses. And you are right this should be documented. Giovanni Bajo From talin at acm.org Sat Apr 22 06:55:52 2006 From: talin at acm.org (Talin) Date: Sat, 22 Apr 2006 04:55:52 +0000 (UTC) Subject: [Python-3000] PEP - string.format References: <444923A3.2050205@colorstudy.com> Message-ID: Ian Bicking colorstudy.com> writes: > ...A lot of good ideas, and I haven't had a chance to consider them all. Let me respond to what I can. > Brace characters ('curly braces') are used to indicate a > replacement field within the string: > > "My name is {0}".format( 'Fred' ) > > While I've argued in an earlier thread that $var is more conventional, > honestly I don't care (except that %(var)s is not very nice). A couple > other people also preferred $var, but I don't know if they have > particularly strong opinions either. I don't care much either; however as you noted, the nested form has some possible uses, which seems harder to do with just a prefix. > Does } have to be escaped? Or just optionally escaped? I assume this > is not a change to string literals, so we're relying on '\{' producing > the same thing as '\\{' (which of course it does). Good points. > Thus you can't nest formatters, e.g., {0:pad(23):xmlquote}, unless the > underlying object understands that. Which is probably unlikely. At this point, I'm thinking not, although I could be convinced otherwise. Remember, that you can accomplish all of the same things by processing the input arguments; The conversion specifiers are a convenience. Also, in your model, there would be a distinction between the first specifier (which converts the object to a string), and subsequent ones (which modify the string). My complexity senses are tingling... > Potentially : could be special, but \: would be pass the ':' to the > underlying formatter. Then {x:pad(23):xmlquote} would mean > format(format(x, 'pad(23)'), 'xmlquote') > > Also, I note that {} doesn't naturally nest in this specification, you > have to quote those as well. E.g.: {0:\{a:b\}}. But I don't really see > why you'd be inclined to use {} in a formatter anyway ([] and () seem > much more likely). Actually, I meant for them to nest, I just didn't say it right. > Also, some parsing will be required in these formatters, e.g., pad(23) > is not parsed in any way and so it's up to the formatter to handle that > (and may use different rules than normal Python syntax). I've been wondering about that too. > When using the 'fformat' variant, it is possible to omit the field > name entirely, and simply include the conversion specifiers: > > "My name is {:pad(23)}" > > This syntax is used to send special instructions to the custom > formatter object (such as instructing it to insert padding > characters up to a given column.) The interpretation of this > 'empty' field is entirely up to the custom formatter; no > standard interpretation will be defined in this PEP. > > If a custom formatter is not being used, then it is an error to > omit the field name. > > This sounds similar to (?i) in a regex. I can't think of a good > use-case, though, since most commands would be localized to a specific > formatter or to the formatting object constructor. {:pad(23)} seems > like a bad example. {:upper}? Also, it applies globally (or does it?); > that is, the formatter can't detect what markers come after the command, > and which come before. So {:upper} seems like a bad example. Right. My thought was that the empty field name would instruct the custom formatter to do something special *at that point in the string*. If you want something done to the entire string, it's probably easier to just pass the string to a function. > > 3) Otherwise, check the internal formatter within > string.format that contains knowledge of certain builtin > types. > > If it is a language change, could all those types have __format__ > methods added? Is there any way for the object to accept or decline to > do formatting? Good question. I suspect that it may be impractical to add __format__ to all built-in types, so we should plan to allow a fallback to an internal formatter. > 4) Otherwise, call str() or unicode() as appropriate. > > Is there a global repr() formatter, like %r? Potentially {0:repr} could > be implemented the same way by convention, including in object.__format__? Good idea. (Should there be a *global* custom formatter? Plugins? Subject of a separate PEP I think.) > The formatter should examine the type of the object and the > specifier string, and decide whether or not it wants to handle > this field. If it decides not to, then it should return False > to indicate that the default formatting for that field should be > used; Otherwise, it should call builder.append() (or whatever > is the appropriate method) to concatenate the converted value > to the end of the string, and return True. > > Well, I guess this is the use case, but it feels a bit funny to me. A > concrete use case would be appreciated. The main use case was that the formatter might need to examine the part of the string that's already been built. For example, it can't handle expansion of tabs unless it knows the current column index. I had originally planned to pass only the column index, but that seemed too special-case to me. > A fairly high degree of convenience for relatively small risk can > be obtained by supporting the getattr (.) and getitem ([]) > operators. While it is certainly possible that these operators > can be overloaded in a way that a maliciously written string could > exploit their behavior in nasty ways, it is fairly rare that those > operators do anything more than retargeting to another container. > On other other hand, the ability of a string to execute function > calls would be quite dangerous by comparison. > > It could be a keyword option to enable this. Though all the keywords > are kind of taken. This itself wouldn't be an issue if ** wasn't going > to be used so often. The keywords are all taken - but there are still plenty of method names available :) That's why "fformat" has a different method name, so that we can distinguish the custom formatter parameter from the rest of the params. Unfortunately, this can't be used too much, or you get a combinatorial explosion of method names: string.format string.fformat string.format_dict string.fformat_dict ... > One other thing that could be done to make the debugging case > more convenient would be to allow the locals() dict to be omitted > entirely. Thus, a format function with no arguments would instead > use the current scope as a dictionary argument: > > print "Error in file {p.file}, line {p.line}".format() > > An alternative would be to dedicate a special method name, other > than 'format' - say, 'interpolate' or 'lformat' - for this > behavior. > > It breaks some conventions to have a method that looks into the parent > frame; but the use cases are very strong for this. Also, if attribute > access was a keyword argument potentially that could be turned on by > default when using the form that pulled from locals(). To be honest, I'd be willing to drop this whole part of the proposal if that's what the folks here would like. I like to present all options, but that doesn't mean that I myself am in favor of all of them. I realize that there are some use cases for it, but I don't know if the use cases are significantly better. > - Backquoting. This method has the benefit of minimal syntactical > clutter, however it lacks many of the benefits of a function > call syntax (such as complex expression arguments, custom > formatters, etc.) > > It doesn't have any natural nesting, nor any way to immediately see the > difference between opening and closing an expression. It also implies a > relation to shell ``, which evaluates the contents. I don't see any > benefit to backquotes. Agreed 100%. > Personally I'm very uncomfortable with using str.format(**args) for all > named substitution. It removes the possibility of non-enumerable > dictionary-like objects, and requires a dictionary copy whenever an > actual dictionary is used. Right. Let me think about this one. > In the case of positional arguments it is currently an error if you > don't use all your positional arguments with %. Would it be an error in > this case? I dunno. It certainly could be implemented that way, but I am not sure why it should be. > Should the custom formatter get any opportunity to finalize the > formatted string (e.g., "here's the finished string, give me what you > want to return")? Easier to just pass the string to a function I think. Great stuff Ian, thanks for spending the time to write such a detailed critique. -- Talin From mike_traynar at credence.com Sat Apr 22 08:09:01 2006 From: mike_traynar at credence.com (Mike Traynar) Date: Sat, 22 Apr 2006 08:09:01 +0200 Subject: [Python-3000] New built-in function: bin() In-Reply-To: References: <4448A3D6.5090602@credence.com> Message-ID: <4449C87D.9060304@credence.com> An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060422/da27e874/attachment.html From tjreedy at udel.edu Sat Apr 22 08:18:42 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 22 Apr 2006 02:18:42 -0400 Subject: [Python-3000] Pronouncement on parameter lists References: Message-ID: "Brett Cannon" wrote in message news:bbaeab100604211341g657270ecu16b2e919dbb37c65 at mail.gmail.com... > On 4/21/06, Guido van Rossum wrote: >> Oh, other standard function attributes probably all ought to be >> renamed from func_whatever to __whatever__. That applies to >> 'func_closure', 'func_code', 'func_defaults', 'func_dict', 'func_doc', >> 'func_globals', 'func_name'. > > That would be nice. Should that be tossed into PEP 3100 for now so it > isn't lost? I would say 'of course' and also leave decided things in 3100, as a summary, even when they appear elsewhere. In fact, the code breaking changes should eventually be PEPed elsewhere (perhaps many together) to discuss and meet the 3002(?) breaking warning requirement. Terry Jan Reedy From ncoghlan at gmail.com Sat Apr 22 09:29:16 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Apr 2006 17:29:16 +1000 Subject: [Python-3000] PEP - string.format In-Reply-To: References: Message-ID: <4449DB4C.6050706@gmail.com> Talin wrote: > No responses? I'm surprised... > > (I assume the PEP editor is on vacation, since I haven't gotten a response > from that channel either.) A couple of days isn't really that long for Py3k stuff, and py-dev has been pretty busy for the last few days :) [...] > Or if you just want me to drop it, I can do that as well...just do so > politely and I will take no offense :) (I'm a pretty hard person to > offend.) Keep plugging away - this is important, IMO, and I think we can do better than the status quo. > PEP: xxx Title: Advanced String Formatting Version: $Revision$ [...] > In any case, string.Template will not be discussed here, > except to say that the this proposal will most likely have some overlapping > functionality with that module. string.Template will either be subsumed by the new mechanism, become an implementation detail of the new mechanism, or else be specialised even further towards il8n use cases. As you say, not something we should get hung up on at this point (although my expectation is that it is the last that will happen - the needs of il8n templating and general debug printing are quite different). > The '%' operator is primarily limited by the fact that it is a binary > operator, and therefore can take at most two arguments. One of those > arguments is already dedicated to the format string, leaving all other > variables to be squeezed into the remaining argument. As Ian noted, the special treatment of tuples and dictionaries makes this even worse (I'm one of those that tends to always use a tuple as the right operand rather than having to ensure that I'm not passing a tuple or dictionary) [...] > The second method, 'fformat', is identical to the first, except that it > takes an additional first argument that is used to specify a 'custom > formatter' object, which can override the normal formatting rules for > specific fields: > > "More on {0}, {1}, and {c}".fformat( formatter, a, b, c=d ) > > Note that the formatter is *not* counted when numbering fields, so 'a' is > still considered argument number zero. I don't like this. Formatting with a different formatter should be done as a method on the formatter object, not as a method on the string. However, there may be implementation strategies to make this easier, such as having str.format mean (sans caching): def format(*args, **kwds): # Oh for the ability to write self, *args = args! self = args[0] args = args[1:] return string.Formatter(self).format(*args, **kwds) Then custom formatters would be invoked via: MyFormatter("More on {0}, {1}, and {c}").format(a, b, c=d) And defined via inheritance from string.Formatter. Yes, this is deliberately modelled on the way string.Template works :) There may be some alternative design decisions to be made in the subclassing API, such as supporting methods on the formatter object that process each field in isolation: def format_field(self, field, fmt_spec, fmt_args, fmt_kwds): # field is x.partition(':')[0], where x is the text between the braces # fmt is x.partition(':')[2] or None if x.partition(':')[1] is false # fmt_args and fmt_kwds are the arguments passed to the # Formatter's format() method val = self.get_value(self, field, fmt_args, fmt_kwds) return self.format_value(self, val, fmt_spec) def get_value(self, field, fmt_args, fmt_kwds): try: pos = int(field) except ValueError: return fmt_kwds[field] return fmt_args[pos] def format_value(self, value, fmt_spec): return value.__format__(fmt_spec) For example, if a string.Template equivalent had these methods, they might look like: def get_value(self, field, fmt_args, fmt_kwds): return fmt_kwds[field] def format_value(self, value, fmt_spec): if format_spec is not None: raise ValueError("field formatting not supported") return value Another example would be a pretty-printer variant that pretty-printed types rather than using their normal string representation. [...] > Brace characters ('curly braces') are used to indicate a replacement field > within the string: I like this, since it provides OOWTDI, rather than "use this way if it's not ambiguous, this other way if it's otherwise unparseable". [...] > Braces can be escaped using a backslash: > > "My name is {0} :-\{\}".format( 'Fred' ) So "My name is 0} :-\{\}".format('Fred') would be an error? I like that - it means you get an immediate exception if you inadvertently leave out a brace, regardless of whether you leave out the left brace or the right brace. [...] > The conversion specifier consists of a sequence of zero or more characters, > each of which can consist of any printable character except for a > non-escaped '}'. Why "conversion specifier" instead of "format specifier"? > The format() method does not attempt to interpret the > conversion specifiers in any way; it merely passes all of the characters > between the first colon ':' and the matching right brace ('}') to the > various underlying formatters (described later.) If we had a subclassing API similar to what I suggest above, a custom formatter could easily support Ian's pipelining idea by doing: def format_value(self, value, fmt_spec): if fmt_spec is None: val = Formatter.format_value(self, value, fmt_spec) else: for fmt in fmt_spec.split(':'): val = Formatter.format_value(self, value, fmt_spec) return val I don't really think that should be the default, though. > When using the 'fformat' variant, it is possible to omit the field name > entirely, and simply include the conversion specifiers: > > "My name is {:pad(23)}" As suggested above, I think this should be invoked as a method on a custom formatter class. It would then be up to the get_value method to decide what to do when "field" was an empty string. (or else the formatter could just add an object with an appropriate __format__ method to the kwd dictionary under '') [...] > - The trailing letter is optional - you don't need to say '2.2d', you can > instead just say '2.2'. If the letter is omitted, then the value will be > converted into its 'natural' form (that is, the form that it take if str() > or unicode() were called on it) subject to the field length and precision > specifiers (if supplied.) I disagree with this. These format specifier do a type coercion before applying the formatting. These specifiers should be retained and should continue to result in coercion to int or float or str, with the relevant TypeErrors when that coercion isn't possible. > - Variable field width specifiers use a nested version of the {} syntax, > allowing the width specifier to be either a positional or keyword argument: > > "{0:{1}.{2}d}".format( a, b, c ) > > (Note: It might be easier to parse if these used a different type of > delimiter, such as parens - avoiding the need to create a regex that > handles the recursive case.) I like this idea, but it shouldn't be handled directly in the regex. Instead, invoke a second pass of the regex directly over the format specifier. [...] > For non-builtin types, the conversion specifiers will be specific to that > type. An example is the 'datetime' class, whose conversion specifiers are > identical to the arguments to the strftime() function: > > "Today is: {0:%x}".format( datetime.now() ) Well, more to the point it's that the format spec gets passed to the object, and its up to the object how to deal with it. This implies the need for an extensible protocol, either via a __format__ method, or via an extensible function (such as string.Formatter.format_value). [...] > The 'specifiers' argument will be either a string object or a unicode > object, depending on the type of the original format string. The __format__ > method should test the type of the specifiers parameter to determine > whether to return a string or unicode object. It is the responsibility of > the __format__ method to return an object of the proper type. Py3k => this problem goes away. This is about text, so the output is always unicode. > The string.format() will format each field using the following steps: > > 1) First, see if there is a custom formatter. If one has been supplied, > see if it wishes to override the normal formatting for this field. If so, > then use the formatter's format() function to convert the field data. As above - custom formatters should be separate objects that either inherit from the standard formatter, or simply expose an equivalent API. > 2) Otherwise, see if the value to be formatted has a __format__ method. If > it does, then call it. So an object can override standard parsing like {0:d} to return something other than an integer? *shudder* Being able to add extra formatting codes for different object types sounds good, but being able to change the meaning of the standard codes sounds (very) bad. If supporting the former also means supporting the latter, then I'd prefer to leave this ability to custom formatter objects or explicit method or function invocations on arguments in the call to format(). > 3) Otherwise, check the internal formatter within string.format that > contains knowledge of certain builtin types. This should happen *before* checking for a custom __format__ method. (If we decide to check for a custom __format__ method at all) > 4) Otherwise, call str() or unicode() as appropriate. As above, this will always be str() (which will actually be equivalent to 2.x's unicode()) > (Note: It may be that in a future version of Python, dynamic dispatch will > be used instead of a magic __format__ method, however that is outside the > scope of this PEP.) What's potentially in scope for this PEP, though, is to ensure that there's a hook that we could potentially hang dynamic dispatch on if we decide to use it, and we decide that __format__ provides worthwhile functionality :) > Custom Formatters: > > If the fformat function is used, a custom formatter object must be > supplied. The only requirement is that it have a format() method with the > following signature: > > def format( self, value, specifier, builder ) As noted above, I think this is a really bad way to implement custom formatters - string.Template provides a better model. [...] > This particular use case could be even more useful if it were possible to > specify attributes directly in the format string: > > print "Error in file {parser.file}, line {parser.line}" \ .format( > **locals() ) I don't think the additional complexity is worthwhile, given that this can be written: print "Error in file {0}, line {1}".format(parser.file, parser.line) [...] > One other thing that could be done to make the debugging case more > convenient would be to allow the locals() dict to be omitted entirely. > Thus, a format function with no arguments would instead use the current > scope as a dictionary argument: > > print "Error in file {p.file}, line {p.line}".format() Again, I don't think this is worth the additional complexity. [...] > Other, more radical proposals include backquoting (`), or a new string > prefix character (let's say 'f' for 'format'): > > print f"Error in file {p.file}, line {p.line}" A method will do the job with far more flexibility - no need to hack the parser. [...] > - Shell variable syntax: $name and $(name) (or in some variants, ${name}). > This is probably the oldest convention out there, and is used by Perl and > many others. When used without the braces, the length of the variable is > determined by lexically scanning until an invalid character is found. [...] > It is my (Talin's) opinion, however, that in cases where the formatting is > explicitly invoked, that less care needs to be taken to prevent accidental > interpolation, in which case a lighter and less unwieldy syntax can be > used. I don't follow this reasoning, but my reason for not liking shell syntax is that it doesn't handle formatting well, and there's a sharp disconnect between cases where the lexical parsing does the right thing, and those where it doesn't. A bracketed syntax handles all cases, and use braces for the purpose has the benefit of rarely being explicitly included in output strings. > - Printf and its cousins ('%'), including variations that add a field > index, so that fields can be interpolated out of order. My objection to these is similar to my objection to shell syntax, except the problem is that these handle formatting, but not cases where you want to name your interpolated values, or reference them by position. The problem with the syntax breaking down when the lexical parsing fails is similar, though - you still need to include a bracketed variant. > - Other bracket-only variations. Various MUDs have used brackets (e.g. > [name]) to do string interpolation. The Microsoft .Net libraries uses > braces {}, and a syntax which is very similar to the one in this proposal, > although the syntax for conversion specifiers is quite different. [2] Simple braces, with a ':' to separate the format specifier works for me. > - Backquoting. This method has the benefit of minimal syntactical clutter, > however it lacks many of the benefits of a function call syntax (such as > complex expression arguments, custom formatters, etc.) And there's already a suggestion on the table to use backquotes to denote "this is a string that must be a legal Python identifier" :) > Backwards Compatibility > > Backwards compatibility can be maintained by leaving the existing > mechanisms in place. The new system does not collide with any of the method > names of the existing string formatting techniques, so both systems can > co-exist until it comes time to deprecate the older system. This reads like a 2.x PEP, not a Py3k PEP :) What I'd suggest putting in here: With this proposal, it may seem to result in 3 different ways to do string formatting, in violation of TOOWTDI. The intent is for the mechanism in this PEP to become the OOW in Py3k, but there are good reasons to keep the other 2 approaches around. Retaining string.Template string.Template is a formatting variant tailored specifically towards simple string substitution. It is designed with a heavy emphasis on the templates being written by il8n translators rather than application programmers. string.Template should be retained as a simplified formatting variant for such problem domains. For simple string substitution, string.Template really is a better tool than the more general mechanism in this PEP - the more limited feature set makes it easier to grasp for non-programmers. The use of shell-style syntax for interpolation variables is appropriate as it is more likely to be familiar to translators without Python programming experience and the problem domain is one where the limitations of shell-syntax aren't a hassle (there is no template-based formatting, and the interpolated text is typically separated from the surrounding text by spaces). Retaining string %-formatting Removing string %-formatting would be a backwards compatibility nightmare. I doubt there's a Python program on the planet that would continue working if it was removed (I know most of mine would break in verbose mode). Even those which continued to work would likely break if all commented out debugging messages were uncommented. python3warn would struggle to find all cases (as it would need to be able to figure out when the left operand was a string) and even an instrumented build would leave significant room for doubt (as debug messages are often in rarely-exercised failure paths, which even decent unit tests might miss) Further, all of the formatting logic would need to be relocated to the implementation of the string formatting method, which while not that hard, would be effort that might be better expended elsewhere. OTOH, if string %-formatting is retained, the new format() method can rely on it as a low-level implementation detail, and %-formatting can continue to exist in that capacity - low-level formatting used for single values that are neither dictionaries nor tuples without having to go through the full regex based formatting machinery. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From fredrik at pythonware.com Sat Apr 22 10:23:50 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 22 Apr 2006 10:23:50 +0200 Subject: [Python-3000] Changing the import machinery References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: Guido van Rossum wrote: > > I'm afraid I disagree. PEP 302 actually has some tremendous advantages > > over a pure objects-on-sys.path approach: > > > > * Strings can be put in any configuration file, and used in .pth files > > > > * Strings can be put in environment variables (like PYTHONPATH). > > > > * Strings can be printed out, with all their contents showing and nothing > > hidden > > > > In short, strings are better for humans. > > I think I like this. I wonder if there's a parallel with my preference > for strings as paths instead of path objects... And strings as exceptions, and Tcl instead of Python ? ;-) Sorry, but I don't buy this argument at all. Of course you need a way to map from external path descriptions (PYTHONPATH, registry entries, etc) to sys.path contents, but ruling that the things you're manipulating *inside* a Python program must be strings so you "can print them out with all their contents showing and nothing hidden" doesn't strike me as very Pythonic. The target audience for this is Python programmers, after all, and Python programmers know how to inspect Python objects -- as long as they can find them, which isn't the case with today's extended import design, which *hides* lots of stuff in *separate* semi-secret registries. If you put all this back on the path, it'll be a lot easier to find and manipulate. I could quote the "If the implementation is hard to explain, it's a bad idea." zen here, but I'll quote Sean McGrath's 20th python zen instead: "Things should be as complex as necessary but not more complex." and offer a "let's get back to the basics and add stuff, instead of assuming that the status quo is complex and complicated because it has to be" solution. Here's an outline, off the top of my head: 1. sys.path can contain strings or import handlers 2. Strings work as today; as paths that a builtin import handler uses to look for packages or modules (flyweight-style). 3. Import handlers are duck-typed objects that implement a simplified version of the PEP 302 protocol. Handlers map dotted module paths to resources, where a resource can be a Python module, a Python package (a module container), or some other resource. Handlers are responsible for creating and populating module objects; whatever they return is stored in sys.modules and bound to the import target. I'm 50/50 on making the import machinery fully type agnostic; that is, allowing the import handler to return *any* kind of object also for ordinary imports. Importing e.g. PIL images and pre- parsed XML resources and Cheetah templates makes perfect sense to me. 4. A support library provides the following mechanisms: - An import handler for builtin/frozen objects (with corresponding hooks on the C API site, so that apps can register things to be treated as builtins). - An import handler for the standard library (*all of it*, minus site-packages!) - An import handler for directory names (used for string path items) - A registry for path specifier syntaxes - A parser for external path descriptions, which uses the registry to map from path components to import handlers - (possibly) Some helpers for user-provided import handlers (to be factored out from such handlers, rather than be designed up front) 5. Remove PTH support, and possibly also a lot of the site- packages related stuff in site.py. I'm 50/50 on *requiring* code to specify what non-core libraries they want to use, before they can import them. I'm also 50/50 on making some additions to the import statement syntax, to make some operations easier (import ... using handler), but I'll leave that for another post. This would cleanly address every deployment scenario and custom importer that I've used with a (near) mininum of core support, and frankly, I fail to see any potential use case that cannot be handled by this mechanism (simply because a handler can do *anything* behind the scenes, without requiring more support from the core import machinery). And it'll let us remove tons of "hard to explain" code from the core. From guido at python.org Sat Apr 22 10:24:27 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 09:24:27 +0100 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: Message-ID: It's an interesting idea; it's been brought up before but nobody AFAIK has ever implemented it. I like the scoping requirement. I suggest you try to implement it and see how well it works. You probably also want to be able to hook list, tuple and dict displays (perhaps by supplying an alternative factory function). Good lock! --Guido On 4/22/06, Michael Urman wrote: > This idea isn't fully fleshed out, but I wanted to air it to see if it > took wind or fell flat. Please forgive inaccuracies between lexing and > parsing. > > It's about being able to override what a given literal is turned into. > It would only take effect in a limited scope, either per module, per > compile/exec, or something similar. When a literal of a registered > token type is parsed, its string would be passed to the provided > function and whose return value would be used for the object. The > following example, when registered, would be used to turn all float > literals into decimal instances. > > def float_literal(literal): > return decimal.Decimal(literal) > > More simply decimal.Decimal could be registered as the literal > constructor if just the literal's string is used. Alternative > signatures could allow a single function to handle multiple token > types, or could make applying modifications to normal types simpler, > but would prevent the obvious use of decimal.Decimal as above. I see > token_type and value (as normally parsed) as obvious candidates for > this. > > This could be used for people who want to create auto-substituting > string literals (which I dislike for I18N reasons) without having to > shoehorn its own syntax into the language. > > There's a lot of holes to this story, including at least how these > functions are registered, and which additional arguments (if any) are > necessary. Shall we try to fill these holes in? > > Michael > -- > Michael Urman http://www.tortall.net/mu/blog > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From andy at neotitans.com Sat Apr 22 10:19:20 2006 From: andy at neotitans.com (Andy Sy) Date: Sat, 22 Apr 2006 16:19:20 +0800 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <20060420134605.clnsox3h1544kwsk@login.werra.lunarpages.com> References: <20060420134605.clnsox3h1544kwsk@login.werra.lunarpages.com> Message-ID: <4449E708.8040301@neotitans.com> Thanks for the discussions on the hows-and-whys of futures and asynchronous approaches in Python... they have been enlightening. [OT stuff below] Michael Chermside wrote: > It is a FUNDAMENTAL PRECONDITION of Py3K that it will BE the existing > CPython codebase, with a few changes and some cruft removed. We are > NOT going to re-write from the ground up. If you don't know why, go > read "http://www.joelonsoftware.com/articles/fog0000000069.html". I found myself agreeing with that essay when I first read it, but hindsight shows us that it was only half-right. It is very unlikely that Firefox could have become the platform it is today had they not completely re-engineered the architecture. Joel's points stand in that from-scratch rewrites are generally disastrous for *companies* which underestimate the length of time involved (almost a given), but because open-source projects do not need to be financially viable in the short run, they can afford much longer gestation periods (and often with superior results). Disclaimer, the above should not be construed as having anything to do with an opinion of whether Py3K should be written from scratch or not. When I asked about Py3K's intended reuse of the existing implementation, it was out of curiousity. From andy at neotitans.com Sat Apr 22 10:37:43 2006 From: andy at neotitans.com (Andy Sy) Date: Sat, 22 Apr 2006 16:37:43 +0800 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4448BD84.5040307@canterbury.ac.nz> References: <20060420151255.o8m8q8xgrnk080g8@login.werra.lunarpages.com> <4448BD84.5040307@canterbury.ac.nz> Message-ID: <4449EB57.5090104@neotitans.com> Greg Ewing wrote: > BTW, if you *really* want to understand continuations, you > need to carry out the following exercise: Write a Scheme > interpreter in Scheme, doing it in a continuation-passing > style. [1] > > You'll find that it's possible to do this using a subset > of Scheme that doesn't itself have continuations, yet it > will be trivially easy to make it so that your interpreter > implements a Scheme that does. > > If you carry out this exercise successfully, and make it > actually work, at some point along the way you will have > a Zen-like flash of insight, and will suddenly understand > exactly what a continuation is and how it works. > > (The other possibility is that you will still be none the > wiser. If this happens, you will never understand > continuations. Give up programming language implementation > and become a potato farmer. :-) Thanks for the suggestion, Greg. Methinks I'll just go get myself hammered on some vodka... a much easier way to get the same kind of headache... From guido at python.org Sat Apr 22 10:39:02 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 09:39:02 +0100 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: References: <00dc01c66591$ed190120$a8b62997@bagio> <4449AAED.3000604@canterbury.ac.nz> Message-ID: On 4/22/06, Talin wrote: > Greg Ewing canterbury.ac.nz> writes: > > > Giovanni Bajo wrote: > > > > > Another (similar) way would be to let the user pay for the high typechecking > > > price in normal cases *but* provide a list[int] class, which is a list > > > decorated with typechecks on modification operations. We could have > > > list[int|float]() to construct a list which can hold either ints or floats. > > > > This is worth thinking about. Recently when pondering the > > question of when it would or would not be appropriate to > > put in type assertions to help catch bugs, I concluded > > that it makes the most sense to do so when building a > > data structure, but not when just passing things around > > between calls. > > This makes total sense to me. You don't check the type of each > entry of the list - you check the type of the list itself. This is > pretty much the way statically typed languages work. > > This means that you can't pass a regular list to a function > that expects a "list of ints". So its an extra step to convert > the list to the proper type. As long as the syntax isn't too > hard to type, that shouldn't be a problem, and this way the > user knows that they are paying the cost for the conversion. > "Explicit is better than implicit". No. This is the kind of thing that would very quickly turn "optional" typing into *mandatory* typing. One library package adds type declarations. Now all packages that use it are required to either add type declarations or explicit conversions or checks. The cost of repeated explicit conversions/checks strongly suggests that adding type declaration is the better choice -- and now you have a ripple effect (similar to the way 'const' declarations, once added, cascade in C or C++ code). Soon enough the entire library uses type declarations and now all user code is faced with the same choice. All style guides will strongly suggest type declarations. And we end up with Java without curly braces. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Sat Apr 22 10:53:44 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Apr 2006 18:53:44 +1000 Subject: [Python-3000] Changing the import machinery In-Reply-To: <44490A47.3000102@colorstudy.com> References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> <44490A47.3000102@colorstudy.com> Message-ID: <4449EF18.3000708@gmail.com> Ian Bicking wrote: > Maybe you wouldn't have to mix it all together... if sys.path, > sys.meta_path, and sys.modules (and maybe some other stuff I don't know > of) were really instance variables of some ModuleEnvironment object, and > you could instantiate a new ModuleEnvironment that was specific to > templates. Probably to be filed under crazy ideas. Not really all that crazy. . . . Think about it - the import mechanism is a bunch of related functions for finding and loading modules. Currently, those are all stored as module level functions, which forces their shared state to be global for the entire interpeter. What's the conventional solution for dealing with a set of functions which need to share complex state? Now, suppose that there was, in sys, the moral equivalent of: sys.importer = imp.Importer() sys.path, sys.meta_path, sys.path_hooks, sys.path_importer_cache and sys.modules would then all be attributes of the Importer object. If sys was changed to be a class rather than a module, then the top-level attributes could be retained as properties accessing sys.importer. Applications wanting to tightly control sys.path would simply replace sys.importer with a different importer object. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Sat Apr 22 10:58:40 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 09:58:40 +0100 Subject: [Python-3000] PEP - string.format In-Reply-To: <4449DB4C.6050706@gmail.com> References: <4449DB4C.6050706@gmail.com> Message-ID: On 4/22/06, Nick Coghlan wrote: > Removing string %-formatting would be a backwards compatibility nightmare. > I doubt there's a Python program on the planet that would continue working if > it was removed (I know most of mine would break in verbose mode). Even those > which continued to work would likely break if all commented out debugging > messages were uncommented. The same is true for some other changes considered, e.g. the new I/O stack, all-unicode strings, and dict views. Py3k exists to *break* backwards compatibility. A format() method added to strings could be added to 2.6. Defining format() in terms of % would be a long-term disaster IMO. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Sat Apr 22 11:36:34 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 22 Apr 2006 11:36:34 +0200 Subject: [Python-3000] Futures in Python 3000 References: <20060420134605.clnsox3h1544kwsk@login.werra.lunarpages.com> <4449E708.8040301@neotitans.com> Message-ID: Andy Sy wrote: > > It is a FUNDAMENTAL PRECONDITION of Py3K that it will BE the existing > > CPython codebase, with a few changes and some cruft removed. We are > > NOT going to re-write from the ground up. If you don't know why, go > > read "http://www.joelonsoftware.com/articles/fog0000000069.html". > > I found myself agreeing with that essay when I first read it, but > hindsight shows us that it was only half-right. It is very unlikely > that Firefox could have become the platform it is today had they not > completely re-engineered the architecture. > > Joel's points stand in that from-scratch rewrites are generally disastrous > for *companies* which underestimate the length of time involved (almost a > given), but because open-source projects do not need to be financially > viable in the short run, they can afford much longer gestation periods > (and often with superior results). given the amount of money that flows into the Mozilla Foundation from various sources (mostly Google), I don't think you can draw any conclusions from the Firefox project wrt. software engineering. (other than "given a large number of volunteers, and large amounts of money from corporate sponsors, and being politically correct in geek and web design circles, you can grab a small chunk of a huge market" ;-) From mail at manuzhai.nl Sat Apr 22 12:09:40 2006 From: mail at manuzhai.nl (Manuzhai) Date: Sat, 22 Apr 2006 12:09:40 +0200 Subject: [Python-3000] Pronouncement on parameter lists In-Reply-To: References: Message-ID: > I already have a rough draft PEP and implementation for a > __signature__ object from a couple of months ago (sparked my thinking > on __introspects__). I will go ahead and start another email thread > with the PEP so that details can get hashed out on that separately. I was wondering if maybe there could be a dict-like thing for the keyword arguments that also supports __setitem__. I'm not sure whether functions/methods currently count as immutable, but it would be nice if there would be a kind of partial() equivalent to deal with setting default arguments on functions (so you can have one function and spawn several functions off of that have other defaults). OTOH, I can't really come up with use cases (but that might be because I'm not too bright today), so this might be a case of YAGNI, but I think it would be a nice parallel to the partial() functionality. Regards, Manuzhai From birchb at tpg.com.au Sat Apr 22 14:00:42 2006 From: birchb at tpg.com.au (Bill Birch) Date: Sat, 22 Apr 2006 22:00:42 +1000 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: <3AE4D4F5-63D5-40F4-8560-ECAF32B09860@gradient.cis.upenn.edu> References: <4448D1D2.3010306@canterbury.ac.nz> <3AE4D4F5-63D5-40F4-8560-ECAF32B09860@gradient.cis.upenn.edu> Message-ID: <200604222200.43090.birchb@tpg.com.au> > Guido van Rossum wrote: > If I have some utterly dynamic > code that comes up with a list of a million ints, and then I pass > that > as an argument to a function that requests the argument type is > list[int], > > you wrap it in something that checks elements for intness > as you access them. [...] Yes you are right. I see some sub-cases. If the value structure is immutable, then as it is constructed you could calculate the Godel strings bottom-up. Even this effectively doubles the work of the runtime since it has to calculate the value _and_ the type information. All this additional calculation just in case a type check might be needed. So not good. And this is the best case. If the data structure is mutable like list[int] then then attempting to keep the Godel strings up-to-date means re-calculating the entire string every time an element changes, again just in case a type check is needed. Last case, what about recursive types? Chances are an algorithm needs to keep track of nodes visited, further adding to the workload. The "Lingua Franca IDL" distributed system was happy to bear this runtime load because of the large-grained method calls over a network. (http://citeseer.ist.psu.edu/27902.html) However within the same executable the costs will be too high. So yes, this idea doesn't tackle the hard problem of the cost of run-time type checking of dynamic data. Is this intractable? I don't see how it can be lightened up unless 'nominal subtyping' aka 'inheritance subtyping' is used. But that is not Pythonic since we want duck typing or 'structural subtyping'. From birchb at tpg.com.au Sat Apr 22 14:03:53 2006 From: birchb at tpg.com.au (Bill Birch) Date: Sat, 22 Apr 2006 22:03:53 +1000 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: References: Message-ID: <200604222203.53303.birchb@tpg.com.au> On Sat, 22 Apr 2006 03:12 am, Jim Jewett wrote: > On 4/20/06, Birch, Bill wrote: > > Type comparison operators would only need a deep > > inspection of the types when the godel strings don't match. > > If most comparisons will be an exact match (not a subtype) > > the lookup should be faster. > > If you're assuming that, then just checking whether > > obj.__class__ is requirement.allowed_class > > should be pretty hard to beat... OK that will work for atomic types. But if obj.__class__ is it does not tell us what is _in_ the list. We want to know it's list[any] or list[int] ot list[str] etc. From birchb at tpg.com.au Sat Apr 22 14:25:10 2006 From: birchb at tpg.com.au (Bill Birch) Date: Sat, 22 Apr 2006 22:25:10 +1000 Subject: [Python-3000] Limit Type Checks to Interfaces (was Type Comparisons with Godel Numbers) In-Reply-To: References: Message-ID: <200604222225.10418.birchb@tpg.com.au> On Sat, 22 Apr 2006 06:39 pm, Guido van Rossum wrote: > > No. This is the kind of thing that would very quickly turn "optional" > typing into *mandatory* typing. One library package adds type > declarations. Now all packages that use it are required to either add > type declarations or explicit conversions or checks. The cost of > repeated explicit conversions/checks strongly suggests that adding > type declaration is the better choice -- and now you have a ripple > effect (similar to the way 'const' declarations, once added, cascade > in C or C++ code). Soon enough the entire library uses type > declarations and now all user code is faced with the same choice. All > style guides will strongly suggest type declarations. And we end up > with Java without curly braces. recapping: What exactly is list[int]? - ?list[int] is shorthand for ????????type(list[0]) <: int and ?type(list[1]) <: int and type(list[2]) <: int ... where "t1 <: t2" means t1 is a structural subtype of t2 otherwise stated as: ???????? ????????for any natural number n, list[n] <: int which is essentially an un-typed viewpoint. So we ?add the type variable to the runtime object. Luca Cardelli's paper on the typed lambda calculus has a lot to say about adding types (http://research.microsoft.com/Users/luca/Papers/OnUnderstanding.A4.pdf) If I understand the math right, we can expressed this as a bounded universal quantifier? ????????all[a <: int] list[a] You would have to construct it specifically with the type attribute: ????????var = list[int] ? ? ? ? ? ? ? # py3k ????????var[0] = 42 ? ? ? ? ? ? ? ? ?# OK ????????var[1] = "foo" ? ? ? # run-time type error Java arrays work this way. ?quote "More formally: an assignment to an element of an array whose type is A[], where A is a reference type, is checked at run-time to ensure that the value assigned can be assigned to the actual element type of the array, where the actual element type may be any reference type that is assignable to A." see (http://java.sun.com/docs/books/jls/second_edition/html/arrays.doc.html) ? ? ? ? The implications of this for the programmer are that type errors would be found closer to the source. But, as was pointed out there will be a proliferation of listOf(T) constructors. ?And a proliferation of performance soaks. so... In my experience deep type checking at runtime is really useful, common, but rarely called type checking. Normally we to write it manually (and call it "input validation" or "checking preconditions"). Usually on major system boundaries. ?Especially in distributed systems. I would be happy to pay the cost of run-time type checking if I could invoke it explicitly. Then I could choose where the performance hit happens and how often. But how can dynamic type checks be made obvious and explicit syntactically? Maybe with an annotation: @enforce_types def func(x: t1, y: t2) -> t3: ? ? ...body... ? or with an explicit check: def func(x, y) : ????????if not x, y <: t1, t2: ????????????????throw "mate, you have a type error" ???????????????? Perhaps we could limit type definitions to interfaces only, and allow preconditions and postconditions there. This would be simple to grasp. If you implement an interface, you'll be paying the cost of runtime type checks. ? After all the whole point of interfaces in Python is to enforce compliance with a standard. Remember this is not Java, we already have multiple inheritance. From birchb at tpg.com.au Sat Apr 22 13:45:46 2006 From: birchb at tpg.com.au (Bill Birch) Date: Sat, 22 Apr 2006 21:45:46 +1000 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: References: Message-ID: <200604222145.47182.birchb@tpg.com.au> On Sat, 22 Apr 2006 06:39 pm, Guido van Rossum wrote: ... > > No. This is the kind of thing that would very quickly turn "optional" > typing into *mandatory* typing. One library package adds type > declarations. Now all packages that use it are required to either add > type declarations or explicit conversions or checks. The cost of > repeated explicit conversions/checks strongly suggests that adding > type declaration is the better choice -- and now you have a ripple > effect (similar to the way 'const' declarations, once added, cascade > in C or C++ code). Soon enough the entire library uses type > declarations and now all user code is faced with the same choice. All > style guides will strongly suggest type declarations. And we end up > with Java without curly braces. recapping: What exactly is list[int]? - list[int] is shorthand for type(list[0]) <: int and type(list[1]) <: int and type(list[2]) <: int ... where "t1 <: t2" means t1 is a structural subtype of t2 otherwise stated as: for any natural number n, list[n] <: int which is essentially an un-typed viewpoint. So we add the type variable to the runtime object. Luca Cardelli's paper on the typed lambda calculus has a lot to say about adding types (http://research.microsoft.com/Users/luca/Papers/OnUnderstanding.A4.pdf) If I understand the math right, we can expressed this as a bounded universal quantifier? all[a <: int] list[a] You would have to construct it specifically with the type attribute: var = list[int] # py3k var[0] = 42 # OK var[1] = "foo" # run-time type error Java arrays work this way. quote "More formally: an assignment to an element of an array whose type is A[], where A is a reference type, is checked at run-time to ensure that the value assigned can be assigned to the actual element type of the array, where the actual element type may be any reference type that is assignable to A." see (http://java.sun.com/docs/books/jls/second_edition/html/arrays.doc.html) The implications of this for the programmer are that type errors would be found closer to the source. But, as was pointed out there will be a proliferation of listOf(T) constructors. And a proliferation of performance soaks. so... In my experience deep type checking at runtime is really useful, common, but rarely called type checking. Normally we to write it manually (and call it "input validation" or "checking preconditions"). Usually on major system boundaries. Especially in distributed systems. I would be happy to pay the cost of run-time type checking if I could invoke it explicitly. Then I could choose where the performance hit happens and how often. But how can dynamic type checks be made obvious and explicit syntactically? Maybe with an annotation: @enforce_types def func(x: t1, y: t2) -> t3: ...body... or with an explicit check: def func(x, y) : if not x, y <: t1, t2: throw "mate, you have a type error" Perhaps we could limit type definitions to interfaces only, and allow preconditions and postconditions there. This would be simple to grasp. If you implement an interface, you'll be paying the cost of runtime type checks. After all the whole point of interfaces in Python is to enforce compliance with a standard. Remember this is not Java, we already have multiple inheritance. From free.condiments at gmail.com Sat Apr 22 14:44:30 2006 From: free.condiments at gmail.com (Sam Pointon) Date: Sat, 22 Apr 2006 13:44:30 +0100 Subject: [Python-3000] Type Expressions In-Reply-To: <20060421212834.C56623@familjen.svensson.org> References: <4447523F.4030504@canterbury.ac.nz> <20060421212834.C56623@familjen.svensson.org> Message-ID: On 22/04/06, Paul Svensson wrote: > At the risk of hypergeneralization... If *arg means "put the rest of the > positional arguments in arg", then if we simply allow arg to be a tuple... > > def foo(*(a, b)): ... # disallow keyword args. > def foo(a, b, *(), x, y): ... # disallow positional args after b. > def foo(*(a, b), x, y): ... # a, b _must_ be positional; x, y keyword. > > That () = () is currently a syntax error might be considered a reason to > use the lone star spelling instead of *(), but I find the tuple more obvious. This is close (ish) to the "def [positional-only arguments](mixed arguments){keyword-only arguments}" syntax I floated earlier, which Guido then promptly rejected. It looks like we won't get a way to force an argument to be given positionally, as he's also already voiced a distaste for nested bracketing in argument lists. --Sam From ncoghlan at gmail.com Sat Apr 22 15:27:22 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Apr 2006 23:27:22 +1000 Subject: [Python-3000] PEP - string.format In-Reply-To: References: <4449DB4C.6050706@gmail.com> Message-ID: <444A2F3A.10603@gmail.com> Guido van Rossum wrote: > On 4/22/06, Nick Coghlan wrote: >> Removing string %-formatting would be a backwards compatibility nightmare. >> I doubt there's a Python program on the planet that would continue working if >> it was removed (I know most of mine would break in verbose mode). Even those >> which continued to work would likely break if all commented out debugging >> messages were uncommented. > > The same is true for some other changes considered, e.g. the new I/O > stack, all-unicode strings, and dict views. > > Py3k exists to *break* backwards compatibility. A format() method > added to strings could be added to 2.6. Defining format() in terms of > % would be a long-term disaster IMO. That (adding .format() in 2.6) occurred to me, but I dismissed it for some reason. It can't have been a very good reason though, since I sure can't remember what it was. In which case, Talin's PEP should probably suggest this as the implementation strategy - python3warn and the instrumented python build can do their best to pick up usage of % formatting, and we can add .format() to 2.6 so that forward compatible code can be written in later 2.x releases. Now that I think about it, simply having python3warn pick up all uses of % in a print statement should pick up many of the uses of string formatting. (Not all, obviously, but a lot of them) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sat Apr 22 15:33:36 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Apr 2006 23:33:36 +1000 Subject: [Python-3000] New built-in function: bin() In-Reply-To: <44490B02.9080704@colorstudy.com> References: <4448A3D6.5090602@credence.com> <44490B02.9080704@colorstudy.com> Message-ID: <444A30B0.3010004@gmail.com> Ian Bicking wrote: > Guido van Rossum wrote: >> This has been brought up many times before. The value of bin() is >> really rather minimal except when you're just learning about binary >> numbers; and then writing it yourself is a useful exercise. >> >> I'm not saying that bin() is useless -- but IMO its (small) value >> doesn't warrant making, maintaining and documenting a new built-in >> function. > > And for some reason no one wants to propose it for any particular stdlib > module... binascii.a2b_bin and binascii.b2a_bin might actually make sense. . . Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sat Apr 22 15:54:12 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 22 Apr 2006 23:54:12 +1000 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: Message-ID: <444A3584.3080102@gmail.com> Michael Urman wrote: > There's a lot of holes to this story, including at least how these > functions are registered, and which additional arguments (if any) are > necessary. Shall we try to fill these holes in? Answering without checking the source (which is undoubtedly a bad idea), but this sounds like something else that could be addressed if it was possible to either register an alternate AST compiler for a scope, or else get hold of an AST and recompile it. (The former works for modules and functions, the latter works only for functions) Even if it was only a matter of some additional keyword arguments to compile and/or exec, it could at least be of benefit for plugin code or an interactive interpreter loop. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Sat Apr 22 16:02:59 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 15:02:59 +0100 Subject: [Python-3000] PEP - string.format In-Reply-To: <444A2F3A.10603@gmail.com> References: <4449DB4C.6050706@gmail.com> <444A2F3A.10603@gmail.com> Message-ID: On 4/22/06, Nick Coghlan wrote: > That (adding .format() in 2.6) occurred to me, but I dismissed it for some > reason. It can't have been a very good reason though, since I sure can't > remember what it was. > > In which case, Talin's PEP should probably suggest this as the implementation > strategy - python3warn and the instrumented python build can do their best to > pick up usage of % formatting, and we can add .format() to 2.6 so that forward > compatible code can be written in later 2.x releases. > > Now that I think about it, simply having python3warn pick up all uses of % in > a print statement should pick up many of the uses of string formatting. (Not > all, obviously, but a lot of them) There's a different approach to discovering future incompatibilities than parsing the source code. I had originally forgotten about this approach even though I'd used the very thing myself years ago! It was pointed out to my by an audience member during my preview of my Python 3000 talk at the ACCU Silicon Valley chapter. I added a brief description to PEP 3000. The approach is to run your app (or a high-coverage test suite) with an instrumented Python 2.x interpreter which spits out warnings (or some other log) for run-time events that are known to break in Python 3.0. I did this years ago to find all instances of int/int in the standard library. It is trivial to do this for things like discovering string formatting. I believe it can also be used to discover cases where the result of dict.keys() etc. is used beyond iteration (we modify keys() to return a subclass of list that spits out a warning for all forms of access except __iter__()). This approach is complementary to the source code inspection approach; we should plan to use both in conjunction to get the maximally successful conversion (either automatic translation or at least automatic indication of issues to be changed). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 22 16:04:36 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 15:04:36 +0100 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: <444A3584.3080102@gmail.com> References: <444A3584.3080102@gmail.com> Message-ID: On 4/22/06, Nick Coghlan wrote: > Michael Urman wrote: > > There's a lot of holes to this story, including at least how these > > functions are registered, and which additional arguments (if any) are > > necessary. Shall we try to fill these holes in? > > Answering without checking the source (which is undoubtedly a bad idea), but > this sounds like something else that could be addressed if it was possible to > either register an alternate AST compiler for a scope, or else get hold of an > AST and recompile it. (The former works for modules and functions, the latter > works only for functions) > > Even if it was only a matter of some additional keyword arguments to compile > and/or exec, it could at least be of benefit for plugin code or an interactive > interpreter loop. Hm... Using the AST seems overkill for this (unless you want to do it without modifying the Python interpreter in any way). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 22 16:16:24 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 15:16:24 +0100 Subject: [Python-3000] New built-in function: bin() In-Reply-To: <444A30B0.3010004@gmail.com> References: <4448A3D6.5090602@credence.com> <44490B02.9080704@colorstudy.com> <444A30B0.3010004@gmail.com> Message-ID: That makes more sense than a builtin. Note that oct() and hex() return something that's a valid Python literal. There are no binary literals (nor should there be IMO). On 4/22/06, Nick Coghlan wrote: > Ian Bicking wrote: > > Guido van Rossum wrote: > >> This has been brought up many times before. The value of bin() is > >> really rather minimal except when you're just learning about binary > >> numbers; and then writing it yourself is a useful exercise. > >> > >> I'm not saying that bin() is useless -- but IMO its (small) value > >> doesn't warrant making, maintaining and documenting a new built-in > >> function. > > > > And for some reason no one wants to propose it for any particular stdlib > > module... > > binascii.a2b_bin and binascii.b2a_bin might actually make sense. . . > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > --------------------------------------------------------------- > http://www.boredomandlaziness.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at gmail.com Sat Apr 22 16:28:06 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Apr 2006 00:28:06 +1000 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: <444A3584.3080102@gmail.com> Message-ID: <444A3D76.2030603@gmail.com> Guido van Rossum wrote: > On 4/22/06, Nick Coghlan wrote: >> Michael Urman wrote: >>> There's a lot of holes to this story, including at least how these >>> functions are registered, and which additional arguments (if any) are >>> necessary. Shall we try to fill these holes in? >> Answering without checking the source (which is undoubtedly a bad idea), but >> this sounds like something else that could be addressed if it was possible to >> either register an alternate AST compiler for a scope, or else get hold of an >> AST and recompile it. (The former works for modules and functions, the latter >> works only for functions) >> >> Even if it was only a matter of some additional keyword arguments to compile >> and/or exec, it could at least be of benefit for plugin code or an interactive >> interpreter loop. > > Hm... Using the AST seems overkill for this (unless you want to do it > without modifying the Python interpreter in any way). Strawman. . . from ast import Compiler from decimal import Decimal class MyCompiler(Compiler): def build_float(self, literal): return Decimal(literal) # Pass this to compile or exec via an ast_compiler argument # It may even be possible to permit it as an argument to __import__ The basic idea would be to have a compiler object that delegated compilation operations to the internal AST compiler, but gave subclasses the ability to easily override certain aspects of the process - like processing different kinds of literal. No matter what, permitting these kinds of hooks is going to require alterations to the process of compiling the AST to the bytecode - and subclassing is an excellent way of allowing some aspects of an implementation to be overridden while leaving other aspects alone. This is just an idea, as I doubt I'd ever use the ability no matter how it was implemented :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From guido at python.org Sat Apr 22 16:38:12 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 15:38:12 +0100 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: <444A3D76.2030603@gmail.com> References: <444A3584.3080102@gmail.com> <444A3D76.2030603@gmail.com> Message-ID: On 4/22/06, Nick Coghlan wrote: > No matter what, permitting these kinds of hooks is going to require > alterations to the process of compiling the AST to the bytecode - and > subclassing is an excellent way of allowing some aspects of an implementation > to be overridden while leaving other aspects alone. Oops, I have to concur. I was thinking that it could all be done later, at run-time. That makes sense for things like [...] and {...} but doesn't really work for floating point or even string literals -- you can't afford the overhead of calling a default hook for each literal at run-time, and passing the default value to the hook doesn't work in the case of floats or strings. Unfortunately a compile-time hook is much more painful to add to the language because it means adding new syntax just to specify the hooks. :-( -- --Guido van Rossum (home page: http://www.python.org/~guido/) From fredrik at pythonware.com Sat Apr 22 16:53:21 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 22 Apr 2006 16:53:21 +0200 Subject: [Python-3000] Brainstorming: literal construction hooks References: Message-ID: Guido van Rossum wrote: > It's an interesting idea; it's been brought up before but nobody AFAIK > has ever implemented it. I like the scoping requirement. can we add support for XML literals while we're at it? http://effbot.org/zone/idea-xml-literal.htm From mail at manuzhai.nl Sat Apr 22 17:15:45 2006 From: mail at manuzhai.nl (Manuzhai) Date: Sat, 22 Apr 2006 17:15:45 +0200 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: Message-ID: > can we add support for XML literals while we're at it? > > http://effbot.org/zone/idea-xml-literal.htm The reason the literals aren't returned as ElementTree Elements is because ElementTree didn't exist back then, right? Because that particular integration would make a lot of sense. Regards, Manuzhai From guido at python.org Sat Apr 22 17:17:59 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 16:17:59 +0100 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: Message-ID: On 4/22/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > It's an interesting idea; it's been brought up before but nobody AFAIK > > has ever implemented it. I like the scoping requirement. > > can we add support for XML literals while we're at it? > > http://effbot.org/zone/idea-xml-literal.htm Javascript has an ECMA standardized extension that supports this (ECMA-357 or E4X). Groovy has a different approach that doesn't blend the two syntaxes, but rather gives you more native syntax for constructing DOM trees (or event streams; it wasn't clear from the description I saw today). That makes perhaps more sense; it avoids the lexical ambiguities and a parentheses-based syntax is easier to type than XML. Maybe this example (which I am making up) suffices: frag1 = element["some content"] frag2 = element["some content", child(attribute='value')["spam & egg"]] -- --Guido van Rossum (home page: http://www.python.org/~guido/) From murman at gmail.com Sat Apr 22 17:34:54 2006 From: murman at gmail.com (Michael Urman) Date: Sat, 22 Apr 2006 10:34:54 -0500 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: Message-ID: On 4/22/06, Guido van Rossum wrote: > It's an interesting idea; it's been brought up before but nobody AFAIK > has ever implemented it. I like the scoping requirement. I suggest you > try to implement it and see how well it works. I'll look into that now that I know it's not a discard idea. > You probably also want to be able to hook list, tuple and dict > displays (perhaps by supplying an alternative factory function). Maybe I've not been awake long enough, but I don't see what I'd want to hook here that isn't already done using a subclass with an overrided __str__ or __repr__ method. On 4/22/06, Nick Coghlan wrote: > class MyCompiler(Compiler): > def build_float(self, literal): > return Decimal(literal) > > # Pass this to compile or exec via an ast_compiler argument > # It may even be possible to permit it as an argument to __import__ I definitely like the look of this for the complie/exec usage, but it definitely makes for a more annoying general usage. I'm not sure about the __import__ way, as it smells of spooky effects, but might be a good practical back door to bring a submodule in line. For in-module effect, maybe some special import could pull in a compiler class instance used only for the current module, allowing its build_xyz methods be replaced. from __ast__ import compiler compiler.build_float = Decimal # on an instance, so no 'self' parameter If something like that sounds reasonable for usage, I'll start looking into how it looks for implementation. (Probably a big ball of mud! And sounds like it depends on the python-side interface to the ast - I've lost track if that's solidified yet.) :) Michael -- Michael Urman http://www.tortall.net/mu/blog From solipsis at pitrou.net Sat Apr 22 17:42:11 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 22 Apr 2006 17:42:11 +0200 Subject: [Python-3000] XML literals In-Reply-To: References: Message-ID: <1145720531.11536.14.camel@fsol> Le samedi 22 avril 2006 ? 16:17 +0100, Guido van Rossum a ?crit : > Groovy has a different approach that doesn't blend the two syntaxes, > but rather gives you more native syntax for constructing DOM trees (or > event streams; it wasn't clear from the description I saw today). That > makes perhaps more sense; it avoids the lexical ambiguities and a > parentheses-based syntax is easier to type than XML. Maybe this > example (which I am making up) suffices: > > frag1 = element["some content"] > > frag2 = element["some content", > child(attribute='value')["spam & egg"]] You might want to take a look at CDuce. It is a functional language dedicated to transformation of XML documents. It has a powerful typing system (including structural pattern matching), and also features a clever syntax for representing XML fragments in-code. Here is a simplified example from the tutorial : let parents : ParentBook = [ [ "Clara" [ [ "P?l Andr?" [] ] ] "clara at lri.fr" "314-1592654" ] [ "Bob" "271828" "66260" ] ] http://www.cduce.org/tutorial.html Regards Antoine. From jcarlson at uci.edu Sat Apr 22 19:07:13 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 22 Apr 2006 10:07:13 -0700 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: Message-ID: <20060422095402.F1E7.JCARLSON@uci.edu> "Guido van Rossum" wrote: > > On 4/22/06, Fredrik Lundh wrote: > > Guido van Rossum wrote: > > > > > It's an interesting idea; it's been brought up before but nobody AFAIK > > > has ever implemented it. I like the scoping requirement. > > > > can we add support for XML literals while we're at it? > > > > http://effbot.org/zone/idea-xml-literal.htm > > Javascript has an ECMA standardized extension that supports this > (ECMA-357 or E4X). > > Groovy has a different approach that doesn't blend the two syntaxes, > but rather gives you more native syntax for constructing DOM trees (or > event streams; it wasn't clear from the description I saw today). That > makes perhaps more sense; it avoids the lexical ambiguities and a > parentheses-based syntax is easier to type than XML. Maybe this > example (which I am making up) suffices: > > frag1 = element["some content"] > > frag2 = element["some content", > child(attribute='value')["spam & egg"]] Nevow.stan has a very similar method for generating *ML with (attribute='value')[content], but I've personally found that using only function-call semantics to create children and attributes for *ML in Python is much more convenient; allowing one to use any of the following and get a reasonable method for *ML object re-use: (attribute='value')(content) (content)(attribute='value') (content, attribute='value') A working system for handling this kind of *ML generation is available in the ASPN Python Cookbook here (the non-closing tags portion may not be correct for any *ML, even HTML): http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440563 - Josiah From guido at python.org Sat Apr 22 19:30:15 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 18:30:15 +0100 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: <20060422095402.F1E7.JCARLSON@uci.edu> References: <20060422095402.F1E7.JCARLSON@uci.edu> Message-ID: On 4/22/06, Josiah Carlson wrote: > Nevow.stan has a very similar method for generating *ML with > (attribute='value')[content], but I've personally found that using only > function-call semantics to create children and attributes for *ML in > Python is much more convenient; allowing one to use any of the following > and get a reasonable method for *ML object re-use: > (attribute='value')(content) > (content)(attribute='value') > (content, attribute='value') That makes sense, so forget the crazy idea of overloading __getitem__. The problem with any of these is that you either have to import (or define, etc.) an object for each element that you want to name, so you can write foo(x=1) to mean (this is what my own parsexml module does, see SF 1337648); or you have to use a helper function or object, e.g. T.foo(x=1) as in the recipe below. > A working system for handling this kind of *ML generation is available > in the ASPN Python Cookbook here (the non-closing tags portion may not > be correct for any *ML, even HTML): > http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440563 Cool. Unfortunately, all of these schemes break down if you have non-identifier characters in attribute or element names, or if you want to use namespaces, or if you want to insert entity references, XML comments, or other funky stuff like DTDs or processing elements. (I'm sure all of these can be hacked in one way or another but by the time you've covered all that ground it won't be as pretty, and the relative advantage over blending XML syntax is diminished.) Also, an important aspect of JavaScript's E4X standard is a standard object model. I believe they use a standard DOM style API. In Python unfortunately we have many different object models for XML -- minidom, ElementTree, and I believe the XML-sig has its own standard (4Suite?). I could handle multiple implementations (quack!) but we'd have to pick an API -- AFAIK ElementTree and the DOM API are incompatible. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From talin at acm.org Sat Apr 22 20:33:32 2006 From: talin at acm.org (Talin) Date: Sat, 22 Apr 2006 18:33:32 +0000 (UTC) Subject: [Python-3000] Open Issues for string.format PEP Message-ID: Here's a condensed list of the open issues that have been raised by people so far: (Well, edited somewhat - I'm including 'major' issues, not minor nits -- we can discuss those once the major issues are settled.) 1) Whether to drop the optional parts of the proposal: -- support for field expressions (was: locals() support) -- support for direct access to current scope (was: locals() support) 2) Should this PEP be targeted at Python 2.6 or 3.0? 3) Role and usage of custom formatters: "string".fformat( formatter, ... ) vs. MyFormat( "string" ).format( ... ) (note: I want to make it clear that the purpose of custom formatters is to override the formatting on a *per-field basis*, but at the same time the custom formatter should have access to the surrounding context.) 4) Should there be a way to pass in a dict argument without flattening it via **args? 5) Should the implementation attempt to detect unused arguments? I'd like to get some sense of the answers before the next revision of the PEP. -- Talin From fredrik at pythonware.com Sat Apr 22 20:49:54 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sat, 22 Apr 2006 20:49:54 +0200 Subject: [Python-3000] Brainstorming: literal construction hooks References: <20060422095402.F1E7.JCARLSON@uci.edu> Message-ID: Guido van Rossum wrote: > Unfortunately, all of these schemes break down if you have > non-identifier characters in attribute or element names, or if you > want to use namespaces, or if you want to insert entity references, > XML comments, or other funky stuff like DTDs or processing elements. a useful design should support non-identifier characters, name- spaces, and comments/processing instructions, but should other- wise work on the infoset level (i.e. CDATA sections and entities doesn't have to be part of this). not sure about DTD:s; they're a whole can of works... (but I'm sure E4X etc can provide some ideas here). > Also, an important aspect of JavaScript's E4X standard is a standard > object model. I believe they use a standard DOM style API. In Python > unfortunately we have many different object models for XML -- minidom, > ElementTree, and I believe the XML-sig has its own standard (4Suite?). > I could handle multiple implementations (quack!) but we'd have to pick > an API -- AFAIK ElementTree and the DOM API are incompatible. why do you think I posted this to the "literal construction hooks" thread ? ;-) From guido at python.org Sat Apr 22 21:12:09 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 22 Apr 2006 20:12:09 +0100 Subject: [Python-3000] Open Issues for string.format PEP In-Reply-To: References: Message-ID: On 4/22/06, Talin wrote: > 1) Whether to drop the optional parts of the proposal: > -- support for field expressions (was: locals() support) > -- support for direct access to current scope (was: locals() support) I'd skip this. KISS etc. > 2) Should this PEP be targeted at Python 2.6 or 3.0? Both equally. We could implement it in the 3.0 branch first. > 3) Role and usage of custom formatters: > > "string".fformat( formatter, ... ) > > vs. > > MyFormat( "string" ).format( ... ) > > (note: I want to make it clear that the purpose of custom formatters is to > override the formatting on a *per-field basis*, but at the same time the > custom formatter should have access to the surrounding context.) I think a single format() method with a standard signature makes more sense. That way you can abstract the choice of formatter out. > 4) Should there be a way to pass in a dict argument without flattening it via > **args? Questionable. TOOWTDI; OTOH the flattening could be expensive. If we can figure out a way to optimize away the flattening as long as the callee is implemented in C, I'm in favor of dropping **kwds. > 5) Should the implementation attempt to detect unused arguments? I think so; the % operator is very strict about this and it's useful in debugging. > I'd like to get some sense of the answers before the next revision of the PEP. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From talin at acm.org Sat Apr 22 21:36:15 2006 From: talin at acm.org (Talin) Date: Sat, 22 Apr 2006 19:36:15 +0000 (UTC) Subject: [Python-3000] Minor hitch writing the Function Signature PEP Message-ID: I realized that the "Keyword-Only Arguments" and the "Function Signature" proposals could be written as two separate PEPs, since there is only minimal interaction between the two. However, I started writing some use case examples for the Function Signature PEP, and ran into a slight snag. My use case was to add preconditions to a function, where the precondition refers to specific arguments by name: @precondition( y=NotNegative ) def power( x, y ): ... ...where 'NotNegative' is a function which simply does an assert( value >= 0 ). The general idea is that 'precondition' would take a **kwds argument, and for each key, it would look up the correspondingly named argument in the function signature. It would then create a wrapper function which would validate each argument against its corresponding precondition before calling the real function. Sounds simple enough, right? Well the problem is that decorator functions don't have access to the machinery that binds input arguments for formal parameters. So the wrapper function has a hard time knowing which input arguments will be bound to which formal params, without having to duplicate the runtime's algorithm to do this. In order to be a generic wrapper, the wrapper will have the form: def wrapper( *args, **kwargs ): ... So where is 'y' in all of that? It could be the second argument in *args; it could be the first argument in *args if x is passed in as a keyword argument; or it could be in the **kwargs dict. I'm not saying that solving this is impossible; what I am saying is that its: -- slow -- makes the example very complicated and hard to understand. And, as we all know, if its hard to understand then its probably wrong. -- Talin From p.f.moore at gmail.com Sat Apr 22 21:40:46 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 22 Apr 2006 20:40:46 +0100 Subject: [Python-3000] Minor hitch writing the Function Signature PEP In-Reply-To: References: Message-ID: <79990c6b0604221240j75f8d2c8x596fe20bec928b6f@mail.gmail.com> On 4/22/06, Talin wrote: > Well the problem is that decorator functions don't have access to the machinery > that binds input arguments for formal parameters. So the wrapper function has a > hard time knowing which input arguments will be bound to which formal params, > without having to duplicate the runtime's algorithm to do this. > > In order to be a generic wrapper, the wrapper will have the form: > > def wrapper( *args, **kwargs ): > ... > > So where is 'y' in all of that? It could be the second argument in *args; it > could be the first argument in *args if x is passed in as a keyword argument; or > it could be in the **kwargs dict. [Wild idea alert!] Maybe the signature object could have a "bind" method sig.bind(args, kwargs) which returns a dictionary mapping argument names to values? In effect, this exposes the internal mechanism for reuse. Paul. From brett at python.org Sat Apr 22 23:07:01 2006 From: brett at python.org (Brett Cannon) Date: Sat, 22 Apr 2006 14:07:01 -0700 Subject: [Python-3000] Minor hitch writing the Function Signature PEP In-Reply-To: <79990c6b0604221240j75f8d2c8x596fe20bec928b6f@mail.gmail.com> References: <79990c6b0604221240j75f8d2c8x596fe20bec928b6f@mail.gmail.com> Message-ID: On 4/22/06, Paul Moore wrote: > On 4/22/06, Talin wrote: > > Well the problem is that decorator functions don't have access to the machinery > > that binds input arguments for formal parameters. So the wrapper function has a > > hard time knowing which input arguments will be bound to which formal params, > > without having to duplicate the runtime's algorithm to do this. > > > > In order to be a generic wrapper, the wrapper will have the form: > > > > def wrapper( *args, **kwargs ): > > ... > > > > So where is 'y' in all of that? It could be the second argument in *args; it > > could be the first argument in *args if x is passed in as a keyword argument; or > > it could be in the **kwargs dict. > > [Wild idea alert!] > > Maybe the signature object could have a "bind" method > > sig.bind(args, kwargs) > > which returns a dictionary mapping argument names to values? In > effect, this exposes the internal mechanism for reuse. I don't see why the signature object has to be created for every function automatically; I don't see it being a constantly needed thing. If something like a signature built-in was created it could be called on objects as needed to create the signature object as well as bind it to __signature__ for future use to avoid duplicated work. This would allow a decorator to assign to the decorator the signature object from the wrapped function:: def decorator(func): def wrapper(*args, **kwargs): # ... stuff wrapper.__signature__ = signature(func) return wrapper Or something along those lines. I am on vacation right now but I am still planning on digging up my signature PEP which covers all of this. -Brett From ianb at colorstudy.com Sat Apr 22 23:18:23 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sat, 22 Apr 2006 16:18:23 -0500 Subject: [Python-3000] PEP - string.format In-Reply-To: References: <444923A3.2050205@colorstudy.com> Message-ID: <444A9D9F.5010409@colorstudy.com> Talin wrote: >> Thus you can't nest formatters, e.g., {0:pad(23):xmlquote}, unless the >> underlying object understands that. Which is probably unlikely. > > At this point, I'm thinking not, although I could be convinced otherwise. > Remember, that you can accomplish all of the same things by processing the input > arguments; The conversion specifiers are a convenience. > > Also, in your model, there would be a distinction between the first specifier > (which converts the object to a string), and subsequent ones (which modify the > string). My complexity senses are tingling... I would assume that any formatting can produce any object, and only at the end will the object (if necessary) be converted to a string with str(). >> 3) Otherwise, check the internal formatter within >> string.format that contains knowledge of certain builtin >> types. >> >> If it is a language change, could all those types have __format__ >> methods added? Is there any way for the object to accept or decline to >> do formatting? > > Good question. I suspect that it may be impractical to add __format__ to all > built-in types, so we should plan to allow a fallback to an internal formatter. Yeah, on further thought while this would be possible for py3k, some form of this could be usefully done as a module before that. >> 4) Otherwise, call str() or unicode() as appropriate. >> >> Is there a global repr() formatter, like %r? Potentially {0:repr} could >> be implemented the same way by convention, including in object.__format__? > > Good idea. (Should there be a *global* custom formatter? Plugins? Subject of a > separate PEP I think.) I don't think there should be any modifiable global formatter. As an implementation detail there may be one, but one module shouldn't be able to change the way formatting works for everyone. But repr() is something of a special case, because it's so widely applicable. >> The formatter should examine the type of the object and the >> specifier string, and decide whether or not it wants to handle >> this field. If it decides not to, then it should return False >> to indicate that the default formatting for that field should be >> used; Otherwise, it should call builder.append() (or whatever >> is the appropriate method) to concatenate the converted value >> to the end of the string, and return True. >> >> Well, I guess this is the use case, but it feels a bit funny to me. A >> concrete use case would be appreciated. > > The main use case was that the formatter might need to examine the part of the > string that's already been built. For example, it can't handle expansion of tabs > unless it knows the current column index. I had originally planned to pass only > the column index, but that seemed too special-case to me. Hmm... so the tab-aligning formatter would look for tab alignment formatting specifications? An actual implementation of a tab aligning custom formatter would probably make this easier to think about. >> A fairly high degree of convenience for relatively small risk can >> be obtained by supporting the getattr (.) and getitem ([]) >> operators. While it is certainly possible that these operators >> can be overloaded in a way that a maliciously written string could >> exploit their behavior in nasty ways, it is fairly rare that those >> operators do anything more than retargeting to another container. >> On other other hand, the ability of a string to execute function >> calls would be quite dangerous by comparison. >> >> It could be a keyword option to enable this. Though all the keywords >> are kind of taken. This itself wouldn't be an issue if ** wasn't going >> to be used so often. > > The keywords are all taken - but there are still plenty of method names > available :) That's why "fformat" has a different method name, so that we can > distinguish the custom formatter parameter from the rest of the params. > > Unfortunately, this can't be used too much, or you get a combinatorial explosion > of method names: > > string.format > string.fformat > string.format_dict > string.fformat_dict > ... Yeah... that's not so pretty ;). Things like .update() and even % handle both cases without too much ambiguity. >> One other thing that could be done to make the debugging case >> more convenient would be to allow the locals() dict to be omitted >> entirely. Thus, a format function with no arguments would instead >> use the current scope as a dictionary argument: >> >> print "Error in file {p.file}, line {p.line}".format() >> >> An alternative would be to dedicate a special method name, other >> than 'format' - say, 'interpolate' or 'lformat' - for this >> behavior. >> >> It breaks some conventions to have a method that looks into the parent >> frame; but the use cases are very strong for this. Also, if attribute >> access was a keyword argument potentially that could be turned on by >> default when using the form that pulled from locals(). > > To be honest, I'd be willing to drop this whole part of the proposal if that's > what the folks here would like. I like to present all options, but that doesn't > mean that I myself am in favor of all of them. > > I realize that there are some use cases for it, but I don't know if the use > cases are significantly better. It's not just *some* use cases -- a substantial fraction of all string interpolation falls into this use case. Really it means that it makes sense to make a sample implementation and try to rewrite some stdlib modules, with and without special treatment for locals(). -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From brett at python.org Sat Apr 22 23:30:31 2006 From: brett at python.org (Brett Cannon) Date: Sat, 22 Apr 2006 14:30:31 -0700 Subject: [Python-3000] Changing the import machinery In-Reply-To: References: <5.1.1.6.0.20060420150508.01e5a300@mail.telecommunity.com> Message-ID: On 4/22/06, Fredrik Lundh wrote: > Guido van Rossum wrote: > > > > I'm afraid I disagree. PEP 302 actually has some tremendous advantages > > > over a pure objects-on-sys.path approach: > > > > > > * Strings can be put in any configuration file, and used in .pth files > > > > > > * Strings can be put in environment variables (like PYTHONPATH). > > > > > > * Strings can be printed out, with all their contents showing and nothing > > > hidden > > > > > > In short, strings are better for humans. > > > > I think I like this. I wonder if there's a parallel with my preference > > for strings as paths instead of path objects... > > And strings as exceptions, and Tcl instead of Python ? ;-) > > Sorry, but I don't buy this argument at all. Of course you need a > way to map from external path descriptions (PYTHONPATH, registry > entries, etc) to sys.path contents, but ruling that the things you're > manipulating *inside* a Python program must be strings so you "can > print them out with all their contents showing and nothing hidden" > doesn't strike me as very Pythonic. > > The target audience for this is Python programmers, after all, and > Python programmers know how to inspect Python objects -- as long as > they can find them, which isn't the case with today's extended import > design, which *hides* lots of stuff in *separate* semi-secret > registries. If you put all this back on the path, it'll be a lot > easier to find and manipulate. > > I could quote the "If the implementation is hard to explain, it's a > bad idea." zen here, but I'll quote Sean McGrath's 20th python zen > instead: > > "Things should be as complex as necessary but not more complex." > > and offer a "let's get back to the basics and add stuff, instead of > assuming that the status quo is complex and complicated because it has > to be" solution. Here's an outline, off the top of my head: > > 1. sys.path can contain strings or import handlers > > 2. Strings work as today; as paths that a builtin import handler > uses to look for packages or modules (flyweight-style). > > 3. Import handlers are duck-typed objects that implement a > simplified version of the PEP 302 protocol. Handlers map dotted > module paths to resources, where a resource can be a Python > module, a Python package (a module container), or some other > resource. Handlers are responsible for creating and populating > module objects; whatever they return is stored in sys.modules and > bound to the import target. > > I'm 50/50 on making the import machinery fully type agnostic; that > is, allowing the import handler to return *any* kind of object > also for ordinary imports. Importing e.g. PIL images and pre- > parsed XML resources and Cheetah templates makes perfect sense > to me. > I say go agnostic. No need to force a specific return type if the import handler knows what it is doing. We are all consenting adults, after all. > 4. A support library provides the following mechanisms: > > - An import handler for builtin/frozen objects (with > corresponding hooks on the C API site, so that apps can > register things to be treated as builtins). > > - An import handler for the standard library (*all of > it*, minus site-packages!) > > - An import handler for directory names (used for string path > items) > > - A registry for path specifier syntaxes > > - A parser for external path descriptions, which uses the > registry to map from path components to import handlers > > - (possibly) Some helpers for user-provided import handlers > (to be factored out from such handlers, rather than be > designed up front) > All sounds good. Basically anything needed for getting a handler for string and zip files probably could be factored out and put here. > 5. Remove PTH support, and possibly also a lot of the site- > packages related stuff in site.py. I'm 50/50 on *requiring* code > to specify what non-core libraries they want to use, before they > can import them. I'm also 50/50 on making some additions to the > import statement syntax, to make some operations easier (import > ... using handler), but I'll leave that for another post. > I don't know about the requirement stuff, but I would not mind seeing .pth files go (but that is partially because I rewrote that code when I partially cleaned up site.py at PyCon years ago and I am still bitter =) . As for specifying external dependencies, if we added a global namespace that wouldn't be an issue since it could be detected by the import machinery automatically. And having something like ``from foo.bar import baz using fancy_import_handler`` seems reasonable, although not necessary. > This would cleanly address every deployment scenario and custom > importer that I've used with a (near) mininum of core support, and > frankly, I fail to see any potential use case that cannot be handled > by this mechanism (simply because a handler can do *anything* behind > the scenes, without requiring more support from the core import > machinery). And it'll let us remove tons of "hard to explain" code > from the core. > Removing complicated stuff is a good thing. =) All sounds good to me, Fredrik! -Brett > > > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/brett%40python.org > From brett at python.org Sat Apr 22 23:47:58 2006 From: brett at python.org (Brett Cannon) Date: Sat, 22 Apr 2006 14:47:58 -0700 Subject: [Python-3000] rough draft signature PEP Message-ID: [I am posting to python-3000 since this is where the parameter list ideas are being discussed, but this is probably generic enough to eventually make it into the 2.x line] Here is a rough draft of a PEP I wrote last summer after I had Guido come for lunch at Google when I was interning there (i.e., a little old =) . Someone (I think Philip) had suggested something like this back then but it didn't go any farther. I liked the idea and I asked Guido at lunch if he was okay for it; he was. So I wrote the following PEP along with a Python implementation (attached, along with test cases). It is still a rough draft since I am on vacation on top of visiting my dad for his birthday and thus cannot put a ton of time into this at the moment, so don't consider this a final version in any way. There is already a list of things to consider and I am sure there are some things discussed during the new parameter list ideas that could stand to be worked in. I am quite happy to work on finishing this PEP so that Talin can focus on the parameter list PEP. So just comment away and I will work on incorporating them as time permits. -Brett ------------------------------------------------------------------ PEP: XXX Title: Introducing the __signature__ Attribute Version: $Revision: 1.5 $ Last-Modified: $Date: 2005/06/07 13:17:37 $ Author: Brett Cannon Status: Draft Type: Standards Track Content-Type: text/x-rst Created: XX-XXX-XXXX Post-History: XXX * Break abstract into Abstract and Rationale * write signature() function - maybe not in built-ins; inspect instead? * look up object identity (issue|crisis) as reference * automatically apply signature when decorator called? * make more general by having an attribute that points to original object being wrapped that introspection checks first? - __wrapping__ - __identity__ - __introspect__ - __metadata__ Abstract ======== Decorators were introduced to Python in version 2.4 . Their introduction provided a syntactically easy way to "decorate" functions and methods. The ease of use has allowed the use of functions that wrap other functions and methods to become more prevalent in Python code. Unfortunately, one side-effect of the increased use of decorators is the loss of introspection on the function being wrapped. Most decorators are not meant to be directly noticed by someone using the code; decorators are meant to be heard, not seen. But when a decorator is used on a function or method that returns a new function that wraps the original, introspection will occur on the wrapping function, not the wrapped one. :: def yell(func): def wrapper(*args, **kwargs): print "Hi!" return func(*args, **kwargs) return wrapper @yell def wrapped_one_arg(x): pass def unwrapped_one_arg(x): pass if __name__ == '__main__': from inspect import getargspec print getargspec(wrapped_one_arg) == getargspec(unwrapped_one_arg) To help deal with this phenomenon, the __signature__ attribute is being proposed. It is to be an optional attribute on objects where introspection of function parameters may be performed. It contains an instance of the Signature class that represents all relevant data that one might want to know about the call signature of a function or method. This allows one to assign to a wrapping function's __signature__ attribute an instance of the Signature class that represents the wrapped function or method, thus allowing introspection on the wrapping function to reflect the call signature of the wrapped function. The Signature also works for classes (representing the call signature for instantiation) and instances (for the ``__call__`` method of an instance). A built-in, aptly named ``signature``, is also being proposed. When passed an object that meets the proper requirements, it will either return a new instance of the Signature class or the pre-existing object for the object passed in. Types of Argument Parameters ============================ Python has a fairly rich set of function parameter possibilities. To start, there are required arguments; ``def positional(x): pass``. These types of parameters require an argument be passed in for them. Next, there are default arguments; ``def defaults(x=42): pass``. They have a default value assigned to them if no argument is passed in for that parameter. They are optional, though, unlike required arguments. Lastly, there are excess arguments; ``def excess(*args, **kwargs): pass``. There are two types of excess arguments. One type is an excess positional argument (``*args``). When this type of parameter is present all positional arguments provided during the call are collected into a tuple and assigned to the excess positional argument. The other type of parameter is an excess keyword argument. All keyword arguments that are not directly assigned to an existing keyword parameter are collected into a dict keyed on the keyword name and containing a value of the argument passed in. This dict then gets assigned to the excess keyword argument parameter. For both types, direct assignment to the parameter is not permitted. Signature Object ================ All types of function parameters mentioned in the section `Types of Argument Parameters`_ must be handled by the Signature object in order for it to be useful. One should be able to view the Signature object as providing enough information to be able to know what is possible in terms of calling a function or method. That said, there must be a way to handle required arguments, default arguments, and both types of excess argument parameters. For required arguments, the attribute ``required_args`` provides a tuple of strings that correspond to the names of the required arguments in the order they are defined in the function. By providing the name and order one can easily find out the calling convention required for the minimal call of the function or method. Default arguments are represented by the ``default_args`` attribute. A tuple is contained within the attribute which itself contains two item tuples of ``(name, value)`` pairs representing the name of the default argument parameter and the value that is given. For instance, for the function ``def defaults(x=42, y=True): pass``, the value held by ``default_args`` would be ``(('x', 42), ('y', True))``. Excess argument parameters each have their own attribute to represent their existence. For the existence of an excess positional argument, ``excess_pos_args`` exists. ``excess_kw_args`` represents an excess keyword argument parameter. Both attributes contain a boolean value representing whether they parameter exists. While changing the attributes to contain the name of the respective excess parameter is technically feasible, it has been deemed unnecessary since that knowledge is not useful when making an actual call to the function or method that is represented by Signature object. One key decision that was made during the development of attribute names is taking into consideration the possibility of optional type checking for function parameters. This meant the names should be general enough to represent what they do without type information. Finally, the Signature object implements a ``__str__`` method. Calling this will return a string representing the parameter as might be seen in the actual code. This is provided for human consumption to easily deduce what is required for calling the object. No parentheses are put around the returned string. The final interface, following the syntax outlined in [#interface-syntax]_, is:: interface Signature: """Specifies the call signature of a class, instance, method, or class""" def __str__(self) -> str: """String representation of the parameters""" required_args: tuple(str) default_args: tuple(tuple(str, object)) excess_pos_args: bool excess_kw_args: bool Implementation ============== XXX Signature object (using 'inspect'), changing pydoc (and thus help()), builtin Example Usage ============= For all examples, assume the variable ``sig`` contains an instance of the Signature object. Making a decorator have its __signature__ set to the function it is wrapping ---------------------------------------------------------------------------- :: def wrapper(func): def inner(*args, **kwargs): # Magic happens here ... return func(*args, **kwargs) inner.__signature__ = signature(func) return inner All positional arguments ------------------------ :: sig.required_args + (name for name, value in sig.default_args) Dictionary of default arguments ------------------------------- :: dict(sig.default_args) References ========== .. [#interface-syntax] Guido van Rossum's blog ("Interfaces or Abstract Base Classes?") http://www.artima.com/weblogs/viewpost.jsp?thread=92662 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.py Type: application/octet-stream Size: 2262 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060422/726eb5a2/attachment-0002.obj -------------- next part -------------- A non-text attachment was scrubbed... Name: test_signature.py Type: application/octet-stream Size: 3969 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060422/726eb5a2/attachment-0003.obj From thomas at python.org Sun Apr 23 01:50:05 2006 From: thomas at python.org (Thomas Wouters) Date: Sun, 23 Apr 2006 01:50:05 +0200 Subject: [Python-3000] Removing __getslice__ et al. Message-ID: <9e804ac0604221650k28f01661m65431ba9e0d9ee8a@mail.gmail.com> A long-standing part of Py3K is removing 'old style slices' (meaning the __get/set/delslice__ methods and the sq_slice/sq_ass_slice sequence-struct functions, but not slice objects.) I started removing them, only to find out that it's damned inconvenient to make all slicing go through tp_as_mapping->mp_subscript, for two reasons: - Many classes don't behave as mapping at all, so the mp_subscript function (and the whole tp_as_mapping struct) would be added just for the benefit of accepting slices, which is really a sequence thing, not a mapping thing. - There's actually a PyMapping_Check that relies on sq_slice: a type is a mapping type when it has tp_as_mapping->mp_subscript but not tp_as_sequence->sq_slice. I'm not sure how to express that if there is no special method for slicing. So, I imagine we need a special (C level) hook for slicing after all, just one that always takes a slice object (or tuple of slice-objects-or-tuples-of-same. Oh, and/or Ellipsis. Did I forget anything? :) Either that, or a set of conveniency functions (which we already have, for the most part) and a mapping/sequence-agnostic getitem/subscript. But if we do that, we need a new way of implementing PyMapping_Check. (While on the subject, howmuch of the C API dealing with slices should be adjusted? There's PySequence_GetSlice() that takes Py_ssize_t's, which is (IMHO) pretty convenient for C code, but it'd have to create a slice object behind the scenes. And maybe creating a slice object manually isn't that much effort, after all.) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060423/15f8fa90/attachment.htm From ncoghlan at gmail.com Sun Apr 23 04:10:24 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Apr 2006 12:10:24 +1000 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: Message-ID: <444AE210.6020504@gmail.com> Michael Urman wrote: > For in-module effect, maybe some special import could pull in a > compiler class instance used only for the current module, allowing its > build_xyz methods be replaced. > > from __ast__ import compiler > compiler.build_float = Decimal # on an instance, so no 'self' parameter > > If something like that sounds reasonable for usage, I'll start looking > into how it looks for implementation. (Probably a big ball of mud! And > sounds like it depends on the python-side interface to the ast - I've > lost track if that's solidified yet.) :) I deliberately left out the in-module effect, because Python compiles the whole module before executing any of it. Having in-module code trying to hook activities that happen before hand is challenging to say the least (that said, 'from __future__ import ...' statements have a double life as normal code and compiler directives, so it's obviously not impossible) OTOH, if you can get something working for the compile/exec/__import__ use cases, then a metapath entry can be used to experiment with applying it auto-magically on module import (and poking around inside the module to figure out what modifications to make). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sun Apr 23 04:27:24 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Apr 2006 12:27:24 +1000 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: <444A3584.3080102@gmail.com> <444A3D76.2030603@gmail.com> Message-ID: <444AE60C.4030300@gmail.com> Guido van Rossum wrote: > On 4/22/06, Nick Coghlan wrote: >> No matter what, permitting these kinds of hooks is going to require >> alterations to the process of compiling the AST to the bytecode - and >> subclassing is an excellent way of allowing some aspects of an implementation >> to be overridden while leaving other aspects alone. > > Oops, I have to concur. > > I was thinking that it could all be done later, at run-time. That > makes sense for things like [...] and {...} but doesn't really work > for floating point or even string literals -- you can't afford the > overhead of calling a default hook for each literal at run-time, and > passing the default value to the hook doesn't work in the case of > floats or strings. > > Unfortunately a compile-time hook is much more painful to add to the > language because it means adding new syntax just to specify the hooks. > :-( This is why I'm suggesting Michael look at the exec/compile plugin execution use case first - then we have control code that runs without the hooks that can ask for other code to be compiled *with* the hooks. (similar to the way the compile statement can already force the use of certain __future__ directives) I think doing that cleanly is a big task in itself, but if it works out, we'll have a much better idea of what compiler directives might be needed so a module or function could request a particular compiler. For example, we may decide that hooking individual bits and pieces from within the module is right out, but provide a general "from __compiler__.my.pkg.namespace import MyCompiler" directive that tells the compiler "retrieve my.pkg.namespace.MyCompiler from the list of registered compilers and use it to compile this module rather than the standard compiler". It would then be up to the application to ensure the appropriate compiler was available before importing the module. If you tried to import such a module before the appropriate compiler has been registered then you'd get an ImportError. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Sun Apr 23 05:03:58 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 23 Apr 2006 13:03:58 +1000 Subject: [Python-3000] Removing __getslice__ et al. In-Reply-To: <9e804ac0604221650k28f01661m65431ba9e0d9ee8a@mail.gmail.com> References: <9e804ac0604221650k28f01661m65431ba9e0d9ee8a@mail.gmail.com> Message-ID: <444AEE9E.2060304@gmail.com> Thomas Wouters wrote: > > A long-standing part of Py3K is removing 'old style slices' (meaning the > __get/set/delslice__ methods and the sq_slice/sq_ass_slice sequence-struct > functions, but not slice objects.) I started removing them, only to find > out that it's damned inconvenient to make all slicing go through > tp_as_mapping->mp_subscript, for two reasons: > > - Many classes don't behave as mapping at all, so the mp_subscript function > (and the whole tp_as_mapping struct) would be added just for the benefit of > accepting slices, which is really a sequence thing, not a mapping thing. These types already provide tp_as_sequence->sq_item. Why would they want to provide tp_as_mapping->mp_subscript as well? > - There's actually a PyMapping_Check that relies on sq_slice: a type is a > mapping type when it has tp_as_mapping->mp_subscript but not > tp_as_sequence->sq_slice. I'm not sure how to express that if there is no > special method for slicing. How about changing it to check tp_as_sequence->sq_item instead? (that will still work for dictionaries - they only define tp_as_sequence because there isn't a slot to hook "__contains__" in the tp_as_mapping structure) On a slightly different note, we should fix the definition of sequence vs mapping so that PySequence_Check and PyMapping_Check work for arbitrary Python classes. For example, a flag "__sequence__ = True" that type() checked when constructing the class. If the flag was missing, both tp_as_mapping->mp_subscript and tp_as_sequence->sq_item would be filled in with __getitem__ (as they are now). If the flag was present and false, only tp_as_mapping would be filled in. If the flag was present and true, only tp_as_sequence would be filled in. > (While on the subject, howmuch of the C API dealing with slices should be > adjusted? There's PySequence_GetSlice() that takes Py_ssize_t's, which is > (IMHO) pretty convenient for C code, but it'd have to create a slice object > behind the scenes. And maybe creating a slice object manually isn't that > much effort, after all.) I think we should leave abstract.h alone - Getting rid of __getslice__ at the Python level is a good thing, but simply fixing these API functions to "do the right thing" is easier than breaking all the code that currently uses them. Should we add a convenience function PySlice_FromIndices to the slice API that accepts Py_ssize_t's rather than PyObjects? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From talin at acm.org Sun Apr 23 08:41:10 2006 From: talin at acm.org (Talin) Date: Sun, 23 Apr 2006 06:41:10 +0000 (UTC) Subject: [Python-3000] rough draft signature PEP References: Message-ID: Brett Cannon python.org> writes: > [I am posting to python-3000 since this is where the parameter list > ideas are being discussed, but this is probably generic enough to > eventually make it into the 2.x line] > > Here is a rough draft of a PEP I wrote last summer after I had Guido > come for lunch at Google when I was interning there (i.e., a little > old =) . Someone (I think Philip) had suggested something like this > back then but it didn't go any farther. I liked the idea and I asked > Guido at lunch if he was okay for it; he was. > > So I wrote the following PEP along with a Python implementation > (attached, along with test cases). It is still a rough draft since I > am on vacation on top of visiting my dad for his birthday and thus > cannot put a ton of time into this at the moment, so don't consider > this a final version in any way. There is already a list of things to > consider and I am sure there are some things discussed during the new > parameter list ideas that could stand to be worked in. I am quite > happy to work on finishing this PEP so that Talin can focus on the > parameter list PEP. So just comment away and I will work on > incorporating them as time permits. Sounds good. A couple of comments: 1) I a still confused (see the "minor hitch" thread) as to how the signature information can be used without either duplicating the interpreter's parameter-binding algorithm, or somehow giving access to it. In other words, having access to signature information isn't very useful unless there is a straightforward and efficient way to map incoming arguments to specific signature slots. Because the decorators execute *before* this step normally happens, they can't just look and see which particular slot an argument got assigned to, they have to calculate it. The actual-to-formal mapping that's used in the interpreter itself can't be directly exposed, because it's not data driven (apparently - this is one of the parts of the intepreter that I don't understand yet.) A duplicate implementation suffers from two drawbacks: First, any implementation will need to be kept in sync with the interpreter. Second, the data structure that the mapping algorithm has to deal with (i.e. the signature object) is going to be in a form that is less efficient (i.e. since it needs to be constructable via Python code, it can't be a simple C structure like argument_ty.) 2) I noticed in your PEP that you followed the same mental model as is currently used in the compiler, which is that *args and **args are treated as special cases. In other words, the description of a function's arguments consists of an array of "regular" arguments, plus a couple of "special" argument fields. My mental model is a little different, as I tend to see the argument as a single list, where arguments have different modifiers, where * and ** are modifiers. In other word, my model follows more closely (IMHO) the way that arguments are read by the programmer. To show the difference between the two models, lets create an algorithm that populates the parameter slots from a set of input arguments. For simplicity, lets just consider positional arguments, including varargs arguments. In the case where the varargs is considered a separate slot: i = 0 while i < numInputArgs: if i <= regularSlotCount: slot[ i ] = args[ i ] i += 1 if i < numInputArgs: slot[ i ] = args[ i: ] In the case where there's a single list: i = 0 s = 0 while i < numInputArgs: if modifiers[ s ] == VARARGS: slot[ s ].append( args[ i ] ) else: slot[ s ] = args[ i ] s += 1 i += 1 (Note that neither of these functions is intended to be optimal.) I'm not saying that my model is necessarily better. Its just how I think about it. The only advantage that I would claim is that it more readily allows for future kinds of "special" arguments, by extending the meaning of the 'modifier' attribute. Here's what I had written in the PEP: Specification The specification defines a new function object attribute, __signature__, which contains a tuple containing a list of parameter descriptions. Each parameter description will be a tuple containing the following elements: [0] - The string name of the parameter. [1] - The Python type of the parameter, if a type has been specified. If none has been specified, this field will contain None. [2] - The parameter modifier (see below.) [3] - The default value of the parameter, if there is one. If there is no default value, then the 4th field will be absent - in other words, it will be a 3-tuple rather than a 4-tuple. This allows discrimination between a parameter whose default is 'None', and a parameter with no default. The second argument, containing the Python type, will always contain None until such time as a convention for specifying explicit types is adopted. The parameter modifier is used to describe any modifier prefix characters attached to the argument, such as '*args'. The following table lists which values are currently defined: 0 - a regular argument (no modifiers) 1 - a varargs argument (one star) 2 - a keyword dict argument (two stars) Other values may be defined later as additions are made to the language. -- Talin From talin at acm.org Sun Apr 23 09:37:46 2006 From: talin at acm.org (Talin) Date: Sun, 23 Apr 2006 07:37:46 +0000 (UTC) Subject: [Python-3000] PEP: Keyword-Only Arguments Message-ID: I've submitted the "Keyword-Only Arguments" PEP that we've been discussing. You can also read it here: http://www.viridia.org/python/doc/PEP_KeywordOnlyArguments.txt -- Talin From guido at python.org Sun Apr 23 10:09:58 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 23 Apr 2006 09:09:58 +0100 Subject: [Python-3000] Removing __getslice__ et al. In-Reply-To: <444AEE9E.2060304@gmail.com> References: <9e804ac0604221650k28f01661m65431ba9e0d9ee8a@mail.gmail.com> <444AEE9E.2060304@gmail.com> Message-ID: On 4/23/06, Nick Coghlan wrote: > Thomas Wouters wrote: > > > > A long-standing part of Py3K is removing 'old style slices' (meaning the > > __get/set/delslice__ methods and the sq_slice/sq_ass_slice sequence-struct > > functions, but not slice objects.) I started removing them, only to find > > out that it's damned inconvenient to make all slicing go through > > tp_as_mapping->mp_subscript, for two reasons: > > > > - Many classes don't behave as mapping at all, so the mp_subscript function > > (and the whole tp_as_mapping struct) would be added just for the benefit of > > accepting slices, which is really a sequence thing, not a mapping thing. Well, we already did that for lists, strings and other built-in sequences that support extended slicing (remember "hello"[::-1] ?:-). So I think it's no fundamental problem, just an inconvenience. > These types already provide tp_as_sequence->sq_item. Why would they want to > provide tp_as_mapping->mp_subscript as well? Ah, you've never tried to implement extended slicing. A[x:y:z] generates bytecode that calls A[slice(x,y,z)] and since a slice object is not an int the GETITEM opcode tries to pass that to the mapping/dict getitem implementation. Same for A[x:y, z:u] -- it generates code to call A[(slice(x,y), slice(z,y))] i.e. the argument is a tuple of slices. Etc.; the tuple can also contain an Ellipsis object if you wrote A[x, ..., z]. (Yes that's valid syntax!) > > - There's actually a PyMapping_Check that relies on sq_slice: a type is a > > mapping type when it has tp_as_mapping->mp_subscript but not > > tp_as_sequence->sq_slice. I'm not sure how to express that if there is no > > special method for slicing. > > How about changing it to check tp_as_sequence->sq_item instead? (that will > still work for dictionaries - they only define tp_as_sequence because there > isn't a slot to hook "__contains__" in the tp_as_mapping structure) The problem with that is that if a Python class defines __getitem__ the C code doesn't know whether it intends to implement the mapping or the sequence protocol, so it fills in both slots. I believe that PyMapping_Check should be gotten rid of, and also any other check, whether in C or in Python, that attempts to distinguish between mappings and sequences. These checks are all vestiges of Python development before user-defined classes were added to the language in late 1990... Ideally (but this is a big refactoring of the C API!) the C API would change so that there are no redundant slots -- no sequence concat/repeat vs. numeric add/mul, and no sequence vs. numeric getitem/setitem/len/anything else. > On a slightly different note, we should fix the definition of sequence vs > mapping so that PySequence_Check and PyMapping_Check work for arbitrary Python > classes. They can't. See above. They should be gotten rid of. (Or they should be used simply as checks whether the AsSequence or AsMapping struct pointer is non-NULL - but that's not very useful and the name would be misleading.) > For example, a flag "__sequence__ = True" that type() checked when > constructing the class. If the flag was missing, both > tp_as_mapping->mp_subscript and tp_as_sequence->sq_item would be filled in > with __getitem__ (as they are now). If the flag was present and false, only > tp_as_mapping would be filled in. If the flag was present and true, only > tp_as_sequence would be filled in. I think this is the wrong approach. Python overloads A[x] so that it could mean either sequence indexing or mapping lookup. That's a fundamental syntactic ambiguity. We should not attempt to write code that depends on whether the object implements one or the other. > > (While on the subject, howmuch of the C API dealing with slices should be > > adjusted? There's PySequence_GetSlice() that takes Py_ssize_t's, which is > > (IMHO) pretty convenient for C code, but it'd have to create a slice object > > behind the scenes. And maybe creating a slice object manually isn't that > > much effort, after all.) > > I think we should leave abstract.h alone I think abstract.[ch] and object.[ch] ought to be merged -- there is no rhyme or reason to whether any particular function is implemented in one or the other. Also there's a lot of redundancy -- there are PySequence_XXX, PyMapping_XXX, PyNumber_XXX and PyObject_XXX functions often with overlapping but non-identical APIs. This is a confusing mess and should be sorted out significantly. (What about 3rd party extensions? Well too bad. We've got to break this stuff some day.) > - Getting rid of __getslice__ at the > Python level is a good thing, but simply fixing these API functions to "do the > right thing" is easier than breaking all the code that currently uses them. Perhaps we could drop the __getslice__ Python API but add new and different support for slicing in the C API. One of the major problems with the current slice API (C and Python) is that it is formulated in terms of ints, *and* that those ints have already been corrected for negative indices. I.e. if you call A[-1:-2] and len(A) == 5, the getslice operation (C or Python) is called with arguments (4, 3). That's a major pain. Extended slicing doesn't do this. > Should we add a convenience function PySlice_FromIndices to the slice API that > accepts Py_ssize_t's rather than PyObjects? Makes sense, but only if there are more than two places in the current code base where it would be used. So, as an overall response to Thomas, if you can get rid of PyMapping_Check, I think you can steam ahead with this one way or another. A special sq_newslice slot that gets called when the getitem arg is a slice is probably not the way to go. Rather, convenience routines to extract the indices as Py_ssize_t values would make the most sense. These can safely be clipped to +/- maxint when the real value is too large, I believe (I think that convenience routine may already exist after Travis Oliphant's patches). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From birchb at tpg.com.au Sun Apr 23 14:11:29 2006 From: birchb at tpg.com.au (Bill Birch) Date: Sun, 23 Apr 2006 22:11:29 +1000 Subject: [Python-3000] Minor hitch writing the Function Signature PEP In-Reply-To: References: Message-ID: <200604232211.29979.birchb@tpg.com.au> On Sun, 23 Apr 2006 05:36 am, Talin wrote: > @precondition( y=NotNegative ) > def power( x, y ): > ? ?... > ...where 'NotNegative' is a function which simply does an assert( value >= 0 ) The phrase "y=NotNegative" is a classic type constraint predicate. Given that type expressions will be dynamic (see GVD blog http://www.artima.com/weblogs/viewpost.jsp?thread=87182) how about: def power(x, y : NotNegative): body... or def power(x, y : not Negative): body... or even: def power(x, y : (lambda t: t > 0) ): body... From baptiste13 at altern.org Sun Apr 23 14:21:55 2006 From: baptiste13 at altern.org (Baptiste Carvello) Date: Sun, 23 Apr 2006 14:21:55 +0200 Subject: [Python-3000] Pronouncement on parameter lists In-Reply-To: <444986D2.5060901@canterbury.ac.nz> References: <444986D2.5060901@canterbury.ac.nz> Message-ID: Greg Ewing a ?crit : > Jim Jewett wrote: > > >>It has not yet been specified what would happen to additional >>positional arguments that get passed in anyway. (Swallow or raise an >>Exception?) > > > I've always intended that there would be an exception. > If so, why not replace the lone star, whose meaning is not clear, with "*forbidden", as in: def f(a, b, *forbidden, c, d=1) The meaning would be clearer: "remaining positional arguments are forbidden, and will thus raise an exception". Baptiste From birchb at tpg.com.au Sun Apr 23 14:28:55 2006 From: birchb at tpg.com.au (Bill Birch) Date: Sun, 23 Apr 2006 22:28:55 +1000 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: References: Message-ID: <200604232228.55500.birchb@tpg.com.au> On Sun, 23 Apr 2006 12:53 am, Fredrik Lundh wrote: > Guido van Rossum wrote: > > It's an interesting idea; it's been brought up before but nobody AFAIK > > has ever implemented it. I like the scoping requirement. > > can we add support for XML literals while we're at it? > > http://effbot.org/zone/idea-xml-literal.htm > Have a look at Comega http://research.microsoft.com/Comega/ which throws in SQL syntax just for good measure. From talin at acm.org Sun Apr 23 21:04:54 2006 From: talin at acm.org (Talin) Date: Sun, 23 Apr 2006 19:04:54 +0000 (UTC) Subject: [Python-3000] Minor hitch writing the Function Signature PEP References: <200604232211.29979.birchb@tpg.com.au> Message-ID: Bill Birch tpg.com.au> writes: > > On Sun, 23 Apr 2006 05:36 am, Talin wrote: > > precondition( y=NotNegative ) > > def power( x, y ): > > ... > > ...where 'NotNegative' is a function which simply does an assert( value >= > 0 ) > The phrase "y=NotNegative" is a classic type constraint predicate. > > Given that type expressions will be dynamic (see GVD blog > http://www.artima.com/weblogs/viewpost.jsp?thread=87182) > how about: > > def power(x, y : NotNegative): > body... > > or > > def power(x, y : not Negative): > body... > > or even: > > def power(x, y : (lambda t: t > 0) ): > body... > That's cool, but it still doesn't solve my problem :) Let me see if I can explain this better. Think of a function as a box. Inside the box, there's code that expects an array of filled parameter slots. Outside the box are decorators, classes, and all the rest of the program. The inputs to the box are always: function( PyTuple *args, PyDict *kwargs ) However, inside the box, the values within 'args' and 'kwargs' get assigned to formal parameter slots, based on the function signature. Now, one peculiarity of this mapping procedure is that it has to be done for all arguments, you can't just do it for a single argument in isolation. You have no idea where any positional argument is going to go until all keyword arguments have been placed, and until all previous positional arguments have been placed. Now, the __signature__ attribute gives a description of what the code inside the box expects. However, that's not what we have to work with. We can't access the code inside the box, we can only feed in values through the parameter mapping mechanism. Which means that any values we want to send to the code have to be represented as '*args, *kwargs'. Now, if we're writing a decorator where we already know what the signature is, its fairly easy to reverse the mapping process in our heads and write code such that the proper values will fall into the proper places. I know that if a function takes '(a, b, c, d)', and I write '(d=1, 2, 3, 4)' I know that the value assignments will be 'a=2, b=3, c=4, d=1'. However in this case there's no need for a __signature__ since I already know what the signature is. But a generic wrapper function has to look at the __signature__ attribue to find this out. But it can't use the signature directly, because the signature only tells it what is going on inside the box, not what to feed the box from the outside. In order to do this, it has to construct a reverse mapping, and it can't do this for just one parameter, it has to do it for all of them. Moreover, the reverse mapping can't be constructed statically, because it changes depending on the values contained in *args, *kwargs. Its not only decorators that have this problem - its *any* entity that attempts to use the __signature__ attribute to examine specific parameter values. (Obviously, code that uses __signature__ for documentation purposes has no problem.) At this point, the best solution seems to be to have some standard library function that duplicates the mapping process - that is, given a args, kwargs, and signature object, it returns a tuple with the values in the proper parameter slots. However, this means that you have to do the whole mapping twice, and it still doesn't let you modify the input parameters to the function. -- Talin From ianb at colorstudy.com Sun Apr 23 21:06:51 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 23 Apr 2006 14:06:51 -0500 Subject: [Python-3000] PEP - string.format In-Reply-To: <4449DB4C.6050706@gmail.com> References: <4449DB4C.6050706@gmail.com> Message-ID: <444BD04B.3030603@colorstudy.com> Nick Coghlan wrote: > [...] >> The second method, 'fformat', is identical to the first, except that it >> takes an additional first argument that is used to specify a 'custom >> formatter' object, which can override the normal formatting rules for >> specific fields: >> >> "More on {0}, {1}, and {c}".fformat( formatter, a, b, c=d ) >> >> Note that the formatter is *not* counted when numbering fields, so 'a' is >> still considered argument number zero. > > I don't like this. Formatting with a different formatter should be done as a > method on the formatter object, not as a method on the string. +1. Plus "fformat" doesn't really mean anything as a name (as least to me). But this should perform pretty well if it's going to replace % (which wouldn't have to happen for Python 2.6, but should for py3k). > [...] >> Braces can be escaped using a backslash: >> >> "My name is {0} :-\{\}".format( 'Fred' ) > > So "My name is 0} :-\{\}".format('Fred') would be an error? I like that - it > means you get an immediate exception if you inadvertently leave out a brace, > regardless of whether you leave out the left brace or the right brace. I kind of like that too, even though } doesn't really have to be quoted. It always weird me out that > doesn't have to be quoted in XML. It occurs to me that the overloading of \ might cause some complications; % and string.Template use doubling (%% and $$). For instance, a little help message: write("To include the username ({username}) use \{username\}\n" "and use \\\{ and \\\} to include literal brackets".format(username=u)) Which I isn't all that bad. I assume other \ sequences won't be interpretted. I.e., "\\n".format() produces "\\n". But I then don't see any way to do "\\%s" % var. >> The format() method does not attempt to interpret the >> conversion specifiers in any way; it merely passes all of the characters >> between the first colon ':' and the matching right brace ('}') to the >> various underlying formatters (described later.) > > If we had a subclassing API similar to what I suggest above, a custom > formatter could easily support Ian's pipelining idea by doing: > > def format_value(self, value, fmt_spec): > if fmt_spec is None: > val = Formatter.format_value(self, value, fmt_spec) > else: > for fmt in fmt_spec.split(':'): > val = Formatter.format_value(self, value, fmt_spec) > return val > > I don't really think that should be the default, though. It won't work if the __format__ conventions don't support it; if all objects return strings then pipelining will become much less useful. > [...] >> - The trailing letter is optional - you don't need to say '2.2d', you can >> instead just say '2.2'. If the letter is omitted, then the value will be >> converted into its 'natural' form (that is, the form that it take if str() >> or unicode() were called on it) subject to the field length and precision >> specifiers (if supplied.) > > I disagree with this. These format specifier do a type coercion before > applying the formatting. These specifiers should be retained and should > continue to result in coercion to int or float or str, with the relevant > TypeErrors when that coercion isn't possible. -1. I think %d is silly in Python, and we shouldn't replicate that here. >> 2) Otherwise, see if the value to be formatted has a __format__ method. If >> it does, then call it. > > So an object can override standard parsing like {0:d} to return something > other than an integer? *shudder* "{0:d}" doesn't scream "integer" to me. %d does because of history, but only because of history. > Being able to add extra formatting codes for different object types sounds > good, but being able to change the meaning of the standard codes sounds (very) > bad. There are no standard codes. There's standard % codes, but this isn't %. > If supporting the former also means supporting the latter, then I'd prefer to > leave this ability to custom formatter objects or explicit method or function > invocations on arguments in the call to format(). > >> 3) Otherwise, check the internal formatter within string.format that >> contains knowledge of certain builtin types. > > This should happen *before* checking for a custom __format__ method. (If we > decide to check for a custom __format__ method at all) How would this change anything? Presumably only the immutable types (int, str, list, etc) would be specially handled anyway. > [...] >> One other thing that could be done to make the debugging case more >> convenient would be to allow the locals() dict to be omitted entirely. >> Thus, a format function with no arguments would instead use the current >> scope as a dictionary argument: >> >> print "Error in file {p.file}, line {p.line}".format() > > Again, I don't think this is worth the additional complexity. I don't really understand the complexity argument; I think you should be more specific about what becomes more complex. > [...] >> Other, more radical proposals include backquoting (`), or a new string >> prefix character (let's say 'f' for 'format'): >> >> print f"Error in file {p.file}, line {p.line}" > > A method will do the job with far more flexibility - no need to hack the parser. In this particular case (where locals() are picked up implicitly) a method has several flaws. I don't think the f"" syntax makes any sense outside of that case; so it's a separate argument (*if* there's support for a special form evaluated in the scope where the string came from, then will it use f""?) -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From martin at v.loewis.de Sun Apr 23 23:11:34 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 23 Apr 2006 23:11:34 +0200 Subject: [Python-3000] Use Py_CLEAR only In-Reply-To: <4445E1D5.5020702@canterbury.ac.nz> References: <4445906C.7060006@comcast.net> <4445D1D4.8030503@canterbury.ac.nz> <4445DD02.9060805@livinglogic.de> <4445E1D5.5020702@canterbury.ac.nz> Message-ID: <444BED86.5020308@v.loewis.de> Greg Ewing wrote: >> Both should be optimized away by the compiler. > > How? I don't see how the compiler can know either of > those things. As you then discussed: it can do that for local variables (it might even *warn* about the redundant assignment, though). OTOH, for local variables, it might be of least interest: it's usually pretty clear whether they will be accessed again or not. For global variables, I'm completely in favour of releasing them through Py_CLEAR always. For pointer that live on the heap, Py_[X]DECREF is usually done because either a) the memory block containing the pointer is about to be released (e.g. when a tuple is deallocated), or b) the pointer is about to be changed to point to something else (e.g. a setitem operation) In either case, there isn't much value to clearing the pointer: In case b), the value will change right away again, and in case a), if anybody still has a pointer to the block, we are in trouble, and clearing the pointer likely doesn't help in finding the trouble. Regards, Martin From talin at acm.org Sun Apr 23 23:19:21 2006 From: talin at acm.org (Talin) Date: Sun, 23 Apr 2006 21:19:21 +0000 (UTC) Subject: [Python-3000] Brainstorming: Python Metaprogramming Message-ID: It seems that the history of the Python mailing lists are littered with the decayed corpses of various ideas related to "metaprogramming", that is, programs that write programs, either at compile time. We've seen proposals for C-style macros, Lisp-style macros, programmable syntax (guilty!), AST access, a first-class symbol type, and much more. Given how many times this has been suggested, I really do think that there is something there; At the same time, however, I recognize that all of these proposals are irrepairably flawed in one way or another. I think that much of the reason for this, is that the various proposals haven't really been distilled down to their absolute minimum essentials, which admittedly is a hard thing to do. Instead, they offer to import a whole, pre-existing architecture from some other language, which transforms Python into something that it is not. As an example, take Lisp-style macros. A macro in Lisp is a function that takes its arguments in unevaluated form. So when you call "mymacro( (add a b) ), you don't get the sum of a and b, you get a list consisting of three elements. The macro can then manipulate that list, and then evaluate it after it has been manipulated. The reason this is possible is because in Lisp, there's no difference between the AST (so to speak) and regular data. This fails in Python for two reasons: 1) For reasons of performance, the compiled code doesn't look very much like regular data, and is hard to manipulate. 2) Most of the things that you might want to do with Lisp macros you can already do in Python using some other technique. Using lambda, generators, operator overloading, and other Python features, we can effectively 'quote' a section of code or an algorithm, and manipulate it as data. No, you can't assemble arbitrary blocks of code, but most of the time you don't want to. Using overloaded Python operators, we can in fact do something very like the Lisp macro - that is, by replacing the '+' operator __add__, we can have it return an AST-like tree of objects, rather than carrying out an actual addition. However, this only works if you actually have control over the types being added. As we've seen in SQLObject, this limitation leads to some interesting syntactical contortions, where you need to insure that at least one of the two objects being added knows about the overloaded operator. So one question to ask is - what can the Lisp macro system do that is (a) useful, and (b) not already doable in Python, and (c) minimal enough that it wouldn't cause a major rethink of the language? And the same question can be asked for all of the other proposals. For some reason, I have stuck in my head the idea that this concept of 'quoting' is central to the whole business. In Lisp the term 'quote' means to supress evaluation of an item. So (quote a) returns to the symbol 'a', not the value stored in 'a'. It is the ability to refer to a thing that would normally be executed in its pre-executed state. In Python, we can already quote expressions, using lambda; we can quote loops, using generators; and so on. However, one piece that seems to be missing is the ability to quote references to global and local variables. In Python, the way to refer to a variable by name is to pass its name as a string. The problem with this, however, is that a string is a type in its own right, and has a whole different set of methods and behaviors than a variable reference. As a hypothetical example, supposed we defined a unary operator '?' such that: ?x was syntactic sugar for: quoted('x') or even: quoted('x', (scope where x is or would be defined) ) Where 'quoted' was some sort of class that behaved like a reference to a variable. So ?x.set( 1 ) is the same as x = 1. Moreover, you would want to customize all of the operators on quoted to return an AST, so that: ?x + 1 produces something like: (add, quoted('x'), 1) ...or whatever data structure is convenient. Of course, one issue that immediately comes to mind is, where does the class 'quoted' come from? Is it a globally defined class, or is it something that is defined for a given scope? What if you need to use two different definitions for 'quoted' within a single scope? For example, I can imagine that in the case of something like SQLObject, the 'quoted' class would transform into an SQL variable reference. But what if you wanted to use that along with the algebraic solver, which defines 'quoted' as something very different? One idea, which is kind of strange (and impractical, but bear with me), is inspired by C++. When you define a member function in C++, the function's formal parameters are considered to be in the same scope as the function body. So for example, instead of having to write: void MyClass::function( MyClass::tListType &l ); ...you can just write: void MyClass::function( tListType &l ); ...because all of the contents of MyClass are visible within the argument list. So the strange idea is that when a function is *called*, the calling parameters are evaluated within a new, nested scope, that inherits from both the function itself, and the calling code's scope. This would allow the called function to overide the meaning of 'quoted' or other operators (in fact, it would solve the SQLObject problem if it allowed built-in operators such as '+' to be redefined.) As an example, the algebraic solver could override the quote operator for its parameters: return solve( ?x + 0 ) --> ?x Of course, this is a silly idea -- for one thing it would be horribly expensive. Anyway, I wanted to throw this out there for discussion. I'm mainly interested in a discussion of ways that we could facilitate additional metaprogramming in Python, with the tiniest possible changes to the language; Don't feel limited to just the discussion of quoting and variable references, but don't let it stray (please) beyond the parameters of what I've set. Thanks :) -- Talin From krstic at fas.harvard.edu Sun Apr 23 23:33:11 2006 From: krstic at fas.harvard.edu (Ivan Krstic) Date: Sun, 23 Apr 2006 17:33:11 -0400 Subject: [Python-3000] Brainstorming: Python Metaprogramming In-Reply-To: References: Message-ID: <444BF297.6050109@fas.harvard.edu> Talin wrote: > Anyway, I wanted to throw this out there for discussion. I'm mainly interested > in a discussion of ways that we could facilitate additional metaprogramming in > Python, with the tiniest possible changes to the language PEP 3099 contains Guido's explicit pronouncement against this, from what I can tell. If this were not so, the defop approach that Logix already implements for Python (http://www.livelogix.net/logix/intro.html) would be a neat macro facility to evaluate. -- Ivan Krstic | GPG: 0x147C722D From crutcher at gmail.com Sun Apr 23 23:40:48 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Sun, 23 Apr 2006 14:40:48 -0700 Subject: [Python-3000] Stackable Blocks Message-ID: Having been shot down so hard on my request to bring for loops into symetry with list comprehensions, I have a new proposal - stackable blocks. This is a very _small_ amount of syntax sugar, the basic idea is that any block headers can be stacked on the same line. This changes no semantics, nor does it change ordering, or add new keywords, it just changes a tiny bit of the grammar for whitespace/block parsing. So for instance this: for file in open_files: if file.readable(): ... can be spelled like this. for file in open_files: if file.readable(): ... And this: for i in range(10): for j in range(20): if i != j: ... can be spelled: for i in range(10): for j in range(20): if i != j: ... and this: for line in open(file): if line.strip(): ... can be this: for line in open(file): if line.strip(): ... -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From thomas at python.org Sun Apr 23 23:42:26 2006 From: thomas at python.org (Thomas Wouters) Date: Sun, 23 Apr 2006 23:42:26 +0200 Subject: [Python-3000] Use Py_CLEAR only In-Reply-To: <444BED86.5020308@v.loewis.de> References: <4445906C.7060006@comcast.net> <4445D1D4.8030503@canterbury.ac.nz> <4445DD02.9060805@livinglogic.de> <4445E1D5.5020702@canterbury.ac.nz> <444BED86.5020308@v.loewis.de> Message-ID: <9e804ac0604231442t18f4280aq1a88cef8e5eeb553@mail.gmail.com> On 4/23/06, "Martin v. L?wis" wrote: > For pointer that live on the heap, Py_[X]DECREF is usually > done because either > a) the memory block containing the pointer is about to be > released (e.g. when a tuple is deallocated), or > b) the pointer is about to be changed to point to something > else (e.g. a setitem operation) Eh, hang on. The reason to use Py_CLEAR() isn't that the pointer may be used after the Py_DECREF(), but that it may be used *as a result of DECREF'ing*. DECREFs can cause cleanup, which can execute arbitrary Python or C code through finalizers or weakref callbacks. So whatever you're doing after the Py_DECREF/Py_CLEAR call is immaterial to the decision to use either. You should use Py_CLEAR when the pointer you are DECREF'ing *might* go away, and is accessible from outside the function (i.e. it isn't only reachable through automatic variables.) However, I agree that it should be a conscious decision, and not an automatism. -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060423/cb3e99a1/attachment.htm From krstic at fas.harvard.edu Sun Apr 23 23:50:58 2006 From: krstic at fas.harvard.edu (Ivan Krstic) Date: Sun, 23 Apr 2006 17:50:58 -0400 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: Message-ID: <444BF6C2.2030507@fas.harvard.edu> Crutcher Dunnavant wrote: > for file in open_files: if file.readable(): > ... > > for line in open(file): if line.strip(): I don't like this. It looks unclean and adds an exception to the conventional Python rules while yielding no particular or significant benefit. -- Ivan Krstic | GPG: 0x147C722D From jcarlson at uci.edu Mon Apr 24 01:08:21 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sun, 23 Apr 2006 16:08:21 -0700 Subject: [Python-3000] Pronouncement on parameter lists In-Reply-To: References: <444986D2.5060901@canterbury.ac.nz> Message-ID: <20060423160522.66C4.JCARLSON@uci.edu> Baptiste Carvello wrote: > > Greg Ewing a ?crit : > > Jim Jewett wrote: > > > > > >>It has not yet been specified what would happen to additional > >>positional arguments that get passed in anyway. (Swallow or raise an > >>Exception?) > > > > > > I've always intended that there would be an exception. > > > If so, why not replace the lone star, whose meaning is not clear, with > "*forbidden", as in: > > def f(a, b, *forbidden, c, d=1) > > The meaning would be clearer: "remaining positional arguments are forbidden, and > will thus raise an exception". It seems as though you are proposing to create a pseudo-keyword embedded inside a function definition, even though 'forbidden', or really any other reasonably lengthed pseudo-keyword has two meanings. While you would say "it will raise an exception", I could say, "it's to tell us what values are forbidden for some other task. -1 on the use of _any_ pseudo-keyword inside of function definitions. - Josiah From talin at acm.org Mon Apr 24 01:42:33 2006 From: talin at acm.org (Talin) Date: Sun, 23 Apr 2006 23:42:33 +0000 (UTC) Subject: [Python-3000] Brainstorming: Python Metaprogramming References: <444BF297.6050109@fas.harvard.edu> Message-ID: Ivan Krstic fas.harvard.edu> writes: > Talin wrote: > > Anyway, I wanted to throw this out there for discussion. I'm mainly > > interested in a discussion of ways that we could facilitate > > additional metaprogramming in Python, with the tiniest possible changes > > to the language > > PEP 3099 contains Guido's explicit pronouncement against this, from what > I can tell. If this were not so, the defop approach that Logix already > implements for Python (http://www.livelogix.net/logix/intro.html) would > be a neat macro facility to evaluate. I may be guilty of "BDFL-lawyering" here, but I believe that what I am talking about falls outside of the 3099 prohibition. Of course, if Guido wants to expand that prohibition to cover the subject, I won't gripe (much) :) Specifically, I am *not* talking about programmable syntax, or completely changing the execution model of Python, or anything of that nature. -- Talin From jimjjewett at gmail.com Mon Apr 24 01:55:56 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Sun, 23 Apr 2006 19:55:56 -0400 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: <00dc01c66591$ed190120$a8b62997@bagio> References: <00dc01c66591$ed190120$a8b62997@bagio> Message-ID: On 4/21/06, Giovanni Bajo wrote: > Another (similar) way would be to let the user pay for the high typechecking > price in normal cases *but* provide a list[int] class, which is a list > decorated with typechecks on modification operations. We could have > list[int|float]() to construct a list which can hold either ints or floats. Remember that the current thought is "annotations" rather than "type specifiers", with any actual verication done by a separate decorator. If you really need to ensure list[int], then you can declare a subclass of list that verifies each element on instance creation or modification, and then use your decorator to replace any incoming sequence with a (possibly newly created) instance of this subclass. There will be some wasted work if another framework uses a different adapter class to say the same thing, but that should be the sort of "major boundary" where it is reasonable to verify things. If you pass an object out to code that doesn't care, they won't see any extra costs (unless they special case CheckExact). What they pass back may even still be the same object, and if not ... you can always remove the decorator for speed. -jJ From crutcher at gmail.com Mon Apr 24 02:12:57 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Sun, 23 Apr 2006 17:12:57 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: <444BF6C2.2030507@fas.harvard.edu> References: <444BF6C2.2030507@fas.harvard.edu> Message-ID: On 4/23/06, Ivan Krstic wrote: > Crutcher Dunnavant wrote: > > for file in open_files: if file.readable(): > > ... > > > > for line in open(file): if line.strip(): > > I don't like this. It looks unclean and adds an exception to the > conventional Python rules while yielding no particular or significant > benefit. It doesn't add any exception, it changes how the block syntax works. -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From jparlar at cogeco.ca Mon Apr 24 05:52:51 2006 From: jparlar at cogeco.ca (Jay Parlar) Date: Sun, 23 Apr 2006 20:52:51 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: <444BF6C2.2030507@fas.harvard.edu> Message-ID: On Apr 23, 2006, at 5:12 PM, Crutcher Dunnavant wrote: > On 4/23/06, Ivan Krstic wrote: >> Crutcher Dunnavant wrote: >>> for file in open_files: if file.readable(): >>> ... >>> >>> for line in open(file): if line.strip(): >> >> I don't like this. It looks unclean and adds an exception to the >> conventional Python rules while yielding no particular or significant >> benefit. > > It doesn't add any exception, it changes how the block syntax works. > I think what Ivan meant is that the normal Python rule is that if you have a full-colon (ANYWHERE), and you put something after it on the same line, then you're done the block. This behaviour is the same for method definitions, IF-blocks, the FOR statement, class definitions, etc. However, your change would be an exception to the rule, in that it would change what a full-colon is allowed to do, but only in one place. Jay P. From crutcher at gmail.com Mon Apr 24 03:32:13 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Sun, 23 Apr 2006 18:32:13 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: <444BF6C2.2030507@fas.harvard.edu> Message-ID: On 4/23/06, Jay Parlar wrote: > > On Apr 23, 2006, at 5:12 PM, Crutcher Dunnavant wrote: > > > On 4/23/06, Ivan Krstic wrote: > >> Crutcher Dunnavant wrote: > >>> for file in open_files: if file.readable(): > >>> ... > >>> > >>> for line in open(file): if line.strip(): > >> > >> I don't like this. It looks unclean and adds an exception to the > >> conventional Python rules while yielding no particular or significant > >> benefit. > > > > It doesn't add any exception, it changes how the block syntax works. > > > > I think what Ivan meant is that the normal Python rule is that if you > have a full-colon (ANYWHERE), and you put something after it on the > same line, then you're done the block. This behaviour is the same for > method definitions, IF-blocks, the FOR statement, class definitions, > etc. > > However, your change would be an exception to the rule, in that it > would change what a full-colon is allowed to do, but only in one place. No, it wouldn't. First, this would apply to method and class defs as well, so it's not an exception, secondly, the 'block ending thing after the colon' would be _extended_ to cover other block defs. -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From jparlar at cogeco.ca Mon Apr 24 06:46:25 2006 From: jparlar at cogeco.ca (Jay Parlar) Date: Sun, 23 Apr 2006 21:46:25 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: <444BF6C2.2030507@fas.harvard.edu> Message-ID: <57f9be0f67b60656c5ba2209459752cb@cogeco.ca> On Apr 23, 2006, at 6:32 PM, Crutcher Dunnavant wrote: > On 4/23/06, Jay Parlar wrote: >> >> I think what Ivan meant is that the normal Python rule is that if you >> have a full-colon (ANYWHERE), and you put something after it on the >> same line, then you're done the block. This behaviour is the same for >> method definitions, IF-blocks, the FOR statement, class definitions, >> etc. >> >> However, your change would be an exception to the rule, in that it >> would change what a full-colon is allowed to do, but only in one >> place. > > No, it wouldn't. First, this would apply to method and class defs as > well, so it's not an exception, secondly, the 'block ending thing > after the colon' would be _extended_ to cover other block defs. > What's the use case for doing this with method and class defs? Jay P. From crutcher at gmail.com Mon Apr 24 03:56:26 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Sun, 23 Apr 2006 18:56:26 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: <57f9be0f67b60656c5ba2209459752cb@cogeco.ca> References: <444BF6C2.2030507@fas.harvard.edu> <57f9be0f67b60656c5ba2209459752cb@cogeco.ca> Message-ID: On 4/23/06, Jay Parlar wrote: > > On Apr 23, 2006, at 6:32 PM, Crutcher Dunnavant wrote: > > > On 4/23/06, Jay Parlar wrote: > >> > >> I think what Ivan meant is that the normal Python rule is that if you > >> have a full-colon (ANYWHERE), and you put something after it on the > >> same line, then you're done the block. This behaviour is the same for > >> method definitions, IF-blocks, the FOR statement, class definitions, > >> etc. > >> > >> However, your change would be an exception to the rule, in that it > >> would change what a full-colon is allowed to do, but only in one > >> place. > > > > No, it wouldn't. First, this would apply to method and class defs as > > well, so it's not an exception, secondly, the 'block ending thing > > after the colon' would be _extended_ to cover other block defs. > > > > What's the use case for doing this with method and class defs? They're blocks, what would be the use case for having the for/while/with/if stuff be treated different from class/def ? Besides, it comes up in platform handling code. if sys.platform == 'foo': class Bar: ... class Foo: if sys.platform == 'foo': def bar(self): ... else: def bar(self): ... -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From talin at acm.org Mon Apr 24 04:04:29 2006 From: talin at acm.org (Talin) Date: Mon, 24 Apr 2006 02:04:29 +0000 (UTC) Subject: [Python-3000] Example implementation for string.format Message-ID: There have been a number of interesting suggestions as to whether string.format should support pipelined conversion specifiers, nested conversion specifiers, and so forth. I'm going to follow in Guido's lead at this point, and say that perhaps these kinds of decisions should be made after looking at a sample implementation. At the same time, I want it make it as easy as possible, so I'm going to post here a sample implementation to use as a starting point. Now, I'm not actually going to post a patch that adds a "format" method to the built-in string type. Instead, I am going to post a function that has the behavior that I am looking for. It's not the greatest Python code in the world, but that's not its purpose. I hacked this up over the course of about an hour, so its probably got a bug or two. In a real implementation, both the string.format and the MyFormatter.format functions would call this underlying 'engine' to do the work of parsing the field names and specifiers. Note: I decided to scan the string character by character rather than using regular expressions because of (a) the recursive nesting of braces, and (b) because something like this may go into the interpreter, and we don't want to add a dependency on re. Anyway, if you have an idea as to how things should behave differently - feel free to hack this, play with it, test out your idea, and then describe what you did. --- Talin ---------------------------------------------------------------------------- # Python string formatting # Except for errors in the format string. class FormatError(StandardError): pass def format(template, format_hook, *args, **kwargs): # Using array types since we're going to be growing # a lot. from array import array array_type = 'c' # Use unicode array if the original string is unicode. if isinstance(template, unicode): array_type = 'u' buffer = array(array_type) # Track which arguments actuallly got used unused_args = set(kwargs.keys()) unused_args.update(range(0, len(args))) # Inner function to format a field from a value and # conversion spec. Most details missing. def format_field(value, cspec, buffer): # See if there's a hook if format_hook and format_hook(value, cspec, buffer): return # See if there's a __format__ method elif hasattr(value, '__format__'): buffer.extend(value.__format__(cspec)) # Example built-in for ints. Probably should be # table driven by type, but oh well. elif isinstance(value, int): if cspec == 'x': buffer.extend(hex(value)) else: buffer.extend(str(value)) # Default to just 'str' else: buffer.extend(str(value)) # Parse a field specification. def parse_field(iterator, buffer): # A separate array for the field name. name = array(array_type) # Consume from the same iterator. for ch in iterator: # A sub-field. We just interpret it # like a normal field, and append to # the name. if ch == '{': parse_field(iterator, name) # End of field. Time to process elif ch == '}': # Convert the array to string or uni if array_type == 'u': name = name.tosunicode() else: name = name.tostring() # Check for conversion spec parts = name.split(':', 1) conversion = 's' if len(parts) > 1: name, conversion = parts # Try to retrieve the field value try: key = int(name) value = args[key] except ValueError: # Keyword args are strings, not uni (so far) key = str(name) value = kwargs[name] # If we got no exception, then remove from # unused args unused_args.remove(key) # Format it format_field(value, conversion, buffer) return elif ch == '\\': # Escape try: name.append(template_iter.next()) except StopIteration: # Backslash at end of string is bad raise FormatError("unmatched open brace") else: name.append(ch) raise FormatError("unmatched open brace") # Construct an iterator from the template template_iter = iter(template) for ch in template_iter: # It's a field! Yay! if ch == '{': parse_field(template_iter, buffer) elif ch == '}': # Unmatch brace raise FormatError("unmatched close brace") elif ch == '\\': # More escapism try: buffer.append(template_iter.next()) except StopIteration: # Backslash at end of string is OK here buffer.append(ch) break else: buffer.append(ch) # Complain about unused args if unused_args: raise FormatError( "Unused arguments: " + ",".join(str(x) for x in unused_args)) # Convert the array to its proper type if isinstance(template, unicode): return buffer.tounicode() else: return buffer.tostring() print format("This is a test of {0:x} {x} {1}\{", None, 1000, 20, x='hex'); From talin at acm.org Mon Apr 24 04:17:09 2006 From: talin at acm.org (Talin) Date: Mon, 24 Apr 2006 02:17:09 +0000 (UTC) Subject: [Python-3000] Stackable Blocks References: <444BF6C2.2030507@fas.harvard.edu> <57f9be0f67b60656c5ba2209459752cb@cogeco.ca> Message-ID: Crutcher Dunnavant gmail.com> writes: > > On 4/23/06, Jay Parlar cogeco.ca> wrote: > > > > What's the use case for doing this with method and class defs? > > They're blocks, what would be the use case for having the > for/while/with/if stuff be treated different from class/def ? Besides, > it comes up in platform handling code. > > if sys.platform == 'foo': class Bar: > ... > > class Foo: > if sys.platform == 'foo': def bar(self): > ... > > else: def bar(self): > ... It seems to me that the end result of what you are trying to accomplish here is to save a level of indentation. I know that it might seem logical to say "Well, if the 'for' keyword in a list comprehension has an 'if' clause, why shouldn't the regular 'for' keyword have an 'if' clause? The answer is that it already does. But because they are statements, rather than simply expressions, they need a little more 'elbow room' instead of trying to cram everything on one line. >From what I can see, the change that you propose doesn't allow you to do anything you can't do already, it just saves a few characters of typing. From what I have seen on these lists, proposals like that generally don't go over well. (Although if you can figure out a way to save a *whole lot* of typing, that might be a different story.) -- Talin From jparlar at cogeco.ca Mon Apr 24 07:16:10 2006 From: jparlar at cogeco.ca (Jay Parlar) Date: Sun, 23 Apr 2006 22:16:10 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: <444BF6C2.2030507@fas.harvard.edu> <57f9be0f67b60656c5ba2209459752cb@cogeco.ca> Message-ID: On Apr 23, 2006, at 6:56 PM, Crutcher Dunnavant wrote: >> > > They're blocks, what would be the use case for having the > for/while/with/if stuff be treated different from class/def ? Besides, > it comes up in platform handling code. > > if sys.platform == 'foo': class Bar: > ... > > class Foo: > if sys.platform == 'foo': def bar(self): > ... > > else: def bar(self): > ... > That's (IMHO) gross to look at. Too much information on one line (especially your sys.platform example). The indent will be all off: class Foo: if sys.platform == 'foo': def bar(self): do stuff Having the method name physically further to the right than the method body just doesn't feel Pythonic. The nice thing about indenting is that everything likes up how you'd except it, quick glance and you know what's what. Jay P. From ncoghlan at gmail.com Mon Apr 24 04:27:01 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 24 Apr 2006 12:27:01 +1000 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: Message-ID: <444C3775.9060108@gmail.com> Crutcher Dunnavant wrote: > Having been shot down so hard on my request to bring for loops into > symetry with list comprehensions, I have a new proposal - stackable > blocks. > > This is a very _small_ amount of syntax sugar, the basic idea is that > any block headers can be stacked on the same line. This changes no > semantics, nor does it change ordering, or add new keywords, it just > changes a tiny bit of the grammar for whitespace/block parsing. What's wrong with a bit of vertical whitespace? If your code blocks are cascading to the right of screen, it's a sign that something needs to be pulled out into a separate function or generator. > can be spelled like this. > > for file in open_files: if file.readable(): > ... def readable_files(files): for f in files: if f.readable(): yield f for f in readable_files(open_files): ... > can be spelled: > > for i in range(10): for j in range(20): if i != j: > ... def cartesian(x_coords, y_coords): for x in x_coords: for y in y_coords: yield x, y for i, j in cartesian(range(10), range(20)): if i != j: ... > can be this: > > for line in open(file): if line.strip(): > ... def stripped(f): for line in f: yield line.strip() for line in stripped(f): ... Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ianb at colorstudy.com Mon Apr 24 04:45:29 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Sun, 23 Apr 2006 21:45:29 -0500 Subject: [Python-3000] Open Issues for string.format PEP In-Reply-To: References: Message-ID: <444C3BC9.7020000@colorstudy.com> Talin wrote: > Here's a condensed list of the open issues that have been raised by people > so far: > > (Well, edited somewhat - I'm including 'major' issues, not minor nits -- we > can discuss those once the major issues are settled.) > > 1) Whether to drop the optional parts of the proposal: > -- support for field expressions (was: locals() support) > -- support for direct access to current scope (was: locals() support) It seems reasonable to implement this as a format() function right now, and maybe test it out on a couple modules. Doing so with or without special locals() support isn't too hard, and then it will be possible to see what change it makes in actual code. The benefit of locals() (and maybe expression) support is going to be when surrounding code can be easily removed -- the actual string substitution isn't going to look much better. The actual benefit is hard to judge without representative chunks of code. > 3) Role and usage of custom formatters: > > "string".fformat( formatter, ... ) > > vs. > > MyFormat( "string" ).format( ... ) MyFormat("string").format(...) doesn't seem right as a convention. I would assume my_formatter("string", ...), where my_formatter was built through subclassing (but doesn't really need to look like a class or instance). I guess I don't see the need for string.Template-style invocation, even if it uses string.Template-style subclassing. > 4) Should there be a way to pass in a dict argument without flattening it via > **args? I think it might not be ambiguous in practice if the first argument could be a dictionary. Because you get: "{foo}".format(a_dict) Since 'foo' isn't an integer and *no* keyword arguments were passed, you assume a_dict is a dictionary. Already "{0}" is going to look for positional arguments even if there is a dictionary with a '0' key, and I think that's just a reasonable limitation. Then there's the case: "{0} is {result}".format(a_dict) Is this an error? I'm not sure; it doesn't really need to be, it'd just evaluate to "{'result': 5} is 5" or something. Potentially this would be a problem, though: "{0} is {result}".format(value, **vars) When vars was {}... statically it would be clear that you intended {result} to come from vars, but at runtime you couldn't detect that. But I can't really imagine where that would happen, especially if there's no support for KeyError. > 5) Should the implementation attempt to detect unused arguments? Of course, just unused positional arguments would be detected. Putting the dict as the first argument might cause a problem here. -- Ian Bicking | ianb at colorstudy.com | http://blog.ianbicking.org From talin at acm.org Mon Apr 24 08:40:15 2006 From: talin at acm.org (Talin) Date: Mon, 24 Apr 2006 06:40:15 +0000 (UTC) Subject: [Python-3000] Open Issues for string.format PEP References: <444C3BC9.7020000@colorstudy.com> Message-ID: Ian Bicking colorstudy.com> writes: > > Talin wrote: > > Here's a condensed list of the open issues that have been raised by people > > so far: > > > > (Well, edited somewhat - I'm including 'major' issues, not minor nits -- we > > can discuss those once the major issues are settled.) > > > > 1) Whether to drop the optional parts of the proposal: > > -- support for field expressions (was: locals() support) > > -- support for direct access to current scope (was: locals() support) > > It seems reasonable to implement this as a format() function right now, > and maybe test it out on a couple modules. Doing so with or without > special locals() support isn't too hard, and then it will be possible to > see what change it makes in actual code. > > The benefit of locals() (and maybe expression) support is going to be > when surrounding code can be easily removed -- the actual string > substitution isn't going to look much better. The actual benefit is > hard to judge without representative chunks of code. Yeah, the more I think about it, the more the issue of supporting locals and field expressions seems like a whole different feature, deserving of its own PEP and its own mailing list flamewars :) I'm going to omit those sections from the next revision of the PEP. > > 3) Role and usage of custom formatters: > > > > "string".fformat( formatter, ... ) > > > > vs. > > > > MyFormat( "string" ).format( ... ) > > MyFormat("string").format(...) doesn't seem right as a convention. I > would assume my_formatter("string", ...), where my_formatter was built > through subclassing (but doesn't really need to look like a class or > instance). I guess I don't see the need for string.Template-style > invocation, even if it uses string.Template-style subclassing. The sample code I posted in another thread is intended to be a base for people to experiment with various calling conventions. For me, the most important aspects of custom formatters are: 1) I don't want to lose the feature of being able to specify alternate per-field formatting. 2) The built-in 'format' method of strings and the custom formatters should share as much code as possible. I'm thinking that both should just call a common formatting engine that is used for both. Other than that, I don't care what the calling syntax looks like. 3) I'd still like to be able to use custom formatters for things like indentation, variable-width padding, and such. Assuming that those constraints can be met, I'm still interested in what kinds of calling conventions people would prefer. (BTW, the name 'fformat' was chosen using the following rule: 'If you can't think of a good name, at least think of something that's easy to type'. You can always backfill an explanation later...) > > 4) Should there be a way to pass in a dict argument without flattening it via > > **args? > > I think it might not be ambiguous in practice if the first argument > could be a dictionary. Because you get: > > "{foo}".format(a_dict) > > Since 'foo' isn't an integer and *no* keyword arguments were passed, you > assume a_dict is a dictionary. Already "{0}" is going to look for > positional arguments even if there is a dictionary with a '0' key, and I > think that's just a reasonable limitation. Then there's the case: > > "{0} is {result}".format(a_dict) > > Is this an error? I'm not sure; it doesn't really need to be, it'd just > evaluate to "{'result': 5} is 5" or something. Potentially this would > be a problem, though: > > "{0} is {result}".format(value, **vars) > > When vars was {}... statically it would be clear that you intended > {result} to come from vars, but at runtime you couldn't detect that. > But I can't really imagine where that would happen, especially if > there's no support for KeyError. Waaaayy too much voodoo. I think that this kind of implicit deduction of "what the programmer meant" is what people are trying to avoid. (And, if you'll forgive me for saying so, it's rather Perl-ish.) I want to go back for a moment to the original question. Whenever someone asks me "how should we solve this problem?" the first thing I always ask is "do we really need to?" Once we've answered that, we can then go on to the issue of coming up with a solution. > > 5) Should the implementation attempt to detect unused arguments? > > Of course, just unused positional arguments would be detected. Putting > the dict as the first argument might cause a problem here. True. In general, if you pass a un-flattened dict, you probably want to disable the unused argument checking altogether, because the whole reason for passing a dict in the first place is so that you can cherry-pick specific values out of it. -- Talin From tjreedy at udel.edu Mon Apr 24 08:57:30 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 24 Apr 2006 02:57:30 -0400 Subject: [Python-3000] Brainstorming: Python Metaprogramming References: Message-ID: "Talin" wrote in message news:loom.20060423T222618-765 at post.gmane.org... > For some reason, I have stuck in my head the idea that this concept of > 'quoting' > is central to the whole business I agree. And in Python, the main means of implicit (unwritten) quoting is to use a statement instead of an expression. For instance, a=b <=> set('a', b) and import xyz <=> xyz = __import__('xyz'). In a def statement, everything is quoted *except* the default arg expressions. But I am not sure how this understanding helps you. Terry Jan Reedy From aurelien.campeas at logilab.fr Mon Apr 24 09:51:04 2006 From: aurelien.campeas at logilab.fr (=?iso-8859-1?Q?Aur=E9lien_Camp=E9as?=) Date: Mon, 24 Apr 2006 09:51:04 +0200 Subject: [Python-3000] Futures in Python 3000 In-Reply-To: <4448935F.20605@canterbury.ac.nz> References: <44454A3F.7070705@neotitans.com> <20060418201344.A820.JCARLSON@uci.edu> <4445CD94.1000409@neotitans.com> <4445DA36.9080509@canterbury.ac.nz> <4446765C.2030306@neotitans.com> <44473F6F.9020809@canterbury.ac.nz> <20060420085750.GD7995@crater.logilab.fr> <4448935F.20605@canterbury.ac.nz> Message-ID: <20060424075104.GA2317@logilab.fr> On Fri, Apr 21, 2006 at 08:10:07PM +1200, Greg Ewing wrote: > Aur?lien Camp?as wrote: > > On Thu, Apr 20, 2006 at 07:59:43PM +1200, Greg Ewing wrote: > > > Andy Sy wrote: > > > > Does this mean that Py3K intends to reuse major portions of > > > > Python 2.x's implementation? > > > I expect that almost all of it will be reused. > > > > Couldn't PyPy be considered an interesting contender ? > > That would be far too big a change. Last time I read > anything about PyPy, it was even more head-exploding > than metaclasses. I just don't understand the relationship with metaclasses. What's so head-exploding about them btw ? They need some thinking for sure ... AFAIK PyPy is not that complicated (I'm working on it btw), it only lacks some documentation currently. It's quite easy to add some features right now (hint : hooks for constraint & logic programming for instance) that I suspect would be no-go with CPython. > > > Sorry to jump in like this, but what about the *current* stackless ? > > Does it also break all existing C extensions ... ? > [...] > > -- > Greg > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/aurelien.campeas%40logilab.fr From birchb at tpg.com.au Mon Apr 24 16:09:03 2006 From: birchb at tpg.com.au (Bill Birch) Date: Tue, 25 Apr 2006 00:09:03 +1000 Subject: [Python-3000] Unsure if I should write a PEP on Types Message-ID: <200604250009.04347.birchb@tpg.com.au> The optional static typing blog http://www.artima.com/weblogs/viewpost.jsp?thread=87182 and many other postings imply a lot of interesting detail about types, type expression, type checking, adapt() and such. I'm considering collecting some of these subjects together in a note for review and perhaps as a PEP. Subjects would include: * Recap of type concepts and motivations: dynamic, subtyping, nominative, structural, duck. * Use cases for types: checking, introspection, test generation, OSTC * Type expressions * Run-time type lattice * Type class abstract interface: type operators * and some working code Has a PEP already been written? Would this be useful? Who else finds this interesting or may have started on these subjects? OK, so I'm fishing for encouragement here..... ;-) -- http://billbirch.wordpress.com/ From thomas at python.org Mon Apr 24 16:27:30 2006 From: thomas at python.org (Thomas Wouters) Date: Mon, 24 Apr 2006 16:27:30 +0200 Subject: [Python-3000] [Python-3000-checkins] r45689 - in python/branches/p3yk: Include/bytesobject.h Lib/test/test_bytes.py Lib/test/test_file.py Objects/bytesobject.c Objects/fileobject.c In-Reply-To: <20060424134706.EAF051E400A@bag.python.org> References: <20060424134706.EAF051E400A@bag.python.org> Message-ID: <9e804ac0604240727g44496509yd5f403e09269c031@mail.gmail.com> On 4/24/06, guido.van.rossum wrote: > + new->ob_size = size; > + if (size == 0) > + new->ob_bytes = NULL; > + else { > + new->ob_bytes = PyMem_Malloc(size); > + if (new->ob_bytes == NULL) { > + Py_DECREF(new); > + return NULL; > + } > + if (bytes != NULL) > + memcpy(new->ob_bytes, bytes, size); Hmmm... Should we see this as official confirmation that PEP 7 is going to go to 4-space indents instead of tab ones, or is it just an editor accidentily left in Google-indent mode? (I think I prefer the former ;) -- Thomas Wouters Hi! I'm a .signature virus! copy me into your .signature file to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060424/e02003d9/attachment.htm From gvwilson at cs.utoronto.ca Mon Apr 24 16:16:42 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Mon, 24 Apr 2006 10:16:42 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: Hi Guido, > On 4/24/06, Greg Wilson wrote: > > On a completely different note, are you willing to consider native > > syntax for sets (as per PEP 218) in Python 3000? > I like to write {1, 2, 3} instead of set([1, 2, 3]) but I don't know > what to do with the ambiguity for {}. I believe you proposed {-} at the > time which is actually hard to parse (because the '-' looks like the > start of an expression and with our LL(1) parser there's no way back). > Ideally perhaps {:} would be an empty dict and {} an empty set -- but > that will take some getting used to (and would probably break > compatibility with JSON). Perhaps {/} for an empty set? I'd hate to have > to say set(). ABC solved this by having a special empty object that > could become either a list or a table, but I don't think that'll work > for us -- the implementation would have to have compatible memory > lay-outs. I'm sure we can work something out --- I agree, {} for empty set and {:} for empty dict would be ideal, were it not for backward compatibility. I liked the "special empty object" idea when I first wrote the PEP (i.e., have {} be something that could turn into either a set or dict), but one of the instructors here convinced me that it would just lead to confusion in newcomers' minds (as well as being a pain to implement). If anyone else on the P3K list is interested in discussing this, please email me directly, and we'll see what we can hammer out. Would getting a trial implementation done on top of P2.5 be a good Google Summer of Code project? Thanks, Greg From barry at python.org Mon Apr 24 17:19:44 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 24 Apr 2006 11:19:44 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: <1145891984.8218.180.camel@resist.wooz.org> On Mon, 2006-04-24 at 10:16 -0400, Greg Wilson wrote: > I'm sure we can work something out --- I agree, {} for empty set and {:} > for empty dict would be ideal, were it not for backward compatibility. I dunno, the more I write dict() and set() the more I kind of like them rather the literal alternatives (at least for the former). -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060424/7eb41084/attachment.pgp From guido at python.org Mon Apr 24 17:35:17 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Apr 2006 08:35:17 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: On 4/24/06, Greg Wilson wrote: > Would getting a > trial implementation done on top of P2.5 be a good Google Summer of Code > project? Mmm, seems a rather small project to me to spend a whole summer... -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Mon Apr 24 17:44:06 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 10:44:06 -0500 Subject: [Python-3000] Brainstorming: Python Metaprogramming In-Reply-To: References: Message-ID: <444CF246.7000104@colorstudy.com> Talin wrote: > It seems that the history of the Python mailing lists are littered with the > decayed corpses of various ideas related to "metaprogramming", that is, programs > that write programs, either at compile time. > > We've seen proposals for C-style macros, Lisp-style macros, programmable syntax > (guilty!), AST access, a first-class symbol type, and much more. > > Given how many times this has been suggested, I really do think that there is > something there; At the same time, however, I recognize that all of these > proposals are irrepairably flawed in one way or another. > > I think that much of the reason for this, is that the various proposals haven't > really been distilled down to their absolute minimum essentials, which > admittedly is a hard thing to do. Instead, they offer to import a whole, > pre-existing architecture from some other language, which transforms Python into > something that it is not. > > As an example, take Lisp-style macros. A macro in Lisp is a function that takes > its arguments in unevaluated form. So when you call "mymacro( (add a b) ), you > don't get the sum of a and b, you get a list consisting of three elements. The > macro can then manipulate that list, and then evaluate it after it has been > manipulated. > > The reason this is possible is because in Lisp, there's no difference between > the AST (so to speak) and regular data. > > This fails in Python for two reasons: > > 1) For reasons of performance, the compiled code doesn't look very much like > regular data, and is hard to manipulate. > > 2) Most of the things that you might want to do with Lisp macros you can already > do in Python using some other technique. > > Using lambda, generators, operator overloading, and other Python features, we > can effectively 'quote' a section of code or an algorithm, and manipulate it as > data. No, you can't assemble arbitrary blocks of code, but most of the time you > don't want to. > > Using overloaded Python operators, we can in fact do something very like the > Lisp macro - that is, by replacing the '+' operator __add__, we can have it > return an AST-like tree of objects, rather than carrying out an actual addition. > However, this only works if you actually have control over the types being > added. As we've seen in SQLObject, this limitation leads to some interesting > syntactical contortions, where you need to insure that at least one of the two > objects being added knows about the overloaded operator. > > So one question to ask is - what can the Lisp macro system do that is (a) > useful, and (b) not already doable in Python, and (c) minimal enough that it > wouldn't cause a major rethink of the language? And the same question can be > asked for all of the other proposals. > > For some reason, I have stuck in my head the idea that this concept of 'quoting' > is central to the whole business. In Lisp the term 'quote' means to supress > evaluation of an item. So (quote a) returns to the symbol 'a', not the value > stored in 'a'. It is the ability to refer to a thing that would normally be > executed in its pre-executed state. In Python, we can already quote expressions, > using lambda; we can quote loops, using generators; and so on. > > However, one piece that seems to be missing is the ability to quote references > to global and local variables. In Python, the way to refer to a variable by name > is to pass its name as a string. The problem with this, however, is that a > string is a type in its own right, and has a whole different set of methods and > behaviors than a variable reference. > > As a hypothetical example, supposed we defined a unary operator '?' such that: > > ?x > > was syntactic sugar for: > > quoted('x') > > or even: > > quoted('x', (scope where x is or would be defined) ) > > Where 'quoted' was some sort of class that behaved like a reference to a > variable. So ?x.set( 1 ) is the same as x = 1. Sounds like lambda x: ... > Moreover, you would want to customize all of the operators on quoted to return > an AST, so that: > > ?x + 1 > > produces something like: > > (add, quoted('x'), 1) > > ...or whatever data structure is convenient. You can match the free variables from the lambda arguments against the variables in the AST to get this same info. > Of course, one issue that immediately comes to mind is, where does the class > 'quoted' come from? Is it a globally defined class, or is it something that is > defined for a given scope? What if you need to use two different definitions for > 'quoted' within a single scope? > > For example, I can imagine that in the case of something like SQLObject, the > 'quoted' class would transform into an SQL variable reference. But what if you > wanted to use that along with the algebraic solver, which defines 'quoted' as > something very different? This isn't actually a problem currently. Right now you use Table.q.column (or Table.column could have worked, had I been aware of descriptors at the time). This has a lot more information in it than just a name. Where it fails is un-overridable operators like "and". > One idea, which is kind of strange (and impractical, but bear with me), is > inspired by C++. When you define a member function in C++, the function's > formal parameters are considered to be in the same scope as the function > body. So for example, instead of having to write: > > void MyClass::function( MyClass::tListType &l ); > > ...you can just write: > > void MyClass::function( tListType &l ); > > ...because all of the contents of MyClass are visible within the argument list. > > So the strange idea is that when a function is *called*, the calling parameters > are evaluated within a new, nested scope, that inherits from both the function > itself, and the calling code's scope. This would allow the called function to > overide the meaning of 'quoted' or other operators (in fact, it would solve the > SQLObject problem if it allowed built-in operators such as '+' to be redefined.) The C++ example is just lexical scoping of the identifiers in the signature. I think what you are suggesting is considerably different, because it involves changing the scoping of the actual runtime call. > As an example, the algebraic solver could override the quote operator for its > parameters: > > return solve( ?x + 0 ) > --> ?x Well... for the SQL case I'm thinking that generators might be a way to resolve this: objs = SQL((person, address) for person, address in [Person, Address] if person.address_id == address.id and address.zip == '50555') There's no natural way to order this, unfortunately, because "person" and "address" aren't available outside the scope of the generator. Instead it would require something rather lame, like "order_by=lambda p, a: (a.zip, p.lname)", and considerable work to merge that in with the generator (so that it could be executed in the database). For equation solving a generator isn't particularly nice... solve(x for x in RealNumbers if x ** 2 + 5 * x - 4 == 0) Well actually that isn't so bad... -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon Apr 24 17:53:39 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 10:53:39 -0500 Subject: [Python-3000] Changing order of class creation Message-ID: <444CF483.6030603@colorstudy.com> Here's something for the crazy idea bin... Right now class creation looks like: class X(base): y = 'z' This gets the things in the () (object), evaluates the body of the class in a new scope and captures that scope (namespace), and looks for a metaclass using a couple rules. Then it does: X = metaclass('X', (base,), {'y': 'z'}) What if instead it did: X = metaclass('X', (base,)) X.y = 'z' ? X could keep track of order if it wanted. Or not. There would be no conflict between metaclasses paying special attention to the constructor namespace, and later attribute setting on the same class (most metaprogramming techniques with metaclasses choose one or the other). It could make rebuilding a class easier (as in reload). If metaclass('X', (bases,)) returned the already-existant X class, attributes will get folded in. (There's still problems there, but maybe this helps a little.) I haven't really thought through all the effects this might have. It does mean the underlying mechanism runs in the same order as the source, instead of the inverted order we currently have (where the body is evaluated before the class is created). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From jimjjewett at gmail.com Mon Apr 24 18:02:27 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 24 Apr 2006 12:02:27 -0400 Subject: [Python-3000] Minor hitch writing the Function Signature PEP In-Reply-To: References: <200604232211.29979.birchb@tpg.com.au> Message-ID: On 4/23/06, Talin wrote: >... you can't just do it for a single argument in isolation. You > have no idea where any positional argument is going to > go until all keyword arguments have been placed, and > until all previous positional arguments have > been placed. Yes you do. All positional arguments go to the first (length of positional arguments) arguments, in exactly that order. If there are leftoevers, they go to *args. If there is a shortage, then the ones at the end are not filled by positional. Then the remaining keyword arguments are filled (or allowed to default). If any remaining keyword attempts to name one of the positional arguments that was already filled, a SyntaxError or TypeError is raised. >>> def f(a, b, c=5): print a, b ,c >>> f(26,a=6) # enough arguments, but a is positional Traceback (most recent call last): File "", line 1, in -toplevel- f(26,a=6) TypeError: f() got multiple values for keyword argument 'a' >>> f(b=15, 4,3) # and the first argument *must* be first SyntaxError: non-keyword arg after keyword arg -jJ From aleaxit at gmail.com Mon Apr 24 18:15:53 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Mon, 24 Apr 2006 09:15:53 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <1145891984.8218.180.camel@resist.wooz.org> References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: On 4/24/06, Barry Warsaw wrote: > On Mon, 2006-04-24 at 10:16 -0400, Greg Wilson wrote: > > > I'm sure we can work something out --- I agree, {} for empty set and {:} > > for empty dict would be ideal, were it not for backward compatibility. > > I dunno, the more I write dict() and set() the more I kind of like them > rather the literal alternatives (at least for the former). I concur -- and I would add list(), too (e.g. to get a copy of somelist's contents, i find list(somelist) *MUCH* more readable than the traditional somelist[:]). I'd love for py3k to focus on 'nailing down [some] builtin names' enough to allow the compiler to specialize many cases of, e.g., dict(a=1,b=2) to make exactly the same bytecode as for {'a':1, 'b':2}, and so on -- I suspect that focusing efforts on that "nailing down" would have much vaster benefits than focusing them on all sorts of punctuation-heavy syntaxes. Alex From jimjjewett at gmail.com Mon Apr 24 18:22:11 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 24 Apr 2006 12:22:11 -0400 Subject: [Python-3000] PEP - string.format In-Reply-To: <4449DB4C.6050706@gmail.com> References: <4449DB4C.6050706@gmail.com> Message-ID: On 4/22/06, Nick Coghlan wrote: > Talin wrote: > > 2) Otherwise, see if the value to be formatted has a __format__ method. If > > it does, then call it. > So an object can override standard parsing like {0:d} to return something > other than an integer? *shudder* There are several unicode characters which serve as digits in other scripts. Even given that {0:d} means the string representing this integer in base 10, that may not specify the exact characters to return. This doesn't make it less scary, but it is a valid use case, if parsing can be overridden. -jJ From ianb at colorstudy.com Mon Apr 24 18:21:11 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 11:21:11 -0500 Subject: [Python-3000] Open Issues for string.format PEP In-Reply-To: References: <444C3BC9.7020000@colorstudy.com> Message-ID: <444CFAF7.2010407@colorstudy.com> Talin wrote: >>>4) Should there be a way to pass in a dict argument without flattening it via >>>**args? >> >>I think it might not be ambiguous in practice if the first argument >>could be a dictionary. Because you get: >> >> "{foo}".format(a_dict) >> >>Since 'foo' isn't an integer and *no* keyword arguments were passed, you >>assume a_dict is a dictionary. Already "{0}" is going to look for >>positional arguments even if there is a dictionary with a '0' key, and I >>think that's just a reasonable limitation. Then there's the case: >> >> "{0} is {result}".format(a_dict) >> >>Is this an error? I'm not sure; it doesn't really need to be, it'd just >>evaluate to "{'result': 5} is 5" or something. Potentially this would >>be a problem, though: >> >> "{0} is {result}".format(value, **vars) >> >>When vars was {}... statically it would be clear that you intended >>{result} to come from vars, but at runtime you couldn't detect that. >>But I can't really imagine where that would happen, especially if >>there's no support for KeyError. > > > Waaaayy too much voodoo. I think that this kind of implicit deduction of "what > the programmer meant" is what people are trying to avoid. (And, if you'll > forgive me for saying so, it's rather Perl-ish.) I don't think it's hard to tell in any specific circumstance what is happening; a programmer can statically determine what is happening easily given the actual method call and the template. > I want to go back for a moment to the original question. Whenever someone asks > me "how should we solve this problem?" the first thing I always ask is "do we > really need to?" Once we've answered that, we can then go on to the issue of > coming up with a solution. Without this all dictionaries must be copied. I think maybe Guido suggested something involving a shortcut that would not covert the dictionary, but that shortcut isn't available to Python so the behavior could be different for custom formatters implemented in Python. Also, only dictionaries are allowed after **; that could be changed (maybe that's part of the signature changes?), but that doesn't change the fact that only a dictionary can be passed into a function with **. In other words, avoiding a dictionary copy when using ** is something that requires terrible hacks and special cases, and *that* seems incredibly Perlish to me. So there's some overhead. That overhead could be considerable depending on the size of the namespace provided by the dictionary. But it's fixed. However, it pretty much ruins wrapping as a generalized strategy. I noted a couple here: http://mail.python.org/pipermail/python-3000/2006-April/000358.html Another wrapper could be a cascading dict: class CascadingDict(object): def __init__(self, *subdicts): self.subdicts = subdicts def __getitem__(self, key): for d in self.subdicts[:-1]: try: return d[key] except KeyError: pass return self.subdicts[-1][key] So you could do ''.format(CascadingDict(locals(), globals())). You could get a dictionary that doesn't require copying globals and locals, and instead delegates to a frame determined dynamically. You could look keys up in a database. You could look keys up in a translation database. You could allow '.', like: class DottedDict(object): def __init__(self, d): self.d = d def __getitem__(self, item): parts = item.split('.') ob = self.d[parts[0]] for attr in parts[1:]: ob = getattr(ob, attr) return ob There's lots and lots of ways of using this pattern. Both % and string.Template allow for this pattern. Python has been removing over time the need for namespaces to be plain dictionaries; adding another place where namespaces are constrained to plain dictionaries is a step backward. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon Apr 24 18:22:40 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 11:22:40 -0500 Subject: [Python-3000] Adding sorting to generator comprehension Message-ID: <444CFB50.6010809@colorstudy.com> In another thread ("Brainstorming: Python Metaprogramming") it occurred to me that many LINQish things are reasonable enough given the AST and generator expressinos. But sorting is hard to do. Sorting also remains a place where lambdas are still frequently needed, like: sorted(list_of_people, key=lambda p: (p.lname, l.fname)) We got rid of the lambda-encouraging map and filter, maybe one more? (p for p in list_of_people orderby (p.lname, p.fname)) I have no particular opinion on the keyword, though I assume a keyword is required; it may be difficult to find a keyword that people are not frequently using (but then this is py3k, so maybe not as big a deal). By including this in the generator expression AST introspection makes it possible to translate that to a SQL ORDER BY clause; I'm sure other out-of-Python query processors (like one of the numeric packages) could use this similarly. But, putting the AST stuff aside, I also think it is just nice syntax for a fairly common case, and a nice compliment for comprehensions. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From aleaxit at gmail.com Mon Apr 24 18:27:06 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Mon, 24 Apr 2006 09:27:06 -0700 Subject: [Python-3000] Changing order of class creation In-Reply-To: <444CF483.6030603@colorstudy.com> References: <444CF483.6030603@colorstudy.com> Message-ID: On 4/24/06, Ian Bicking wrote: > Here's something for the crazy idea bin... > > Right now class creation looks like: > > class X(base): > y = 'z' > > This gets the things in the () (object), evaluates the body of the class > in a new scope and captures that scope (namespace), and looks for a > metaclass using a couple rules. Then it does: > > X = metaclass('X', (base,), {'y': 'z'}) > > What if instead it did: > > X = metaclass('X', (base,)) > X.y = 'z' > > ? X could keep track of order if it wanted. Or not. There would be no > conflict between metaclasses paying special attention to the constructor > namespace, and later attribute setting on the same class (most > metaprogramming techniques with metaclasses choose one or the other). > It could make rebuilding a class easier (as in reload). If > metaclass('X', (bases,)) returned the already-existant X class, > attributes will get folded in. (There's still problems there, but maybe > this helps a little.) > > I haven't really thought through all the effects this might have. It > does mean the underlying mechanism runs in the same order as the source, > instead of the inverted order we currently have (where the body is > evaluated before the class is created). It appears to me that, in order to allow recreation of today's functionality in all cases, we *also* need a way to tell X, or probably better, X's metaclass, that ``the class statement is finished, please complete whatever it is you need to do to build X''. And it does seem perfectly reasonable to me that a metaclass may want to treat some X.a=... assignment coming "from" the class body, differently from one happening much later on. So, presumably, we need two ways to call the metaclass (or methods thereon), one for the initial creation and one for the "end of class statement" -- perhaps just two 'overloads' of the metaclass's __call__, one with two arguments (name and bases) for initial creation, one with just one argument (the X object as prepared so far) for end of class statement. Also, it seems we need some other way to pick the metaclass, to substitute for today's "assign to __metaclass__ in the classbody". And, I'd like (though it's not a must) some way to replicate today's functionality to "call the metaclass on the fly" to make a class. But if all of these nits can be solved, I do see the advantage on letting X "keep track of the order if it wants to". Alex From jack at performancedrivers.com Mon Apr 24 18:32:47 2006 From: jack at performancedrivers.com (Jack Diederich) Date: Mon, 24 Apr 2006 12:32:47 -0400 Subject: [Python-3000] Changing order of class creation In-Reply-To: <444CF483.6030603@colorstudy.com> References: <444CF483.6030603@colorstudy.com> Message-ID: <20060424163247.GA6301@performancedrivers.com> On Mon, Apr 24, 2006 at 10:53:39AM -0500, Ian Bicking wrote: > Here's something for the crazy idea bin... > > Right now class creation looks like: > > class X(base): > y = 'z' > > This gets the things in the () (object), evaluates the body of the class > in a new scope and captures that scope (namespace), and looks for a > metaclass using a couple rules. Then it does: > > X = metaclass('X', (base,), {'y': 'z'}) > > What if instead it did: > > X = metaclass('X', (base,)) > X.y = 'z' This would make overriding __new__ in a metaclass useless because if you are defining __new__ instead of __init__ you presumably want to return different things depending on the content of the namespace. > ? X could keep track of order if it wanted. Or not. There would be no > conflict between metaclasses paying special attention to the constructor > namespace, and later attribute setting on the same class (most > metaprogramming techniques with metaclasses choose one or the other). I've been working on a dict-alike that is used just for namespaces (py3k list[1]). I'm hoping it can be faster than a generic dict and also keep track of the order the keys are defined but faster is the main goal. Slow going, dicts have _a lot_ of methods to cover. > It could make rebuilding a class easier (as in reload). If > metaclass('X', (bases,)) returned the already-existant X class, > attributes will get folded in. (There's still problems there, but maybe > this helps a little.) Can't you do this already? -Jack [1] "Specializing the dicts in __dict__" http://article.gmane.org/gmane.comp.python.python-3000.devel/641/ From jimjjewett at gmail.com Mon Apr 24 18:44:44 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 24 Apr 2006 12:44:44 -0400 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: <444BF6C2.2030507@fas.harvard.edu> Message-ID: On 4/23/06, Jay Parlar wrote: > ... the normal Python rule is that if you have a full-colon > (ANYWHERE), and you put something after it on the > same line, then you're done the block. That would already be a slight improvement, from my perspective. Today, being done is required, but not sufficient. The improvement is only slight, because the use case is quick examples, and the extra regularity would be nice, but isn't required. >>> class C: def m(self): pass but >>> class C: def m(self): pass SyntaxError: invalid syntax -jJ From aleaxit at gmail.com Mon Apr 24 19:05:25 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Mon, 24 Apr 2006 10:05:25 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: On 4/24/06, Greg Wilson wrote: > Interesting --- I think that being able to write down a data structure > using the same sort of notation you'd use on a whiteboard in a high school > math class is one of the great strengths of scripting languages, and one > of the things that makes it possible to use Python, Perl, and Ruby as > configuration languages (instead of the XML that Java/C# users have to put > up with). I think most newcomers will find: > > x = {2, 3, 5, 7} > > more appealing than: > > x = set(2, 3, 5, 7) > > though I don't have any data to support that. It's exactly because we have no data that we can have a cool debate;-). Python doesn't mean to support set-theory notation -- it doesn't even have the most elementary operators, such as the epsilon-like thingy for membership test, the big-funky-U for union and reverse-that for intersection. Anybody expecting to use set-theory notation will be disappointed nearly instantly, just as soon as they try to DO anything with their sets, they'll have to use words rather than funky mathlike glyphs. So, since you have to code anyway "if z in " rather than "if z ", for example, then even for the "funky mathlike glyphlover" there's little added value if can be expressed with funky glyphs rather than spelled out into readable words. When it comes to operation, we support operator-glyphs such as & and |, which deviate from set-theoretical notation (and towards Boolean notation instead) in a way that I suspect might be even more irking to the set-theory-glyphs lover, and confusing to the high school student (who might get an F for confusing & with reverse-U-intersection, if his or her classes cover both settheory and Boolean logic -- though in the latter case they're more likely to have to use reverse-V for ``and'', and, once again, Python can't do that). IOW, it doesn't seem to me that the "high school whiteboard" test can be even very roughly approximated, so focusing on it for literals-notation only may do more harm than good; while introducing "nice readable words with a little punctuation to help out" from the very start (literals of most types, except numbers and strings) keeps things and expectations a bit more consistent with each other, as well as producing code that's easier to read out loud. Alex From jimjjewett at gmail.com Mon Apr 24 19:15:50 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 24 Apr 2006 13:15:50 -0400 Subject: [Python-3000] symbols? In-Reply-To: References: <4D1115E6-E5B4-4E15-B927-60E48C035492@monkeyfist.com> <443F4403.8090202@gmail.com> Message-ID: On 4/18/06, Guido van Rossum wrote: > On 4/18/06, Jim Jewett wrote: > Do you realize that you are > describing that the expression > .foo > is syntactic sugar for > > lambda self: self._foo() Somehow I had it in my head that I needed to also call that lambda, though I can't see why now. > That's not what I would call a "symbol type" by any stretch of the > imagination. Roughly, it would do the same as your property-with-strings enhancement, except that .foo would make it obvious that foo was a name (found on .), rather than an arbitrary string. I do see that as symbol-like, because it is a cross between a string (the method isn't resolved immediately) and a name (unquoted, limited to values that are valid for a name). -jJ From gvwilson at cs.utoronto.ca Mon Apr 24 18:44:45 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Mon, 24 Apr 2006 12:44:45 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: Interesting --- I think that being able to write down a data structure using the same sort of notation you'd use on a whiteboard in a high school math class is one of the great strengths of scripting languages, and one of the things that makes it possible to use Python, Perl, and Ruby as configuration languages (instead of the XML that Java/C# users have to put up with). I think most newcomers will find: x = {2, 3, 5, 7} more appealing than: x = set(2, 3, 5, 7) though I don't have any data to support that. Thanks, Greg On Mon, 24 Apr 2006, Alex Martelli wrote: > On 4/24/06, Barry Warsaw wrote: > > On Mon, 2006-04-24 at 10:16 -0400, Greg Wilson wrote: > > > > > I'm sure we can work something out --- I agree, {} for empty set and {:} > > > for empty dict would be ideal, were it not for backward compatibility. > > > > I dunno, the more I write dict() and set() the more I kind of like them > > rather the literal alternatives (at least for the former). > > I concur -- and I would add list(), too (e.g. to get a copy of > somelist's contents, i find list(somelist) *MUCH* more readable than > the traditional somelist[:]). > > I'd love for py3k to focus on 'nailing down [some] builtin names' > enough to allow the compiler to specialize many cases of, e.g., > dict(a=1,b=2) to make exactly the same bytecode as for {'a':1, 'b':2}, > and so on -- I suspect that focusing efforts on that "nailing down" > would have much vaster benefits than focusing them on all sorts of > punctuation-heavy syntaxes. > > > Alex > > From jcarlson at uci.edu Mon Apr 24 19:25:49 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 24 Apr 2006 10:25:49 -0700 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: <444CFB50.6010809@colorstudy.com> References: <444CFB50.6010809@colorstudy.com> Message-ID: <20060424102056.66CA.JCARLSON@uci.edu> Ian Bicking wrote: > > In another thread ("Brainstorming: Python Metaprogramming") it occurred > to me that many LINQish things are reasonable enough given the AST and > generator expressinos. But sorting is hard to do. Sorting also remains > a place where lambdas are still frequently needed, like: > > sorted(list_of_people, key=lambda p: (p.lname, l.fname)) > > We got rid of the lambda-encouraging map and filter, maybe one more? > > (p for p in list_of_people orderby (p.lname, p.fname)) > > I have no particular opinion on the keyword, though I assume a keyword > is required; it may be difficult to find a keyword that people are not > frequently using (but then this is py3k, so maybe not as big a deal). > By including this in the generator expression AST introspection makes it > possible to translate that to a SQL ORDER BY clause; I'm sure other > out-of-Python query processors (like one of the numeric packages) could > use this similarly. > > But, putting the AST stuff aside, I also think it is just nice syntax > for a fairly common case, and a nice compliment for comprehensions. One of the features of generator expressions which makes it desireable instead of list comprehensions is that generator expressions may use less memory *now*, and may be able to start returning results *now*. Using ( orderby ...) as a replacement for sorted((genexp), key=...) is a bit misleading because while the original generator expression could have been space and time efficient, the orderby version certainly may not be. - Josiah From skip at pobox.com Mon Apr 24 19:28:18 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 24 Apr 2006 12:28:18 -0500 Subject: [Python-3000] Unsure if I should write a PEP on Types In-Reply-To: <200604250009.04347.birchb@tpg.com.au> References: <200604250009.04347.birchb@tpg.com.au> Message-ID: <17485.2738.36966.171466@montanaro.dyndns.org> Bill> Has a PEP already been written? Would this be useful? Who else Bill> finds this interesting or may have started on these subjects? Bill> OK, so I'm fishing for encouragement here..... ;-) To very loosely paraphrase the late (some would say great) Johnny Cochran: When in doubt, a PEP will out. Skip From jimjjewett at gmail.com Mon Apr 24 19:31:35 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Mon, 24 Apr 2006 13:31:35 -0400 Subject: [Python-3000] Open Issues for string.format PEP In-Reply-To: References: Message-ID: On 4/22/06, Talin wrote: > Here's a condensed list of the open issues that have been raised by people > so far: I'm not sure whether this was raised explicitly, but Ian at least mentioned it: Are they keys to the dictionary in some way restricted? For instance, I think it would be reasonable to say that only valid names (or possibly expressions starting with a valid name) would be looked up in the dictionary, as a way of explaining why {0} means position zero, rather than dict['0']. -jJ From ianb at colorstudy.com Mon Apr 24 19:42:46 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 12:42:46 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: <444D0E16.9000508@colorstudy.com> Greg Wilson wrote: > Interesting --- I think that being able to write down a data structure > using the same sort of notation you'd use on a whiteboard in a high school > math class is one of the great strengths of scripting languages, and one > of the things that makes it possible to use Python, Perl, and Ruby as > configuration languages (instead of the XML that Java/C# users have to put > up with). I think most newcomers will find: > > x = {2, 3, 5, 7} > > more appealing than: > > x = set(2, 3, 5, 7) That looks fine to me, except of course it doesn't work :( Instead you get set([2, 3, 5, 7]), which is much less attractive and introduces an unneeded intermediate data structure. Or set((2, 3, 5, 7))... which is typographically prettier, but probably more confusing to a newbie. Generator comprehensions + dict() were a nice alternative to dict comprehension, and also replace the need for set comprehension. I feel like there might be some clever way to constructing sets? Not that there's any direct relation to generator expressions that I can see, but maybe something in the same vein. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From talin at acm.org Mon Apr 24 19:47:10 2006 From: talin at acm.org (Talin) Date: Mon, 24 Apr 2006 17:47:10 +0000 (UTC) Subject: [Python-3000] Open Issues for string.format PEP References: Message-ID: Jim Jewett gmail.com> writes: > > On 4/22/06, Talin acm.org> wrote: > > Here's a condensed list of the open issues that have been raised by people > > so far: > > I'm not sure whether this was raised explicitly, but Ian at least > mentioned it: Are they keys to the dictionary in some way restricted? > > For instance, I think it would be reasonable to say that only valid > names (or possibly expressions starting with a valid name) would be > looked up in the dictionary, as a way of explaining why {0} means > position zero, rather than dict['0']. Think 'keyword arguments', not 'dict keys'. You can't call a function with '0' as a keyword. -- Talin From guido at python.org Mon Apr 24 19:48:27 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Apr 2006 10:48:27 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: On 4/24/06, Alex Martelli wrote: > On 4/24/06, Greg Wilson wrote: > > Interesting --- I think that being able to write down a data structure > > using the same sort of notation you'd use on a whiteboard in a high school > > math class is one of the great strengths of scripting languages, and one > > of the things that makes it possible to use Python, Perl, and Ruby as > > configuration languages (instead of the XML that Java/C# users have to put > > up with). I think most newcomers will find: > > > > x = {2, 3, 5, 7} > > > > more appealing than: > > > > x = set(2, 3, 5, 7) > > > > though I don't have any data to support that. > > It's exactly because we have no data that we can have a cool debate;-). > > Python doesn't mean to support set-theory notation -- it doesn't even > have the most elementary operators, such as the epsilon-like thingy > for membership test, the big-funky-U for union and reverse-that for > intersection. Anybody expecting to use set-theory notation will be > disappointed nearly instantly, just as soon as they try to DO anything > with their sets, they'll have to use words rather than funky mathlike > glyphs. I totally disagree. There are different conventions for set-theory notation, and mapping the non-ASCII symbols on other operators is a standard convention (hey, we use * to mean multiplication!). Python has 'in' for ', '&' for and '|' for . That's a very sensible mapping (and I believe the latter two are common in programming languages, not just for Booleans, but for sets). I think that being able to write {1,2,3} instead of funky set([1,2,3]) would be a decent addition to the list. > So, since you have to code anyway "if z in " rather than "if > z ", for example, then even for the "funky > mathlike glyphlover" there's little added value if can be > expressed with funky glyphs rather than spelled out into readable > words. When it comes to operation, we support operator-glyphs such as > & and |, which deviate from set-theoretical notation (and towards > Boolean notation instead) in a way that I suspect might be even more > irking to the set-theory-glyphs lover, and confusing to the high > school student (who might get an F for confusing & with > reverse-U-intersection, if his or her classes cover both settheory and > Boolean logic -- though in the latter case they're more likely to have > to use reverse-V for ``and'', and, once again, Python can't do that). > > IOW, it doesn't seem to me that the "high school whiteboard" test can > be even very roughly approximated, so focusing on it for > literals-notation only may do more harm than good; while introducing > "nice readable words with a little punctuation to help out" from the > very start (literals of most types, except numbers and strings) keeps > things and expectations a bit more consistent with each other, as well > as producing code that's easier to read out loud. Reading out loud is a lost cause anyway -- you explicitly have to read all the squiggly marks to be the least unambiguous. Nobody said anything about high-school whiteboards. It's simply that sets will remain second-class citizens as long as we don't have a literal notation for them. {1,2,3} is certainly unambiguously the best notation for set literals -- and it isn't even ambiguous with dictionaries (nor hard to parse with our LL1 parser). The *only* point of contention (and AFAIK the only issue that Barry commented on) is that there's an ambiguity for the empty set, which *could* be resolved by continuing to use set() for that. I believe that mathematicians use a crossed-out little oh to indicate the empty set, so a notational discontinuity is not unheard of. OTOH mathematicians (whether in high school or not) write things like {x | 2 < x < 10}, which is of course the origin of our list comprehensions and generator expressions. Therefor I think it makes sense that {F(x) for x in S if P(x)}, ough to be valid syntax if we support {1, 2, 3} -- IOW the form {} should mean the same as set(). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ianb at colorstudy.com Mon Apr 24 19:51:53 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 12:51:53 -0500 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: <20060424102056.66CA.JCARLSON@uci.edu> References: <444CFB50.6010809@colorstudy.com> <20060424102056.66CA.JCARLSON@uci.edu> Message-ID: <444D1039.8050007@colorstudy.com> Josiah Carlson wrote: > One of the features of generator expressions which makes it desireable > instead of list comprehensions is that generator expressions may use > less memory *now*, and may be able to start returning results *now*. > > Using ( orderby ...) as a replacement for sorted((genexp), key=...) > is a bit misleading because while the original generator expression > could have been space and time efficient, the orderby version certainly may > not be. Certainly it changes the performance substantially (of course if the expression is translated and executed elsewhere the performance can actually be improved, so it can go both ways). Since list comprehensions are planned to just be syntactic sugar for generator comprehension, generator comprehensions are now the more fundamental construct. But yeah, it is a little awkward, since something that is sorted can be returned as a list anyway, except for the fact that the expression itself could be ported off elsewhere which isn't normal Python. (But should be normal Python!) -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Mon Apr 24 19:52:09 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Mon, 24 Apr 2006 12:52:09 -0500 Subject: [Python-3000] Open Issues for string.format PEP In-Reply-To: References: Message-ID: <444D1049.3060904@colorstudy.com> Talin wrote: >>>Here's a condensed list of the open issues that have been raised by people >>>so far: >> >>I'm not sure whether this was raised explicitly, but Ian at least >>mentioned it: Are they keys to the dictionary in some way restricted? >> >>For instance, I think it would be reasonable to say that only valid >>names (or possibly expressions starting with a valid name) would be >>looked up in the dictionary, as a way of explaining why {0} means >>position zero, rather than dict['0']. > > > Think 'keyword arguments', not 'dict keys'. You can't call a function with '0' > as a keyword. Sure you can, func(**{'0': 'x'}). Anytime you use ** as a first-class way of passing in a dictionary argument, keywords that are not valid Python identifiers will get passed in, often intentionally. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From aleaxit at gmail.com Mon Apr 24 20:03:48 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Mon, 24 Apr 2006 11:03:48 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: On 4/24/06, Guido van Rossum wrote: > On 4/24/06, Alex Martelli wrote: > > On 4/24/06, Greg Wilson wrote: > > > Interesting --- I think that being able to write down a data structure > > > using the same sort of notation you'd use on a whiteboard in a high school ... > > IOW, it doesn't seem to me that the "high school whiteboard" test can > > be even very roughly approximated, so focusing on it for ... > Nobody said anything about high-school whiteboards. Greg just did, and I had quoted that phrase -- see above. > sense that {F(x) for x in S if P(x)}, ough to be valid syntax if we > support {1, 2, 3} -- IOW the form {} should mean the same as > set(). I dislike that as much as I dislike [] as a shorthand for list(), but I have no trouble admitting that if we have the [...] form, it's consistent to have the {...} one too. Alex From guido at python.org Mon Apr 24 20:06:50 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Apr 2006 11:06:50 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: On 4/24/06, Alex Martelli wrote: > I dislike that as much as I dislike [] as a shorthand for > list(), but I have no trouble admitting that if we have the > [...] form, it's consistent to have the {...} one too. I think you're atypical in that dislike. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon Apr 24 20:18:11 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Apr 2006 11:18:11 -0700 Subject: [Python-3000] [Python-3000-checkins] r45689 - in python/branches/p3yk: Include/bytesobject.h Lib/test/test_bytes.py Lib/test/test_file.py Objects/bytesobject.c Objects/fileobject.c In-Reply-To: <9e804ac0604240727g44496509yd5f403e09269c031@mail.gmail.com> References: <20060424134706.EAF051E400A@bag.python.org> <9e804ac0604240727g44496509yd5f403e09269c031@mail.gmail.com> Message-ID: On 4/24/06, Thomas Wouters wrote: > > On 4/24/06, guido.van.rossum wrote: > > + new->ob_size = size; > > + if (size == 0) > + new->ob_bytes = NULL; > + else { > + new->ob_bytes = PyMem_Malloc(size); > + if (new->ob_bytes == NULL) { > + Py_DECREF(new); > + return NULL; > + } > + if (bytes != NULL) > + memcpy(new->ob_bytes, bytes, size); > > Hmmm... Should we see this as official confirmation that PEP 7 is going to > go to 4-space indents instead of tab ones, or is it just an editor > accidentily left in Google-indent mode? (I think I prefer the former ;) The former. I thought it was already decided? Of course, only for *new* code so far, to ease merges. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Mon Apr 24 20:34:43 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 24 Apr 2006 14:34:43 -0400 Subject: [Python-3000] [Python-3000-checkins] r45689 - in python/branches/p3yk: Include/bytesobject.h Lib/test/test_bytes.py Lib/test/test_file.py Objects/bytesobject.c Objects/fileobject.c In-Reply-To: References: <20060424134706.EAF051E400A@bag.python.org> <9e804ac0604240727g44496509yd5f403e09269c031@mail.gmail.com> Message-ID: <1145903683.13310.230.camel@resist.wooz.org> On Mon, 2006-04-24 at 11:18 -0700, Guido van Rossum wrote: > > Hmmm... Should we see this as official confirmation that PEP 7 is going to > > go to 4-space indents instead of tab ones, or is it just an editor > > accidentily left in Google-indent mode? (I think I prefer the former ;) > > The former. I thought it was already decided? Yes, it was. > Of course, only for > *new* code so far, to ease merges. Definitely, although there should be a flag day at some point where we reformat the old code, which I definitely think we should do, as painful as it might be. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060424/4b59c058/attachment.pgp From barry at python.org Mon Apr 24 20:43:45 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 24 Apr 2006 14:43:45 -0400 Subject: [Python-3000] [Python-3000-checkins] r45689 - in python/branches/p3yk: Include/bytesobject.h Lib/test/test_bytes.py Lib/test/test_file.py Objects/bytesobject.c Objects/fileobject.c In-Reply-To: References: <20060424134706.EAF051E400A@bag.python.org> <9e804ac0604240727g44496509yd5f403e09269c031@mail.gmail.com> Message-ID: <1145904225.13311.235.camel@resist.wooz.org> On Mon, 2006-04-24 at 11:18 -0700, Guido van Rossum wrote: > The former. I thought it was already decided? Of course, only for > *new* code so far, to ease merges. Here's a tiny bit of Emacs Lisp that should support the py3k style. Only moderately tested in XEmacs 21.5. -Barry (defconst py3k-style '("python" (indent-tabs-mode . nil) (c-basic-offset . 4) )) (if (not (assoc "py3k" c-style-alist)) (c-add-style "py3k" py3k-style)) -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060424/d06d0bdb/attachment.pgp From guido at python.org Mon Apr 24 20:44:17 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Apr 2006 11:44:17 -0700 Subject: [Python-3000] [Python-3000-checkins] r45689 - in python/branches/p3yk: Include/bytesobject.h Lib/test/test_bytes.py Lib/test/test_file.py Objects/bytesobject.c Objects/fileobject.c In-Reply-To: <1145903683.13310.230.camel@resist.wooz.org> References: <20060424134706.EAF051E400A@bag.python.org> <9e804ac0604240727g44496509yd5f403e09269c031@mail.gmail.com> <1145903683.13310.230.camel@resist.wooz.org> Message-ID: On 4/24/06, Barry Warsaw wrote: > On Mon, 2006-04-24 at 11:18 -0700, Guido van Rossum wrote: > > > > Hmmm... Should we see this as official confirmation that PEP 7 is going to > > > go to 4-space indents instead of tab ones, or is it just an editor > > > accidentily left in Google-indent mode? (I think I prefer the former ;) > > > > The former. I thought it was already decided? > > Yes, it was. If it wasn't written down it will be forgotten. I'll update various PEPs. > > Of course, only for > > *new* code so far, to ease merges. > > Definitely, although there should be a flag day at some point where we > reformat the old code, which I definitely think we should do, as painful > as it might be. Doesn't have to be a flag day -- we can reformat code opportunistically whenever there's no hope of ever doing merges from 2.x. I do plan that within one file we'll stick to style consistency. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Mon Apr 24 20:49:24 2006 From: barry at python.org (Barry Warsaw) Date: Mon, 24 Apr 2006 14:49:24 -0400 Subject: [Python-3000] [Python-3000-checkins] r45689 - in python/branches/p3yk: Include/bytesobject.h Lib/test/test_bytes.py Lib/test/test_file.py Objects/bytesobject.c Objects/fileobject.c In-Reply-To: References: <20060424134706.EAF051E400A@bag.python.org> <9e804ac0604240727g44496509yd5f403e09269c031@mail.gmail.com> <1145903683.13310.230.camel@resist.wooz.org> Message-ID: <1145904564.8217.242.camel@resist.wooz.org> On Mon, 2006-04-24 at 11:44 -0700, Guido van Rossum wrote: > Doesn't have to be a flag day -- we can reformat code > opportunistically whenever there's no hope of ever doing merges from > 2.x. > > I do plan that within one file we'll stick to style consistency. Sounds good to me. I guess when someone is about to do significant work on a C file, we should ask on this list about that possibility, and if it's low, then we can reformat the file at that time. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060424/b0938296/attachment.pgp From jcarlson at uci.edu Mon Apr 24 20:55:20 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Mon, 24 Apr 2006 11:55:20 -0700 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: <444D1039.8050007@colorstudy.com> References: <20060424102056.66CA.JCARLSON@uci.edu> <444D1039.8050007@colorstudy.com> Message-ID: <20060424115248.66D3.JCARLSON@uci.edu> Ian Bicking wrote: > > Josiah Carlson wrote: > > One of the features of generator expressions which makes it desireable > > instead of list comprehensions is that generator expressions may use > > less memory *now*, and may be able to start returning results *now*. > > > > Using ( orderby ...) as a replacement for sorted((genexp), key=...) > > is a bit misleading because while the original generator expression > > could have been space and time efficient, the orderby version certainly may > > not be. > > Certainly it changes the performance substantially (of course if the > expression is translated and executed elsewhere the performance can > actually be improved, so it can go both ways). I think that discussions over farming out sorted(genexp) computation is a little premature, and I would hope that such would be explicit in any case. - Josiah From skip at pobox.com Mon Apr 24 21:28:58 2006 From: skip at pobox.com (skip at pobox.com) Date: Mon, 24 Apr 2006 14:28:58 -0500 Subject: [Python-3000] [Python-3000-checkins] r45689 - in python/branches/p3yk: Include/bytesobject.h Lib/test/test_bytes.py Lib/test/test_file.py Objects/bytesobject.c Objects/fileobject.c In-Reply-To: <1145904564.8217.242.camel@resist.wooz.org> References: <20060424134706.EAF051E400A@bag.python.org> <9e804ac0604240727g44496509yd5f403e09269c031@mail.gmail.com> <1145903683.13310.230.camel@resist.wooz.org> <1145904564.8217.242.camel@resist.wooz.org> Message-ID: <17485.9978.103055.242869@montanaro.dyndns.org> Barry> Sounds good to me. I guess when someone is about to do Barry> significant work on a C file, we should ask on this list about Barry> that possibility, and if it's low, then we can reformat the file Barry> at that time. I presume with the reformatting happening in a separate checkin so as not to obscure the real work... Skip From talin at acm.org Mon Apr 24 21:51:53 2006 From: talin at acm.org (Talin) Date: Mon, 24 Apr 2006 19:51:53 +0000 (UTC) Subject: [Python-3000] Open Issues for string.format PEP References: <444D1049.3060904@colorstudy.com> Message-ID: Ian Bicking colorstudy.com> writes: > > Talin wrote: > >>>Here's a condensed list of the open issues that have been raised by people > >>>so far: > >> > >>I'm not sure whether this was raised explicitly, but Ian at least > >>mentioned it: Are they keys to the dictionary in some way restricted? > >> > >>For instance, I think it would be reasonable to say that only valid > >>names (or possibly expressions starting with a valid name) would be > >>looked up in the dictionary, as a way of explaining why {0} means > >>position zero, rather than dict['0']. > > > > > > Think 'keyword arguments', not 'dict keys'. You can't call a function with '0' > > as a keyword. > > Sure you can, func(**{'0': 'x'}). Anytime you use ** as a first-class > way of passing in a dictionary argument, keywords that are not valid > Python identifiers will get passed in, often intentionally. > Well, let me phrase it another way. The question was, what limits should be placed on field identifiers, and more specifically, what is the rationale for distinguishing between '0' and 'name'? And my answer to this is, that the interpretation of field names should be the same as the interpretation of keyword arguments by the Python parser. -- Talin From tjreedy at udel.edu Mon Apr 24 22:22:59 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 24 Apr 2006 16:22:59 -0400 Subject: [Python-3000] sets in P3K? References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604241106x6719551eo5318aae9da8075b1 at mail.gmail.com... > On 4/24/06, Alex Martelli wrote: >> I dislike that as much as I dislike [] as a shorthand for >> list(), but I have no trouble admitting that if we have the >> [...] form, it's consistent to have the {...} one too. > > I think you're atypical in that dislike. Like Greg and you, I would like to see sets treated the same way they might have been if included from the beginning in 1.0. The current situation is only understandable knowing their history as an add-on reflecting a slight relaxation of initial minimalism. Terry Jan Reedy From brett at python.org Mon Apr 24 22:26:45 2006 From: brett at python.org (Brett Cannon) Date: Mon, 24 Apr 2006 13:26:45 -0700 Subject: [Python-3000] rough draft signature PEP In-Reply-To: References: Message-ID: On 4/22/06, Talin wrote: > Brett Cannon python.org> writes: > > > [I am posting to python-3000 since this is where the parameter list > > ideas are being discussed, but this is probably generic enough to > > eventually make it into the 2.x line] > > > > Here is a rough draft of a PEP I wrote last summer after I had Guido > > come for lunch at Google when I was interning there (i.e., a little > > old =) . Someone (I think Philip) had suggested something like this > > back then but it didn't go any farther. I liked the idea and I asked > > Guido at lunch if he was okay for it; he was. > > > > So I wrote the following PEP along with a Python implementation > > (attached, along with test cases). It is still a rough draft since I > > am on vacation on top of visiting my dad for his birthday and thus > > cannot put a ton of time into this at the moment, so don't consider > > this a final version in any way. There is already a list of things to > > consider and I am sure there are some things discussed during the new > > parameter list ideas that could stand to be worked in. I am quite > > happy to work on finishing this PEP so that Talin can focus on the > > parameter list PEP. So just comment away and I will work on > > incorporating them as time permits. > > Sounds good. A couple of comments: > > 1) I a still confused (see the "minor hitch" thread) as to how the signature > information can be used without either duplicating the interpreter's > parameter-binding algorithm, or somehow giving access to it. > > In other words, having access to signature information isn't very useful unless > there is a straightforward and efficient way to map incoming arguments to > specific signature slots. Because the decorators execute *before* this step > normally happens, they can't just look and see which particular slot an argument > got assigned to, they have to calculate it. > > The actual-to-formal mapping that's used in the interpreter itself can't be > directly exposed, because it's not data driven (apparently - this is one of the > parts of the intepreter that I don't understand yet.) > > A duplicate implementation suffers from two drawbacks: First, any implementation > will need to be kept in sync with the interpreter. Second, the data structure > that the mapping algorithm has to deal with (i.e. the signature object) is going > to be in a form that is less efficient (i.e. since it needs to be constructable > via Python code, it can't be a simple C structure like argument_ty.) > Honestly, I don't know how important this kind of functionality is. Is it really that important to know ahead of time how arguments will bind into the parameters? I originally came up with this for documentation/introspection purposes, not to provide a way for people to detect how bindings would end up. As for the duplication of code if a function to do bindings was to come into existence, that is just life. The C code used by the interpreter could possibly be factored out if needed, but the binding rules are not that complex as to need to worry too much about duplicating it, especially if the signature object is coded in Python. > 2) I noticed in your PEP that you followed the same mental model as is currently > used in the compiler, which is that *args and **args are treated as special > cases. In other words, the description of a function's arguments consists of an > array of "regular" arguments, plus a couple of "special" argument fields. > > My mental model is a little different, as I tend to see the argument as a single > list, where arguments have different modifiers, where * and ** are modifiers. In > other word, my model follows more closely (IMHO) the way that arguments are read > by the programmer. > Yeah, I see what you are saying. But I actually held this view long before Iearned how Python did things underneath the covers. I have no clue how people tend to view things. Anyone else care to comment on how they tend to see arguments? > To show the difference between the two models, lets create an algorithm that > populates the parameter slots from a set of input arguments. For simplicity, > lets just consider positional arguments, including varargs arguments. > > In the case where the varargs is considered a separate slot: > > i = 0 > while i < numInputArgs: > if i <= regularSlotCount: > slot[ i ] = args[ i ] > i += 1 > > if i < numInputArgs: > slot[ i ] = args[ i: ] > > In the case where there's a single list: > > i = 0 > s = 0 > while i < numInputArgs: > if modifiers[ s ] == VARARGS: > slot[ s ].append( args[ i ] ) > else: > slot[ s ] = args[ i ] > s += 1 > i += 1 > > (Note that neither of these functions is intended to be optimal.) > > I'm not saying that my model is necessarily better. Its just how I think about > it. The only advantage that I would claim is that it more readily allows for > future kinds of "special" arguments, by extending the meaning of the 'modifier' > attribute. > > Here's what I had written in the PEP: > > Specification > > The specification defines a new function object attribute, > __signature__, which contains a tuple containing a list of > parameter descriptions. > > Each parameter description will be a tuple containing the > following elements: > > [0] - The string name of the parameter. Don't forget that can be a tuple as well. > [1] - The Python type of the parameter, if a type has been > specified. If none has been specified, this field will > contain None. > [2] - The parameter modifier (see below.) > [3] - The default value of the parameter, if there is one. > > If there is no default value, then the 4th field will be absent - > in other words, it will be a 3-tuple rather than a 4-tuple. This > allows discrimination between a parameter whose default is 'None', > and a parameter with no default. > I would change it to have a boolean value to flag if there is a default and then have the field contain the default args since I would like it all to work nicely with a genexp and filtering on whether a field is True or False seems nicer than basing it on the length of the tuple. > The second argument, containing the Python type, will always > contain None until such time as a convention for specifying > explicit types is adopted. > > The parameter modifier is used to describe any modifier prefix > characters attached to the argument, such as '*args'. The > following table lists which values are currently defined: > > 0 - a regular argument (no modifiers) > 1 - a varargs argument (one star) > 2 - a keyword dict argument (two stars) > > Other values may be defined later as additions are made to the > language. As I said, I would like to hear how other people think. I have no problem wrapping my head around to think like the rest of you (if it happens to be the case) and view each argument as its own thing and rework how the object represents things. -Brett From tjreedy at udel.edu Mon Apr 24 22:36:12 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 24 Apr 2006 16:36:12 -0400 Subject: [Python-3000] Changing order of class creation References: <444CF483.6030603@colorstudy.com> Message-ID: "Ian Bicking" wrote in message news:444CF483.6030603 at colorstudy.com... metaclass using a couple rules. Then it does: > > X = metaclass('X', (base,), {'y': 'z'}) > > What if instead it did: > > X = metaclass('X', (base,)) > X.y = 'z' What if X is inmutable, or intended to be so, either by inheritance or by method overriding? tjr From steven.bethard at gmail.com Mon Apr 24 23:43:40 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 24 Apr 2006 15:43:40 -0600 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: Guido van Rossum wrote: > OTOH mathematicians (whether in high school or not) write things like > {x | 2 < x < 10}, which is of course the origin of our list > comprehensions and generator expressions. Therefor I think it makes > sense that {F(x) for x in S if P(x)}, ough to be valid syntax if we > support {1, 2, 3} -- IOW the form {} should mean the same as > set(). So then we'd have list comprehensions and set comprehensions, but no dict-comprehensions? Or was the intention to introduce one of the previously rejected dict-comprehension syntaxes? STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From rhettinger at ewtllc.com Tue Apr 25 00:09:09 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Mon, 24 Apr 2006 15:09:09 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: <444D4C85.3080404@ewtllc.com> Greg Wilson wrote: >I'm sure we can work something out --- I agree, {} for empty set and {:} >for empty dict would be ideal, were it not for backward compatibility. > For the record, I am strong -1 on adding a custom syntax for sets. IMO, most of the interesting uses of sets start with some external data source or some function which builds-up a dataset. Accordingly, the notation should be biased toward construction from iterables rather than for writing hard-coded literals. IOW, it is more important to be able to write set(data) than set(0,1,2). Proposals for set syntax all have some problems. It is a wart to invent a kludge to distinguish between an empty set and an empty dict with {} and {:}. Also, it is not obvious whether {0,1,2} would construct a set or a frozenset. In contrast, the current approach is unambiguous and completely clear even if you've been away from the language for a long time: frozenset(range(3)). Further, punctuation approaches face uncomfortable choices with respect to iterables. Would {range(3)} be allowed? If not, then you've lost a key bit of functionality and imposed an arbitrary restriction. If so, then do you allow something similar for dicts? If not, then you've introduced another arbitrary inconsistency. If so, then you face more challenges distinguishing between sets and dicts -- is {somegen()} a set or dict -- there are valid interpretations either way if somegen() returns tuples. With the collections module, we can expect a few more datatypes to be added over time (with ordered dicts being the next most likely addition). IMO, we gain greater consistency, learnability, and substitutability by sticking with uniform constructors not based on punctuation: datatype(iterable). I understand the temptation to save five characters but think that would be a foolish savings, trading readability and clarity for brevity. Also, I concur with Alex's thought that it is a lost cause to twist ourselves into knots trying to emulate the high-school-whiteboard. Likewise, I understand being bugged by bracketed set representation; however, it is unambiguous and not without precedent: array('i', [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]). IMO, adding punctuation will unnecessarily burden the learning curve. IIRC, this was previously discussed and settled on python-dev. Also, I believe similar thoughts were behind the effort to remove backticks as syntactic sugar for repr(). gratuitous-punctuation-is-the-road-to-hell-ly yours, Raymond From steven.bethard at gmail.com Tue Apr 25 00:48:35 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Mon, 24 Apr 2006 16:48:35 -0600 Subject: [Python-3000] sets in P3K? In-Reply-To: <444D4C85.3080404@ewtllc.com> References: <444D4C85.3080404@ewtllc.com> Message-ID: On 4/24/06, Raymond Hettinger wrote: > Greg Wilson wrote: > > >I'm sure we can work something out --- I agree, {} for empty set and {:} > >for empty dict would be ideal, were it not for backward compatibility. > > [snip] > Further, punctuation approaches face uncomfortable choices with respect > to iterables. Would {range(3)} be allowed? If not, then you've lost a > key bit of functionality and imposed an arbitrary restriction. If so, > then do you allow something similar for dicts? If not, then you've > introduced another arbitrary inconsistency. If so, then you face more > challenges distinguishing between sets and dicts -- is {somegen()} a set > or dict -- there are valid interpretations either way if somegen() > returns tuples. While I agree with all your other arguments (and I'm perfectly happy to write dict(), set() and list() when necessary), I think there's no reason to be confused about something like ``{range(3)}``. It would work exactly like list literals do: >>> [range(3)] [[0, 1, 2]] This is not an "arbitrary restriction" -- unless you believe that lists also have such an arbitrary restricion. STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From aahz at pythoncraft.com Tue Apr 25 01:05:10 2006 From: aahz at pythoncraft.com (Aahz) Date: Mon, 24 Apr 2006 16:05:10 -0700 Subject: [Python-3000] PEP3100 update: chained exceptions Message-ID: <20060424230510.GA6793@panix.com> The recent discussion about the with statement on python-dev reminded me that at one point we talked about making chained exceptions automatic. Could someone update PEP3100 to list chained exceptions as one of the features that needs discussion? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From guido at python.org Tue Apr 25 01:51:20 2006 From: guido at python.org (Guido van Rossum) Date: Mon, 24 Apr 2006 16:51:20 -0700 Subject: [Python-3000] PEP3100 update: chained exceptions In-Reply-To: <20060424230510.GA6793@panix.com> References: <20060424230510.GA6793@panix.com> Message-ID: On 4/24/06, Aahz wrote: > The recent discussion about the with statement on python-dev reminded me > that at one point we talked about making chained exceptions automatic. > Could someone update PEP3100 to list chained exceptions as one of the > features that needs discussion? It's not mentioned there now, is it? (If it is, I can't find it so a hint on the line number would be appreciated. :-) IMNSHO this feature (PEP 344) is dead unless someone can find a way around the excessive circular references involving locals that this feature will necessarily cause. (And no, handwaving solutions are not accepted -- you need to understand the cause of the circularities in the proposal and explain a detailed plan for getting rid of them. Relying on GC is unacceptable because of the effect on local variables -- while reference counting is technically a CPython implementation detail, many people have come to rely on the predictable finalization semantics and I don't want to brfeak this wholesale. We had this in early Python versions and it was a big pain.) -- --Guido van Rossum (home page: http://www.python.org/~guido/) From birchb at tpg.com.au Tue Apr 25 02:31:20 2006 From: birchb at tpg.com.au (Bill Birch) Date: Tue, 25 Apr 2006 10:31:20 +1000 Subject: [Python-3000] Unsure if I should write a PEP on Types Message-ID: <200604251031.20849.birchb@tpg.com.au> You wrote: > Bill> Has a PEP already been written? Would this be useful? Who else > Bill> finds this interesting or may have started on these subjects? > Bill> OK, so I'm fishing for encouragement here..... ;-) > >To very loosely paraphrase the late (some would say great) Johnny Cochran: > > When in doubt, a PEP will out. > >Skip "I'm a big believer in the fact that life is about peparation, peparation, peparation." Johnnie Cochran -- http://billbirch.wordpress.com/ From gvwilson at cs.utoronto.ca Tue Apr 25 03:35:21 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Mon, 24 Apr 2006 21:35:21 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: <444D0E16.9000508@colorstudy.com> References: <1145891984.8218.180.camel@resist.wooz.org> <444D0E16.9000508@colorstudy.com> Message-ID: > Ian Bicking: > Instead you get set([2, 3, 5, 7]), which is much less attractive and > introduces an unneeded intermediate data structure. Or set((2, 3, 5, > 7))... which is typographically prettier, but probably more confusing to > a newbie. > > Generator comprehensions + dict() were a nice alternative to dict > comprehension, and also replace the need for set comprehension. I feel > like there might be some clever way to constructing sets? Not that > there's any direct relation to generator expressions that I can see, but > maybe something in the same vein. One of the reasons I'd like native syntax for sets is that I'd like set comprehensions: a = {b for b in c where b > 0} may not quite be as beautiful as using epsilon for membership (sorry, Alex ;-), but it's still quite nice. Greg From gvwilson at cs.utoronto.ca Tue Apr 25 03:39:35 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Mon, 24 Apr 2006 21:39:35 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> Message-ID: > > Guido: > > sense that {F(x) for x in S if P(x)}, ough to be valid syntax if we > > support {1, 2, 3} -- IOW the form {} should mean the same as > > set(). > Alex: > I dislike that as much as I dislike [] as a shorthand for > list(), but I have no trouble admitting that if we have the > [...] form, it's consistent to have the {...} one too. Greg: So it really does come down to finding an unlikely-to-confuse notation for the empty set? If people would like to send me suggestions, I can try an experiment in class (I start teaching a new crop of second-year students Python in May). Thanks, Greg From aleaxit at gmail.com Tue Apr 25 04:17:45 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Mon, 24 Apr 2006 19:17:45 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> <444D0E16.9000508@colorstudy.com> Message-ID: <73927E8F-3CD0-4DF6-8DF7-FB6094C69C03@gmail.com> On Apr 24, 2006, at 6:35 PM, Greg Wilson wrote: >> Ian Bicking: >> Instead you get set([2, 3, 5, 7]), which is much less attractive and >> introduces an unneeded intermediate data structure. Or set((2, 3, 5, >> 7))... which is typographically prettier, but probably more >> confusing to >> a newbie. >> >> Generator comprehensions + dict() were a nice alternative to dict >> comprehension, and also replace the need for set comprehension. I >> feel >> like there might be some clever way to constructing sets? Not that >> there's any direct relation to generator expressions that I can >> see, but >> maybe something in the same vein. > > One of the reasons I'd like native syntax for sets is that I'd like > set > comprehensions: > > a = {b for b in c where b > 0} > > may not quite be as beautiful as using epsilon for membership > (sorry, Alex > ;-), but it's still quite nice. I think set(b for b in c if b > 0) is much nicer -- two separate issues, using 'where' instead of 'if' doesn't seem to buy anything, and the distinct one of using {} rather than set(), and on EACH of them I strongly prefer today's syntax. Alex From kayschluehr at gmx.de Tue Apr 25 07:20:13 2006 From: kayschluehr at gmx.de (Kay Schluehr) Date: Tue, 25 Apr 2006 07:20:13 +0200 Subject: [Python-3000] Give power to __getattr__ Message-ID: <444DB18D.1030004@gmx.de> While observing heated discussions about adaption and generic functions I wonder if this is a good opportunity for a modest complaint about the castration of __getattr__ in new style classes. What are you doing if you want to adapt a new style class X to a class that has implemented e.g. __add__ or any other special function, you might not yet know? You can't simply delegate the call to the wrapper but you have to implement __add__ in X by hand and then delegate ( or create it explicitely using the metaclass protocol but that's less flexible and if the target class of your adaption is changed at runtime you get lost ) otherwise you keep a type-error. Special methods become very special so to say. I would like to ask for a little un-specialization of special methods in this respect and making the language a little more symmetric. From ncoghlan at gmail.com Tue Apr 25 07:38:48 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 25 Apr 2006 15:38:48 +1000 Subject: [Python-3000] Give power to __getattr__ In-Reply-To: <444DB18D.1030004@gmx.de> References: <444DB18D.1030004@gmx.de> Message-ID: <444DB5E8.10706@gmail.com> Kay Schluehr wrote: > While observing heated discussions about adaption and generic functions > I wonder if this > is a good opportunity for a modest complaint about the castration of > __getattr__ in new style classes. If you want to get full control back, that's what __getattribute__ is for. . . Or are you asking for better access to the bits and pieces of object.__getattribute__ so that it's easier to customise the operation? Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From kendall at monkeyfist.com Tue Apr 25 14:39:03 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Tue, 25 Apr 2006 08:39:03 -0400 Subject: [Python-3000] Brainstorming: literal construction hooks In-Reply-To: <200604232228.55500.birchb@tpg.com.au> References: <200604232228.55500.birchb@tpg.com.au> Message-ID: <19F83ED2-DBE5-4F72-9AD8-74D4B94A23B0@monkeyfist.com> On Apr 23, 2006, at 8:28 AM, Bill Birch wrote: > On Sun, 23 Apr 2006 12:53 am, Fredrik Lundh wrote: >> Guido van Rossum wrote: >>> It's an interesting idea; it's been brought up before but nobody >>> AFAIK >>> has ever implemented it. I like the scoping requirement. >> >> can we add support for XML literals while we're at it? >> >> http://effbot.org/zone/idea-xml-literal.htm >> > Have a look at Comega > http://research.microsoft.com/Comega/ > which throws in SQL syntax just for good measure. Actually, I believe that if any project is especially interesting in re: Py3k and XML literals, it's LINQ: http://msdn.microsoft.com/netframework/future/linq/ As I understand things (since I'm really a Unix guy), the Comega stuff got folded into LINQ. Cheers, Kendall From bioinformed at gmail.com Tue Apr 25 15:07:02 2006 From: bioinformed at gmail.com (Kevin Jacobs ) Date: Tue, 25 Apr 2006 09:07:02 -0400 Subject: [Python-3000] Special object to return from functions that return only None In-Reply-To: <4445C9AB.5090004@canterbury.ac.nz> References: <44459540.1040709@comcast.net> <4445C9AB.5090004@canterbury.ac.nz> Message-ID: <2e1434c10604250607h571922a7w1c46cfdd4c2d6dd4@mail.gmail.com> On 4/19/06, Greg Ewing wrote: > > Edward C. Jones wrote: > > > I suggest a singleton object "UseForbidden" ("Py_UseForbidden" in C). > > It should be called "Mu". :-) > > Although such an object could cause a lot of pain to > debuggers and other such introspective code. > Actually this sounds a lot like a signaling Not-a-number (sNaN) from IEEE754. One can pass around a sNaN, but any use will raise a (numeric) exception. Similarly, you want a signaling Not-an-Object (sNaO) that also doesn't like to be passed as a function argument. It is the last bit that makes it a pain, since what if your function is log_error_value(sNaO)? -Kevin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060425/640395bb/attachment.htm From rhettinger at ewtllc.com Tue Apr 25 17:34:05 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Tue, 25 Apr 2006 08:34:05 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <1145891984.8218.180.camel@resist.wooz.org> <444D0E16.9000508@colorstudy.com> Message-ID: <444E416D.2040205@ewtllc.com> Greg Wilson wrote: >One of the reasons I'd like native syntax for sets is that I'd like set >comprehensions: > > a = {b for b in c where b > 0} > > > > This is no good. That form could as easily represent a frozenset comprehension or dict comprehension. It buys us nothing over the clear and unambiguous form we have now: set(b for b in collection if b > 0) Distinguishing between sets, frozensets, and dicts is an insurmountable hurdle for punctuation based proposals (because you can't make the distinctions without introducing complexity). Besides, there are no real gains to be had -- adding punctuation seems to have become a quest unto itself. Embrace genexps -- they are a path to uniform and clear expression across a wide range of datatypes. Avoid punctuation and funky syntax -- they burden the learning curve, they do not generalize well, they challenge parsers (both human and electronic), and they throw-away the clarity affored by spelled-out datatype names. Raymond From bborcic at gmail.com Tue Apr 25 15:30:29 2006 From: bborcic at gmail.com (Boris Borcic) Date: Tue, 25 Apr 2006 15:30:29 +0200 Subject: [Python-3000] sets in P3K? Message-ID: <444E2475.7050001@gmail.com> On 4/24/06, Guido van Rossum wrote: > [...] Python has 'in' for ', Isn't it an inconsistency that Python admits 'in' as a concise english equivalent to ? (and 'not in' for ?) while ? requires painful spelling out as 'lambda' ? NB : this mail means to bring to mind the possibility of supporting math glyphs in source code. With unicode, non-ascii glyphs tend to get supported to surprising levels by usual tools. So I use a few in this utf-8 mail, if they don't make it unscathed through the moderator's mail to the list's website and back to browsers, no need to worry further I'd say. But they may well make it. > [...] {1,2,3} is certainly > unambiguously the best notation for set literals Nit : when I was initiated to sets in primary school ("new math" generation), we had to write it exactly as {1;2;3}. > [...] I believe > that mathematicians use a crossed-out little oh to indicate the empty > set, ? you mean ? > so a notational discontinuity is not unheard of. > > OTOH mathematicians (whether in high school or not) write things like > {x | 2 < x < 10}, which is of course the origin of our list > comprehensions and generator expressions [...] Most of the time they use it as "filter" and write something like {x ? S st. P(x)} which is the cause that I regularly do the mistake in Python, to write (x in S if p(x)) when I should write (x for x in S if p(x)) but I guess the advent of the ternary if-else operator buried the last slimmer of hope to see that syntax evolve in the context of 1-token-lookahead parsing :( Regards, Boris Borcic -- assert "304" in "340343", "P424D15E M15M47CH" From gvwilson at cs.utoronto.ca Tue Apr 25 17:44:43 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Tue, 25 Apr 2006 11:44:43 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: <444E416D.2040205@ewtllc.com> References: <1145891984.8218.180.camel@resist.wooz.org> <444D0E16.9000508@colorstudy.com> <444E416D.2040205@ewtllc.com> Message-ID: > On Tue, 25 Apr 2006, Raymond Hettinger wrote: > Embrace genexps -- they are a path to uniform and clear expression > across a wide range of datatypes. > ... > Avoid punctuation and funky syntax -- they burden the learning curve, > they do not generalize well, they challenge parsers (both human and > electronic), and they throw-away the clarity affored by spelled-out > datatype names. Hi Raymond, Do you have any data to back up these assertions? Something like http://mail.python.org/pipermail/python-dev/2000-July/006427.html? Thanks, Greg From aahz at pythoncraft.com Tue Apr 25 18:35:21 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 25 Apr 2006 09:35:21 -0700 Subject: [Python-3000] Special object to return from functions that return only None In-Reply-To: <2e1434c10604250607h571922a7w1c46cfdd4c2d6dd4@mail.gmail.com> References: <44459540.1040709@comcast.net> <4445C9AB.5090004@canterbury.ac.nz> <2e1434c10604250607h571922a7w1c46cfdd4c2d6dd4@mail.gmail.com> Message-ID: <20060425163521.GA23595@panix.com> On Tue, Apr 25, 2006, Kevin Jacobs wrote: > > Actually this sounds a lot like a signaling Not-a-number (sNaN) > from IEEE754. One can pass around a sNaN, but any use will raise a > (numeric) exception. Similarly, you want a signaling Not-an-Object > (sNaO) that also doesn't like to be passed as a function argument. It > is the last bit that makes it a pain, since what if your function is > log_error_value(sNaO)? Note that Python does make it fairly straightforward to define such objects on a per-domain basis. E.g.: sentinel = object() The biggest problem from my POV with widespread adoption of this technique is that it doesn't play well with pickle. Perhaps someone wants to take a whack at defining a more generic mechanism for sentinels? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From jcarlson at uci.edu Tue Apr 25 18:45:26 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Tue, 25 Apr 2006 09:45:26 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <444E416D.2040205@ewtllc.com> Message-ID: <20060425093223.66E8.JCARLSON@uci.edu> Greg Wilson wrote: > > > On Tue, 25 Apr 2006, Raymond Hettinger wrote: > > Embrace genexps -- they are a path to uniform and clear expression > > across a wide range of datatypes. > > ... > > Avoid punctuation and funky syntax -- they burden the learning curve, > > they do not generalize well, they challenge parsers (both human and > > electronic), and they throw-away the clarity affored by spelled-out > > datatype names. > > Hi Raymond, > > Do you have any data to back up these assertions? Something like > http://mail.python.org/pipermail/python-dev/2000-July/006427.html? Assuming programmer sanity, how can naming the data type in the construction of the data type be ambiguous? Or really, how could learning a name be more difficult than learning a syntax? It seems to be quite clear, if one bothers to read the name, precisely what is being constructed (assuming the reader understands what a 'set', 'frozenset', etc. is). If list comprehensions didn't come first (and even though list comprehensions came first), I would argue that there should only be generator expressions. If one wants a list comprehension, one should use list(genexp). Obviously it would have clear implications of the non-starting of {genexp} for set, frozenset, and/or dict comprehensions. This would also make it clear that the Python language isn't going the way of "a new syntax every release", but rather a language with minimal syntax to learn and/or get in your way. - Josiah From gvwilson at cs.utoronto.ca Tue Apr 25 18:50:06 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Tue, 25 Apr 2006 12:50:06 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: <20060425093223.66E8.JCARLSON@uci.edu> References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> Message-ID: > > Greg Wilson wrote: > > Do you have any data to back up these assertions? Something like > > http://mail.python.org/pipermail/python-dev/2000-July/006427.html? > Josiah Carlson: > Assuming programmer sanity, how can naming the data type in the > construction of the data type be ambiguous? Or really, how could > learning a name be more difficult than learning a syntax? Greg Wilson: Again, do we have any data --- however imperfectly gathered --- to back up claims either way? As I said in a previous post, I'm going to start teaching a new bunch of never-seen-Python-before undergrads on May 15; if there are proposals for new syntax (not just sets) that people want me to try out on paper, I'd be happy to talk about it. Thanks, Greg From aleaxit at gmail.com Tue Apr 25 19:49:28 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Tue, 25 Apr 2006 10:49:28 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <20060425093223.66E8.JCARLSON@uci.edu> References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> Message-ID: On 4/25/06, Josiah Carlson wrote: ... > If list comprehensions didn't come first (and even though list > comprehensions came first), I would argue that there should only be > generator expressions. If one wants a list comprehension, one should > use list(genexp). Obviously it would have clear implications of the > non-starting of {genexp} for set, frozenset, and/or dict comprehensions. I entirely agree, as, it appears to me from his posts to this thread, does Raymond; however, Guido thinks our shared preference is atypical, as he posted on this thread 24 hours ago: """ On 4/24/06, Alex Martelli wrote: > I dislike that as much as I dislike [] as a shorthand for > list(), but I have no trouble admitting that if we have the > [...] form, it's consistent to have the {...} one too. I think you're atypical in that dislike. """ I'm not sure if it really matters at all (to Guido's decision) whether this preference is more widespread than he thinks, or just an atypical quirk shared just by Carlson, Hettinger and Martelli; if it does matter, I guess that organizing some kind of "straw popularity poll" is not too hard. Alex From jimjjewett at gmail.com Tue Apr 25 20:29:32 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 25 Apr 2006 14:29:32 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <444E2475.7050001@gmail.com> References: <444E2475.7050001@gmail.com> Message-ID: On 4/25/06, Boris Borcic wrote: > On 4/24/06, Guido van Rossum wrote: > > OTOH mathematicians (whether in high school or not) write things like > > {x | 2 < x < 10}, which is of course the origin of our list > > comprehensions and generator expressions [...] > Most of the time they use it as "filter" and write something like > {x ? S st. P(x)} That is the formal form, because then you can specify (for example) whether x is limited to Real, Rational, or Integers. But what actually got used most of the time (in classes I took and texts and papers I read) was the shorter form Guido mentioned. On the other hand, one reason the "|" form was preferred is that it leant itself to defining oddball functions, which in turns suggests a mapping rather than a set. -jJ From tjreedy at udel.edu Tue Apr 25 20:48:54 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 25 Apr 2006 14:48:54 -0400 Subject: [Python-3000] sets in P3K? References: <444E416D.2040205@ewtllc.com><20060425093223.66E8.JCARLSON@uci.edu> Message-ID: "Alex Martelli" wrote in message news:e8a0972d0604251049x29a39d61l690608b1c005ce2b at mail.gmail.com... > On 4/25/06, Josiah Carlson wrote: > ... >> If list comprehensions didn't come first (and even though list >> comprehensions came first), I would argue that there should only be >> generator expressions. If one wants a list comprehension, one should >> use list(genexp). Obviously it would have clear implications of the >> non-starting of {genexp} for set, frozenset, and/or dict comprehensions. > > I entirely agree, as, it appears to me from his posts to this thread, > does Raymond; however, Guido thinks our shared preference is atypical, > as he posted on this thread 24 hours ago: > """ > On 4/24/06, Alex Martelli wrote: >> I dislike that as much as I dislike [] as a shorthand for >> list(), but I have no trouble admitting that if we have the >> [...] form, it's consistent to have the {...} one too. > > I think you're atypical in that dislike. > """ When Guido posted that, I replied in support of more syntax. While I don't share your dislike, Raymond and Josiah have somewhat neutralized my support. I can see Raymond's point that we now have too many types to support all with syntax. And I can see an argument that the appropriate response to more types is a compensating decrease in syntax (as in dropping [xx] as a synonym for list(xx)) rather than adding more. Terry Jan Reedy From jimjjewett at gmail.com Tue Apr 25 20:49:29 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 25 Apr 2006 14:49:29 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <20060425093223.66E8.JCARLSON@uci.edu> References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> Message-ID: On 4/25/06, Josiah Carlson wrote: > Greg Wilson wrote: > > > On Tue, 25 Apr 2006, Raymond Hettinger wrote: > > > Avoid punctuation and funky syntax -- they burden the learning curve, > > > they do not generalize well, they challenge parsers (both human and > > > electronic), and they throw-away the clarity affored by spelled-out > > > datatype names. > > Do you have any data to back up these assertions? Something like > > http://mail.python.org/pipermail/python-dev/2000-July/006427.html? I think that actually provides some indirect evidence there. In the abstract, there is nothing about "zip" to suggest a crossproduct, but seeing it used with 3x2 somehow distracted people. It would probably have misdirected me, if the questions had been "What does this mean?", rather than "What should this mean?", because I would have been reluctant to say error. The punctuation creates groups, but what to do with that group is just like any other random association -- at best, it seems similar to something you've seen before. > Assuming programmer sanity, how can naming the data type in the > construction of the data type be ambiguous? Or really, how could > learning a name be more difficult than learning a syntax? It seems to > be quite clear, if one bothers to read the name, precisely what is being > constructed (assuming the reader understands what a 'set', 'frozenset', > etc. is). set, Set. frozenset, frozen_set, FrozenSet, frozenSet, forzenset. (I left in a typo, because I actually typed three of them while writing this.) If you don't see frozenset (and I generally haven't, even with what it effectively a literal constant), then you need to wonder whether or not set is mutable. And this all assumes that you know sets are collections (rather than data binding functions) which can hold arbitrary objects. {} seems less likely to hide something tricky, precisely because it is anonymous. Technically, this is a false confidence, but it can be helpful anyhow. > If list comprehensions didn't come first (and even though list > comprehensions came first), I would argue that there should only be > generator expressions. If one wants a list comprehension, one should > use list(genexp). It isn't just about generators; it is also about literals. "mixed feelings" -jJ From jimjjewett at gmail.com Tue Apr 25 20:59:49 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Tue, 25 Apr 2006 14:59:49 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> Message-ID: On 4/25/06, Terry Reedy wrote: > I can see an argument that the appropriate > response to more types is a compensating decrease in syntax > (as in dropping [xx] as a synonym for list(xx)) rather than adding > more. Great. Except then how would you create a collection in the first place? Right now, I use the awkward set(["asdf", "asdf2"]) set(list("asdf", "asdf2")) #syntax error set(tuple("asdf", "asdf2")) #syntax error set(tuple(("asdf", "asdf2"))) # just add parens? Or does that go too? We need at least one literal for sequences. We currently have list (ordered, mutable), tuple (ordered, immutable), and dict (unordered, mutable, associative). The question is whether we should also have one for unordered collections. This is the datatype that we really want for most "x in y" tests, but getting rid of the extra information may not be important enough to use up literals. -jJ From tim.peters at gmail.com Tue Apr 25 21:20:51 2006 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 25 Apr 2006 15:20:51 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <444E2475.7050001@gmail.com> Message-ID: <1f7befae0604251220i35f41893r4847e732100588b1@mail.gmail.com> [Guido] >>> OTOH mathematicians (whether in high school or not) write things like >>> {x | 2 < x < 10}, which is of course the origin of our list >>> comprehensions and generator expressions [...] [Boris Borcic] >> Most of the time they use it as "filter" and write something like >> >> {x ? S st. P(x)} [Jim Jewett] > That is the formal form, because then you can specify (for example) > whether x is limited to Real, Rational, or Integers. Not that it matters to Python, but it's the formal form because Guido's form isn't allowed at all in mainstream modern set theory: it directly yields Russell's Paradox (let R = {x | x not in x}; then is R in R?; both "yes" and "no" lead to contradiction). So in modern work, "unrestricted comprehension" isn't allowed -- you can't just make up a predicate and assert that there's a set consisting precisely of the things satisfying the predicate. For another example, {x | True} (the set containing "everything") is also paradoxical. Now you can only use a set if you can prove it exists, and one way to prove a set exists is via what was once called the Axiom of Comprehsion (but seems to be called the Axiom of Subsets more often these days). That says that if you have a set S, then for any one-place predicate P, {x ? S | P(x)} exists. So you can still make up any predicate you like, but you can only use it to "separate out" (why it's also sometimes, but I think rarely, called the Axiom of Separation) the satisfiyng elements from another set. > But what actually got used most of the time (in classes I took and texts and > papers I read) was the shorter form Guido mentioned. For example, in a text on number theory, it would be a major PITA to keep repeating that the set from which predicates "separate out" is the set of integers. Nothing wrong with stating that that (or any other known-to-exist) set is to be assumed in subsequent comprehensions. It would be a dubious thing for Python to, e.g., a grow sys.universal_set global default though ;-) From brett at python.org Tue Apr 25 21:36:06 2006 From: brett at python.org (Brett Cannon) Date: Tue, 25 Apr 2006 13:36:06 -0600 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> Message-ID: On 4/25/06, Alex Martelli wrote: > On 4/25/06, Josiah Carlson wrote: > ... > > If list comprehensions didn't come first (and even though list > > comprehensions came first), I would argue that there should only be > > generator expressions. If one wants a list comprehension, one should > > use list(genexp). Obviously it would have clear implications of the > > non-starting of {genexp} for set, frozenset, and/or dict comprehensions. > > I entirely agree, as, it appears to me from his posts to this thread, > does Raymond; however, Guido thinks our shared preference is atypical, > as he posted on this thread 24 hours ago: > """ > On 4/24/06, Alex Martelli wrote: > > I dislike that as much as I dislike [] as a shorthand for > > list(), but I have no trouble admitting that if we have the > > [...] form, it's consistent to have the {...} one too. > > I think you're atypical in that dislike. > """ > > I'm not sure if it really matters at all (to Guido's decision) whether > this preference is more widespread than he thinks, or just an atypical > quirk shared just by Carlson, Hettinger and Martelli; if it does > matter, I guess that organizing some kind of "straw popularity poll" > is not too hard. > I'll toss in my straw; I have no issue losing listcomps and favoring ``list(genexp)``. TOOWTDI since there is not that huge of a "simple over complex" savings. Probably never thought about this since I really got into Python when listcomps were introduced so they happen to be heavily ingrained into my Python psyche. -Brett From rrr at ronadam.com Tue Apr 25 21:56:47 2006 From: rrr at ronadam.com (Ron Adam) Date: Tue, 25 Apr 2006 14:56:47 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: <444E416D.2040205@ewtllc.com> References: <1145891984.8218.180.camel@resist.wooz.org> <444D0E16.9000508@colorstudy.com> <444E416D.2040205@ewtllc.com> Message-ID: Raymond Hettinger wrote: > Greg Wilson wrote: > >> One of the reasons I'd like native syntax for sets is that I'd like set >> comprehensions: >> >> a = {b for b in c where b > 0} >> >> >> >> > > This is no good. That form could as easily represent a frozenset > comprehension or dict comprehension. It buys us nothing over the clear > and unambiguous form we have now: > > set(b for b in collection if b > 0) > > Distinguishing between sets, frozensets, and dicts is an insurmountable > hurdle for punctuation based proposals (because you can't make the > distinctions without introducing complexity). Besides, there are no > real gains to be had -- adding punctuation seems to have become a quest > unto itself. > > Embrace genexps -- they are a path to uniform and clear expression > across a wide range of datatypes. > > Avoid punctuation and funky syntax -- they burden the learning curve, > they do not generalize well, they challenge parsers (both human and > electronic), and they throw-away the clarity affored by spelled-out > datatype names. I agree with Raymond on these points, but maybe not for the same reasons. Punctuation can be clear *if* it has a strong underlying consistency. This just isn't possible with the current data types without making quite a few changes that probably wouldn't be liked. For example. (not a suggestion) set {} dict {:} list [] ordered multi dict [:] ; *A *A - Yes, this is ambiguous with slicing and the ordered multi-dict isn't a part of the language. Then some way to indicate each of these is immutable or frozen would be needed, where a frozen list is equivalent to a tuple, but it then gets messy and looses the nice symmetry. There just isn't enough symbols. And if there were, you wouldn't want to do more than a handful of types with shorthand syntax anyway. I like being able to create data structures by explicitly typing them out. I also like being able to convert a repr() of a data structure back to it's original for simple data types. But these are not big things I can live without. (and aren't the best way to do it in most cases) The obvious (to me) kiss compromise would be to continue to let (), and [] be used as they are and have all other data types explicitly spelled out. Since dict has a convenient creation syntax dict(a=val, b=val), I am ok with spelling it out. It may also be possible to optimize named data types that have are all literals as components at compile time... set(1,2,3), list('a','b','c'), etc... Then it would be equivalent to using the shorthand forms in these cases. Yes, these cause errors presently, but maybe they shouldn't. Cheers, Ron From mcherm at mcherm.com Tue Apr 25 23:16:30 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Tue, 25 Apr 2006 14:16:30 -0700 Subject: [Python-3000] sets in P3K? Message-ID: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> I think perhaps I'm the the only one who's missing the big joke. At any rate, I'm going to go ahead and speak up: The Emperor Has No Cloths! Adding a literal notation for sets may or may not be a good idea (it's certainly not *necessary*). Adding a literal notation for every built-in data structure (or every possible data structure) is completely batty. REMOVING the literal notation for tuples, lists, or dicts is also completely batty. No, I retract that... it's quite sensible, but it's just not Python. Python has simple literal syntax for lists, tuples, and dicts. That's one of the key ideas that makes it Python. -- Michael Chermside PS: Sorry about spoiling the joke. From aahz at pythoncraft.com Wed Apr 26 00:48:03 2006 From: aahz at pythoncraft.com (Aahz) Date: Tue, 25 Apr 2006 15:48:03 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> Message-ID: <20060425224803.GA9983@panix.com> On Tue, Apr 25, 2006, Brett Cannon wrote: > > I'll toss in my straw; I have no issue losing listcomps and favoring > ``list(genexp)``. TOOWTDI since there is not that huge of a "simple > over complex" savings. Probably never thought about this since I > really got into Python when listcomps were introduced so they happen > to be heavily ingrained into my Python psyche. There are [at least] two separate issues being discussed here, and I think that's confusing things. From my POV, the issue about whether listcomps per se go away in Python 3.0 is completely separate from the issue of whether sets get special literal syntax. My opinion: if we were designing Python from scratch right now, we might well consider having only set literals and not dict literals. However, I don't think we can have both set and dict literals, and I think that removing dict literals (or list literals) counts as gratuitous breakage. Losing listcomps won't be a big problem because they can be automatically fixed and they aren't much loss. -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From rhettinger at ewtllc.com Wed Apr 26 03:11:01 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Tue, 25 Apr 2006 18:11:01 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <20060425224803.GA9983@panix.com> References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> <20060425224803.GA9983@panix.com> Message-ID: <444EC8A5.7010406@ewtllc.com> Aahz wrote: >There are [at least] two separate issues being discussed here, and I >think that's confusing things. From my POV, the issue about whether >listcomps per se go away in Python 3.0 is completely separate from the >issue of whether sets get special literal syntax. > > For the record, I favor keeping list comps for P3K. Those should be the only type specific form. Everything else (including sets) should use: datatype(genexp). >My opinion: if we were designing Python from scratch right now, we might >well consider having only set literals and not dict literals. However, >I don't think we can have both set and dict literals, and I think that >removing dict literals (or list literals) counts as gratuitous breakage. > > > > Rightly said. From edloper at gradient.cis.upenn.edu Wed Apr 26 04:33:12 2006 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Tue, 25 Apr 2006 22:33:12 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> Message-ID: <444EDBE8.3020303@gradient.cis.upenn.edu> I think that a significant motivation for people that propose set literals is that the following is just plain ugly: s = set([1,2,3]) It seems much more natural to write: s = set(1, 2, 3) However, it is fairly common to want to build a set from a collection, an iterator, or a genexp. With the current notation, all three of these are easy. But with this "more natural" notation, they're no longer possible without resorting to varargs (which would create an unnecessary imtermediary tuple). I.e., we don't want to lose the ability to do any of the following: s = set(my_list) s = set(enumerate(my_list)) s = set(x for x in collection if x>10) One way around this conflict might be to define a new factory function, that generates a set from an iterable. I.e., something like: s = set.from_iter(my_list) s = set.from_iter(enumerate(my_list)) s = set.from_iter(x for x in collection if x>10) (But with a better name than 'from_iter' :) ) The disadvantage, of course, is that it makes building sets from iterables less simple. And this certainly might be enough of a disadvantage to rule out this solution. But I thought I'd throw it out, anyway, to see what people think. I'm personally +0 on this proposal if a good name can be thought of for the factory function; and -0 if a good name can't be thought of. If this proposal does gain support, then it should presumably be applied to other data types as well, for consistency. I.e.: l = list(1, 2, 3) l = list.from_iter([1, 2, 3]) d = dict(('x', 'y'), ('a', 'b')) d = dict.from_iter((x,i) for (i,x) in enumerate(collection)) t = tuple(1, 2, 3) t = tuple.from_iter(collection) -Edward From kayschluehr at gmx.de Wed Apr 26 08:27:25 2006 From: kayschluehr at gmx.de (Kay Schluehr) Date: Wed, 26 Apr 2006 08:27:25 +0200 Subject: [Python-3000] Give power to __getattr__ In-Reply-To: <444DB5E8.10706@gmail.com> References: <444DB18D.1030004@gmx.de> <444DB5E8.10706@gmail.com> Message-ID: <444F12CD.9020408@gmx.de> Nick Coghlan wrote: > Kay Schluehr wrote: > >> While observing heated discussions about adaption and generic >> functions I wonder if this >> is a good opportunity for a modest complaint about the castration of >> __getattr__ in new style classes. > > > If you want to get full control back, that's what __getattribute__ is > for. . . > > Or are you asking for better access to the bits and pieces of > object.__getattribute__ so that it's easier to customise the operation? > > Cheers, > Nick. > I have likely overestimated the usefullness of __getattr__ for old syle classes in case of calling an operator. I have wrongly asserted that having an instance x of an old style class X and an implementation of __getattr__ on X would pass e.g. '__add__' on a call >>> x + 7 to __getattr__. Unlike __getattribute__ for new style classes the __getattr__ method actually responds, but it is '__coerce__' that is passed instead of '__add__' and that's not much usefull information. What I'm looking for is capturing the call of an arbitrary special method without implementing it explicitely. This would enable delegating a call on __add__ to another class that implements __add__. This could either be done by empowering __getattribute__ or use the now residual __getattr__ that has no value in case of an __getattribute__ implementation. Of course if we are free to choose a name it would be better to use a non-confusing name e.g. __getspecialattr__ that is used when the attribute is the name of a special method and the method accordingly is neither an instance attribute nor could it be found in the class tree. Ciao, Kay From talin at acm.org Wed Apr 26 09:47:00 2006 From: talin at acm.org (Talin) Date: Wed, 26 Apr 2006 07:47:00 +0000 (UTC) Subject: [Python-3000] Adding sorting to generator comprehension References: <444CFB50.6010809@colorstudy.com> <20060424102056.66CA.JCARLSON@uci.edu> <444D1039.8050007@colorstudy.com> Message-ID: Ian Bicking colorstudy.com> writes: > > Josiah Carlson wrote: > > One of the features of generator expressions which makes it desireable > > instead of list comprehensions is that generator expressions may use > > less memory *now*, and may be able to start returning results *now*. > > > > Using ( orderby ...) as a replacement for sorted((genexp), key=...) > > is a bit misleading because while the original generator expression > > could have been space and time efficient, the orderby version certainly may > > not be. > > Certainly it changes the performance substantially (of course if the > expression is translated and executed elsewhere the performance can > actually be improved, so it can go both ways). Since list > comprehensions are planned to just be syntactic sugar for generator > comprehension, generator comprehensions are now the more fundamental > construct. > > But yeah, it is a little awkward, since something that is sorted can be > returned as a list anyway, except for the fact that the expression > itself could be ported off elsewhere which isn't normal Python. (But > should be normal Python!) > I think that I am (mostly) in agreement with Ian on this one, but perhaps with different reasoning. There seems to be a common feeling on py-dev that "list comprehensions are just a special case of generators expressions, so really, we don't need them". But then we turn around and say "Well, this feature might be handy for list comprehensions, but since list comprehensions are based on generator expressions, and since this feature would make generator expressions inefficient, we won't do it." In other words, we dismiss any evidence that disproves our initial assumption. This is sort of like saying that we don't need the 'def' statement, since you can define functions with "func = lambda"; and also we shouldn't add multi-line statements, since lambda doesn't support them. (Of course, I am overstating the argument so that I can make the point.) There are a number of tendancies which I would caution programmers to be wary of: 'Overfactoring' - also known as 'excessive reductionism'. It is the tendancy to try and boil down everything into one mathematically perfect form, of which all real programs are just special cases. Often you run into two problems which are superficially similar, but have a subtle distinction; overfactoring tends to blur and hide that distinguishing line. The result is an algorithm that solves neither problem well. 'Simplifying the implementation design instead of the usage design' - The subtle distinction between list comprehensions and generator expressions has a not-so-subtle effect on how they are used. Our primary goal should be to optimize those usage patterns; Minimizing the number of special cases in the interpreter should take second or third place behind that goal, IMHO. The fact is that sorting happens a *lot*. Sorting with key= happens a lot. A large proportion of my list comprehensions involve sorting or ordering of one form or another, and the number of ugly sorted + key + lambda expressions is enough to convince me that an "orderby" or "ascending" or whatever clause would be a welcome addition. -- Talin From talin at acm.org Wed Apr 26 10:03:18 2006 From: talin at acm.org (Talin) Date: Wed, 26 Apr 2006 08:03:18 +0000 (UTC) Subject: [Python-3000] sets in P3K? References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> <20060425224803.GA9983@panix.com> Message-ID: Aahz pythoncraft.com> writes: > > On Tue, Apr 25, 2006, Brett Cannon wrote: > > > > I'll toss in my straw; I have no issue losing listcomps and favoring > > ``list(genexp)``. TOOWTDI since there is not that huge of a "simple > > over complex" savings. Probably never thought about this since I > > really got into Python when listcomps were introduced so they happen > > to be heavily ingrained into my Python psyche. > > There are [at least] two separate issues being discussed here, and I > think that's confusing things. From my POV, the issue about whether > listcomps per se go away in Python 3.0 is completely separate from the > issue of whether sets get special literal syntax. > > My opinion: if we were designing Python from scratch right now, we might > well consider having only set literals and not dict literals. However, > I don't think we can have both set and dict literals, and I think that > removing dict literals (or list literals) counts as gratuitous breakage. > > Losing listcomps won't be a big problem because they can be automatically > fixed and they aren't much loss. In my experience dict literals are far more useful than set literals. In fact, I don't think I've ever made a set literal. The only syntactical short cut that I would find useful for a set would be "new empty set", for which I think "set()" is probably short enough already. As someone stated earlier, sets are most often built up incrementally, or are the result of a transformation of other data. Dicts, on the other hand, are very frequently specified in literal form. I think that the syntactical shortcuts we have for Python are just right - if I were to start over, I doubt I would change anything. Part of the big win that scripting languages have over statically compiled languages such as C++ is their ability to declare complex, dynamic data structures as literals; Syntactic shortcuts make that strength even stronger. And Python takes the 3 most commonly used container types - dicts, tuples, and lists - and gives them easy-to-type shortcuts. Actually, there is *one* thing that I would change about the current syntax, which is that I would change dict literals to have the same internal syntax as function calls, so that { } would be equivalent to dict( ). This means using '=' instead of ':' and allowing key names to be bare words instead of quoted strings. So you could write: { name="Frodo", race="Hobbit" } instead of: { 'name':'Frodo', 'race':'Hobbit' } (Only 4 characters longer, but *much* harder to type. Try typing it if you don't believe me.) -- Talin From talin at acm.org Wed Apr 26 10:12:33 2006 From: talin at acm.org (Talin) Date: Wed, 26 Apr 2006 08:12:33 +0000 (UTC) Subject: [Python-3000] Brainstorming: Python Metaprogramming References: <444CF246.7000104@colorstudy.com> Message-ID: Ian Bicking colorstudy.com> writes: > > As a hypothetical example, supposed we defined a unary operator '?' such that: > > > > ?x > > > > was syntactic sugar for: > > > > quoted('x') > > > > or even: > > > > quoted('x', (scope where x is or would be defined) ) > > > > Where 'quoted' was some sort of class that behaved like a reference to a > > variable. So ?x.set( 1 ) is the same as x = 1. > > Sounds like lambda x: ... More differences than similarities, I think. For one thing, you can't use lambda x: ... to assign to x. And while you could possibly overload the arithmetic operators on a lambda, you'd have to do it for each lambda individually, since lambdas don't have classes. > > Moreover, you would want to customize all of the operators on quoted to return > > an AST, so that: > > > > ?x + 1 > > > > produces something like: > > > > (add, quoted('x'), 1) > > > > ...or whatever data structure is convenient. > > You can match the free variables from the lambda arguments against the > variables in the AST to get this same info. I don't understand what this means. Look, folks, I don't have a concrete proposal here. I'm actually fishing for ideas, not looking to have my own ideas validated (or not.) My gut tells me that there's something here worth looking into, but I haven't put my finger on it yet. (Isn't it interesting how often we programmers, who pride ourselves on our use of logic and reason, so often use intuition and hunches to solve problems?) -- Talin From talin at acm.org Wed Apr 26 10:37:35 2006 From: talin at acm.org (Talin) Date: Wed, 26 Apr 2006 08:37:35 +0000 (UTC) Subject: [Python-3000] Open Issues for string.format PEP References: <444C3BC9.7020000@colorstudy.com> <444CFAF7.2010407@colorstudy.com> Message-ID: Ian Bicking colorstudy.com> writes: > > I want to go back for a moment to the original question. Whenever someone asks > > me "how should we solve this problem?" the first thing I always ask is "do we > > really need to?" Once we've answered that, we can then go on to the issue of > > coming up with a solution. > > Without this all dictionaries must be copied. I think maybe Guido > suggested something involving a shortcut that would not covert the > dictionary, but that shortcut isn't available to Python so the behavior > could be different for custom formatters implemented in Python. Also, > only dictionaries are allowed after **; that could be changed (maybe > that's part of the signature changes?), but that doesn't change the fact > that only a dictionary can be passed into a function with **. I think you are missing some alternatives. You see, the question I was asking wasn't "How important is it that we be able to efficiently pass whole dictionaries to the format function?" The question was "How important is it to be able to pass whole dictionaries at all?" Remember, that anything that you can do with **args you can also do by explicitly passing the individual args you want. Another alternative is just to fix the efficiency problems with **args. The reason that **args is so expensive is because of the need to funnel all calls through the existing C API, which requires that all function calls have the signature of ( PyTuple *args, PyDict *kwargs ). If you were somehow able to bypass or extend the C API, you could take the **args from the function call, and directly hook it up to the **args in the formal parameter list, without any flattening. Assumming that neither of those options are available, I would probably then do one of the following: 1) Create a different method name for the dict case: "{name}".vformat( mydict ) 2) Have some sort of reserved, keyword-only argument for dicts: "{name}".format( indirect_args=mydict ) 3) Bring back field expressions: "{0.name}".format( mydict ) (Perhaps the '.' operator can mean both '.' and '[]' depending on the type of the object.) -- Talin From tdickenson at devmail.geminidataloggers.co.uk Wed Apr 26 10:56:00 2006 From: tdickenson at devmail.geminidataloggers.co.uk (Toby Dickenson) Date: Wed, 26 Apr 2006 09:56:00 +0100 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <444E416D.2040205@ewtllc.com> <20060425224803.GA9983@panix.com> Message-ID: <200604260956.00287.tdickenson@devmail.geminidataloggers.co.uk> On Wednesday 26 April 2006 09:03, Talin wrote: > In my experience dict literals are far more useful than set literals. > In fact, I don't think I've ever made a set literal. The only > syntactical short cut that I would find useful for a set would > be "new empty set", for which I think "set()" is probably short > enough already. I quite often write.... if state in (case1,case2,case3): as a shorter form of.... if state==case1 or state==case2 or state==case3: That is definitely a set membership test, although there probably is little advantage in using a Set object rather than a tuple for this. -- Toby Dickenson From talin at acm.org Wed Apr 26 11:26:00 2006 From: talin at acm.org (Talin) Date: Wed, 26 Apr 2006 09:26:00 +0000 (UTC) Subject: [Python-3000] PEP: Advanced String Formatted, Rev 2 Message-ID: I've submitted the most recent revisions of the string formatting PEP to the pep editor; In the mean time, you can read it here: http://www.viridia.org/python/doc/PEP_AdvancedStringFormatting2.txt -- Talin From ncoghlan at gmail.com Wed Apr 26 12:51:57 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Apr 2006 20:51:57 +1000 Subject: [Python-3000] sets in P3K? In-Reply-To: <444EDBE8.3020303@gradient.cis.upenn.edu> References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> Message-ID: <444F50CD.6080903@gmail.com> Edward Loper wrote: > One way around this conflict might be to define a new factory function, > that generates a set from an iterable. I.e., something like: > > s = set.from_iter(my_list) > s = set.from_iter(enumerate(my_list)) > s = set.from_iter(x for x in collection if x>10) > > (But with a better name than 'from_iter' :) ) Some might object to this as being too magical, but one option is a metaclass that overrides __getitem__ as an alternative constructor to the usual __call__. Numpy uses this trick to good effect for building matrices. A single metaclass could cover most builtin and standard library containers: def __getitem__(self, args): # This just defines the semantics. A real implementation would # avoid the temporary objects wherever possible. if not isinstance(args, tuple): args = (args,) arglist = [] for item in args: if isinstance(item, slice): if item.step is None: item = (item.start, item.stop) else: item = (item.start, item.stop, item.step) arglist.append(item) return self(*arglist) Then you could have: s = set[1, 2, 3] # == set((1, 2, 3)) f = frozenset[1, 2, 3] # == frozenset((1, 2, 3)) d = dict[1:'x', 2:'y'] # == {1:'x', 2:'y'} == dict(((1,'x'), (2,'y'))) l = list[1, 2, 3] # == [1, 2, 3] == list((1, 2, 3)) t = tuple[1, 2, 3] # == (1, 2, 3) == tuple((1, 2, 3)) This can also be spelt more explicitly, but doing so pretty much misses the point of this discussion (we're only talking about avoiding a couple of parentheses, after all). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From molasses0 at gmail.com Wed Apr 26 12:50:06 2006 From: molasses0 at gmail.com (molasses) Date: Wed, 26 Apr 2006 20:50:06 +1000 Subject: [Python-3000] More batteries included! Message-ID: <5384bedc0604260350k1c92a212s8825c680516624bd@mail.gmail.com> Hi all, How about including wxPython[1] and Wax[2] in the Py3k standard libs and getting rid of the TK gui stuff? Also, I think pywin32[3] should be included. Or is there already a way to use Windows COM objects in Python that I missed? PIL[4] would be great to have in the standard libs too. But that's probably asking too much ;) [1] http://www.wxpython.org [2] http://sourceforge.net/projects/waxgui [3] http://sourceforge.net/projects/pywin32 [4] http://www.pythonware.com/products/pil Keep up the great work everyone, Mark From ncoghlan at gmail.com Wed Apr 26 13:02:56 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Apr 2006 21:02:56 +1000 Subject: [Python-3000] Give power to __getattr__ In-Reply-To: <444F12CD.9020408@gmx.de> References: <444DB18D.1030004@gmx.de> <444DB5E8.10706@gmail.com> <444F12CD.9020408@gmx.de> Message-ID: <444F5360.4070303@gmail.com> Kay Schluehr wrote: > What I'm looking for is capturing the call of an arbitrary special > method without implementing it explicitely. This would enable delegating > a call on __add__ to another class that implements __add__. This could > either be done by empowering __getattribute__ or use the now residual > __getattr__ that has no value in case of an __getattribute__ > implementation. Of course if we are free to choose a name it would be > better to use a non-confusing name e.g. __getspecialattr__ that is used > when the attribute is the name of a special method and the method > accordingly is neither an instance attribute nor could it be found in > the class tree. Ah, I see what you're getting at - currently the interpreter doesn't fall back to the generic __getattribute__ slot for most special methods, so it makes it harder to write proxy objects, and those proxy objects aren't future proof. I suspect the right answer to that is to provide better metaprogramming tools to make writing proxy objects easier, rather than providing additional fallback paths for the special methods in the interpreter (as the latter slows down the common case to benefit the comparatively rare case of proxying). Exactly what those tools might be, I'm not sure :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Wed Apr 26 13:06:44 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 26 Apr 2006 21:06:44 +1000 Subject: [Python-3000] More batteries included! In-Reply-To: <5384bedc0604260350k1c92a212s8825c680516624bd@mail.gmail.com> References: <5384bedc0604260350k1c92a212s8825c680516624bd@mail.gmail.com> Message-ID: <444F5444.4090002@gmail.com> molasses wrote: > Hi all, > > How about including wxPython[1] and Wax[2] in the Py3k standard libs > and getting rid of the TK gui stuff? > > Also, I think pywin32[3] should be included. Or is there already a way > to use Windows COM objects in Python that I missed? > > PIL[4] would be great to have in the standard libs too. But that's > probably asking too much ;) > > [1] http://www.wxpython.org > [2] http://sourceforge.net/projects/waxgui > [3] http://sourceforge.net/projects/pywin32 > [4] http://www.pythonware.com/products/pil Since this is still well into the future, by the time Py3k rolls around it might be possible to include the option of easy-installing various packages from the Python installer. That's far more likely than too many big external libraries getting rolled into the reference interpreter. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From barry at python.org Wed Apr 26 14:19:48 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 26 Apr 2006 08:19:48 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <20060425224803.GA9983@panix.com> References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> <20060425224803.GA9983@panix.com> Message-ID: <1146053988.11946.72.camel@geddy.wooz.org> On Tue, 2006-04-25 at 15:48 -0700, Aahz wrote: > My opinion: if we were designing Python from scratch right now, we might > well consider having only set literals and not dict literals. However, > I don't think we can have both set and dict literals, and I think that > removing dict literals (or list literals) counts as gratuitous breakage. Agreed. The current status quo seems fine. Losing listcomps won't be a big problem because they can be automatically > fixed and they aren't much loss. Totally half-baked way-pre-morning-coffee thought: isn't [f(x) for x in seq] just another way to spell list(genexp) where genexp == f(x) for x in eq? So, if there are no list comps any more, but list literal notation could take a genexp, doesn't that give us today's syntactic convenience with tomorrow's consistence world view? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060426/354b28ba/attachment.pgp From barry at python.org Wed Apr 26 14:25:42 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 26 Apr 2006 08:25:42 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> <20060425224803.GA9983@panix.com> Message-ID: <1146054342.11946.78.camel@geddy.wooz.org> On Wed, 2006-04-26 at 08:03 +0000, Talin wrote: > In my experience dict literals are far more useful than set literals. In fact, I > don't think I've ever made a set literal. The only syntactical short cut that I > would find useful for a set would be "new empty set", for which I think "set()" > is probably short enough already. I agree that empty sets are created much more often than filled sets, and I have no problem writing "set()" for that. I have very occasionally written a pre-filled set, and the only thing about that is that the TypeError resulting from "set(1, 2, 3)" is at first confusing. But it's one of those things you 'get' after the first trip-up so again, I think it's fine. I forgot who wrote yesterday about literal notation being "about right" in Python 2, and I agree. List, tuple, and dict literals are one of the things that make Python Python. No reason to get rid of those, and I don't think filled-sets are common enough to warrant literal notation. > Actually, there is *one* thing that I would change about the current syntax, > which is that I would change dict literals to have the same internal syntax as > function calls, so that { } would be equivalent to dict( > ). This means using '=' instead of ':' and allowing key names to be bare words > instead of quoted strings. So you could write: > > { name="Frodo", race="Hobbit" } > > instead of: > > { 'name':'Frodo', 'race':'Hobbit' } > > (Only 4 characters longer, but *much* harder to type. Try typing it if you don't > believe me.) Very interesting. That's probably the primary reason why I've started preferring dict(...) over {...}. Of course it only works when your keys are identifier-like strings. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060426/688aee77/attachment.pgp From gvwilson at cs.utoronto.ca Wed Apr 26 14:40:11 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Wed, 26 Apr 2006 08:40:11 -0400 (EDT) Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 In-Reply-To: References: Message-ID: > From: Talin > In my experience dict literals are far more useful than set literals. In fact, I > don't think I've ever made a set literal. I think this is because: a) sets are new to Python --- habit and examples both encourage us to use lists, even when the collections in question aren't intrinsically ordered; and b) the current notation is unpleasant --- I appreciate Raymond et al's dislike of egregious punctuation (it's why we all abandoned Perl, right?), but {1, 2, 3} *is* a lot cleaner than set([1, 2, 3]). I think the second half of (a) is important: we all use lists (or tuples) to hold unordered collections because that's what we're used to, and because until recently, that's all we had. In contrast, flip through a copy of Aho, Hopcroft, and Ullman (if you're old enough to have it on your shelf ;-). I didn't see any examples of maps (dicts) outside the sections that are devoted to their design and implementation, but dozens of algorithms use sets. I think that the smaller we make the gap between what people write when they aren't constrained by implementation, and what they write when implementing, the better off we'll all be. > From: Aahz > My opinion: if we were designing Python from scratch right now, we might > well consider having only set literals and not dict literals. However, > I don't think we can have both set and dict literals, and I think that > removing dict literals (or list literals) counts as gratuitous breakage. I agree that we can't remove dict or list literals; I don't see why we can't have both sets and dicts. I'll bet a bottle of Macallan that newcomers will stumble over the set/dict distinction and notation less often than they currently do over the list/tuple distinction --- given an implementation, it's something we could actually measure by looking at error rates on introductory programming exercises. Thanks, Greg From aleaxit at gmail.com Wed Apr 26 16:43:15 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Wed, 26 Apr 2006 07:43:15 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <1146053988.11946.72.camel@geddy.wooz.org> References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> <20060425224803.GA9983@panix.com> <1146053988.11946.72.camel@geddy.wooz.org> Message-ID: On Apr 26, 2006, at 5:19 AM, Barry Warsaw wrote: > On Tue, 2006-04-25 at 15:48 -0700, Aahz wrote: > >> My opinion: if we were designing Python from scratch right now, we >> might >> well consider having only set literals and not dict literals. >> However, >> I don't think we can have both set and dict literals, and I think >> that >> removing dict literals (or list literals) counts as gratuitous >> breakage. > > Agreed. The current status quo seems fine. > > Losing listcomps won't be a big problem because they can be > automatically >> fixed and they aren't much loss. > > Totally half-baked way-pre-morning-coffee thought: isn't > > [f(x) for x in seq] > > just another way to spell > > list(genexp) where genexp == f(x) for x in eq? > > So, if there are no list comps any more, but list literal notation > could > take a genexp, doesn't that give us today's syntactic convenience with > tomorrow's consistence world view? If I read you right, that is exactly the reason I would prefer that Python 3k *NOT* have both notations that work today, i.e.: [ f(x) for x in seq ] as an exact synonym of list( f(x) for x in seq ) Today, among the reasons for preferring the list(......) form is that it doesn't leak name x into the enclosing scope. But even once py3k fixes that leakage, the list(...) form remains IMHO more readable -- and nobody's proposing to take it away, in any case, so the "one obvious way" mantra suggest taking away the alternate [...] syntax instead. Among other things, this will place lists, sets and other (builtin or user-coded) collection types on even syntactic footing for comprehension purposes: list( f(x) for x in seq ) set( f(x) for x in seq ) deque( f(x) for x in seq ) ... rather than singling one or two types out for special, different alternative comprehension syntax. On literals as opposed to comprehensions I have no strong opinions, although I don't think the status quo is actually ``fine'' (it IS a tiny bother, but a bother nevertheless, to have to write set ((23,42,77)) vs [23, 42, 77]...!), neither is it "badly broken". In fact the most common beginner error in writing literals is using (23) instead of (23,) when trying to make a singleton tuple, and I have no great idea on how to fix that, anyway;-). Alex From solipsis at pitrou.net Wed Apr 26 17:00:10 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Apr 2006 17:00:10 +0200 Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 In-Reply-To: References: Message-ID: <1146063611.5751.24.camel@fsol> Le mercredi 26 avril 2006 ? 08:40 -0400, Greg Wilson a ?crit : > I think this is because: > > a) sets are new to Python --- habit and examples both encourage us to use > lists, even when the collections in question aren't intrinsically > ordered; and > > b) the current notation is unpleasant --- I appreciate Raymond et al's > dislike of egregious punctuation (it's why we all abandoned Perl, > right?), but {1, 2, 3} *is* a lot cleaner than set([1, 2, 3]). I think Raymond H is right: 1) sets are most often built dynamically rather than statically (in my own experience) 2) set([1,2,3]) makes little sense anyway, since it probably isn't significantly more efficient than [1,2,3] The real benefit of sets is when you have at least tens or hundreds of elements; in that case you won't use a literal to build the set. Another remark is that I often use sets to hold my own objects rather than simple values (ints or strings). In a sense, I use sets as iterable containers where arbitrary remove() is fast. From kendall at monkeyfist.com Wed Apr 26 17:08:29 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Wed, 26 Apr 2006 11:08:29 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <444E416D.2040205@ewtllc.com> <20060425093223.66E8.JCARLSON@uci.edu> <20060425224803.GA9983@panix.com> Message-ID: <45964986-42D4-4552-A984-D3EEA781FE50@monkeyfist.com> On Apr 26, 2006, at 4:03 AM, Talin wrote: > This means using '=' instead of ':' and allowing key names to be > bare words > instead of quoted strings. For my money, this is another use case for symbols in Py3K. I very often mistype string keys in dicts, where I think I would mistype symbols less often (Guido's !symbol syntax isn't half-bad), and where upon access would be much nicer too: compare my_dict["the_key"] with my_dict[!the_key]. Only one character saved (for me) but that adds up and you have fewer variants (my_dict['the_key'] or my_dict["the_key"] versus my_dict[!the_key]) -- of course I'm not saying that strings shoudln't be keys in dicts, only that for a lot of the common uses of dicts, symbols are better. Cheers, Kendall Clark From jimjjewett at gmail.com Wed Apr 26 17:38:22 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 26 Apr 2006 11:38:22 -0400 Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 In-Reply-To: <1146063611.5751.24.camel@fsol> References: <1146063611.5751.24.camel@fsol> Message-ID: On 4/26/06, Antoine Pitrou wrote: > 2) set([1,2,3]) makes little sense anyway, since it > probably isn't significantly more efficient than [1,2,3] In the current implementation, is is less efficient. But a set is what you really intend; using a list simply because it is currently more efficient is a bad thing to encourage. The questions are (1) How to make set usage at least as easy as list/tuple. (2) How to make set usage at least as efficient. (3) Whether the tradeoffs are worth it. {1, 2, 3} as literal notation for a set makes it as easy (though there will still be habit to break). Making it as efficient would require changes under the cover. For example, sets small enough to fit in smalltable might use a linear scan and not bother with hashing. Both of these add complication to the langauge. Is doing the "correct" thing instead of "close enough" important enough to justify the changes? -jJ From gmccaughan at synaptics-uk.com Wed Apr 26 17:39:35 2006 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Wed, 26 Apr 2006 16:39:35 +0100 Subject: [Python-3000] sets in P3K? In-Reply-To: <1f7befae0604251220i35f41893r4847e732100588b1@mail.gmail.com> References: <444E2475.7050001@gmail.com> <1f7befae0604251220i35f41893r4847e732100588b1@mail.gmail.com> Message-ID: <200604261639.35991.gmccaughan@synaptics-uk.com> On Tuesday 2006-04-25 20:20, Tim Peters wrote: > Now you can only use a set if you can prove it exists, and one way to > prove a set exists is via what was once called the Axiom of > Comprehsion (but seems to be called the Axiom of Subsets more often > these days). That says that if you have a set S, then for any > one-place predicate P, {x ? S | P(x)} exists. So you can still make > up any predicate you like, but you can only use it to "separate out" > (why it's also sometimes, but I think rarely, called the Axiom of > Separation) the satisfiyng elements from another set. When I was last studying this stuff, "Comprehension" meant the unrestricted schema (which yields inconsistency with the rest of the ZF axioms) and "Separation" meant the restricted one. I've not heard "axiom of subsets", which I quite like (except that, like comprehension and separation, it's really an axiom schema, not a single axiom) despite the possibility of confusing it with the power set axiom. The Web of a Million Lies suggests that maybe terminological fashions have changed a bit in this area over the last decade or so. -- g From ianb at colorstudy.com Wed Apr 26 17:48:33 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 26 Apr 2006 10:48:33 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: <200604260956.00287.tdickenson@devmail.geminidataloggers.co.uk> References: <444E416D.2040205@ewtllc.com> <20060425224803.GA9983@panix.com> <200604260956.00287.tdickenson@devmail.geminidataloggers.co.uk> Message-ID: <444F9651.6050203@colorstudy.com> Toby Dickenson wrote: > On Wednesday 26 April 2006 09:03, Talin wrote: > > >>In my experience dict literals are far more useful than set literals. >>In fact, I don't think I've ever made a set literal. The only >>syntactical short cut that I would find useful for a set would >>be "new empty set", for which I think "set()" is probably short >>enough already. > > > I quite often write.... > > if state in (case1,case2,case3): > > as a shorter form of.... > > if state==case1 or state==case2 or state==case3: Yes, this is what I do; for small static sets inlined into source code lists or tuples are slightly more convenient, and so I'd never use a set in that situation without a set literal. I don't know if this points one way or the other, but that sets are only built up out of sequences is a byproduct of the (lack of) syntax, which makes people use lists for sets; it's not that literal sets aren't useful. OTOH, lists and tuples are a perfectly workable alternative to sets, so maybe it doesn't matter much. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From solipsis at pitrou.net Wed Apr 26 17:54:29 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 26 Apr 2006 17:54:29 +0200 Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 In-Reply-To: References: <1146063611.5751.24.camel@fsol> Message-ID: <1146066869.5751.57.camel@fsol> Le mercredi 26 avril 2006 ? 11:38 -0400, Jim Jewett a ?crit : > (2) How to make set usage at least as efficient. > (3) Whether the tradeoffs are worth it. Well, testing for inclusion in a container of 3 ints may not be significant vs. all the overhead of the interpreter and vs. what you are actually doing inbetween the inclusion tests. That's why I said "*significantly* more efficient" : I think both 3-int lists and 3-int sets are fast enough, aren't they ? > Making it as efficient would require changes under the cover. For > example, sets small enough to fit in smalltable might use a linear > scan and not bother with hashing. This optimization is sane with set literals built out of constants for which you know the comparison is fast. But searching for an element without hashing may also involve costly __eq__ calls. Or am I mistaken ? From jimjjewett at gmail.com Wed Apr 26 18:07:59 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 26 Apr 2006 12:07:59 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <444EDBE8.3020303@gradient.cis.upenn.edu> References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> Message-ID: On 4/25/06, Edward Loper wrote: > I think that a significant motivation for people that propose set > literals is that the following is just plain ugly: > > s = set([1,2,3]) > > It seems much more natural to write: > > s = set(1, 2, 3) > > s = set.from_iter(my_list) > s = set.from_iter(enumerate(my_list)) > s = set.from_iter(x for x in collection if x>10) Alternatively, define a classmethod for the literal form. For me, even the much longer s = set.literal(1, 2, 3) would be much better than s = set([1,2,3]) Nick's NumPy-inspired suggestion of s = set[1, 2, 3] is probably even better, though I'm not sure I would like what it does to the rest of the language. (The [] vs () would be arbitrary (historical), and Class.__getitem__() would be almost unrelated to Class().__getitem__(). ) -jJ From ianb at colorstudy.com Wed Apr 26 18:06:30 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 26 Apr 2006 11:06:30 -0500 Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 In-Reply-To: References: <1146063611.5751.24.camel@fsol> Message-ID: <444F9A86.3020106@colorstudy.com> Jim Jewett wrote: > On 4/26/06, Antoine Pitrou wrote: > >>2) set([1,2,3]) makes little sense anyway, since it >>probably isn't significantly more efficient than [1,2,3] > > > In the current implementation, is is less efficient. > > But a set is what you really intend; using a list simply because it is > currently more efficient is a bad thing to encourage. I my not-at-all-scientific tests, sets were faster when the set is created ahead of time, but just barely, except when you were testing something that was at the front of the list. Tuples were always a bit slower than sets or lists. But if you factor in the expense of actually creating the set/list/tuple, then the different is more significant; a little less than a factor of three between tuples and lists (since tuples can be allocated once), and a little less than a factor of three between lists and sets when building the set from a list (which is what everyone has been using as examples), or a little less than a factor of two when building from a tuple. I imagine a set literal would improve the set case a little, but since it hashes its arguments it's always going to have more overhead to create than a list. If you could create a *frozen* set with a literal, that should outperform lists and tuples. But I should note that I didn't do very thorough testing at all, just tested loops that used "in" on a three-item container (and in py2.4). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Wed Apr 26 18:06:33 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 26 Apr 2006 11:06:33 -0500 Subject: [Python-3000] Brainstorming: Python Metaprogramming In-Reply-To: References: <444CF246.7000104@colorstudy.com> Message-ID: <444F9A89.50201@colorstudy.com> Talin wrote: >>>Where 'quoted' was some sort of class that behaved like a reference to a >>>variable. So ?x.set( 1 ) is the same as x = 1. >> >>Sounds like lambda x: ... > > > More differences than similarities, I think. For one thing, you can't use > lambda x: ... to assign to x. And while you could possibly overload the > arithmetic operators on a lambda, you'd have to do it for each lambda > individually, since lambdas don't have classes. You mean, you can modify the value of a variable in an a calling function using this? Is that really what you mean? Because that's crazy Tcl talk. In the case of lambda, you'd have to transform it somehow, like make_query(lambda x: ...). One transformation is filling, so (lambda x: x+1)(1) seems (from what you describe) to be like (?x + 1); ?x.set(1)... though I'm unclear how you actually use that for something useful. >>>Moreover, you would want to customize all of the operators on quoted to return >>>an AST, so that: >>> >>> ?x + 1 >>> >>>produces something like: >>> >>> (add, quoted('x'), 1) >>> >>>...or whatever data structure is convenient. >> >>You can match the free variables from the lambda arguments against the >>variables in the AST to get this same info. > > > I don't understand what this means. I mean you can look at "lambda x: x+1" and get the AST, and see that "x" is an argument to the lambda, and construct something like (add, quoted('x'), 1). > Look, folks, I don't have a concrete proposal here. I'm actually fishing > for ideas, not looking to have my own ideas validated (or not.) My gut > tells me that there's something here worth looking into, but I haven't > put my finger on it yet. > > (Isn't it interesting how often we programmers, who pride ourselves on our > use of logic and reason, so often use intuition and hunches to solve > problems?) That's what everyone does, isn't it? Mathematicians do this and then pretend they didn't by writing up a "proof", but that's just a big lie. And don't even get me started on philosophers... -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From jimjjewett at gmail.com Wed Apr 26 18:14:00 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Wed, 26 Apr 2006 12:14:00 -0400 Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 In-Reply-To: <1146066869.5751.57.camel@fsol> References: <1146063611.5751.24.camel@fsol> <1146066869.5751.57.camel@fsol> Message-ID: On 4/26/06, Antoine Pitrou wrote: > That's why I said "*significantly* more efficient" : I think both 3-int > lists and 3-int sets are fast enough, aren't they ? Raymond H should answer this; my memory is that he changed some library code (or the compiler as a peephole optimization?) to use frozensets instead of listeral lists or tuples -- and then changed it back, because for 3 item sets, it was a loss. -jJ From ianb at colorstudy.com Wed Apr 26 18:13:42 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 26 Apr 2006 11:13:42 -0500 Subject: [Python-3000] Open Issues for string.format PEP In-Reply-To: References: <444C3BC9.7020000@colorstudy.com> <444CFAF7.2010407@colorstudy.com> Message-ID: <444F9C36.6030003@colorstudy.com> Talin wrote: > Ian Bicking colorstudy.com> writes: >>>I want to go back for a moment to the original question. Whenever someone asks >>>me "how should we solve this problem?" the first thing I always ask is "do we >>>really need to?" Once we've answered that, we can then go on to the issue of >>>coming up with a solution. >> >>Without this all dictionaries must be copied. I think maybe Guido >>suggested something involving a shortcut that would not covert the >>dictionary, but that shortcut isn't available to Python so the behavior >>could be different for custom formatters implemented in Python. Also, >>only dictionaries are allowed after **; that could be changed (maybe >>that's part of the signature changes?), but that doesn't change the fact >>that only a dictionary can be passed into a function with **. > > > I think you are missing some alternatives. You see, the question I was asking > wasn't "How important is it that we be able to efficiently pass whole > dictionaries to the format function?" The question was "How important is it to > be able to pass whole dictionaries at all?" Remember, that anything that you > can do with **args you can also do by explicitly passing the individual args > you want. If you allow for {0.name}, yes. Otherwise, no, not at all. > Another alternative is just to fix the efficiency problems with **args. The > reason that **args is so expensive is because of the need to funnel all calls > through the existing C API, which requires that all function calls have the > signature of ( PyTuple *args, PyDict *kwargs ). If you were somehow able to > bypass or extend the C API, you could take the **args from the function call, > and directly hook it up to the **args in the formal parameter list, without > any flattening. That seems like a tricky optimization that disappear in ways that are hard for people to understand. So: def x(**kw): return kw You can pass through the dictionary directly, right? What about: def x(**kw): v = kw.pop('template') return v.format(**kw) Well, you need a copy there, unless you want the caller to have their dictionary modified. Or what about: def x(template, **kw): return template.format(**kw) Now, if you call it as x('...', **d) then it can be optimized. But if you call it as x('...', var1='foo', **d) it can't? What about x(**d)? To me any optimization just feels like a corner case. > Assumming that neither of those options are available, I would probably then > do one of the following: > > 1) Create a different method name for the dict case: > > "{name}".vformat( mydict ) That would be fine. > 2) Have some sort of reserved, keyword-only argument for dicts: > > "{name}".format( indirect_args=mydict ) That seems not as nice, but maybe okay. > 3) Bring back field expressions: > > "{0.name}".format( mydict ) That would be fine sometimes. > (Perhaps the '.' operator can mean both '.' and '[]' depending on the type of > the object.) As an option that could be nice regardless; I'd be a little nervous about enabling that everywhere. It's a lot less powerful (and less of a security issue) than full evaluation, but still leans a bit in that direction. But without auto-calling it's not actually that bad. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From ianb at colorstudy.com Wed Apr 26 18:15:45 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 26 Apr 2006 11:15:45 -0500 Subject: [Python-3000] Give power to __getattr__ In-Reply-To: <444F5360.4070303@gmail.com> References: <444DB18D.1030004@gmx.de> <444DB5E8.10706@gmail.com> <444F12CD.9020408@gmx.de> <444F5360.4070303@gmail.com> Message-ID: <444F9CB1.90009@colorstudy.com> Nick Coghlan wrote: > I suspect the right answer to that is to provide better metaprogramming tools > to make writing proxy objects easier, rather than providing additional > fallback paths for the special methods in the interpreter (as the latter slows > down the common case to benefit the comparatively rare case of proxying). +1. But I guess also unrelated to py3k at that point. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From mcherm at mcherm.com Wed Apr 26 18:17:17 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Wed, 26 Apr 2006 09:17:17 -0700 Subject: [Python-3000] rough draft signature PEP Message-ID: <20060426091717.kzb6e411kw00800g@login.werra.lunarpages.com> Talin writes: > 2) I noticed in your PEP that you followed the same mental model as is currently > used in the compiler, which is that *args and **args are treated as special > cases. In other words, the description of a function's arguments consists of an > array of "regular" arguments, plus a couple of "special" argument fields. > > My mental model is a little different, as I tend to see the argument as a single > list, where arguments have different modifiers, where * and ** are modifiers. In > other word, my model follows more closely (IMHO) the way that arguments are read > by the programmer. Brett responds: > Yeah, I see what you are saying. But I actually held this view long > before Iearned how Python did things underneath the covers. > > I have no clue how people tend to view things. Anyone else care to > comment on how they tend to see arguments? My mental model matches Brett's. I think of the argument signature as defining what arguments are permitted/required by the caller -- more a static process than a dynamic one. So instead of imagining *args as meaning "now bind all the rest of the positional arguments" I think of it more as "and then you're allowed as many positional arguments as you like". -- Michael Chermside From barry at python.org Wed Apr 26 18:59:29 2006 From: barry at python.org (Barry Warsaw) Date: Wed, 26 Apr 2006 12:59:29 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> Message-ID: <1146070769.10733.49.camel@resist.wooz.org> On Wed, 2006-04-26 at 12:07 -0400, Jim Jewett wrote: > Alternatively, define a classmethod for the literal form. For me, > even the much longer > > s = set.literal(1, 2, 3) Or how about just set.new(1, 2, 3) ? not-sure-it's-worth-it-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060426/48728ea0/attachment.pgp From rhettinger at ewtllc.com Wed Apr 26 20:07:39 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Wed, 26 Apr 2006 11:07:39 -0700 Subject: [Python-3000] Iterables and scalars was: sets in P3K? In-Reply-To: <444EDBE8.3020303@gradient.cis.upenn.edu> References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> Message-ID: <444FB6EB.5050407@ewtllc.com> Edward Loper wrote: >I think that a significant motivation for people that propose set >literals is that the following is just plain ugly: > > s = set([1,2,3]) > >It seems much more natural to write: > > s = set(1, 2, 3) > >However, it is fairly common to want to build a set from a collection, >an iterator, or a genexp. With the current notation, all three of these >are easy. But with this "more natural" notation, they're no longer >possible without resorting to varargs (which would create an unnecessary >imtermediary tuple). I.e., we don't want to lose the ability to do any >of the following: > > s = set(my_list) > s = set(enumerate(my_list)) > s = set(x for x in collection if x>10) > >One way around this conflict might be to define a new factory function, >that generates a set from an iterable. I.e., something like: > > s = set.from_iter(my_list) > s = set.from_iter(enumerate(my_list)) > s = set.from_iter(x for x in collection if x>10) > > We're no longer talking about an issue unique to the set and frozenset datatypes. Many elements of the language are challenged by conflicting desires for a function or type choose between taking an iterable argument or to have them already unwound. IMO, the worst solution is trying to accommodate both within a single callable. The problematic %-formatting operator is one example. The min() and max() functions are another: >>> min(3,4,5) 3 >>> min([3,4,5]) 3 The current solution for sets is to provide only the iterable form and to require the user to create a list when he/she wants to write-out a literal: set([1,2,3]). The advantage of this approach is that it has a small footprint in terms of human memory because it takes advantage of pre-existing knowledge about how to write list literals and because it doesn't double the number of forms for every function and type. Edward's solution is to provide an alternate classmethod or factory function. IMO, this doesn't solve the general problem unless it is applied throughout the language: list(1) list.from_iter([1]) tuple(1) tuple.from_iter([1]) set(1) set.from_iter([1]) If those examples make your stomach churn, then let's not go down the path of adding all these classmethods and factory functions. The existing approach using list literals is dirt simple and universally applicable. While writing set([1,2,3]) is not a perfect solution, it is much less complex than non-generalizable alternatives involving classmethods, factory functions, or punctuation. Raymond From jcarlson at uci.edu Wed Apr 26 20:34:42 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 26 Apr 2006 11:34:42 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <444EDBE8.3020303@gradient.cis.upenn.edu> References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> Message-ID: <20060426112227.66F7.JCARLSON@uci.edu> Edward Loper wrote: > > I think that a significant motivation for people that propose set > literals is that the following is just plain ugly: > > s = set([1,2,3]) > > It seems much more natural to write: > > s = set(1, 2, 3) I agree. > However, it is fairly common to want to build a set from a collection, > an iterator, or a genexp. With the current notation, all three of these > are easy. But with this "more natural" notation, they're no longer > possible without resorting to varargs (which would create an unnecessary > imtermediary tuple). Right now you are manually creating an intermediate list or tuple to pass to set, so this wouldn't seem to be a significant change to me (it's just an argument semantic addition). - Josiah From jcarlson at uci.edu Wed Apr 26 20:53:07 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 26 Apr 2006 11:53:07 -0700 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: References: <444D1039.8050007@colorstudy.com> Message-ID: <20060426113617.66FA.JCARLSON@uci.edu> Talin wrote: > Ian Bicking colorstudy.com> writes: > > Josiah Carlson wrote: > > > One of the features of generator expressions which makes it desireable > > > instead of list comprehensions is that generator expressions may use > > > less memory *now*, and may be able to start returning results *now*. > > > > > > Using ( orderby ...) as a replacement for sorted((genexp), key=...) > > > is a bit misleading because while the original generator expression > > > could have been space and time efficient, the orderby version certainly may > > > not be. > > > > Certainly it changes the performance substantially (of course if the > > expression is translated and executed elsewhere the performance can > > actually be improved, so it can go both ways). Since list > > comprehensions are planned to just be syntactic sugar for generator > > comprehension, generator comprehensions are now the more fundamental > > construct. > > > > But yeah, it is a little awkward, since something that is sorted can be > > returned as a list anyway, except for the fact that the expression > > itself could be ported off elsewhere which isn't normal Python. (But > > should be normal Python!) > > > > I think that I am (mostly) in agreement with Ian on this one, but perhaps with > different reasoning. > > There seems to be a common feeling on py-dev that "list comprehensions are just > a special case of generators expressions, so really, we don't need them". > > But then we turn around and say "Well, this feature might be handy for list > comprehensions, but since list comprehensions are based on generator > expressions, and since this feature would make generator expressions > inefficient, we won't do it." No. I was only responding to the question of orderby in relation to generator expressions. In generator expressions, it is further unnecessary because one can always wrap the generator expression up with a sorted(genexp, ...) to get your sorted version of the generator (in list form). In the case of list comprehensions, it is doubly unnecessary, because you can again use sorted([genexp], ...) or even list.sort(...) . > In other words, we dismiss any evidence that disproves our initial assumption. > This is sort of like saying that we don't need the 'def' statement, since you > can define functions with "func = lambda"; and also we shouldn't add multi-line > statements, since lambda doesn't support them. You are letting the tail wag the dog. From what I understand, lambda was a post-facto introduction to the language along with map and filter to allow for a more functional approach with Python. def already exists. I would also point out that all of those modifications to lambda (multi-line function bodies, statements, etc.) have all been proposed and *rejected*. Indeed, it is a similar case for list comprehensions and generator expressions. Generator expressions did come after list comprehensions; but note that Ian's original request was in relation to generator expressions, _not_ list comprehensions, and in the realm of generator expressions, I believe, orderby is a non-starter (for the reasons I've already expressed). > The fact is that sorting happens a *lot*. Sorting with key= happens a lot. A > large proportion of my list comprehensions involve sorting or ordering of one > form or another, and the number of ugly sorted + key + lambda expressions is > enough to convince me that an "orderby" or "ascending" or whatever clause would > be a welcome addition. Sorting may happen "a *lot*" with what you write, but in the software that I write, sorting *rarely* happens. If you would like to lessen your use of lambda, you should consider discovering the operator.attrgetter() and operator.itemgetter() functions to be passed to sorted(). - Josiah From ianb at colorstudy.com Wed Apr 26 21:15:19 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 26 Apr 2006 14:15:19 -0500 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: <20060426113617.66FA.JCARLSON@uci.edu> References: <444D1039.8050007@colorstudy.com> <20060426113617.66FA.JCARLSON@uci.edu> Message-ID: <444FC6C7.3060702@colorstudy.com> Josiah Carlson wrote: > Talin wrote: > >>Ian Bicking colorstudy.com> writes: >> >>>Josiah Carlson wrote: >>> >>>>One of the features of generator expressions which makes it desireable >>>>instead of list comprehensions is that generator expressions may use >>>>less memory *now*, and may be able to start returning results *now*. >>>> >>>>Using ( orderby ...) as a replacement for sorted((genexp), key=...) >>>>is a bit misleading because while the original generator expression >>>>could have been space and time efficient, the orderby version certainly may >>>>not be. >>> >>>Certainly it changes the performance substantially (of course if the >>>expression is translated and executed elsewhere the performance can >>>actually be improved, so it can go both ways). Since list >>>comprehensions are planned to just be syntactic sugar for generator >>>comprehension, generator comprehensions are now the more fundamental >>>construct. >>> >>>But yeah, it is a little awkward, since something that is sorted can be >>>returned as a list anyway, except for the fact that the expression >>>itself could be ported off elsewhere which isn't normal Python. (But >>>should be normal Python!) >>> >> >>I think that I am (mostly) in agreement with Ian on this one, but perhaps with >>different reasoning. >> >>There seems to be a common feeling on py-dev that "list comprehensions are just >>a special case of generators expressions, so really, we don't need them". >> >>But then we turn around and say "Well, this feature might be handy for list >>comprehensions, but since list comprehensions are based on generator >>expressions, and since this feature would make generator expressions >>inefficient, we won't do it." > > > No. I was only responding to the question of orderby in relation to > generator expressions. In generator expressions, it is further > unnecessary because one can always wrap the generator expression up with > a sorted(genexp, ...) to get your sorted version of the generator (in > list form). > > In the case of list comprehensions, it is doubly unnecessary, because > you can again use sorted([genexp], ...) or even list.sort(...) . Using sorted is syntactically different: [(p.fname, p.lname) for p in person orderby (p.lname, p.fname)] vs: sorted((p.fname, p.lname) for p in person, key=lambda name: (name[1], name[0])) It's not a huge difference, but to argue that it's unnecessary because sorted() exists is the same as arguing that list/generator comprehensions are unnecessary because of map/imap and filter/ifilter. >>The fact is that sorting happens a *lot*. Sorting with key= happens a lot. A >>large proportion of my list comprehensions involve sorting or ordering of one >>form or another, and the number of ugly sorted + key + lambda expressions is >>enough to convince me that an "orderby" or "ascending" or whatever clause would >>be a welcome addition. > > > Sorting may happen "a *lot*" with what you write, but in the software > that I write, sorting *rarely* happens. If you would like to lessen > your use of lambda, you should consider discovering the > operator.attrgetter() and operator.itemgetter() functions to be passed > to sorted(). Well now you are just being silly... foo.sort(key=lambda x: x.name) or import operator foo.sort(key=operator.attrgetter('name')) ? That second one is just there for people who are irrationally opposed to lambda, it's not a serious alternative. If someone asked me how to sort based on an attribute, I would *never* tell them to use operator.attrgetter(). -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From jcarlson at uci.edu Wed Apr 26 22:13:00 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 26 Apr 2006 13:13:00 -0700 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: <444FC6C7.3060702@colorstudy.com> References: <20060426113617.66FA.JCARLSON@uci.edu> <444FC6C7.3060702@colorstudy.com> Message-ID: <20060426123824.6703.JCARLSON@uci.edu> Ian Bicking wrote: >Josiah Carlson wrote: > >Talin wrote: > >>Ian Bicking colorstudy.com> writes: > >>>Josiah Carlson wrote: > >>> > >>>>One of the features of generator expressions which makes it desireable > >>>>instead of list comprehensions is that generator expressions may use > >>>>less memory *now*, and may be able to start returning results *now*. > >>>> > >>>>Using ( orderby ...) as a replacement for sorted((genexp), key=...) > >>>>is a bit misleading because while the original generator expression > >>>>could have been space and time efficient, the orderby version certainly may > >>>>not be. > >>> > >>>Certainly it changes the performance substantially (of course if the > >>>expression is translated and executed elsewhere the performance can > >>>actually be improved, so it can go both ways). Since list > >>>comprehensions are planned to just be syntactic sugar for generator > >>>comprehension, generator comprehensions are now the more fundamental > >>>construct. > >>> > >>>But yeah, it is a little awkward, since something that is sorted can be > >>>returned as a list anyway, except for the fact that the expression > >>>itself could be ported off elsewhere which isn't normal Python. (But > >>>should be normal Python!) > >>> > >> > >>I think that I am (mostly) in agreement with Ian on this one, but perhaps with > >>different reasoning. > >> > >>There seems to be a common feeling on py-dev that "list comprehensions are just > >>a special case of generators expressions, so really, we don't need them". > >> > >>But then we turn around and say "Well, this feature might be handy for list > >>comprehensions, but since list comprehensions are based on generator > >>expressions, and since this feature would make generator expressions > >>inefficient, we won't do it." > > > > > > No. I was only responding to the question of orderby in relation to > > generator expressions. In generator expressions, it is further > > unnecessary because one can always wrap the generator expression up with > > a sorted(genexp, ...) to get your sorted version of the generator (in > > list form). > > > > In the case of list comprehensions, it is doubly unnecessary, because > > you can again use sorted([genexp], ...) or even list.sort(...) . > > Using sorted is syntactically different: > > [(p.fname, p.lname) for p in person > orderby (p.lname, p.fname)] > > vs: > > sorted((p.fname, p.lname) for p in person, > key=lambda name: (name[1], name[0])) I could have sworn we were talking about a combination of syntax and semantics differences, so a difference in syntax, I would assume, is to be expected ;). > It's not a huge difference, but to argue that it's unnecessary because > sorted() exists is the same as arguing that list/generator > comprehensions are unnecessary because of map/imap and filter/ifilter. You can draw parallels as to why my argument vis-a-vis sorted doesn't work, and I will point out my original argument; orderby takes O(nlogn) time and O(n) space (given a generator that would have returned n items). Fundamentally, one of the reasons to use generator expressions rather than list comprehensions was that you wouldn't have to wait for all of the items, and not all items would need to be in memory simultaneously. This would be a not-insignificant change in time/memory use *semantics*. The use of orderby in generator expressions is fundamentally equivalent to the Decorate-Sort-Undecorate pattern on lists. While I believe that such would be convenient, I also believe that it is inherantly dishonest from a time/space expectation perspective to allow them in generator expressions. In list comprehensions? Sure. At least one isn't being quite so dishonest from a time/space perspective, but then one is getting the equivalent of an automatic syntax conversion from... [... orderby XXX] to... sorted((...), key=lambda x: XXX) Is this desireable? As I said above, yes. Is it a compelling syntax addition? I don't believe so. Do Ian and Talin believe it compelling? It would seem so. The question remains whether others also find it a compelling (or not) addition to the list comprehension or generator expression syntax, vis-a-vis the change in memory and time use of orderby in generator expressions. > Well now you are just being silly... Indeed, but only if you use operator.attrgetter and operator.itemgetter as their full names. One can always bind those functions to other names in any namespace. > ? That second one is just there for people who are irrationally opposed > to lambda, it's not a serious alternative. If someone asked me how to > sort based on an attribute, I would *never* tell them to use > operator.attrgetter(). Interesting, I didn't realize that any portion of the standard library was influenced due to 'irrational opposition to lambda'. I could have sworn the opposite (given lambda's use in so many of the standard library modules). - Josiah From ianb at colorstudy.com Wed Apr 26 22:24:14 2006 From: ianb at colorstudy.com (Ian Bicking) Date: Wed, 26 Apr 2006 15:24:14 -0500 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: <20060426123824.6703.JCARLSON@uci.edu> References: <20060426113617.66FA.JCARLSON@uci.edu> <444FC6C7.3060702@colorstudy.com> <20060426123824.6703.JCARLSON@uci.edu> Message-ID: <444FD6EE.9080702@colorstudy.com> Josiah Carlson wrote: >>It's not a huge difference, but to argue that it's unnecessary because >>sorted() exists is the same as arguing that list/generator >>comprehensions are unnecessary because of map/imap and filter/ifilter. > > > You can draw parallels as to why my argument vis-a-vis sorted doesn't > work, and I will point out my original argument; orderby takes O(nlogn) > time and O(n) space (given a generator that would have returned n items). > Fundamentally, one of the reasons to use generator expressions rather > than list comprehensions was that you wouldn't have to wait for all of > the items, and not all items would need to be in memory simultaneously. > This would be a not-insignificant change in time/memory use *semantics*. > > The use of orderby in generator expressions is fundamentally equivalent > to the Decorate-Sort-Undecorate pattern on lists. While I believe that > such would be convenient, I also believe that it is inherantly dishonest > from a time/space expectation perspective to allow them in generator > expressions. In list comprehensions? Sure. At least one isn't being > quite so dishonest from a time/space perspective, but then one is > getting the equivalent of an automatic syntax conversion from... > > [... orderby XXX] > > to... > > sorted((...), key=lambda x: XXX) > > Is this desireable? As I said above, yes. Is it a compelling syntax > addition? I don't believe so. Do Ian and Talin believe it compelling? > It would seem so. The question remains whether others also find it a > compelling (or not) addition to the list comprehension or generator > expression syntax, vis-a-vis the change in memory and time use of > orderby in generator expressions. My ulterior motive for introducing this into generator expressions is that then generator expressions could be used to do LINQ-ish things. This isn't required for that, but it pulls the entire idea together better than a generator expression plus a lambda. List comprehensions, being immediately evaluated, cannot be used in this fashion. This is admittedly a rather roundabout reason for orderby, but it's also a relatively nonintrusive way to introducing a feature for which there is demonstrable demand (since people are already doing this in Python in using even more hackish techniques). Ignoring this ulterior motive, I also think it's a nice feature for list comprehension. >>Well now you are just being silly... > > > Indeed, but only if you use operator.attrgetter and operator.itemgetter > as their full names. One can always bind those functions to other names > in any namespace. > > > >>? That second one is just there for people who are irrationally opposed >>to lambda, it's not a serious alternative. If someone asked me how to >>sort based on an attribute, I would *never* tell them to use >>operator.attrgetter(). > > > Interesting, I didn't realize that any portion of the standard library > was influenced due to 'irrational opposition to lambda'. I could have > sworn the opposite (given lambda's use in so many of the standard > library modules). Offering operator.attrgetter as an alternative to a simple lambda is either a disingenous argument (because the person so arguing is very unlikely to actually make that choice in their own code, or seriously suggest it to other people), or is based on a desire to use a highly functional style that borders on the irrational. There are some reasons you would use attrgetter, though not many; there's no good reason to use that function with a string literal argument. -- Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org From guido at python.org Thu Apr 27 00:10:02 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Apr 2006 15:10:02 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <20060426112227.66F7.JCARLSON@uci.edu> References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> <20060426112227.66F7.JCARLSON@uci.edu> Message-ID: On 4/26/06, Josiah Carlson wrote: > > Edward Loper wrote: > > > > I think that a significant motivation for people that propose set > > literals is that the following is just plain ugly: > > > > s = set([1,2,3]) > > > > It seems much more natural to write: > > > > s = set(1, 2, 3) > > I agree. > > > However, it is fairly common to want to build a set from a collection, > > an iterator, or a genexp. With the current notation, all three of these > > are easy. But with this "more natural" notation, they're no longer > > possible without resorting to varargs (which would create an unnecessary > > imtermediary tuple). > > Right now you are manually creating an intermediate list or tuple to > pass to set, so this wouldn't seem to be a significant change to me > (it's just an argument semantic addition). I have no idea what you mean by "argument semantic addition". I don't think this form (set(1, 2, 3)) will ever fly, because it would makes set(x) ambiguous. Currently, this interprets x as an iterable and produces a set with the elements produced by iterating over x. But if set(x, y) is a set of two elements, then set(x) should be a set of one element, *regardless of the type of x*. Curiously, min() and max() successfully use the same overloading. However, these don't have the same problem, because min() or max() of one value makes no sense, so it's safe to define these such that if they are called with a single argument, the argument is assumed to be an iterable. But constructing a set() of one value *does* make sense (and already has a meaning) so the overloading fails here. I find the many attempts to come up with a reasonable way to construct a set from a list of given values; I hope that {x, y} can still come out as a winner, with set(x) useful to construct a set from a pre-existing iterable. Note that set(range(10)) is a set with 10 elements; {range(10)} of course would be a set with one element. The same differences exists between list(range(10)) and [range(10)]. Sets will become more "mainstream" in Python 3000 if the views proposal is accepted; once dict.keys() and dict.items() return set views (i.e. objects that implement most of the set API but whose contents are linked to the underlying dict instance), the awareness of sets is bound to increase. Regarding set vs. frozenset: the Google query ``frozenset filetype:py'' returns only 160 results. It's not fair to compare this to the 230K results for "set filetype:py" because set is also a common method name and a common word, but still it's an underwhelming number of results. I found 70K results for tuple and dict each, 200K+ for list, 50K for unicode. Frozen sets just aren't all that popular. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 27 00:15:35 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Apr 2006 15:15:35 -0700 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: <444CFB50.6010809@colorstudy.com> References: <444CFB50.6010809@colorstudy.com> Message-ID: Let me be the one to say no for a change. Python 3000 will *not* add syntax to get generator expressions or list (or set) comprehensions sorted. There is exactly one sort algorithm, and it is list.sort(). There are exactly two sort APIs, the other being the built-in sorted(). That's enough. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Thu Apr 27 00:19:41 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 26 Apr 2006 15:19:41 -0700 Subject: [Python-3000] Adding sorting to generator comprehension In-Reply-To: <444FD6EE.9080702@colorstudy.com> References: <20060426123824.6703.JCARLSON@uci.edu> <444FD6EE.9080702@colorstudy.com> Message-ID: <20060426145421.6706.JCARLSON@uci.edu> Ian Bicking wrote: > Josiah Carlson wrote: > > Is this desireable? As I said above, yes. Is it a compelling syntax > > addition? I don't believe so. Do Ian and Talin believe it compelling? > > It would seem so. The question remains whether others also find it a > > compelling (or not) addition to the list comprehension or generator > > expression syntax, vis-a-vis the change in memory and time use of > > orderby in generator expressions. > > My ulterior motive for introducing this into generator expressions is > that then generator expressions could be used to do LINQ-ish things. > This isn't required for that, but it pulls the entire idea together > better than a generator expression plus a lambda. List comprehensions, > being immediately evaluated, cannot be used in this fashion. That is quite a circuitous route to get some LINQ syntax into Python. Personally, I've never been terribly enamoured with SQL-like syntaxes, so the idea of orderby (when we can always post-process) isn't terribly appealing to me. > This is admittedly a rather roundabout reason for orderby, but it's also > a relatively nonintrusive way to introducing a feature for which there > is demonstrable demand (since people are already doing this in Python in > using even more hackish techniques). I guess I haven't seen any of these hackish techniques. > >>? That second one is just there for people who are irrationally opposed > >>to lambda, it's not a serious alternative. If someone asked me how to > >>sort based on an attribute, I would *never* tell them to use > >>operator.attrgetter(). > > > > Interesting, I didn't realize that any portion of the standard library > > was influenced due to 'irrational opposition to lambda'. I could have > > sworn the opposite (given lambda's use in so many of the standard > > library modules). > > Offering operator.attrgetter as an alternative to a simple lambda is > either a disingenous argument (because the person so arguing is very > unlikely to actually make that choice in their own code, or seriously > suggest it to other people), or is based on a desire to use a highly > functional style that borders on the irrational. There are some reasons > you would use attrgetter, though not many; there's no good reason to use > that function with a string literal argument. I would also argue that not using Python's already built-in metaprogramming features to solve this particular problem is also disingenous. More specicifically... class Getters: def __getattr__(self, attr): return operator.attrgetter(attr) def __getitem__(self, item): return operator.itemgetter(item) G = Getters() class tpl: def __init__(self, *items): self.items = items def __call__(self, item): if len(self.items) == 1: return self.items[0](item) return tuple(i(item) for i in self.items) x = sorted(((i.firstname, i.lastname) for i in names), key=tpl(G.lastname, G.firstname)) Now that I've gone ahead and both made operator.*getter() more equivalent from a syntax standpoint, as well as made multi-key sorting more convenient I'm hard-pressed to find any benefit to orderby instead of sorted, except for perhaps a speed improvement due to the reduction in function calls, and as a first step towards a LINQ-ish syntax. Now that we've gotten that out of the way, do we want LINQ-ish syntax for Python? As I said earlier in this email, I never much liked SQL syntax, and LINQ seems to be a variant of SQL embedded in certain parts of other languages, so I would answer: no, I don't want LINQ-ish syntax for Python. - Josiah From guido at python.org Thu Apr 27 00:22:49 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Apr 2006 15:22:49 -0700 Subject: [Python-3000] Iterables and scalars was: sets in P3K? In-Reply-To: <444FB6EB.5050407@ewtllc.com> References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> <444FB6EB.5050407@ewtllc.com> Message-ID: Agreed. set(), like list() and tuple(), should take exactly one argument, which is an iterable providing the set elements. Class methods are pretty ugly and should be limited to cases that are rare but nevertheless useful -- dict.fromkeys saves a builtin and that's about all there is to say in its favor. In general dict should not be looked at as an example: the dict constructor signature is already too far overloaded, giving a different meaning to keyword args, and distinguishing between a mapping and a sequence is iffy: it requires a "keys" method which is kind of sub-optimal compared to using "iteritems" if it existed... --Guido On 4/26/06, Raymond Hettinger wrote: > Edward Loper wrote: > > >I think that a significant motivation for people that propose set > >literals is that the following is just plain ugly: > > > > s = set([1,2,3]) > > > >It seems much more natural to write: > > > > s = set(1, 2, 3) > > > >However, it is fairly common to want to build a set from a collection, > >an iterator, or a genexp. With the current notation, all three of these > >are easy. But with this "more natural" notation, they're no longer > >possible without resorting to varargs (which would create an unnecessary > >imtermediary tuple). I.e., we don't want to lose the ability to do any > >of the following: > > > > s = set(my_list) > > s = set(enumerate(my_list)) > > s = set(x for x in collection if x>10) > > > >One way around this conflict might be to define a new factory function, > >that generates a set from an iterable. I.e., something like: > > > > s = set.from_iter(my_list) > > s = set.from_iter(enumerate(my_list)) > > s = set.from_iter(x for x in collection if x>10) > > > > > We're no longer talking about an issue unique to the set and frozenset > datatypes. Many elements of the language are challenged by conflicting > desires for a function or type choose between taking an iterable > argument or to have them already unwound. > > IMO, the worst solution is trying to accommodate both within a single > callable. The problematic %-formatting operator is one example. The > min() and max() functions are another: > > >>> min(3,4,5) > 3 > >>> min([3,4,5]) > 3 > > The current solution for sets is to provide only the iterable form and > to require the user to create a list when he/she wants to write-out a > literal: set([1,2,3]). The advantage of this approach is that it has a > small footprint in terms of human memory because it takes advantage of > pre-existing knowledge about how to write list literals and because it > doesn't double the number of forms for every function and type. > > Edward's solution is to provide an alternate classmethod or factory > function. IMO, this doesn't solve the general problem unless it is > applied throughout the language: > > list(1) > list.from_iter([1]) > tuple(1) > tuple.from_iter([1]) > set(1) > set.from_iter([1]) > > If those examples make your stomach churn, then let's not go down the > path of adding all these classmethods and factory functions. The > existing approach using list literals is dirt simple and universally > applicable. While writing set([1,2,3]) is not a perfect solution, it is > much less complex than non-generalizable alternatives involving > classmethods, factory functions, or punctuation. > > > Raymond > > > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jcarlson at uci.edu Thu Apr 27 00:55:43 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Wed, 26 Apr 2006 15:55:43 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <20060426112227.66F7.JCARLSON@uci.edu> Message-ID: <20060426152838.6709.JCARLSON@uci.edu> "Guido van Rossum" wrote: > On 4/26/06, Josiah Carlson wrote: > > > > Edward Loper wrote: > > > > > > I think that a significant motivation for people that propose set > > > literals is that the following is just plain ugly: > > > > > > s = set([1,2,3]) > > > > > > It seems much more natural to write: > > > > > > s = set(1, 2, 3) > > > > I agree. > > > > > However, it is fairly common to want to build a set from a collection, > > > an iterator, or a genexp. With the current notation, all three of these > > > are easy. But with this "more natural" notation, they're no longer > > > possible without resorting to varargs (which would create an unnecessary > > > imtermediary tuple). > > > > Right now you are manually creating an intermediate list or tuple to > > pass to set, so this wouldn't seem to be a significant change to me > > (it's just an argument semantic addition). > > I have no idea what you mean by "argument semantic addition". Arguments to a function have a specified semantic result. Allow all of all the previously working argument passing semantics to work as they always did (sequences), then add a new allowed argument passing semantic. > I don't think this form (set(1, 2, 3)) will ever fly, because it would > makes set(x) ambiguous. Currently, this interprets x as an iterable > and produces a set with the elements produced by iterating over x. But > if set(x, y) is a set of two elements, then set(x) should be a set of > one element, *regardless of the type of x*. The ambiguity isn't when using set(1), it's when using set((1,2)). set (1) has one very clear semantic meaning: create a set of one item where that item is 1. However, set((1,2)) has two meanings: create a set of two items, one of those items being 1, the other item being 2 - or create a set of a single item, where that single item is a tuple. > Curiously, min() and max() successfully use the same overloading. > However, these don't have the same problem, because min() or max() of > one value makes no sense, so it's safe to define these such that if > they are called with a single argument, the argument is assumed to be > an iterable. But constructing a set() of one value *does* make sense > (and already has a meaning) so the overloading fails here. Indeed, but only with immutable iterable sequences. Mutable iterable sequences are unambiguous (they don't hash(), so can't be an entry in and of themselves). def _set(*args): if len(args) > 1: #one unambiguous case return set(args) arg = args[0] try: #catches non-sequences _ = iter(arg) except TypeError: #another unambiguous case return set((arg,)) try: #catches mutable sequences _ = hash(arg) except: #unambiguous for non-hashables return set(arg) else: #ambiguous for any immutable iterable sequence... return set(arg) However, being that this is the Py3k list, and the only ambiguous case is that of a single immutable sequence argument to set(), we could state an execution semantic and call it sufficient. Being that set() right now results in the insertion of each item from that sequence into a new set (set((1,2)) -> set([1,2]) ), we wouldn't need to change the obvious (at least to some) meaning, or really "the way it works now" meaning. On the other hand, I note that both Raymond and Guido are saying -1, so I'll not push it (though I'm also still against more literal syntax), even though one can come to an almost obvious signature. > I find the many attempts to come up with a reasonable way to construct > a set from a list of given values; I hope that {x, y} can still come > out as a winner, with set(x) useful to construct a set from a > pre-existing iterable. Note that set(range(10)) is a set with 10 > elements; {range(10)} of course would be a set with one element. Or really {tuple(range(10))}? - Josiah From tjreedy at udel.edu Thu Apr 27 00:54:17 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 26 Apr 2006 18:54:17 -0400 Subject: [Python-3000] sets in P3K? References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com><444EDBE8.3020303@gradient.cis.upenn.edu><20060426112227.66F7.JCARLSON@uci.edu> Message-ID: "Guido van Rossum" wrote in message news:ca471dc20604261510j6c85563el25af280d2e2b0980 at mail.gmail.com... > Curiously, min() and max() successfully use the same overloading. > However, these don't have the same problem, because min() or max() of > one value makes no sense, The expressions 'min(x)' and 'max(x)', x an ordered set member, both make perfect sense. (And indeed, min([x])==max([x])==x.) But since the result is simply x, the act of *writing* the expression in code (as opposed to expository text) does not make sense. Thanks for the clarification on when this type of overloading can and can't work. Terry Jan Reedy From guido at python.org Thu Apr 27 01:08:06 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Apr 2006 16:08:06 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <20060426152838.6709.JCARLSON@uci.edu> References: <20060426112227.66F7.JCARLSON@uci.edu> <20060426152838.6709.JCARLSON@uci.edu> Message-ID: On 4/26/06, Josiah Carlson wrote: > Indeed, but only with immutable iterable sequences. Mutable iterable > sequences are unambiguous (they don't hash(), so can't be an entry in > and of themselves). Unfortunately, generator expressions happen to have a (useless) hash value. > def _set(*args): > if len(args) > 1: > #one unambiguous case > return set(args) > arg = args[0] > try: > #catches non-sequences > _ = iter(arg) > except TypeError: > #another unambiguous case > return set((arg,)) > try: > #catches mutable sequences > _ = hash(arg) > except: > #unambiguous for non-hashables > return set(arg) > else: > #ambiguous for any immutable iterable sequence... > return set(arg) This is extremely distasteful. Catching exceptions in order to detect what to do is an approach of last resort, when you've otherwise painted yourself into a bad corner; it has a high probability of at some point hiding a genuine exception that should have triggered a traceback instead of the code taking a different behavior. This is well-documented. Also, it's deadly for code readability(). When we see set(x) we won't know what was intended unless we know a lot about the type of x. We'd get the same problems we have today with "...%s..." % x where x happens to be a tuple under certain circumstances. This kind of proposal might fly in a statically typed language. It doesn't work in Python. > However, being that this is the Py3k list, and the only ambiguous case > is that of a single immutable sequence argument to set(), we could > state an execution semantic and call it sufficient. Choosing a semantic doesn't help if it is confusing or likely to hide bugs. > Being that set() right now results in the > insertion of each item from that sequence into a new set (set((1,2)) -> > set([1,2]) ), we wouldn't need to change the obvious (at least to some) > meaning, or really "the way it works now" meaning. > > On the other hand, I note that both Raymond and Guido are saying -1, so > I'll not push it (though I'm also still against more literal syntax), > even though one can come to an almost obvious signature. Right. I'll stop now too; we all know where we stand and in *this* case the BDFL's opinion still prevails. :-) > > I find the many attempts to come up with a reasonable way to construct > > a set from a list of given values; I hope that {x, y} can still come > > out as a winner, with set(x) useful to construct a set from a > > pre-existing iterable. Note that set(range(10)) is a set with 10 > > elements; {range(10)} of course would be a set with one element. > > Or really {tuple(range(10))}? I didn't say what the type of that element would be. Even today, [xrange(10), xrange(20)] is a totally valid expression. In Python 3000, {range(10)} would mean what you'd write today as set([xrange(10)]). -- --Guido van Rossum (home page: http://www.python.org/~guido/) From edloper at gradient.cis.upenn.edu Thu Apr 27 02:41:59 2006 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Wed, 26 Apr 2006 20:41:59 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> <20060426112227.66F7.JCARLSON@uci.edu> Message-ID: <44501357.5020402@gradient.cis.upenn.edu> Guido van Rossum wrote: > I don't think this form (set(1, 2, 3)) will ever fly, because it would > makes set(x) ambiguous. For what it's worth, under the changes I was proposing set(x) would be unambiguous -- it would mean a set containing the single element x. A separate class method would be required to create a set containing the elements of an iterable x. I would be -1 on any proposal that tries to make set(x) depend on the value of x. But I was only +0 on the proposal to begin with; and so if Guido's intuitions say that class method factories are "ugly and should be limited to cases that are rare but nevertheless useful," then it's fairly clear to me that this isn't the right way to go. (Our BDFL's intuitions about what's ugly seem to be quite trustworthy.) -Edward From guido at python.org Thu Apr 27 03:09:28 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Apr 2006 18:09:28 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <44501357.5020402@gradient.cis.upenn.edu> References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> <20060426112227.66F7.JCARLSON@uci.edu> <44501357.5020402@gradient.cis.upenn.edu> Message-ID: On 4/26/06, Edward Loper wrote: > Guido van Rossum wrote: > > I don't think this form (set(1, 2, 3)) will ever fly, because it would > > makes set(x) ambiguous. > > For what it's worth, under the changes I was proposing set(x) would be > unambiguous -- it would mean a set containing the single element x. Unfortunately this would be (a) different from what it means today, (b) different from what tuple(x) and list(x) mean; (c) make casts from one collection type to another more cumbersome. > A separate class method would be required to create a set containing the > elements of an iterable x. I would be -1 on any proposal that tries to > make set(x) depend on the value of x. Great. > But I was only +0 on the proposal to begin with; and so if Guido's > intuitions say that class method factories are "ugly and should be > limited to cases that are rare but nevertheless useful," then it's > fairly clear to me that this isn't the right way to go. (Our BDFL's > intuitions about what's ugly seem to be quite trustworthy.) Thanks for the statement of trust. I needed that! -- --Guido van Rossum (home page: http://www.python.org/~guido/) From talin at acm.org Thu Apr 27 04:03:02 2006 From: talin at acm.org (Talin) Date: Thu, 27 Apr 2006 02:03:02 +0000 (UTC) Subject: [Python-3000] sets in P3K? References: Message-ID: Greg Wilson cs.utoronto.ca> writes: > Hi Guido, > > > On 4/24/06, Greg Wilson cs.utoronto.ca> wrote: > > > On a completely different note, are you willing to consider native > > > syntax for sets (as per PEP 218) in Python 3000? > > > I like to write {1, 2, 3} instead of set([1, 2, 3]) but I don't know > > what to do with the ambiguity for {}. I believe you proposed {-} at the > > time which is actually hard to parse (because the '-' looks like the > > start of an expression and with our LL(1) parser there's no way back). > > Ideally perhaps {:} would be an empty dict and {} an empty set -- but > > that will take some getting used to (and would probably break > > compatibility with JSON). Perhaps {/} for an empty set? I'd hate to have > > to say set(). ABC solved this by having a special empty object that > > could become either a list or a table, but I don't think that'll work > > for us -- the implementation would have to have compatible memory > > lay-outs. > > I'm sure we can work something out --- I agree, {} for empty set and {:} > for empty dict would be ideal, were it not for backward compatibility. I > liked the "special empty object" idea when I first wrote the PEP (i.e., > have {} be something that could turn into either a set or dict), but one > of the instructors here convinced me that it would just lead to confusion > in newcomers' minds (as well as being a pain to implement). At the risk of creating further "ugliness", here is my humble proposal for a notation for set literals: As suggested, the nominal notation is a comma-separated list, surrounded by braces: {1, 2, 3} A set with a single element is: {1,} In other words, we steal the idea from tuples - a trailing comma is used as a way to signal that this is a comma-separated list of one item. A set with no elements is: set() Alternatively, you could take the tuple idea one step further: {,} However, I'm not sure I like this very much. -- Talin From collinw at gmail.com Thu Apr 27 04:24:03 2006 From: collinw at gmail.com (Collin Winter) Date: Wed, 26 Apr 2006 22:24:03 -0400 Subject: [Python-3000] Type Comparisons with Godel Numbers In-Reply-To: <4449AAED.3000604@canterbury.ac.nz> References: <00dc01c66591$ed190120$a8b62997@bagio> <4449AAED.3000604@canterbury.ac.nz> Message-ID: <43aa6ff70604261924i1279584cu1e08cf43e54d8fe6@mail.gmail.com> On 4/22/06, Greg Ewing wrote: > This is worth thinking about. Recently when pondering the > question of when it would or would not be appropriate to > put in type assertions to help catch bugs, I concluded > that it makes the most sense to do so when building a > data structure, but not when just passing things around > between calls. > > The most difficult-to-debug type errors are the ones > where you've put something of the wrong type into a > data structure, and it's sat there for a while not > causing any obvious problem until something else > tries to use it, by which time most evidence of where > it came from is gone. Inspired by this post, I've implemented this functionality in my typecheck package[1]. Traditionally, the package has offered only decorator-based type assertions for a function's arguments and return values and a generator's yield values. To complement this, I've coded up an assert_type function that's intended to be used just as Greg has described. Documentation for assert_type is available[2] from the project's website, as is a tarball of the latest SVN revision[3] (assert_type will go into the upcoming 0.4 release) so all interested parties can play around with this. [1] - http://oakwinter.com/code/typecheck/ [2] - http://oakwinter.com/code/typecheck/dev/tutorial/assert_type.html [3] - http://oakwinter.com/code/typecheck/dist/typecheck-svn-latest.tar.gz Collin Winter From guido at python.org Thu Apr 27 04:47:58 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Apr 2006 19:47:58 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: On 4/26/06, Talin wrote: > A set with a single element is: > > {1,} > > In other words, we steal the idea from tuples - a trailing comma is used as a > way to signal that this is a comma-separated list of one item. There's no need for this; there's no ambiguity. The ambiguity only starts for empty sets (unlike for tuples). > A set with no elements is: > > set() If we don't come up with something better I can live with this. > Alternatively, you could take the tuple idea one step further: > > {,} > > However, I'm not sure I like this very much. I'm sure I don't. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 27 05:43:39 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Apr 2006 20:43:39 -0700 Subject: [Python-3000] Unsure if I should write a PEP on Types In-Reply-To: <200604250009.04347.birchb@tpg.com.au> References: <200604250009.04347.birchb@tpg.com.au> Message-ID: Sorry for the belated response. Yes, I'd like to see this PEP. I'll probably eventually end up rewriting it :-), but right now I'm terribly overcommitted and really appreciate the help!!! --Guido On 4/24/06, Bill Birch wrote: > The optional static typing blog > http://www.artima.com/weblogs/viewpost.jsp?thread=87182 and many other > postings imply a lot of interesting detail about types, type expression, type > checking, adapt() and such. I'm considering collecting some of these > subjects together in a note for review and perhaps as a PEP. > Subjects would include: > * Recap of type concepts and motivations: dynamic, subtyping, nominative, > structural, duck. > * Use cases for types: checking, introspection, test generation, OSTC > * Type expressions > * Run-time type lattice > * Type class abstract interface: type operators > * and some working code > > Has a PEP already been written? Would this be useful? Who else finds this > interesting or may have started on these subjects? > > OK, so I'm fishing for encouragement here..... ;-) > > -- > http://billbirch.wordpress.com/ > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Thu Apr 27 06:26:40 2006 From: guido at python.org (Guido van Rossum) Date: Wed, 26 Apr 2006 21:26:40 -0700 Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 In-Reply-To: <1146063611.5751.24.camel@fsol> References: <1146063611.5751.24.camel@fsol> Message-ID: On 4/26/06, Antoine Pitrou wrote: > 1) sets are most often built dynamically rather than statically (in my > own experience) The same is true to the same extent for lists and dicts. And yet there are many uses for list and dict literals. I've seen a lot of list and tuple literals that should have been sets, because the main usage was to test whether some value was "in" the list/tuple or not. That's an O(N) operation -- fortunately it's pretty fast for tiny N. > 2) set([1,2,3]) makes little sense anyway, since it probably isn't > significantly more efficient than [1,2,3] Here's a benchmark: guido at pythonic:guido$ python2.4 -m timeit -s 'x = (1,2,3)' '(4 in x)' 1000000 loops, best of 3: 0.243 usec per loop guido at pythonic:guido$ python2.4 -m timeit -s 'x = set((1,2,3))' '(4 in x)' 10000000 loops, best of 3: 0.147 usec per loop The tuple version is slightly faster for (1 in x); but it's slower for (2 in x), slower still for (3 in x). You can easily extrapolate from this. > The real benefit of sets is when you have at least tens or hundreds of > elements; in that case you won't use a literal to build the set. Have a look at keyword.py in the stdlib. > Another remark is that I often use sets to hold my own objects rather > than simple values (ints or strings). In a sense, I use sets as iterable > containers where arbitrary remove() is fast. Sure. That's not an argument against set literals. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From kay.schluehr at gmx.net Thu Apr 27 08:27:35 2006 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Thu, 27 Apr 2006 08:27:35 +0200 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: <44506457.1040607@gmx.net> Talin schrieb: >A set with no elements is: > > set() > >Alternatively, you could take the tuple idea one step further: > > {,} > > If you want to simulate the mathematical notion of an empty set I would say that {/} has a stronger mnemonic appeal. Furthermore: what about notions of infinite sets? >However, I'm not sure I like this very much. > > I don't know if I like to be forced mixing functional and non-functional syntax to express the same kind of entities. Kay From rasky at develer.com Thu Apr 27 12:09:11 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 27 Apr 2006 12:09:11 +0200 Subject: [Python-3000] More batteries included! References: <5384bedc0604260350k1c92a212s8825c680516624bd@mail.gmail.com> <444F5444.4090002@gmail.com> Message-ID: <004601c669e2$a21183a0$8d472597@bagio> Nick Coghlan wrote: >> How about including wxPython[1] and Wax[2] in the Py3k standard libs >> and getting rid of the TK gui stuff? >> >> Also, I think pywin32[3] should be included. Or is there already a >> way to use Windows COM objects in Python that I missed? >> >> PIL[4] would be great to have in the standard libs too. But that's >> probably asking too much ;) >> >> [1] http://www.wxpython.org >> [2] http://sourceforge.net/projects/waxgui >> [3] http://sourceforge.net/projects/pywin32 >> [4] http://www.pythonware.com/products/pil > > Since this is still well into the future, by the time Py3k rolls > around it might be possible to include the option of easy-installing > various packages from the Python installer. As usually said, this would be miss the *main* point of having a standard library: consistency across Python installations. Giovanni Bajo From g.brandl at gmx.net Thu Apr 27 12:42:49 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Apr 2006 12:42:49 +0200 Subject: [Python-3000] sets in P3K? In-Reply-To: <44506457.1040607@gmx.net> References: <44506457.1040607@gmx.net> Message-ID: Kay Schluehr wrote: > Talin schrieb: > >>A set with no elements is: >> >> set() >> >>Alternatively, you could take the tuple idea one step further: >> >> {,} >> >> > If you want to simulate the mathematical notion of an empty set I would > say that {/} has a stronger mnemonic appeal. It also has a strong feeling of arbitraryness. > Furthermore: what about notions of infinite sets? Can you elaborate on that? Georg From gvwilson at cs.utoronto.ca Thu Apr 27 12:59:03 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Thu, 27 Apr 2006 06:59:03 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: > A set with no elements is: > set() I prefer Guido's earlier suggestion of {/} for the empty set (assuming no parsing ambiguities?): a) requiring 'set()' for empty sets is inconsistent with (), [], {} for empty collections of other kinds b) {/} kind of looks like phi (circle with a slash through it, to quote my engineering students ;-), which some of the books on my shelves use for empty set. Greg From gmccaughan at synaptics-uk.com Thu Apr 27 13:17:58 2006 From: gmccaughan at synaptics-uk.com (Gareth McCaughan) Date: Thu, 27 Apr 2006 12:17:58 +0100 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <44506457.1040607@gmx.net> Message-ID: <200604271217.59323.gmccaughan@synaptics-uk.com> On Thursday 2006-04-27 11:42, Georg Brandl wrote: > Kay Schluehr wrote: ... > > Furthermore: what about notions of infinite sets? > > Can you elaborate on that? Seems to me that if you want infinite sets, you want some type other than "set". Ah, the wonders of duck typing. (Just as if you want mappings with infinite domain, you want a type other than "dict".) Still, there's one generalization that's not 100% crazy (though quite possibly 90%): allow *cofinite* sets; that is, ones whose complement is finite. Then you can express "everything except 1" as, say, ~{1}. The nice thing is that { finite and cofinite sets } is, just like { finite sets }, closed under the usual operations, so you can still do things like unions and intersections, and you also get the complement operator (which I was proposing should be ~ rather than "not", by analogy with bitwise ops). Two objections, which between them I think are probably sufficient to make this a Bad Idea. Firstly, you can't iterate over a cofinite set. Secondly and, I think, more importantly, in practical use any given set is going to be known in advance to be finite, or known in advance to be cofinite, and your algorithm is unlikely to be made less clear by working with the complement of that set when it's cofinite. So the benefit isn't altogether clear. -- g From ncoghlan at gmail.com Thu Apr 27 14:35:33 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Apr 2006 22:35:33 +1000 Subject: [Python-3000] rough draft signature PEP In-Reply-To: <20060426091717.kzb6e411kw00800g@login.werra.lunarpages.com> References: <20060426091717.kzb6e411kw00800g@login.werra.lunarpages.com> Message-ID: <4450BA95.3000002@gmail.com> Michael Chermside wrote: > Brett responds: >> Yeah, I see what you are saying. But I actually held this view long >> before Iearned how Python did things underneath the covers. >> >> I have no clue how people tend to view things. Anyone else care to >> comment on how they tend to see arguments? > > My mental model matches Brett's. I think of them as separate things, too. Separate attributes also makes it easier to add additional metadata later, and the structure of each attribute can be well-defined. For example, given: def f(req:int, opt=val, *args, reqkwd, optkwd=val2, **kwds): pass A signature object like the following would then be relatively easy to grasp: sig = f.__signature__ assert sig.required_args == ('req',) assert sig.optional_args == ('opt',) assert sig.extra_args == 'args' assert sig.required_keywords == ('reqkwd',) assert sig.optional_keywords == ('optkwd',) assert sig.extra_keywords == 'kwds' assert sig.defaults == dict(opt=val, optkwd=val2) assert sig.argtypes == dict(req=int) Note that this can easily be adjusted for changes in signature expressiveness. Without keyword only arguments or argument type, the example might look like: def f(req, opt=val, *args, **kwds): pass sig = f.__signature__ assert sig.required_args == ('req',) assert sig.optional_args == ('opt',) assert sig.extra_args == 'args' assert sig.extra_keywords == 'kwds' assert sig.defaults == dict(opt=val, optkwd=val2) The simplest example: def f(): pass sig = f.__signature__ assert sig.required_args == () assert sig.optional_args == () assert sig.extra_args == '' assert sig.extra_keywords == '' assert sig.defaults == {} A sequence+enum approach is really hard to adapt to changes in signature expressiveness, whereas an attribute based approach can just add more tuples and string-keyed dictionaries for all relevant information. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Thu Apr 27 14:50:22 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Apr 2006 22:50:22 +1000 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: <4450BE0E.8090903@gmail.com> Greg Wilson wrote: >> A set with no elements is: >> set() > > I prefer Guido's earlier suggestion of {/} for the empty set (assuming no > parsing ambiguities?): I believe the leading '{' should be enough to clue the parser in to what's going on (i.e. '{/' is currently a syntax error) I think I'm a convert - +1 for set literals, with {/} for the empty set. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Thu Apr 27 15:11:13 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Apr 2006 23:11:13 +1000 Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 In-Reply-To: References: <1146063611.5751.24.camel@fsol> Message-ID: <4450C2F1.6030600@gmail.com> Guido van Rossum wrote: > On 4/26/06, Antoine Pitrou wrote: >> 1) sets are most often built dynamically rather than statically (in my >> own experience) > > The same is true to the same extent for lists and dicts. > > And yet there are many uses for list and dict literals. > > I've seen a lot of list and tuple literals that should have been sets, > because the main usage was to test whether some value was "in" the > list/tuple or not. That's an O(N) operation -- fortunately it's pretty > fast for tiny N. Another point in favour of set literals is that they let an optimising compiler play games because the compiler knows that the contents of that literal are supposed to be hashable, even if it can't resolve them right now, and it knows that the result is going to be an actual builtin set object. So the constant-folding part of the optimiser could create the actual set and tuck it away in co_consts if the literal was only used for an "in" test. Those kinds of tricks can't even be contemplated with 'set' because the compiler can't trust that the name is going to resolve to the actual builtin. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From krstic at fas.harvard.edu Thu Apr 27 15:11:45 2006 From: krstic at fas.harvard.edu (Ivan Krstic) Date: Thu, 27 Apr 2006 09:11:45 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <4450BE0E.8090903@gmail.com> References: <4450BE0E.8090903@gmail.com> Message-ID: <4450C311.8040603@fas.harvard.edu> Nick Coghlan wrote: > I think I'm a convert - +1 for set literals, with {/} for the empty set. I'm +1 on set literals, but -1 on {/} for the empty set. I much prefer set() -- {/} seems too Perlesque and magical-looking. I don't think we have three subsequent punctuation marks as an atom anywhere else in Python, do we? (Except for """, but those are three of the same, so it doesn't look that strange to me.) -- Ivan Krstic | GPG: 0x147C722D From ncoghlan at iinet.net.au Thu Apr 27 15:33:01 2006 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu, 27 Apr 2006 23:33:01 +1000 Subject: [Python-3000] Automatically invoking str() in str.join() Message-ID: <4450C80D.50002@iinet.net.au> Georg just added the idea of automatically invoking str() in str.join() to PEP 3100 with a question mark on the end. I thought I'd responded when Talin first brought it up, but couldn't find a record of that in the archive (this is why one suggestion per message is a good idea ;) I tried to implement this a couple of years ago for 2.4, and ended up giving up. The sticking point was the str->unicode promotion code that meant the implementation had to keep the un-str-ified objects around so they could be passed to unicode.join if it later proved necessary. Or something like that. Whatever the details were, the code necessary to support it without slowing down the common case of a sequence of all strings was horribly contorted and ugly and the idea was (rightly) knocked on the head. In Py3k though, that problem won't exist, as we only have to worry about supporting it in the function that is currently unicode.join. And fixing it there should be relatively straightforward. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From ncoghlan at gmail.com Thu Apr 27 15:46:30 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 27 Apr 2006 23:46:30 +1000 Subject: [Python-3000] sets in P3K? In-Reply-To: <4450C311.8040603@fas.harvard.edu> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> Message-ID: <4450CB36.7090504@gmail.com> Ivan Krstic wrote: > Nick Coghlan wrote: >> I think I'm a convert - +1 for set literals, with {/} for the empty set. > > I'm +1 on set literals, but -1 on {/} for the empty set. I much prefer > set() -- {/} seems too Perlesque and magical-looking. I could certainly live with set(). The ASCII-art phi might be trying to be overly cute :) > I don't think we > have three subsequent punctuation marks as an atom anywhere else in > Python, do we? (Except for """, but those are three of the same, so it > doesn't look that strange to me.) ... is the only other atom I can think of. Although there's always siht[::-1] :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From g.brandl at gmx.net Thu Apr 27 16:31:46 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Apr 2006 16:31:46 +0200 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <4450C80D.50002@iinet.net.au> References: <4450C80D.50002@iinet.net.au> Message-ID: Nick Coghlan wrote: > Georg just added the idea of automatically invoking str() in str.join() to PEP > 3100 with a question mark on the end. I thought I'd responded when Talin first > brought it up, but couldn't find a record of that in the archive (this is why > one suggestion per message is a good idea ;) As a side note, I named the thing "string.join()" because I didn't know what to write. The str type will be dead, and I assume that the new string type will have a different name. Will it be "unicode"? Or "text"? Or "string"? Georg From fredrik at pythonware.com Thu Apr 27 16:34:18 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 27 Apr 2006 16:34:18 +0200 Subject: [Python-3000] More batteries included! References: <5384bedc0604260350k1c92a212s8825c680516624bd@mail.gmail.com><444F5444.4090002@gmail.com> <004601c669e2$a21183a0$8d472597@bagio> Message-ID: Giovanni Bajo wrote: > > Since this is still well into the future, by the time Py3k rolls > > around it might be possible to include the option of easy-installing > > various packages from the Python installer. > > As usually said, this would be miss the *main* point of having a standard > library: consistency across Python installations. hopefully, the Py3K distributors will use semi-automatic tools to download all they need to build a battery-included release. pushing all that over to the end users would be a major mistake. (python-dev and others could provide "profiles" for well-known and tested configurations; e.g. "psf standard", "scientific", "sumo", ...) From g.brandl at gmx.net Thu Apr 27 16:36:23 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Thu, 27 Apr 2006 16:36:23 +0200 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: Message-ID: Crutcher Dunnavant wrote: > Having been shot down so hard on my request to bring for loops into > symetry with list comprehensions, I have a new proposal - stackable > blocks. > > This is a very _small_ amount of syntax sugar, the basic idea is that > any block headers can be stacked on the same line. This changes no > semantics, nor does it change ordering, or add new keywords, it just > changes a tiny bit of the grammar for whitespace/block parsing. > > So for instance this: > > for file in open_files: > if file.readable(): > ... > > can be spelled like this. > > for file in open_files: if file.readable(): > ... Gak. I think it was a Freudian anti-slip that you used "readable" in the above example ;) Also, you know that Guido at one time said he wanted to forbid the "if x: y" suite style? Georg From fredrik at pythonware.com Thu Apr 27 16:38:03 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 27 Apr 2006 16:38:03 +0200 Subject: [Python-3000] Automatically invoking str() in str.join() References: <4450C80D.50002@iinet.net.au> Message-ID: Georg Brandl wrote: > As a side note, I named the thing "string.join()" because I didn't know what to > write. join() it's time to make this a builtin. the "it's the separator that's doing the joining" idea is silly and unintuitive, and should be fixed. From krstic at fas.harvard.edu Thu Apr 27 16:44:08 2006 From: krstic at fas.harvard.edu (Ivan Krstic) Date: Thu, 27 Apr 2006 10:44:08 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <4450C80D.50002@iinet.net.au> Message-ID: <4450D8B8.40408@fas.harvard.edu> Fredrik Lundh wrote: > it's time to make this a builtin. the "it's the separator that's doing the > joining" idea is silly and unintuitive, and should be fixed. +1 on this; I've thought this for a long time. -- Ivan Krstic | GPG: 0x147C722D From rhettinger at ewtllc.com Thu Apr 27 18:15:46 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Thu, 27 Apr 2006 09:15:46 -0700 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <4450D8B8.40408@fas.harvard.edu> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> Message-ID: <4450EE32.7000909@ewtllc.com> Ivan Krstic wrote: >Fredrik Lundh wrote: > > >>it's time to make this a builtin. the "it's the separator that's doing the >>joining" idea is silly and unintuitive, and should be fixed. >> >> > >+1 on this; I've thought this for a long time. > > > Strong +1. This fixes a long-standing micro-wart. Raymond From talin at acm.org Thu Apr 27 18:29:15 2006 From: talin at acm.org (Talin) Date: Thu, 27 Apr 2006 16:29:15 +0000 (UTC) Subject: [Python-3000] rough draft signature PEP References: <20060426091717.kzb6e411kw00800g@login.werra.lunarpages.com> <4450BA95.3000002@gmail.com> Message-ID: Nick Coghlan gmail.com> writes: > > My mental model matches Brett's. > > I think of them as separate things, too. Separate attributes also makes it > easier to add additional metadata later, and the structure of each attribute > can be well-defined. > > For example, given: > > def f(req:int, opt=val, *args, reqkwd, optkwd=val2, **kwds): > pass > > A signature object like the following would then be relatively easy to grasp: > > sig = f.__signature__ > assert sig.required_args == ('req',) > assert sig.optional_args == ('opt',) > assert sig.extra_args == 'args' > assert sig.required_keywords == ('reqkwd',) > assert sig.optional_keywords == ('optkwd',) > assert sig.extra_keywords == 'kwds' > assert sig.defaults == dict(opt=val, optkwd=val2) > assert sig.argtypes == dict(req=int) Sounds like I'm out-voted here. All right :) My only other comment is that I think that the format of the signature attribute should be use-case driven - that is, we should come up with some example use cases, implement them in code, and see what kind of signature format is most convenient for those use cases. I would suggest the following use cases as sample points within a broader range of possible uses: 1) trace decorator: Print the names of their arguments and their values before calling the function. 2) precondition decorator: Allow arbitrary preconditions to be added to specific parameters. 3) alternative dispatch decorator: Using the signature API, implement an alternative dispatch method. This could be something as simple as matching argument names - so if you call "func( a=3 )" it searches for a version of func that has an argument named 'a'. -- Talin From aahz at pythoncraft.com Thu Apr 27 19:08:03 2006 From: aahz at pythoncraft.com (Aahz) Date: Thu, 27 Apr 2006 10:08:03 -0700 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <4450EE32.7000909@ewtllc.com> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> Message-ID: <20060427170803.GB17595@panix.com> On Thu, Apr 27, 2006, Raymond Hettinger wrote: > Ivan Krstic wrote: >>Fredrik Lundh wrote: >>> >>>it's time to make this a builtin. the "it's the separator that's doing the >>>joining" idea is silly and unintuitive, and should be fixed. >> >>+1 on this; I've thought this for a long time. > > Strong +1. > This fixes a long-standing micro-wart. -0 While I hate the way it looks, I never have gotten mixed up about the order of arguments since switching to ''.join(l). -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From barry at python.org Thu Apr 27 19:12:35 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 27 Apr 2006 13:12:35 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <20060427170803.GB17595@panix.com> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> Message-ID: <1146157955.10752.161.camel@resist.wooz.org> On Thu, 2006-04-27 at 10:08 -0700, Aahz wrote: > While I hate the way it looks, I never have gotten mixed up about the > order of arguments since switching to ''.join(l). Which is why EMPTYSTRING = '' ... EMPTYSTRING.join(seq) looks much better. But hey, yeah, a join() builtin would be fine if it took the string arg first, so that ''.join(seq) == join('', seq) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060427/5f321fab/attachment.pgp From rasky at develer.com Thu Apr 27 19:10:29 2006 From: rasky at develer.com (Giovanni Bajo) Date: Thu, 27 Apr 2006 19:10:29 +0200 Subject: [Python-3000] Python-3000 Digest, Vol 2, Issue 151 References: <1146063611.5751.24.camel@fsol> <4450C2F1.6030600@gmail.com> Message-ID: <058501c66a1d$7be8ad80$bf03030a@trilan> Nick Coghlan wrote: > Another point in favour of set literals is that they let an optimising > compiler play games because the compiler knows that the contents of > that literal are supposed to be hashable, even if it can't resolve > them right now, and it knows that the result is going to be an actual > builtin set object. So the constant-folding part of the optimiser > could create the actual set and tuck it away in co_consts if the > literal was only used for an "in" test. > > Those kinds of tricks can't even be contemplated with 'set' because > the compiler can't trust that the name is going to resolve to the > actual builtin. I wouldn't go in that direction. There ahead lies the usage of punctuation for everything, just because punctuation can't be rebound while builtin names can. What needs to be done in Py3K is to change the language so that it's possible to know whether a builtin was rebound and not, thus allowing to optimize their lookup away. I believe there were already proposals for this, but I'm not sure there is a PEP already. We shouldn't recur to using literals/punctuation just because we can't optimize builtins. -- Giovanni Bajo From aleaxit at gmail.com Thu Apr 27 19:18:23 2006 From: aleaxit at gmail.com (Alex Martelli) Date: Thu, 27 Apr 2006 10:18:23 -0700 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <1146157955.10752.161.camel@resist.wooz.org> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> Message-ID: On 4/27/06, Barry Warsaw wrote: > On Thu, 2006-04-27 at 10:08 -0700, Aahz wrote: > > > While I hate the way it looks, I never have gotten mixed up about the > > order of arguments since switching to ''.join(l). > > Which is why > > EMPTYSTRING = '' > > ... > > EMPTYSTRING.join(seq) > > looks much better. But hey, yeah, a join() builtin would be fine if it > took the string arg first, so that > > ''.join(seq) == join('', seq) I think I would prefer a signature of: join(seq, joiner='') Rationale: an emptystring joiner is the most frequent cases, but several others (space, newline, space-comma, ...) occur often enough to be worth allowing the joiner to be optionally specified. The ability to omit the joiner in most cases works particularly well with genexps, e.g. join(str(x) for x in whatever) ve join('', (str(x) for x in whatever)) Alex From tim.hochberg at ieee.org Thu Apr 27 19:24:43 2006 From: tim.hochberg at ieee.org (Tim Hochberg) Date: Thu, 27 Apr 2006 10:24:43 -0700 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <1146157955.10752.161.camel@resist.wooz.org> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> Message-ID: <4450FE5B.4010208@ieee.org> Barry Warsaw wrote: > On Thu, 2006-04-27 at 10:08 -0700, Aahz wrote: >> -0 >> >> >>While I hate the way it looks, I never have gotten mixed up about the >>order of arguments since switching to ''.join(l). Me too on all counts including the -0. > > > Which is why > > EMPTYSTRING = '' > > ... > > EMPTYSTRING.join(seq) > > looks much better. But hey, yeah, a join() builtin would be fine if it > took the string arg first, so that > > ''.join(seq) == join('', seq) > Isn't that the same as str.join? >>> str.join(', ', 'abcdefg') 'a, b, c, d, e, f, g' In which case why would we need a builtin? -tim From guido at python.org Thu Apr 27 19:37:24 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Apr 2006 10:37:24 -0700 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <4450C80D.50002@iinet.net.au> Message-ID: On 4/27/06, Georg Brandl wrote: > As a side note, I named the thing "string.join()" because I didn't know what to > write. The str type will be dead, and I assume that the new string type will > have a different name. > > Will it be "unicode"? Or "text"? Or "string"? "str" of course. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.peters at gmail.com Thu Apr 27 19:46:52 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 27 Apr 2006 13:46:52 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <4450FE5B.4010208@ieee.org> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> <4450FE5B.4010208@ieee.org> Message-ID: <1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com> [Barry Warsaw] >> ... >> While I hate the way it looks, I never have gotten mixed up about the >> order of arguments since switching to ''.join(l). [Tim Hochberg] > Me too on all counts including the -0. >> ... >> But hey, yeah, a join() builtin would be fine if it >> took the string arg first, so that >> >> ''.join(seq) == join('', seq) > Isn't that the same as str.join? Well, join(a_string, seq), or join(seq, a_string) (depending on who you ask) , would be the same as the current a_string.join(seq). Note that from string import join already supplies a functional interface (with the separator second). That's what was used before strings _had_ methods, and it's still there. > In which case why would we need a builtin? Because people enjoy arguing about the best order for the arguments, and about the best default separator value, neither of which you _can_ argue about in the method spelling ;-) From steven.bethard at gmail.com Thu Apr 27 20:02:27 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 27 Apr 2006 12:02:27 -0600 Subject: [Python-3000] the dict constructor Message-ID: On 4/26/06, Guido van Rossum wrote: > ... the dict constructor signature is > already too far overloaded, giving a different meaning to keyword > args, and distinguishing between a mapping and a sequence is iffy: it > requires a "keys" method which is kind of sub-optimal compared to > using "iteritems" if it existed... Yes, this is a pain to emulate, as the code in UserDict.DictMixin.update shows. Is there something we can do in Python 3000 to eliminate the need to check for "keys"? It bothers me that the language reference [1] says that to be a mapping type, you just need to provide __len__, __getitem__, etc. and that "keys", "values" etc. are just reccommended methods, but yet the implementation seems to imply otherwise. I guess maybe we should consider restricting the dict constructor in Python 3000? I'd like to keep keyword args if possible -- I find the syntax: dict(foo='spam', bar=42) incredibly convenient. If dict(), like set(), list() and tuple() just took a sequence argument, we would break the nice consistency that for most collections type(obj)(obj) copies the object, e.g. set(set_obj) # copies the set_obj list(list_obj) # copies the list_obj I guess that's not horrible given the existence of copy(), but it makes me uneasy. If the dict constructor just took a mapping though, you couldn't do use generator comprehensions with the dict constructor anymore, e.g. the following would break: dict((foo(x), bar(x)) for x in items) I guess if we went this route, I'd want to push for adding a dict comprehension syntax to cover this use case. It's probably clear that I don't have any solutions to this problem, but I thought I'd bring it up in case anyone out there does. [1]http://docs.python.org/ref/sequence-types.html STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From fredrik at pythonware.com Thu Apr 27 20:06:43 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 27 Apr 2006 20:06:43 +0200 Subject: [Python-3000] Automatically invoking str() in str.join() References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu><4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com><1146157955.10752.161.camel@resist.wooz.org><4450FE5B.4010208@ieee.org> <1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com> Message-ID: Tim Peters wrote: > > In which case why would we need a builtin? > > Because people enjoy arguing about the best order for the arguments, > and about the best default separator value, neither of which you _can_ > argue about in the method spelling ;-) no, because people enjoy writing readable code. doing things by exe- cuting methods attached to literals isn't very readable, and isn't used for anything else. I don't think anyone on this list can take the "but if there's more than one argument, *I* am going to be confused" argument seriously. From pje at telecommunity.com Thu Apr 27 20:15:08 2006 From: pje at telecommunity.com (Phillip J. Eby) Date: Thu, 27 Apr 2006 14:15:08 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: Message-ID: <5.1.1.6.0.20060427141125.03bf0878@mail.telecommunity.com> At 07:16 PM 4/27/2006 +0200, Barry Warsaw wrote: > But hey, yeah, a join() builtin would be fine if it >took the string arg first, so that > >''.join(seq) == join('', seq) And thanks to the time machine, str.join('', seq) does this already: Python 2.2.2 (#1, Feb 24 2003, 19:13:11) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> str.join('',['a','b','c']) 'abc' Who needs a new builtin? :) From rhettinger at ewtllc.com Thu Apr 27 20:49:27 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Thu, 27 Apr 2006 11:49:27 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <4450CB36.7090504@gmail.com> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> Message-ID: <44511237.5000609@ewtllc.com> [pep-3100 checkin] > {F(x) for x in S if P(x)} means set(F(x) for x in S if P(x)). I presume this means that there will never be dictionary comprehensions (as they would aspire to have an identical notation). > There's no frozenset literal; they are too rarely needed. Actually, they are one of the most common use cases for set literals. Whenever someone writes code like, " if ext in {'html', 'xml, 'xhtml'}: do_something()", they are intending to create an immutable set that could possibly be optimized into a constant by the compiler. Currently, folks don't bother writing-in the frozen-part because they know that no optimization currently takes place. > The {/} part is still controversial. I suspect this is going to be a wart. One of the reasons for the class-keyword to now accept an emtpy list of bases is the difficulties arising from deleting a base class and leaving an invalid syntax. The same is true for sets. Start with {1,2}, delete an element leaving {1}, and delete another element leaving {}, oops the type just changed. I forsee {} vs {/} as being a prominent entry on everyone's list of Python pitfalls, faqs, and common errors. Maybe PyLint or PyChecker will be able to scan to see how the object is used and determine whether the two were mixed-up. Raymond From tim.peters at gmail.com Thu Apr 27 20:54:19 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 27 Apr 2006 14:54:19 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> <4450FE5B.4010208@ieee.org> <1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com> Message-ID: <1f7befae0604271154w6ba78d3p3b6851e85d4dc68d@mail.gmail.com> [Fredrik Lundh] > no, because people enjoy writing readable code. doing things by exe- > cuting methods attached to literals isn't very readable, and isn't used > for anything else. As Barry often says, he spells it TAB.join() or BLANK.join() (etc) instead. That's very readable. > I don't think anyone on this list can take the "but if there's more than > one argument, *I* am going to be confused" argument seriously. Then you don't remember that the order of arguments here _was_ a frequent confusion in the old days. If you like a callable with separator first, you can also do join = str.join today, and if you like a callable with separator second, you can also do from string import join today. My bet is that nobody here uses either, because they don't really find the method spelling distasteful enough to endure the one-line bother to set up an alternative that would flood their soul with joy ;-) If you want a builtin instead in Py3K, I'm just -0 on that (but -1 if that's in _addition_ to the three spellings Python already has). From steven.bethard at gmail.com Thu Apr 27 20:57:30 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 27 Apr 2006 12:57:30 -0600 Subject: [Python-3000] sets in P3K? In-Reply-To: <44511237.5000609@ewtllc.com> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> Message-ID: On 4/27/06, Raymond Hettinger wrote: > [pep-3100 checkin] > > {F(x) for x in S if P(x)} means set(F(x) for x in S if P(x)). > > I presume this means that there will never be dictionary > comprehensions (as they would aspire to have an identical > notation). Why would that be necessary? Wouldn't {F(x):G(x) for x in S if P(x)} be unambiguous? (Not that I'm pushing for dict comprehensions -- I just don't see how if Guido wants both dict and set comprehensions, he couldn't have them.) STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From fredrik at pythonware.com Thu Apr 27 21:00:20 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Thu, 27 Apr 2006 21:00:20 +0200 Subject: [Python-3000] Automatically invoking str() in str.join() References: <5.1.1.6.0.20060427141125.03bf0878@mail.telecommunity.com> Message-ID: Phillip J. Eby wrote: > And thanks to the time machine, str.join('', seq) does this already: > > Python 2.2.2 (#1, Feb 24 2003, 19:13:11) > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> str.join('',['a','b','c']) > 'abc' > > Who needs a new builtin? :) >>> str.join(sep, seq) Traceback (most recent call last): File "", line 1, in ? TypeError: descriptor 'join' requires a 'str' object but received a 'unicode' From jack at performancedrivers.com Thu Apr 27 21:19:00 2006 From: jack at performancedrivers.com (Jack Diederich) Date: Thu, 27 Apr 2006 15:19:00 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> Message-ID: <20060427191900.GF13633@performancedrivers.com> On Thu, Apr 27, 2006 at 10:18:23AM -0700, Alex Martelli wrote: > On 4/27/06, Barry Warsaw wrote: > > On Thu, 2006-04-27 at 10:08 -0700, Aahz wrote: > > > > > While I hate the way it looks, I never have gotten mixed up about the > > > order of arguments since switching to ''.join(l). > > > > Which is why > > > > EMPTYSTRING = '' > > > > ... > > > > EMPTYSTRING.join(seq) > > > > looks much better. But hey, yeah, a join() builtin would be fine if it > > took the string arg first, so that > > > > ''.join(seq) == join('', seq) > > I think I would prefer a signature of: > join(seq, joiner='') > > Rationale: an emptystring joiner is the most frequent cases, but > several others (space, newline, space-comma, ...) occur often enough > to be worth allowing the joiner to be optionally specified. Grepping through my own for joins that use a literal: Join type Count ------------------- ','.join 83 ''.join 61 # the default suggested above str.join 35 ' '.join 28 string.join 10 OTHER.join 45 # everything else with a literal ' AND '.join, etc So I'd prefer if the seperator was explicit and up front where I can see it instead of hiding at the end. Also, if join() is builtin does that mean split() will be too? -Jack From steven.bethard at gmail.com Thu Apr 27 21:21:07 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Thu, 27 Apr 2006 13:21:07 -0600 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <5.1.1.6.0.20060427141125.03bf0878@mail.telecommunity.com> Message-ID: On 4/27/06, Fredrik Lundh wrote: > Phillip J. Eby wrote: > > > And thanks to the time machine, str.join('', seq) does this already: > > > > Python 2.2.2 (#1, Feb 24 2003, 19:13:11) > > [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-4)] on linux2 > > Type "help", "copyright", "credits" or "license" for more information. > > >>> str.join('',['a','b','c']) > > 'abc' > > > > Who needs a new builtin? :) > > >>> str.join(sep, seq) > Traceback (most recent call last): > File "", line 1, in ? > TypeError: descriptor 'join' requires a 'str' object but received a 'unicode' I was going to point out this problem too, but of course in Python 3000, str and unicode will be the same thing. I guess you could run into the problem if you tried to use a "sep" that was some user-defined string-like object, but I'm not sure how likely that would be. STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From tjreedy at udel.edu Thu Apr 27 22:02:14 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 27 Apr 2006 16:02:14 -0400 Subject: [Python-3000] More batteries included! References: <5384bedc0604260350k1c92a212s8825c680516624bd@mail.gmail.com><444F5444.4090002@gmail.com><004601c669e2$a21183a0$8d472597@bagio> Message-ID: "Fredrik Lundh" wrote in message news:e2qkpd$8j4$1 at sea.gmane.org... > hopefully, the Py3K distributors will use semi-automatic tools to > download > all they need to build a battery-included release. pushing all that over > to > the end users would be a major mistake. > (python-dev and others could provide "profiles" for well-known and tested > configurations; e.g. "psf standard", "scientific", "sumo", ...) I think this worth considering. The initial download for Internet Explorer updates is less than a megabyte. It looks at what you have and displays a menu of optional components, and then downloads and installs the 15-30 megabytes actually needed. Terry Jan Reedy From rrr at ronadam.com Thu Apr 27 22:05:08 2006 From: rrr at ronadam.com (Ron Adam) Date: Thu, 27 Apr 2006 15:05:08 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: <44511237.5000609@ewtllc.com> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> Message-ID: Raymond Hettinger wrote: > [pep-3100 checkin] >> {F(x) for x in S if P(x)} means set(F(x) for x in S if P(x)). > > I presume this means that there will never be dictionary comprehensions (as they would aspire to have an identical notation). > > > >> There's no frozenset literal; they are too rarely needed. > > Actually, they are one of the most common use cases for set literals. Whenever someone writes code like, " > if ext in {'html', 'xml, 'xhtml'}: do_something()", they are intending to create an immutable set that could possibly be optimized into a constant by the compiler. Currently, folks don't bother writing-in the frozen-part because they know that no optimization currently takes place. > > > >> The {/} part is still controversial. > > I suspect this is going to be a wart. One of the reasons for the class-keyword to now accept an emtpy list of bases is the difficulties arising from deleting a base class and leaving an invalid syntax. The same is true for sets. Start with {1,2}, delete an element leaving {1}, and delete another element leaving {}, oops the type just changed. > > I forsee {} vs {/} as being a prominent entry on everyone's list of Python pitfalls, faqs, and common errors. Maybe PyLint or PyChecker will be able to scan to see how the object is used and determine whether the two were mixed-up. > > > Raymond Yes, -1 on the {/} I think I really prefer the {:} for an empty dictionary, and {} for an empty set. I don't see where breaking this is any more of a problem than many of the other items that are changing in P3K. Yes it's a more commonly used item, but that is the point of the P3K project. To make those changes that just aren't possible without breakage. For those who want to write P3K compatible code in 2.x. They could continue to write set() as they do now, and start writing dict() where empty containers are needed. So this isn't an impossible situation. It's also would not be difficult to run a search and replace on {} to change it to {:} when the time comes. I also think it would not be a problem to learn to use {:} for an empty dict. An error would result fairly soon when someone tries to use {} as a dict and it will be easy to remember dicts always contain at least one ':' in their spelling. I think it's been suggested before that there should be a PY2.x to PY3.x conversion utility to facilitate the move to PY3K. So maybe a sticking point as when to *not* do something would be, if it's not possible to translate a changed 2.x feature to a PY3K equivalent automatically rather than if it will break 2.x code or not. Cheers, Ron From guido at python.org Thu Apr 27 22:17:15 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Apr 2006 13:17:15 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> Message-ID: On 4/27/06, Ron Adam wrote: > I think it's been suggested before that there should be a PY2.x to PY3.x > conversion utility to facilitate the move to PY3K. So maybe a sticking > point as when to *not* do something would be, if it's not possible to > translate a changed 2.x feature to a PY3K equivalent automatically > rather than if it will break 2.x code or not. Please read the section "Compatibility and Transition" in PEP 3000. Perfect automatic translation is not a goal. A combination of tools should be able to diagnose all areas of a program that require changes but may not necessarily be able to tell you how to change it. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Thu Apr 27 22:44:51 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 27 Apr 2006 16:44:51 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <44511237.5000609@ewtllc.com> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> Message-ID: <1146170691.10791.20.camel@resist.wooz.org> On Thu, 2006-04-27 at 11:49 -0700, Raymond Hettinger wrote: > [pep-3100 checkin] > > {F(x) for x in S if P(x)} means set(F(x) for x in S if P(x)). > > I presume this means that there will never be dictionary > comprehensions (as they would aspire to have an identical notation). > > > > > There's no frozenset literal; they are too rarely needed. > > Actually, they are one of the most common use cases for set literals. > Whenever someone writes code like, " > if ext in {'html', 'xml, 'xhtml'}: do_something()", they are intending > to create an immutable set that could possibly be optimized into a > constant by the compiler. Currently, folks don't bother writing-in > the frozen-part because they know that no optimization currently takes > place. Okay, radical/insane/heretical question time: do we really need literal tuple syntax? IOW, what if (1, 2, 3) created a frozenset instead of a tuple? Or perhaps an even stupider idea: what if the compiler could recognize certain use cases and generate different types depending on how the object is used. For example: if key in ('one', 'two', 'three') for x in (1, 2, 3, 5, 7, 11) In both these cases, you don't really care if the literal syntax builds a tuple or a frozenset. If you had duplicate entries you'd care, but wouldn't that be something the compiler could figure out? Now, in a case like the following, you /would/ want a tuple, but only to preserve order: x = (1, 2, 3) foo(*x) are tuple literals used enough this way, and what if you were forced to use a list literal instead? x = [1, 2, 3] foo(*x) So, again, what if (...) created a set/frozen set instead of a tuple? It's py3k, so broken code be damned. :) put-down-that-pitchfork-ly y'rs, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060427/a921e442/attachment.pgp From barry at python.org Thu Apr 27 22:46:45 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 27 Apr 2006 16:46:45 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> Message-ID: <1146170805.10790.22.camel@resist.wooz.org> On Thu, 2006-04-27 at 12:57 -0600, Steven Bethard wrote: > On 4/27/06, Raymond Hettinger wrote: > > [pep-3100 checkin] > > > {F(x) for x in S if P(x)} means set(F(x) for x in S if P(x)). > > > > I presume this means that there will never be dictionary > > comprehensions (as they would aspire to have an identical > > notation). > > Why would that be necessary? Wouldn't > {F(x):G(x) for x in S if P(x)} > be unambiguous? (Not that I'm pushing for dict comprehensions -- I > just don't see how if Guido wants both dict and set comprehensions, he > couldn't have them.) Can I unwithdraw PEP 274 now? -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060427/aec41e86/attachment.pgp From barry at python.org Thu Apr 27 22:50:33 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 27 Apr 2006 16:50:33 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <1146170691.10791.20.camel@resist.wooz.org> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <1146170691.10791.20.camel@resist.wooz.org> Message-ID: <1146171033.10791.24.camel@resist.wooz.org> On Thu, 2006-04-27 at 16:44 -0400, Barry Warsaw wrote: > for x in (1, 2, 3, 5, 7, 11) Uh, ignore that. Order matters. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060427/91866835/attachment-0001.pgp From rrr at ronadam.com Thu Apr 27 23:10:57 2006 From: rrr at ronadam.com (Ron Adam) Date: Thu, 27 Apr 2006 16:10:57 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> Message-ID: Guido van Rossum wrote: > On 4/27/06, Ron Adam wrote: >> I think it's been suggested before that there should be a PY2.x to PY3.x >> conversion utility to facilitate the move to PY3K. So maybe a sticking >> point as when to *not* do something would be, if it's not possible to >> translate a changed 2.x feature to a PY3K equivalent automatically >> rather than if it will break 2.x code or not. > > Please read the section "Compatibility and Transition" in PEP 3000. > Perfect automatic translation is not a goal. A combination of tools > should be able to diagnose all areas of a program that require changes > but may not necessarily be able to tell you how to change it. > > -- > --Guido van Rossum (home page: http://www.python.org/~guido/) Thanks, I just read it. In that light I would say to be able to translate easily would still be a plus, which is the case of {} vs {:}. I didn't mean to suggest it as an absolute goal. I am suggesting that those items that can be translated automatically have a bit less resistance to change than those items that can't. In a way it's only the next lower (or higher depending on how you look at it) rung on the ladder as far as being backwards compatibility goes. Cheers, Ron From guido at python.org Thu Apr 27 23:43:21 2006 From: guido at python.org (Guido van Rossum) Date: Thu, 27 Apr 2006 14:43:21 -0700 Subject: [Python-3000] PEP 3002 (Procedure for Backwards-Incompatible Changes) Message-ID: I just read PEP 3002 and while I like most of what I see (thanks Steven Bethard!) there's one section that stands out as problematic. I'm quoting it here: """ Optional Extensions =================== Instead of the python3warn.py script, a branch of Python 3000 could be maintained that added warnings at all the appropriate points in the code-base. PEPs proposing backwards-incompatible changes would then provide patches to the Python-3000-warn branch instead of to python3warn.py. With such a branch, the warnings issued could be near-perfect and Python users could be confident that their code was correct Python 3000 code by first running it on the Python-3000-warn branch and fixing all the warnings. At the moment, however, this PEP opts for the weaker measure (python3warn.py) as it is expected that maintaining a Python-3000-warn branch will be too much of a time drain. """ I believe that this is aiming at a similar idea I put in PEP 3000: """ Another kind of tool could be an instrumented version of 2.x which produces run-time warnings about constructs that will get a different meaning in 3.0. This can't be used for all incompatibilities, but it's likely to help reach a larger percentage of correct translations. (This approach is already in place for detecting reliance on '/' to do integer division; see Tools/scripts/fixdiv.py, which finds occurrences of int/int and long/long that were flagged by running your program with -Qwarnall.) """ I expect that the PEP 3002 proposal won't quite work; a branch of Python 3000 to identify backwards incompatibilities would have a hard time parsing code that uses the Python 2.x features that are completely cut out of Python 3000, such as u"..." literals, or different except clause syntax, etc. Rather than a version of Python 3000, I suggest to add such warning code to Python 2.x, and not as a branch, but as a permanent feature (that is however only enabled by a command-line flag). I suppose the PEP 3002 proposal was written with the assumption that a *syntactic* conversion would be doable purely mechanically, and that the semantic issues would be sorted out using a second phase tool. But I would rather see a single (integrated) tool that looks at unconverted 2.x code and suggests or makes all changes necessary. This would require a combination of techniques, but would be more useful to 2.x users. I guess this would also affect the use of the python3warn.py script. Perhaps Steven or someone else could propose specific edits to PEP 3002 to reflect this? -- --Guido van Rossum (home page: http://www.python.org/~guido/) From baptiste13 at altern.org Fri Apr 28 00:39:52 2006 From: baptiste13 at altern.org (Baptiste Carvello) Date: Fri, 28 Apr 2006 00:39:52 +0200 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> Message-ID: Ron Adam a ?crit : > Raymond Hettinger wrote: >>I forsee {} vs {/} as being a prominent entry on everyone's list of Python pitfalls, faqs, and common errors. Maybe PyLint or PyChecker will be able to scan to see how the object is used and determine whether the two were mixed-up. >> > > I think I really prefer the {:} for an empty dictionary, and {} for an > empty set. Actually I think we need both {/} as the empty set and {:} as the empty dict. {} should raise SyntaxError, '{} is ambiguous, use {/} for set, {:} for dict' If we decide to have both dict and set litteral, {} will *always* be ambiguous. Lots of bugs can be avoided if we just forbid it. Baptiste From listsub at wickedgrey.com Fri Apr 28 01:05:13 2006 From: listsub at wickedgrey.com (Eli Stevens (WG.c)) Date: Thu, 27 Apr 2006 16:05:13 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <1146170691.10791.20.camel@resist.wooz.org> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <1146170691.10791.20.camel@resist.wooz.org> Message-ID: <44514E29.4080406@wickedgrey.com> Barry Warsaw wrote: > Okay, radical/insane/heretical question time: do we really need literal > tuple syntax? IOW, what if (1, 2, 3) created a frozenset instead of a > tuple? > > Or perhaps an even stupider idea: what if the compiler could recognize > certain use cases and generate different types depending on how the > object is used. I think you'd have to create a tuple, and convert it to a set on the fly. Which should x be in the following: x = (1, 2, 1) dictOfCallables["foo"](x) My gut feeling is that you'd need to have PyPy's RPython to even begin taking a guess (and AFAIK, the above isn't valid RPython). > So, again, what if (...) created a set/frozen set instead of a tuple? > It's py3k, so broken code be damned. :) How would you have the following code be written? I'm curious what the syntax transformation would be. >>> a = (3,0) >>> b = (0,4) >>> c = (3,4) >>> def addpoint(a, b): ... return (a[0] + b[0], a[1] + b[1]) ... >>> tri[a] = "a" >>> tri[b] = "b" >>> tri[c] = "c" >>> assert tri[addpoint(a, b)] == "c" >>> > put-down-that-pitchfork-ly y'rs, > -Barry just-long-enough-to-light-my-torch-ly y'rs, ;) Eli From talin at acm.org Fri Apr 28 01:47:45 2006 From: talin at acm.org (Talin) Date: Thu, 27 Apr 2006 23:47:45 +0000 (UTC) Subject: [Python-3000] sets in P3K? References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <1146170691.10791.20.camel@resist.wooz.org> Message-ID: Barry Warsaw python.org> writes: > Or perhaps an even stupider idea: what if the compiler could recognize > certain use cases and generate different types depending on how the > object is used. Hey! If anyone is going to post stupid ideas to this list, it's gonne be ME! Got that? Speaking of which, while we are on the subject of literals, lets add a "range literal". We can use the mathematical notation of square brackets to represent inclusion, while parens represent exclusion. So range( 1, 10 ) would be written as: [1,10) Or alternatively, the Pascal notation: 1..10 Also, we should us the greek symbol 'theta' (unicode 03D1) to represent a new empty set. For compatibility with existing editors that do not support unicode editing, we will allow unicode characters to be inserted into the text using the escape sequence \u03d1. So instead of having to write: s = set() You can now write: s = \u03d1 :) :) :) :) :) :) -- Talin From barry at python.org Fri Apr 28 03:42:23 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 27 Apr 2006 21:42:23 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: <44514E29.4080406@wickedgrey.com> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <1146170691.10791.20.camel@resist.wooz.org> <44514E29.4080406@wickedgrey.com> Message-ID: <1146188543.10790.65.camel@resist.wooz.org> On Thu, 2006-04-27 at 16:05 -0700, Eli Stevens (WG.c) wrote: > I think you'd have to create a tuple, and convert it to a set on the > fly. Which should x be in the following: > > x = (1, 2, 1) > dictOfCallables["foo"](x) Well here's another nutty idea. We actually have two ways to create literal tuples, right? x = 1, 2, 3 x = (1, 2, 3) So let's keep the first and hijack the second to be set constructor notation. As for your other questions, I dunno, it was a dumb idea. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060427/60972e59/attachment.pgp From barry at python.org Fri Apr 28 04:00:22 2006 From: barry at python.org (Barry Warsaw) Date: Thu, 27 Apr 2006 22:00:22 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> Message-ID: <1146189622.10789.77.camel@resist.wooz.org> On Thu, 2006-04-27 at 10:18 -0700, Alex Martelli wrote: > > ''.join(seq) == join('', seq) > > I think I would prefer a signature of: > join(seq, joiner='') Except that I think it would be a mistake to have that and keep the .join() method on strings because that would increase the confusion in the argument order. For example, Guido pointed out a while ago that a handy way to remember the argument order in re.search() and friends is to think about pattern.search(string). That's a useful symmetry to keep. > Rationale: an emptystring joiner is the most frequent cases, but > several others (space, newline, space-comma, ...) occur often enough > to be worth allowing the joiner to be optionally specified. If I look at the ones I write a lot it's emptystring, space, newline, comma-space and the Unicode versions of them occasionally. There are others, but those are probably the majority of them. I'm not sure what that tells us, except that I don't think there's an obvious default. You could probably argue for the emptystring just as well as for the space. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 309 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-3000/attachments/20060427/7249fa1f/attachment.pgp From crutcher at gmail.com Fri Apr 28 04:09:26 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Thu, 27 Apr 2006 19:09:26 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: Message-ID: On 4/27/06, Georg Brandl wrote: > Crutcher Dunnavant wrote: > > Having been shot down so hard on my request to bring for loops into > > symetry with list comprehensions, I have a new proposal - stackable > > blocks. > > > > This is a very _small_ amount of syntax sugar, the basic idea is that > > any block headers can be stacked on the same line. This changes no > > semantics, nor does it change ordering, or add new keywords, it just > > changes a tiny bit of the grammar for whitespace/block parsing. > > > > So for instance this: > > > > for file in open_files: > > if file.readable(): > > ... > > > > can be spelled like this. > > > > for file in open_files: if file.readable(): > > ... > > Gak. I think it was a Freudian anti-slip that you used "readable" in the > above example ;) Ha. ha-ha. No. > Also, you know that Guido at one time said he wanted to forbid the > "if x: y" suite style? I don't really care. What I really want is to be able to say: CONTEXTUAL-TRAVERSAL-SPEC: TRAVERSAL CODE I don't really care how it gets spelled. In the current system, I _have_ to break the context spec up on multiple lines, _despite_ the fact that we have list comprehension semantics. I'd like to see a solution. This one seems much simpler than the list comprehension semantic loops I proposed earlier, so I thought it would recieve some sympathy. Aparently it doesn't, which kinda bums me. -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From skip at pobox.com Fri Apr 28 04:09:33 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 27 Apr 2006 21:09:33 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> Message-ID: <17489.31069.526965.186604@montanaro.dyndns.org> Baptiste> Actually I think we need both {/} as the empty set and {:} as Baptiste> the empty dict. Baptiste> {} should raise SyntaxError, '{} is ambiguous, use {/} for Baptiste> set, {:} for dict' Piggybacking on this post and Barry's heretical ideas, (/) looks more like the null set to me... Actually, if Py3k is going to be Unicode only, why not use ? as the null set? ;-) Skip From skip at pobox.com Fri Apr 28 04:11:05 2006 From: skip at pobox.com (skip at pobox.com) Date: Thu, 27 Apr 2006 21:11:05 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: <1146188543.10790.65.camel@resist.wooz.org> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <1146170691.10791.20.camel@resist.wooz.org> <44514E29.4080406@wickedgrey.com> <1146188543.10790.65.camel@resist.wooz.org> Message-ID: <17489.31161.189430.872862@montanaro.dyndns.org> Barry> x = 1, 2, 3 Barry> x = (1, 2, 3) Barry> So let's keep the first and hijack the second to be set Barry> constructor notation. This is a fairly common error: x = 1, For that reason I prefer to always use parens for tuple literals. Skip From tim.peters at gmail.com Fri Apr 28 04:41:59 2006 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 27 Apr 2006 22:41:59 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <1146189622.10789.77.camel@resist.wooz.org> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> <1146189622.10789.77.camel@resist.wooz.org> Message-ID: <1f7befae0604271941p2871190cu5e4e578902305007@mail.gmail.com> [Alex Martelli, suggests that a functional join's default separator be the empty string] >> Rationale: an emptystring joiner is the most frequent cases, but >> several others (space, newline, space-comma, ...) occur often enough >> to be worth allowing the joiner to be optionally specified. [Barry Warsaw] > If I look at the ones I write a lot it's emptystring, space, newline, > comma-space and the Unicode versions of them occasionally. There are > others, but those are probably the majority of them. I'm not sure what > that tells us, except that I don't think there's an obvious default. > You could probably argue for the emptystring just as well as for the > space. The existing string.join's default is a space, and-- back when that was the only way to join() --I remember that suprised me about half, and only half, the time I used it <0.5 wink>. This really isn't like split(), where split-on-whitespace is overwhelmingly most common. In the absence of an obvious default, better not to supply a default at all. From brett at python.org Fri Apr 28 05:41:01 2006 From: brett at python.org (Brett Cannon) Date: Thu, 27 Apr 2006 21:41:01 -0600 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <1f7befae0604271154w6ba78d3p3b6851e85d4dc68d@mail.gmail.com> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> <4450FE5B.4010208@ieee.org> <1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com> <1f7befae0604271154w6ba78d3p3b6851e85d4dc68d@mail.gmail.com> Message-ID: On 4/27/06, Tim Peters wrote: > [Fredrik Lundh] > > no, because people enjoy writing readable code. doing things by exe- > > cuting methods attached to literals isn't very readable, and isn't used > > for anything else. > > As Barry often says, he spells it TAB.join() or BLANK.join() (etc) > instead. That's very readable. > > > I don't think anyone on this list can take the "but if there's more than > > one argument, *I* am going to be confused" argument seriously. > > Then you don't remember that the order of arguments here _was_ a > frequent confusion in the old days. If you like a callable with > separator first, you can also do > > join = str.join > > today, and if you like a callable with separator second, you can also do > > from string import join > > today. My bet is that nobody here uses either, because they don't > really find the method spelling distasteful enough to endure the > one-line bother to set up an alternative that would flood their soul > with joy ;-) > Right, but I am thinking of the newbies for this. I remember when I learned Python and I thought the way str.join() worked was weird (and I still do, although I understand why it how it is). Granted we could just let string concatenation through '+' be more prominent, but we all know the performance issues of looping through an iterator to concatenate strings. > If you want a builtin instead in Py3K, I'm just -0 on that (but -1 if > that's in _addition_ to the three spellings Python already has). I wouldn't want it added without ditching the other two versions as well. Personally I would be fine if string.join() stayed and we considered removing str.join() and just made the string module more prominent (after removing all the deprecated stuff). -Brett From crutcher at gmail.com Fri Apr 28 05:50:42 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Thu, 27 Apr 2006 20:50:42 -0700 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> Message-ID: On 4/27/06, Alex Martelli wrote: > On 4/27/06, Barry Warsaw wrote: > > On Thu, 2006-04-27 at 10:08 -0700, Aahz wrote: > > > > > While I hate the way it looks, I never have gotten mixed up about the > > > order of arguments since switching to ''.join(l). > > > > Which is why > > > > EMPTYSTRING = '' > > > > ... > > > > EMPTYSTRING.join(seq) > > > > looks much better. But hey, yeah, a join() builtin would be fine if it > > took the string arg first, so that > > > > ''.join(seq) == join('', seq) > > I think I would prefer a signature of: > join(seq, joiner='') Just because I have to, why not consider a join operator? Lets look at a simple case, the one matching SEP.join(SEQ), and try using | as the join op (though others, such as <>, ~, @, ::, |>, etc. are available): SEQ | SEP := SEQ[1] + SEP + SEQ[2] + SEP + ... Perhaps though, we'd like a family similar to OP=, so we'd have OP|, and we could say: SEQ +| SEP := SEQ[1] + SEP + SEQ[2] + SEP + ... Though this is a bit unwieldy. Perhaps it is time to support something like the Infix class from the cookbook: http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/384122 This would let us decorate a join builtin with the infix decorator, and then say: SEQ |join| SEP, or, depending upon the join arg order, SEP |join| SEQ > Rationale: an emptystring joiner is the most frequent cases, but > several others (space, newline, space-comma, ...) occur often enough > to be worth allowing the joiner to be optionally specified. > > The ability to omit the joiner in most cases works particularly well > with genexps, e.g. > > join(str(x) for x in whatever) > > ve > > join('', (str(x) for x in whatever)) > > > Alex > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/crutcher%40gmail.com > -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From bwinton at latte.ca Fri Apr 28 04:28:46 2006 From: bwinton at latte.ca (Blake Winton) Date: Thu, 27 Apr 2006 22:28:46 -0400 Subject: [Python-3000] Make it easier to port to small devices. Message-ID: <44517DDE.5040306@latte.ca> Specifically, I'm thinking of the 300MHz Palm TX sitting on the table beside me, which should be more than powerful enough to run Python, but in general I think it should be easier to cross-compile Python for new architectures. I've taken a stab at it, and perhaps it's because I don't have enough experience porting large projects, but it turned out to be a non-starter for me. It might be worth talking to the Nokia and Pippy people to see what the major challenges were, and how we could make it easier for them. I see smaller devices as a new field, without much competition from other languages, in which Python could come to dominate. (Python on the Palm at least exists, even if it is based on 1.5. I haven't heard anything about anyone attempting to put Ruby on a Palm.) Thanks, Blake. From nnorwitz at gmail.com Fri Apr 28 06:10:42 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Thu, 27 Apr 2006 21:10:42 -0700 Subject: [Python-3000] Make it easier to port to small devices. In-Reply-To: <44517DDE.5040306@latte.ca> References: <44517DDE.5040306@latte.ca> Message-ID: Blake, What problems did you have? There is a patch or two up on SF that address some of the issues. I know of some problems, but we don't have a volunteer to work on these issues. So if you could be that volunteer, maybe we could solve this problem. Not for 3.0 in the future, but for 2.5 today. Cheers, n -- On 4/27/06, Blake Winton wrote: > Specifically, I'm thinking of the 300MHz Palm TX sitting on the table > beside me, which should be more than powerful enough to run Python, but > in general I think it should be easier to cross-compile Python for new > architectures. I've taken a stab at it, and perhaps it's because I > don't have enough experience porting large projects, but it turned out > to be a non-starter for me. It might be worth talking to the Nokia and > Pippy people to see what the major challenges were, and how we could > make it easier for them. > > I see smaller devices as a new field, without much competition from > other languages, in which Python could come to dominate. (Python on the > Palm at least exists, even if it is based on 1.5. I haven't heard > anything about anyone attempting to put Ruby on a Palm.) > > Thanks, > Blake. > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/nnorwitz%40gmail.com > From brett at python.org Fri Apr 28 06:14:32 2006 From: brett at python.org (Brett Cannon) Date: Thu, 27 Apr 2006 22:14:32 -0600 Subject: [Python-3000] rough draft signature PEP In-Reply-To: References: <20060426091717.kzb6e411kw00800g@login.werra.lunarpages.com> <4450BA95.3000002@gmail.com> Message-ID: On 4/27/06, Talin wrote: > Nick Coghlan gmail.com> writes: > > > > My mental model matches Brett's. > > > > I think of them as separate things, too. Separate attributes also makes it > > easier to add additional metadata later, and the structure of each attribute > > can be well-defined. > > > > For example, given: > > > > def f(req:int, opt=val, *args, reqkwd, optkwd=val2, **kwds): > > pass > > > > A signature object like the following would then be relatively easy to grasp: > > > > sig = f.__signature__ > > assert sig.required_args == ('req',) > > assert sig.optional_args == ('opt',) > > assert sig.extra_args == 'args' > > assert sig.required_keywords == ('reqkwd',) > > assert sig.optional_keywords == ('optkwd',) > > assert sig.extra_keywords == 'kwds' > > assert sig.defaults == dict(opt=val, optkwd=val2) > > assert sig.argtypes == dict(req=int) > > Sounds like I'm out-voted here. All right :) > > My only other comment is that I think that the format of the signature attribute > should be use-case driven - that is, we should come up with some example use > cases, implement them in code, and see what kind of signature format is most > convenient for those use cases. > I am still on vacation at the moment, so I don't have a ton of time to dive into this right now, but when I have the time I will take a look at attribute naming with a pair of fresh eyes (I have not looked at the PEP in months =) . I also want to add an attribute that points to the function that the signature represents. This would basically give us __decorates__ for free. Last thing I am not sure about is having __signature__ defined automatically, or on a as-needed basis. I originally was thinking the former, but realizing that function creation is not really a big bottleneck, I am leaning towards the latter. > I would suggest the following use cases as sample points within a broader range > of possible uses: > > 1) trace decorator: Print the names of their arguments and their values before > calling the function. > This would be a good sample case. Not so sure about the other two, but this one should be easy to do for any signature object. -Brett > 2) precondition decorator: Allow arbitrary preconditions to be added to specific > parameters. > > 3) alternative dispatch decorator: Using the signature API, implement an > alternative dispatch method. This could be something as simple as matching > argument names - so if you call "func( a=3 )" it searches for a version of func > that has an argument named 'a'. From janssen at parc.com Fri Apr 28 06:50:46 2006 From: janssen at parc.com (Bill Janssen) Date: Thu, 27 Apr 2006 21:50:46 PDT Subject: [Python-3000] Add a standard GUI system Message-ID: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> I mentioned this a couple of weeks ago, and here it is again. The biggest wart in Python 2, the one that forces me to use another language, is the lack of a standard GUI. By standard, I mean that when you install Python on a system, the GUI toolkit is also installed. I suggest that for Python 3K, we adopt Greg Ewing's PyGUI into the standard library. It's been around for years, works on every major platform (using GTK+ for Windows and X11, and Cocoa for Mac OS X), is Pythonic in flavor, and is aimed at eventual inclusion into the standard library. http://www.cosc.canterbury.ac.nz/~greg/python_gui/ Might want to add another port of it, to a lower-level multi-platform framework like cairo (which is now being used in Mono, and Firefox, and GTK+). To function properly in a plug-in in a Web browser, you probably need to take over an existing window, and use it. Bill From tjreedy at udel.edu Fri Apr 28 07:02:44 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 Apr 2006 01:02:44 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com><20060427170803.GB17595@panix.com><1146157955.10752.161.camel@resist.wooz.org><4450FE5B.4010208@ieee.org><1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com><1f7befae0604271154w6ba78d3p3b6851e85d4dc68d@mail.gmail.com> Message-ID: "Brett Cannon" wrote in message news:bbaeab100604272041r2062f339je99e7a2b4ee6ca17 at mail.gmail.com... > I wouldn't want it added without ditching the other two versions as > well. Personally I would be fine if string.join() stayed and we > considered removing str.join() and just made the string module more > prominent (after removing all the deprecated stuff). -1 Get rid of the string module instead. I consider str.join an advance. Had trouble remembering arg order of string.join, never do with str.join or actual .join. tjr From tjreedy at udel.edu Fri Apr 28 07:13:31 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 Apr 2006 01:13:31 -0400 Subject: [Python-3000] Stackable Blocks References: Message-ID: "Crutcher Dunnavant" wrote in message news:d49fe110604271909i2b4b068aq3230021f4abca524 at mail.gmail.com... > I don't really care how it gets spelled. In the current system, I > _have_ to break the context spec up on multiple lines, I consider that a plus. It is part of Python's basic indented statement structure, which makes it easy to read. Also, it is possibly more common to have more that one statement in outer context. >_despite_ the fact that we have list comprehension semantics. The problem and danger with lcs is that they can be used to turn perfectly readable code into something much less readable. They are useful for compact expressions that can be easily comprehended in one reading, but can be pushed too far. Terry Jan Reedy From crutcher at gmail.com Fri Apr 28 07:44:36 2006 From: crutcher at gmail.com (Crutcher Dunnavant) Date: Thu, 27 Apr 2006 22:44:36 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: Message-ID: On 4/27/06, Terry Reedy wrote: > > "Crutcher Dunnavant" wrote in message > news:d49fe110604271909i2b4b068aq3230021f4abca524 at mail.gmail.com... > > I don't really care how it gets spelled. In the current system, I > > _have_ to break the context spec up on multiple lines, > > I consider that a plus. It is part of Python's basic indented statement > structure, which makes it easy to read. Also, it is possibly more common > to have more that one statement in outer context. > > >_despite_ the fact that we have list comprehension semantics. > > The problem and danger with lcs is that they can be used to turn perfectly > readable code into something much less readable. They are useful for > compact expressions that can be easily comprehended in one reading, but can > be pushed too far. Of course they can be pushed to far, any programming construct can be pushed to far such as: very deep: nesting: because: it makes me: a better person -- Crutcher Dunnavant littlelanguages.com monket.samedi-studios.com From kay.schluehr at gmx.net Fri Apr 28 08:00:20 2006 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Fri, 28 Apr 2006 08:00:20 +0200 Subject: [Python-3000] sets in P3K? In-Reply-To: <17489.31069.526965.186604@montanaro.dyndns.org> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> Message-ID: <4451AF74.8040702@gmx.net> skip at pobox.com schrieb: >Piggybacking on this post and Barry's heretical ideas, (/) looks more like >the null set to me... Actually, if Py3k is going to be Unicode only, why >not use ? as the null set? ;-) > > Yes, Py3K + Unicode rulez! I'm +1 on this but not before having a Py3K keyboard :) Musing about flexible/loadable keyboard displays I sometimes believe that all those excersises in 127Bit-ASCII syntax scrabbling are retro. As long as we deal with ASCII flavors I'd vote for {/} and {:} as the initial objects of set and dict. From kay.schluehr at gmx.net Fri Apr 28 08:43:19 2006 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Fri, 28 Apr 2006 08:43:19 +0200 Subject: [Python-3000] PEP 3002 (Procedure for Backwards-Incompatible Changes) In-Reply-To: References: Message-ID: <4451B987.1030804@gmx.net> Guido van Rossum schrieb: >I suppose the PEP 3002 proposal was written with the assumption that a >*syntactic* conversion would be doable purely mechanically, and that >the semantic issues would be sorted out using a second phase tool. > I have a question towards the syntax/semantics distinction that is involved here in migration issues. Is this boundary a political or a technical one? I'm aware that semantical differences cannot be detected using a static analyzer because type-information is not available at compile time but runtime-type checks could be performed instead. So a "critical function" call can be replaced by a call on a wrapper function that branches along the detected runtime types. Example: In Py3K floor integer divison is expressed using the double slash operator // that replaces the single slash operator used in Python 2.X. We cannot naively translate a/b in Python 2.X to a//b because no type information is available but we can relpace a/b by the wrapper newdiv(a,b) where def newdiv(a,b): # Python2.X wrapper of a/b to obtain Py3K behaviour if isinstance(a, int) and isinstance(b, int): return a//b else: return a/b I don't see yet severe limitations to this procedure since Python has good reflective capabilities ( even where they don't spring off naturally you can tune them: for example if you want to reflect on the module origin of a certain function it is possible to perform automatical annotations using decorators that are merged with the Python 2.X legacy code on AST level ). On the other hand this automatical translations are done likely for the purpose of letting old Python code just work on Py3K not extending it as translated source. Regards, Kay From mike.klaas at gmail.com Fri Apr 28 09:07:19 2006 From: mike.klaas at gmail.com (Mike Klaas) Date: Fri, 28 Apr 2006 00:07:19 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <200604271217.59323.gmccaughan@synaptics-uk.com> References: <44506457.1040607@gmx.net> <200604271217.59323.gmccaughan@synaptics-uk.com> Message-ID: <3d2ce8cb0604280007q303261bbu6bbc7634d6a2a9b0@mail.gmail.com> On 4/27/06, Gareth McCaughan wrote: > On Thursday 2006-04-27 11:42, Georg Brandl wrote: > > Kay Schluehr wrote: > ... > > > Furthermore: what about notions of infinite sets? > > > > Can you elaborate on that? > > Seems to me that if you want infinite sets, you want some type > other than "set". Ah, the wonders of duck typing. (Just as if > you want mappings with infinite domain, you want a type other > than "dict".) <> This discussion has concentrated too heavily on the correspondence with mathematical sets and has proved to be somewhat of a chimaera. Python sets are _data structures_, and while they share some aesthetic similarities with the mathematical homonym, drawing on the latter for more than initial inspiration is flawed. It is easy to illustrate why: Assume someone wanted to work with math sets in python. Would set() cover any, let alone most use cases? Of interest is reasoning about their existence, emptiness, finitude, etc. How often do you want to iterate over a mathematical set? I've used set()s extensively since they were introduced. Many of my uses ignore the non-duplicative aspects of sets entirely, and use them for their algorithmic properties (O(1) insertion/containment tests). I particularly dislike any notion of using a phi or phi-like symbol to represent python set construction. The mathematical empty set is a totally different beast. True: ? is ?. False: set() is set(). Grepping through my source code. I find many uses of literal set construction. About half are frozenset([...]), half set([...]), but almost all the set([...]) construction calls should have been frozensets (but weren't due to laziness in unittest, etc.). there are also several instances where I would have used (frozen) sets over lists/tuples had a syntactic shortcut existed. I'm ambivalent on the need for an empty set literal. 'set()' is fine to type. Nor do I think {} has any advantage over set(). But {} notation for frozensets would be a boon. Interestingly, I noticed that most of my used of { :, ... } dict construction expressions are used as "frozen" dicts. Despite that, I'm not bothered by these dicts being mutable. As so, were a {} set construction syntax to create sets rather than frozensets, I would likely still use it in place of frozenset. -Mike From jcarlson at uci.edu Fri Apr 28 09:19:30 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 28 Apr 2006 00:19:30 -0700 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> References: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> Message-ID: <20060428000801.671E.JCARLSON@uci.edu> Bill Janssen wrote: > > I mentioned this a couple of weeks ago, and here it is again. > > The biggest wart in Python 2, the one that forces me to use another > language, is the lack of a standard GUI. By standard, I mean that > when you install Python on a system, the GUI toolkit is also > installed. > > I suggest that for Python 3K, we adopt Greg Ewing's PyGUI into the > standard library. It's been around for years, works on every major > platform (using GTK+ for Windows and X11, and Cocoa for Mac OS X), is > Pythonic in flavor, and is aimed at eventual inclusion into the > standard library. > > http://www.cosc.canterbury.ac.nz/~greg/python_gui/ > > Might want to add another port of it, to a lower-level multi-platform > framework like cairo (which is now being used in Mono, and Firefox, > and GTK+). To function properly in a plug-in in a Web browser, you > probably need to take over an existing window, and use it. Python already comes with a standard GUI toolkit; tk. Also, while I generally like Greg Ewing's work (Pyrex has saved me so much brain power interfacing with C extensions), PyGUI doesn't have nearly the market penetration (into the Python GUI toolkit area) to justify its inclusion into the python standard library. Also, it would require shipping a Python GTK binding, which would basically necessitate shipping GTK+ with Python. If we are going to go to the bother of shipping a GUI toolkit with Python (in addition to or in replacement of tcl/tk), we may as well go whole hog and take wxPython; it's market penetration (into the Python GUI toolkit area) is growing steadily, and it offers an "upgrade" path towards more "Pythonic" additions like Pythoncard, Wax, or even Dabo. Add to all this the amazing support that Robin Dunn offers, and you are not likely find any better option (if we are going to change). Ultimately, I think the work to get everything included is not insignificant, so I'm not going to push for it (because I'm probably not going to be doing any of that work). +0 on switching included toolkits -1 for anything that isn't wxPython +0 for wxPython - Josiah From phd at mail2.phd.pp.ru Fri Apr 28 10:07:29 2006 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Fri, 28 Apr 2006 12:07:29 +0400 Subject: [Python-3000] More batteries included! In-Reply-To: References: Message-ID: <20060428080729.GA7746@phd.pp.ru> On Thu, Apr 27, 2006 at 04:02:14PM -0400, Terry Reedy wrote: > The initial download for Internet Explorer > updates is less than a megabyte. It looks at what you have and displays a > menu of optional components, and then downloads and installs the 15-30 > megabytes actually needed. Thus making it impossible to upgrade a LAN (intranet) that's not connected to Internet. Administrator's nightamre. :( Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From skip at pobox.com Fri Apr 28 12:10:09 2006 From: skip at pobox.com (skip at pobox.com) Date: Fri, 28 Apr 2006 05:10:09 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: <3d2ce8cb0604280007q303261bbu6bbc7634d6a2a9b0@mail.gmail.com> References: <44506457.1040607@gmx.net> <200604271217.59323.gmccaughan@synaptics-uk.com> <3d2ce8cb0604280007q303261bbu6bbc7634d6a2a9b0@mail.gmail.com> Message-ID: <17489.59905.992631.381588@montanaro.dyndns.org> Mike> I particularly dislike any notion of using a phi or phi-like Mike> symbol to represent python set construction. You did notice the ;-) in my post on that topic, right? That was meant to imply it wasn't a serious proposal. Being a US programmer myself, I understand that typing ? would be a challenge for most of us on the US side of the pond. In my case it's Fn-Shift-F3 / O though I note now that my Mac supports Shift-Option-O as well. In any case, neither is all that straightforward, and ? does look a lot like zero in some fonts. Skip From fredrik at pythonware.com Fri Apr 28 13:19:38 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 28 Apr 2006 13:19:38 +0200 Subject: [Python-3000] Automatically invoking str() in str.join() References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu><4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com><1146157955.10752.161.camel@resist.wooz.org><4450FE5B.4010208@ieee.org><1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com> <1f7befae0604271154w6ba78d3p3b6851e85d4dc68d@mail.gmail.com> Message-ID: Tim Peters wrote: >> I don't think anyone on this list can take the "but if there's more than >> one argument, *I* am going to be confused" argument seriously. > > Then you don't remember that the order of arguments here _was_ a > frequent confusion in the old days. nope. any pointers ? From fredrik at pythonware.com Fri Apr 28 13:26:36 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Fri, 28 Apr 2006 13:26:36 +0200 Subject: [Python-3000] More batteries included! References: <5384bedc0604260350k1c92a212s8825c680516624bd@mail.gmail.com><444F5444.4090002@gmail.com><004601c669e2$a21183a0$8d472597@bagio> Message-ID: Terry Reedy wrote: >> hopefully, the Py3K distributors will use semi-automatic tools to download >> all they need to build a battery-included release. pushing all that over to >> the end users would be a major mistake. >> >> (python-dev and others could provide "profiles" for well-known and tested >> configurations; e.g. "psf standard", "scientific", "sumo", ...) > > I think this worth considering. The initial download for Internet Explorer updates is less than a > megabyte. It looks at what you have and displays a menu of optional components, and then > downloads and installs the 15-30 megabytes actually needed. you misread my argument: I'm saying that the core Python distribution should be reasonably small (=smaller than today), but that we should provide a list of well-known/well-tested recommended packages that a distributor may want to include in their Python configuration, and a mechanism that they can use to get the packages, so they can include them in their distribution. we should also allow others to publish additional or alternative lists. (setuptools already contains most of the mechanisms needed for this, of course) if the distributor wants to ship the result as a sumo-installer, a highly granular set of RPM packages, an intelligent mini-installer, a bunch of eggs, or use some other approach is up to the distributor. From gvwilson at cs.utoronto.ca Fri Apr 28 14:09:24 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Fri, 28 Apr 2006 08:09:24 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: OK, OK, just so I can get it off my chest: - introduce '@' as a prefix operator meaning 'freeze'. - [1, 2, 3] is a mutable sequence (our old friend the list) - {1, 2, 3} is a mutable set - @{1, 2, 3} is an immutable set (i.e., a frozen set --- frozen at creation, so all current optimizations possible) - @[1, 2, 3] is an immutable sequence (i.e., a tuple) Now, if only I can figure out whether "@x" means "bind the name x permanently to its current value to create a symbolic constant" or "freeze the object to which x currently refers"... ;-) Thanks, Greg From kendall at monkeyfist.com Fri Apr 28 14:26:22 2006 From: kendall at monkeyfist.com (Kendall Clark) Date: Fri, 28 Apr 2006 08:26:22 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <1146170691.10791.20.camel@resist.wooz.org> Message-ID: <7E6F4352-BAE5-42E5-85C0-46F1A9E07368@monkeyfist.com> On Apr 27, 2006, at 7:47 PM, Talin wrote: > Or alternatively, the Pascal notation: > > 1..10 +1..2 :> Cheers, Kendall From goodger at python.org Fri Apr 28 16:01:38 2006 From: goodger at python.org (David Goodger) Date: Fri, 28 Apr 2006 10:01:38 -0400 Subject: [Python-3000] empty set & dict syntax Message-ID: <4335d2c40604280701p77cf8c2bl23288e00e6f51d32@mail.gmail.com> Sets and dicts are similar beasts, right? "A set object is an unordered collection of immutable values." Dicts can be described as sets with lookup values associated with the keys. Sets can be described as dicts with only keys, no values. So how about creating a new base set/mapping type (call it 'baseset' for now), use "{}" as the literal syntax of baseset instances, and specialize dynamically on the first unambiguous use? Empty sets and empty dicts would *both* be spelled "{}". For example: >>> x = {} >>> type(x) >>> bool(x) # common operation, doesn't alter type False >>> x['a'] = 1 # dict-only operation >>> type(x) >>> y = {} >>> y.add('a') # set-only operation >>> type(y) Any specialized operation on an empty baseset instance determines its type, even if the operation doesn't alter the instance: >>> z = {} >>> 'a' in z # common operation, doesn't affect type False >>> type(z) >>> z.items() # dict-only operation [] >>> type(z) Initializing the baseset is an unambiguous operation: >>> type({'a'}) >>> type({'a': 1}> -- David Goodger From guido at python.org Fri Apr 28 16:32:48 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Apr 2006 07:32:48 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <4451AF74.8040702@gmx.net> References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> Message-ID: On 4/27/06, Kay Schluehr wrote: > Yes, Py3K + Unicode rulez! I'm +1 on this but not before having a Py3K > keyboard :) I know you were only joking in a dreamy sort of way; but I have a serious point to make against Unicode in the language source code. *Supporting* Unicode is very important for equality amongst nations. But *requiring* it in order to type your programs would just create *inequality*. Think of the $100 laptop. Think of other refurbished laptops being sent to 3rd world countries. Think of people whose keyboard is already full with the characters of their native language. We're lucky enough that the latin alphabet and ASCII are historically dominant so there will always be a way to type "A" or "{" (although I believe Swiss keyboards already require an option key for one or two common Python punctuation characters). I'm also not in favor of supporting a Unicode version and an ASCII alternative for certain characters; this would just create more inequality, and everybody would have to learn to read both and know the equivalence. I want good Unicode support for string literals and comments. Everything else in the language ought to be ASCII. Hopefully Python 4000 will be designed in a different world. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jack at performancedrivers.com Fri Apr 28 16:52:29 2006 From: jack at performancedrivers.com (Jack Diederich) Date: Fri, 28 Apr 2006 10:52:29 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <4450C80D.50002@iinet.net.au> Message-ID: <20060428145229.GI13633@performancedrivers.com> On Fri, Apr 28, 2006 at 01:02:44AM -0400, Terry Reedy wrote: > > "Brett Cannon" wrote in message > news:bbaeab100604272041r2062f339je99e7a2b4ee6ca17 at mail.gmail.com... > > I wouldn't want it added without ditching the other two versions as > > well. Personally I would be fine if string.join() stayed and we > > considered removing str.join() and just made the string module more > > prominent (after removing all the deprecated stuff). > -1 > Get rid of the string module instead. I consider str.join an advance. Had > trouble remembering arg order of string.join, never do with str.join or > actual .join. > I use string.ascii_letters and string.printable() occasionally. Many of those members could be dropped though (atoi, lowercase, etc) [GCC 3.3.5 (Debian 1:3.3.5-8ubuntu2)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import string >>> set(dir(string)) - set(dir(str)) set(['ascii_lowercase', '_re', 'letters', 'uppercase', 'ascii_letters', 'capwords', 'index_error', 'Template', 'ascii_uppercase', 'octdigits', '_idmapL', '_int', '_float', 'atol_error', '__builtins__', 'splitfields', 'punctuation', '_multimap', 'joinfields', 'printable', '__file__', 'atof_error', 'atoi', 'hexdigits', 'atol', '__name__', 'atof', 'whitespace', 'digits', 'lowercase', '_long', 'atoi_error', 'maketrans', '_TemplateMetaclass', '_idmap']) >>> -Jack From mcherm at mcherm.com Fri Apr 28 18:15:20 2006 From: mcherm at mcherm.com (Michael Chermside) Date: Fri, 28 Apr 2006 09:15:20 -0700 Subject: [Python-3000] sets in P3K? Message-ID: <20060428091520.22eiofbwxa4gc8g4@login.werra.lunarpages.com> Barry writes: > Well here's another nutty idea. We actually have two ways to create > literal tuples, right? > > x = 1, 2, 3 > x = (1, 2, 3) > > So let's keep the first and hijack the second to be set constructor > notation. Nutty and wrong (but don't let that stop you from continuing to think up interesting alternative approaches in the future!). Those are not two different literals... both use the comma literal to create a tuple, and the second has parentheses (used for grouping) around it. For instance, this is all legal Python: # warning: untested code x = "123" y = ("123") x = {1:2, 3:4} y = ({1:2, 3:4}) x = [1, 2, 3] y = ([1, 2, 3]) In each case, x and y are set to the same value. The fact that '()' is special notation for the empty tuple is just misleading -- parentheses have nothing to do with tuple notation for non-empty tuples. -- Michael Chermside From jimjjewett at gmail.com Fri Apr 28 18:45:23 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 28 Apr 2006 12:45:23 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: On 4/28/06, Greg Wilson wrote: > - introduce '@' as a prefix operator meaning 'freeze'. I mostly like the concept, but not the operator. Using @ limits it to syntax-created literals (because of ambiguity with decorators). Since numbers are already immutable, that basically means lists, dicts, and maybe sets, but not even user-defined subclasses. (And you don't have to do it with tuples, because ... uh ...) I'm not sure what to use instead, except maybe for the backwards quote, which means I have to duck now too. ("frozen" or "snapshot" might be too long to be useful.) I am slightly concerned that it *might* be a bad idea to encourage immutable containers, in the same way that it can be a bad idea to prevent subclassing or even to use a __slots__ field. > - [1, 2, 3] is a mutable sequence (our old friend the list) > > - {1, 2, 3} is a mutable set > > - @{1, 2, 3} is an immutable set (i.e., a frozen set --- frozen > at creation, so all current optimizations possible) > > - @[1, 2, 3] is an immutable sequence (i.e., a tuple) Are there still plans to push the claim that lists hold a homogenous collection, and tuples don't? A frozen list (and a mutable tuple) would make that claim easier to defend, but it still wouldn't be easy. I'm inclined to go with real objects and named attributes... > Now, if only I can figure out whether "@x" means "bind the name x > permanently to its current value to create a symbolic constant" or "freeze > the object to which x currently refers"... ;-) If it is really limited to literals, then I'm not sure it matters; the freeze should have the only reference. Unless you were thinking of supporting horrible things like x=@[1,2,3] y=x y.append(4) # a different *name*, so we can now treat it mutably -jJ From gvwilson at cs.utoronto.ca Fri Apr 28 19:00:34 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Fri, 28 Apr 2006 13:00:34 -0400 (EDT) Subject: [Python-3000] sets in P3K? (off-topic joke) In-Reply-To: References: Message-ID: > Guido van Rossum: > Hopefully Python 4000 will be designed in a different world. When the first (and only) Cray-3 was delivered, Seymour Cray announced that he had already started designing the Cray-4, and that he was planning to use the planet Neptune as a cooling system... ;-) Greg From janssen at parc.com Fri Apr 28 19:11:15 2006 From: janssen at parc.com (Bill Janssen) Date: Fri, 28 Apr 2006 10:11:15 PDT Subject: [Python-3000] Add a standard GUI system In-Reply-To: Your message of "Fri, 28 Apr 2006 00:19:30 PDT." <20060428000801.671E.JCARLSON@uci.edu> Message-ID: <06Apr28.101121pdt."58641"@synergy1.parc.xerox.com> > Python already comes with a standard GUI toolkit; tk. With a decidedly unpythonic bad interface. And it's not standard; I frequently find Python installations where no one has installed Tk. Pardon me for being blunt: Tkinter is a half-assed "solution" that's set Python back for years. > Also, while I > generally like Greg Ewing's work (Pyrex has saved me so much brain power > interfacing with C extensions), PyGUI doesn't have nearly the market > penetration (into the Python GUI toolkit area) to justify its inclusion > into the python standard library. Doesn't matter. Putting it into the stdlib will solve the penetration problem. > Also, it would require shipping a Python GTK binding, which would > basically necessitate shipping GTK+ with Python. No, it wouldn't. It runs natively on Mac OS X, and could be made to do on Windows. I'd suggest using the existing GTK+ binding for Linux, and only in the interim for Windows. I certainly wouldn't ship GTK+ with Python, or wxWindows either. As I suggested, I think that it would be a good idea to do a PyGUI port to a lower-level system, such as cairo, for more generic use. I wouldn't object to shipping cairo to those few systems where it won't be pre-installed. > it's market penetration (into the Python GUI toolkit area) > is growing steadily Pointless consideration. Tkinter has a great penetration, but it's a bad solution. Bill From rrr at ronadam.com Fri Apr 28 19:27:14 2006 From: rrr at ronadam.com (Ron Adam) Date: Fri, 28 Apr 2006 12:27:14 -0500 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: Jim Jewett wrote: > On 4/28/06, Greg Wilson wrote: >> - introduce '@' as a prefix operator meaning 'freeze'. > > I mostly like the concept, but not the operator. Using @ limits it to > syntax-created literals (because of ambiguity with decorators). Since > numbers are already immutable, that basically means lists, dicts, and > maybe sets, but not even user-defined subclasses. (And you don't have > to do it with tuples, because ... uh ...) I'm not sure what to use > instead, except maybe for the backwards quote, which means I have to > duck now too. ("frozen" or "snapshot" might be too long to be > useful.) How about the '!' which is currently used indicate a python long and an integer. Here it would indicate the difference between a frozen container and a mutable container. frozen_set = {1, 2, 3}! frozen_dict = {1:'a', 2:'b', 3:'c'}! > I am slightly concerned that it *might* be a bad idea to encourage > immutable containers, in the same way that it can be a bad idea to > prevent subclassing or even to use a __slots__ field. We can always discourage something in the docs by giving reasons not to use that which isn't a good idea in the context for which it isn't a good idea. Cheers, Ron From nnorwitz at gmail.com Fri Apr 28 19:37:11 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 28 Apr 2006 10:37:11 -0700 Subject: [Python-3000] Summer of Code mailing list Message-ID: There's a new SoC mailing list. soc2006 at python.org You can sign up here: http://mail.python.org/mailman/listinfo/soc2006 This list is for any SoC discussion: mentors, students, idea, etc. Student can submit applications starting May 1, so now is the time to get students interested in your ideas! Please pass this information along. Cheers, n From jimjjewett at gmail.com Fri Apr 28 20:14:51 2006 From: jimjjewett at gmail.com (Jim Jewett) Date: Fri, 28 Apr 2006 14:14:51 -0400 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: On 4/28/06, Greg Wilson wrote: > > If it is really limited to literals, then I'm not sure it matters; the > > freeze should have the only reference. > I'd like to say: > x = 123 and x's value will never ever change > Opens up lots of optimization possibilities, and makes code much safer. So now were back to looking for a const marker. Do you want to limit it to literals, or do you want to also say (magic constant) x = y meaning that x is bound do the current value of y? If so, do you want to freeze the object itself to some level of recursion? "V will always be this dict" isn't as useful as "V will always be the same dict and that dict itself will never change again" As to what the marker should be: const x=123 # No worse than global, but ... that is faint praise. @const # decorators can wrap assignments? Allows other type specifiers x = 123 # ... but a local-variable *slot* isn't currently a full-fledged object const(x) = 123 # variation of the decorator -jJ From gvwilson at cs.utoronto.ca Fri Apr 28 20:21:02 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Fri, 28 Apr 2006 14:21:02 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: > > Greg Wilson: > > I'd like to say: > > x = 123 and x's value will never ever change > > Opens up lots of optimization possibilities, and makes code much safer. > Jim Jewett: > So now were back to looking for a const marker. Greg Wilson: Yep. But I'm happy to stick to arguing about set literals... ;-) > Do you want to limit it to literals, or do you want to also say > (magic constant) x = y > meaning that x is bound do the current value of y? I know, I know: "const type * p" vs. "type const * p"... *sigh* Thanks, Greg From tjreedy at udel.edu Fri Apr 28 21:12:59 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 28 Apr 2006 15:12:59 -0400 Subject: [Python-3000] Add a standard GUI system References: <20060428000801.671E.JCARLSON@uci.edu> <06Apr28.101121pdt."58641"@synergy1.parc.xerox.com> Message-ID: "Bill Janssen" wrote in message news:06Apr28.101121pdt."58641"@synergy1.parc.xerox.com... >> PyGUI doesn't have nearly the market >> penetration (into the Python GUI toolkit area) to justify its inclusion >> into the python standard library. > > Doesn't matter. Putting it into the stdlib will solve the penetration > problem. >> [wxPython]'s market penetration (into the Python GUI toolkit area) >> is growing steadily > > Pointless consideration. Not. >Tkinter has a great penetration, but it's a bad solution. To me, your logic is backwards. wxWidgets/wxPython are popular and have perhaps the highest usage (not counting IDLE) by voluntary choice. Tkinter's apparent penetration is pretty clearly boosted by being in the standard lib. Non-analogy. So you want to artificially boost the penetration of your minority favorite by putting it in the standard lib. To me, that is a bad solution. The generic solution is better package fetching, such as apparently done (I have not used them yet) by setuptools and easy_install. The installing an app needing wxPython should fetch and install that also and installing an app needing PyGUI would do the same for PyGUI. Terry Jan Reedy From guido at python.org Fri Apr 28 21:53:09 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Apr 2006 12:53:09 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: On 4/28/06, Jim Jewett wrote: > On 4/28/06, Greg Wilson wrote: > > - introduce '@' as a prefix operator meaning 'freeze'. > > I mostly like the concept, but not the operator. Let me go on record as not liking the concept. IMO it's usually a mistake to want to take some data structure and ask for it to be frozen. (Especially since there's usually a way to thaw it.) I'm not going to argue the point further (I'm sure others will on behalf) and I know there are two sides to this coin; but I think you can save your time. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri Apr 28 22:03:49 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Apr 2006 13:03:49 -0700 Subject: [Python-3000] Stackable Blocks In-Reply-To: References: Message-ID: On 4/23/06, Crutcher Dunnavant wrote: > So for instance this: > > for file in open_files: > if file.readable(): > ... > > can be spelled like this. > > for file in open_files: if file.readable(): > ... Since this discussion is still going on, let me add my -1 in the hope that this thread will die. It ain't gonna happen; it reduces code readability. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From paul-python at svensson.org Fri Apr 28 22:10:25 2006 From: paul-python at svensson.org (Paul Svensson) Date: Fri, 28 Apr 2006 16:10:25 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: <20060428160312.O87427@familjen.svensson.org> > On 4/28/06, Greg Wilson wrote: >> - introduce '@' as a prefix operator meaning 'freeze'. Nice idea, but is it possible to be more radical ? How about having all the existing literal syntaxes produce frozen containers, using explicit calls to dict(), set(), list(), etc, to create mutable copies. How much code would that break ? /Paul From martin at v.loewis.de Fri Apr 28 22:24:23 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 28 Apr 2006 22:24:23 +0200 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: References: <4450BE0E.8090903@gmail.com> <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> Message-ID: <445279F7.8080306@v.loewis.de> Guido van Rossum wrote: > I want good Unicode support for string literals and comments. > Everything else in the language ought to be ASCII. Does that include restricting identifiers to ASCII as well? I was hoping to propose a PEP on non-ASCII identifiers some day; that would (of course) include a requirement that the standard library would always be restricted to ASCII-only identifiers as a style-guide. Regards, Martin From martin at v.loewis.de Fri Apr 28 22:30:30 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri, 28 Apr 2006 22:30:30 +0200 Subject: [Python-3000] Make it easier to port to small devices. In-Reply-To: References: <44517DDE.5040306@latte.ca> Message-ID: <44527B66.5070305@v.loewis.de> Neal Norwitz wrote: > What problems did you have? There is a patch or two up on SF that > address some of the issues. > > I know of some problems, but we don't have a volunteer to work on > these issues. So if you could be that volunteer, maybe we could solve > this problem. Not for 3.0 in the future, but for 2.5 today. Right. Portability isn't a Python 3000 topic: Python was always portable to the platforms people care about, and will always be. If it doesn't run on your system, it is because nobody cared about porting it, even though it is likely very easy to port Python, if you know how to do porting in the first place. There might be other issues with memory foot-print, and they certainly deserve discussion, and there might be other issues with embedded systems (e.g. how to expose the funny peripheral devices that these systems often have), but all of this could happen today if there were only contributions. Regards, Martin From guido at python.org Fri Apr 28 22:44:53 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Apr 2006 13:44:53 -0700 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: <445279F7.8080306@v.loewis.de> References: <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> Message-ID: On 4/28/06, "Martin v. L?wis" wrote: > Guido van Rossum wrote: > > I want good Unicode support for string literals and comments. > > Everything else in the language ought to be ASCII. > > Does that include restricting identifiers to ASCII as well? Yes, that's my intent. > I was hoping to propose a PEP on non-ASCII identifiers some > day; that would (of course) include a requirement that the > standard library would always be restricted to ASCII-only > identifiers as a style-guide. IMO communication about code becomes much more cumbersome if there are non-ASCII letters in identifiers, and the rules about what's a letter, what's a digit, and what separates two identifiers become murky. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From ferringb at gmail.com Fri Apr 28 23:32:07 2006 From: ferringb at gmail.com (Brian Harring) Date: Fri, 28 Apr 2006 14:32:07 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: <20060428213207.GA22469@nightcrawler> On Fri, Apr 28, 2006 at 08:09:24AM -0400, Greg Wilson wrote: > OK, OK, just so I can get it off my chest: > > - introduce '@' as a prefix operator meaning 'freeze'. > > - [1, 2, 3] is a mutable sequence (our old friend the list) > > - {1, 2, 3} is a mutable set > > - @{1, 2, 3} is an immutable set (i.e., a frozen set --- frozen > at creation, so all current optimizations possible) > > - @[1, 2, 3] is an immutable sequence (i.e., a tuple) > > Now, if only I can figure out whether "@x" means "bind the name x > permanently to its current value to create a symbolic constant" or "freeze > the object to which x currently refers"... ;-) Potentially a stupid question, but I've been following this thread for a while and I'm still not seeing the real gain of extending the syntax for frozen markers/set literals. Someone care to recap what actually is gained from having a set literal? ~harring -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://mail.python.org/pipermail/python-3000/attachments/20060428/a94bf0b4/attachment.pgp From mike.klaas at gmail.com Fri Apr 28 23:37:26 2006 From: mike.klaas at gmail.com (Mike Klaas) Date: Fri, 28 Apr 2006 14:37:26 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <17489.59905.992631.381588@montanaro.dyndns.org> References: <44506457.1040607@gmx.net> <200604271217.59323.gmccaughan@synaptics-uk.com> <3d2ce8cb0604280007q303261bbu6bbc7634d6a2a9b0@mail.gmail.com> <17489.59905.992631.381588@montanaro.dyndns.org> Message-ID: <3d2ce8cb0604281437p679ce344vde080bcc49ca89ca@mail.gmail.com> On 4/28/06, skip at pobox.com wrote: > You did notice the ;-) in my post on that topic, right? That was meant to > imply it wasn't a serious proposal. Sure did . I'd be shocked if non-ascii punctuation was ever seriously considered. But ascii phi-like things have been proposed. cheers, -Mike From martin at v.loewis.de Sat Apr 29 00:04:31 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 29 Apr 2006 00:04:31 +0200 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: References: <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> Message-ID: <4452916F.1030304@v.loewis.de> Guido van Rossum wrote: >> I was hoping to propose a PEP on non-ASCII identifiers some >> day; that would (of course) include a requirement that the >> standard library would always be restricted to ASCII-only >> identifiers as a style-guide. > > IMO communication about code becomes much more cumbersome if there are > non-ASCII letters in identifiers, and the rules about what's a letter, > what's a digit, and what separates two identifiers become murky. It depends on the language you use to communicate. In English, it is certainly cumbersome to talk about Chinese identifiers. OTOH, I believe it is cumbersome to communicate about English identifiers in Chinese, either, because the speakers might not even know what the natural-language concept behind the identifiers is, and because they can't pronounce the identifier. As for lexical aspects: these are really straight-forward. In principal, it would be possible to allow any non-ASCII character as part of an identifier: all punctuation is ASCII, so anything non-ASCII can't possibly be punctuation for the language. However, that much freedom would be confusing; the Unicode consortium has established rules of what characters should be allowed in identifiers, and these rules intend to match the intuition of the users of these characters. The distinction of letters and digits is also straight-forward: a digit is ASCII [0-9]; it's a separate lexical class only because it plays a special role in (number) literals. More generally, there is the distinction of starter and non-starter characters. An identifier ends when the first non-identifier character is encountered (although I don't think there are many places in Python where you can have two identifiers immediately following each other). Regards, Martin From tomerfiliba at gmail.com Sat Apr 29 00:11:27 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 29 Apr 2006 00:11:27 +0200 Subject: [Python-3000] suggestion for a new socket io stack Message-ID: <1d85506f0604281511i3829ce1dq5b2aa7e9e3a96044@mail.gmail.com> i've seen discussion about improving the IO stack of python, to rely less on the low-level lib-c implementation. so, i wanted to share my ideas in that niche. i feel today's sockets are way *outdated* and overloaded. python's sockets are basically a wrapper for the low-level BSD sockets... but IMHO it would be much nicer to alleviate this dependency: expose a more high-level interface to socket programming. the *BSD-socket methodology* does not sit well with pythonic paradigms. let's start with {set/get}sockopt... that's one of the ugliest things in python, i believe most would agree. it's basically C programming in python. so, indeed, it's a way to overcome differences between platforms and protocols, but i believe it's not the way python should handle it. my suggestion is nothing "revolutionary". it's basically taking the existing socket module and extending it for most common use cases. there are two types of sockets, streaming and datagram. the underlying protocols don't matter. and these two types of sockets have different semantics to them: send/recv vs. sendto/recvfrom. so why not introduce a StreamSocket and DgramSocket types? and of course RawSocket should be introduced to completement them. you can argue that recvfrom and sendto can be used on streaming sockets as well, but did anyone ever use it? i never saw such code, and i can't think why you would want to use it. next, all the socket options would become properties or methods (i prefer properties). each protocol would subclass {Stream/Dgram/Raw}Socket and add its protocol-specific options. here's an example for a hierarchy: Socket RawSocket DgramSocket UDPSocket StreamSocket TCPSocket SSLSocket the above tree is only partial of course. but it needn't be complete, either. less used protocols, like X25 or ICMP could be constructed directly with the Socket class, in the old fashion of passing parameters. after all, the suggested class hierarchy only wraps the existing socket constructor and adds a more python API to its options. here's an example: s = TCPSocket(AF_INET6) s.reuse_address = True # this option is inherited from Socket s.no_delay = True # this is a TCP-level option s.bind(("", 12345)) s.listen(1) s2 = s.accept() s2.send("hello") or s = UDPSocket() s.allow_broadcast = True s.sendto("hello everybody", ("255.255.255.255", 12345)) perhaps we should consider adding an "options" namespace, in order to keep the root level of the instance simpler. for example: s.options.reuse_address = True it clarifies that reuse_address is an option. is it necessary? donno. and since we can override bind(), perhaps we should override it to provide a more specific interface, i.e. def bind(self, addr, port): super(self, ...).bind((addr, port)) because we *know* it's a tcp socket, so we don't need to *retain support* for all addressing forms: it's an IP address and a port. --- i would also want to replace the current BSD semantics for *client sockets*, of first creating a socket and then connecting it, i.e., s = socket() s.connect(("localhost", 80)) i would prefer s = ConnectedSocket(("localhost", 80)) because a *connecting the socket* is part of *initiallizing* it, hence it should be part of the class' constructor, and not a separate phase of the socket's life. perhaps the syntax should be s = TCPSocket.connect(("localhost", 80)) # or s = TCPSocket.connect("localhost", 80) # if we override connect() where .connect would be a classmethod, which returns a new instance of the class, connected to the server. of course DgramSockets don't need such a mechanism. i would like to suggest the same about connection-oriented server sockets, but the case with those is a little more complicated, and possibly asynchronous (select()ing before accept()ing), so i would retain the existing semantics. --- another thing i find quite silly is the way sockets behave on shutdown and in non-blocking mode. when the connection breaks, i would expect recv() to raise EOFError, or some sort of socket.error, instead of returning "". moreover, when i'm using a non-blocking recv(), and there's no data to return, i would expect "", not a socket.timeout exception. to sum it up: * no data = "" * connection breaks = EOFError the situation, however, is *exactly the opposite*. which is quite not intuitive or logical, and i remember having to write this code: def recv(s): try: data = s.recv(1000) if not data: # socket closed raise EOFError except socket.timeout: data = "" # timeout return data to accumulate data from non-blocking sockets, in a friendly way. so yeah, the libsocket version of recv returns 0 on EOF and -1 with some errno when there's no data, but the pythonic version shouldn't just *copy* this behavior -- it should *translate* it to pythonic standards. you have to remember that libsocket and the rest where written in the 80's, and are very platform-dependent. plus, C doesn't allow multiple return values or exceptions, so they had to do it this way. the question that should guide you is, "if you where to write pythonic sockets, how would they look?" rather than "how do i write a 1:1 wrapper for libsocket?" --- by the way, a little cleanup: * why does accept return a tuple? instead of newsock, sockname = sock.accept() why not do newsock = sock.accept() sockname = newsock.getsockname() i'm always having strange bugs because i forget accept gives me a tuple rather than just a socket... and you don't generally need the sockname, especially since you can get it later with getsockname. * the host-to-network functions, are they needed? can't you just use struct.pack and unpack? why not throw them away? what do you say? -tomer From rhettinger at ewtllc.com Sat Apr 29 00:13:54 2006 From: rhettinger at ewtllc.com (Raymond Hettinger) Date: Fri, 28 Apr 2006 15:13:54 -0700 Subject: [Python-3000] sets in P3K? In-Reply-To: <20060428213207.GA22469@nightcrawler> References: <20060428213207.GA22469@nightcrawler> Message-ID: <445293A2.7020301@ewtllc.com> Brian Harring wrote: >Potentially a stupid question, but I've been following this thread for >a while and I'm still not seeing the real gain of extending the syntax >for frozen markers/set literals. > >Someone care to recap what actually is gained from having a set >literal? >~harring > > Good question. To see the answer, look at a code tranformation from: if file_ext.lower() in set(['html', 'xml', 'xhtml']): handle(filename) into: if file_ext.lower() in {'html', 'xml', 'xhtml'}: handle(filename) The latter form builds a set directly and does not construct a temporary intermediate list. Also, the latter form is potentially optimizable so that the set is built at compile-time and referrenced as a constant rather than being rebuilt from scratch everytime the code is executed. In terms of user-friendliness, the latter form is closer to the way sets are expressed in mathematics. The only bump in the road is that the {} notation is so similar to dictionaries that care must be taken to keep the two as distinct as possible. Raymond From guido at python.org Sat Apr 29 00:21:11 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Apr 2006 15:21:11 -0700 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: <4452916F.1030304@v.loewis.de> References: <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> Message-ID: On 4/28/06, "Martin v. L?wis" wrote: > Guido van Rossum wrote: > >> I was hoping to propose a PEP on non-ASCII identifiers some > >> day; that would (of course) include a requirement that the > >> standard library would always be restricted to ASCII-only > >> identifiers as a style-guide. > > > > IMO communication about code becomes much more cumbersome if there are > > non-ASCII letters in identifiers, and the rules about what's a letter, > > what's a digit, and what separates two identifiers become murky. > > It depends on the language you use to communicate. In English, > it is certainly cumbersome to talk about Chinese identifiers. > OTOH, I believe it is cumbersome to communicate about English > identifiers in Chinese, either, because the speakers might > not even know what the natural-language concept behind the > identifiers is, and because they can't pronounce the identifier. True; but (and I realize we're just pitting our beliefs against each other here) I believe that Chinese computer users are more likely to be able (and know how) to type characters in the Latin alphabet than Western programmers are able to type Chinese. For example, I notice that baidu.cn (a Chinese search engine) spells its own name (a big brand in China) using the Latin alphabet. I expect that Chinese users are used to typing "baidu.cn" in their browser's search bar, rather than Chinese characters. > As for lexical aspects: these are really straight-forward. > In principal, it would be possible to allow any non-ASCII > character as part of an identifier: all punctuation is ASCII, > so anything non-ASCII can't possibly be punctuation for the > language. However, that much freedom would be confusing; > the Unicode consortium has established rules of what characters > should be allowed in identifiers, and these rules intend to > match the intuition of the users of these characters. Right; for example the "nice" quotes, dashes and ellipsis that Word likes to generate are not ASCII, but they feel like punctuation to me, and it would be very confusing if they were allowed in identifiers. > The distinction of letters and digits is also straight-forward: > a digit is ASCII [0-9]; it's a separate lexical class only > because it plays a special role in (number) literals. More > generally, there is the distinction of starter and non-starter > characters. But Unicode has many alternative sets digits for which "isdigit" is true. > An identifier ends when the first non-identifier character > is encountered (although I don't think there are many places > in Python where you can have two identifiers immediately following > each other). You can as far a the lexer is concerned because the lexer treats keywords as "just" identifiers. Only the parser knows which ones are really keywords. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Sat Apr 29 00:36:49 2006 From: guido at python.org (Guido van Rossum) Date: Fri, 28 Apr 2006 15:36:49 -0700 Subject: [Python-3000] suggestion for a new socket io stack In-Reply-To: <1d85506f0604281511i3829ce1dq5b2aa7e9e3a96044@mail.gmail.com> References: <1d85506f0604281511i3829ce1dq5b2aa7e9e3a96044@mail.gmail.com> Message-ID: Good ideas. Sockets follow the C API to the letter; this was a good idea when I first created the socket extension module, but nowadays it doesn't feel right. Also, the socket.py module contains a lot of code (a whole buffering file implementation!) that ought to be subsumed into the new I/O stack. The resolver APIs (gethostname() etc.) also need to be rethought. And then there's the select module, which has a very close relationship with sockets (e.g. on Windows, only sockets are acceptable to select) -- and the new BSD kqueue API which ought to be provided when available. (Preferably there should be a way to wait for multiple sockets without having to decide whether to use select, poll or kqueue.) And then there's the exception model. It's a pain that we have separate OSError, IOError and socket.error classes, and the latter doesn't even inherit from EnvironmentError. I propose that these all be unified (and the new class called IOError). Perhaps a few people could get together and design a truly minimal socket API along the lines suggested here? The goal ought to be to support the same set of use cases as supported by the current socket and select modules but using more appropriate APIs. Here's how it could interface to the new I/O stack: at the very bottom of the new I/O stack will be an object supporting unbuffered read and write operations. o.read(n) returns a bytes object of length at most n. o.write(b) takes a bytes object and writes some bytes, returning the number of bytes actually written. (There are also seek/tell/truncate ops but these aren't relevant here.) On top of this we can build buffering, encoding, and line ending translation (covering all the functionality we have in 2.x) but we could also build alternative functionality on top of buffered byte streams, for example record-based I/O. The bottom of the stack maps directly onto Unix read/write system calls but also onto recv/send socket system calls (and on Windows, the filedescriptor spaces are different so a filedescriptor-based common cass can't work there). I wouldn't bother mapping datagram sockets to the I/O stack; datagrams ough to be handled on a packet basis. --Guido On 4/28/06, tomer filiba wrote: > i've seen discussion about improving the IO stack of python, to rely less on > the low-level lib-c implementation. so, i wanted to share my ideas in > that niche. > > i feel today's sockets are way *outdated* and overloaded. python's sockets are > basically a wrapper for the low-level BSD sockets... but IMHO it would be much > nicer to alleviate this dependency: expose a more high-level interface to socket > programming. the *BSD-socket methodology* does not sit well with pythonic > paradigms. > > let's start with {set/get}sockopt... that's one of the ugliest things > in python, i > believe most would agree. it's basically C programming in python. so, indeed, > it's a way to overcome differences between platforms and protocols, but i > believe it's not the way python should handle it. > > my suggestion is nothing "revolutionary". it's basically taking the existing > socket module and extending it for most common use cases. > > there are two types of sockets, streaming and datagram. the underlying > protocols don't matter. and these two types of sockets have different semantics > to them: send/recv vs. sendto/recvfrom. so why not introduce a StreamSocket > and DgramSocket types? and of course RawSocket should be introduced > to completement them. > > you can argue that recvfrom and sendto can be used on streaming sockets > as well, but did anyone ever use it? i never saw such code, and i can't think > why you would want to use it. > > next, all the socket options would become properties or methods (i prefer > properties). each protocol would subclass {Stream/Dgram/Raw}Socket > and add its protocol-specific options. > > here's an example for a hierarchy: > Socket > RawSocket > DgramSocket > UDPSocket > StreamSocket > TCPSocket > SSLSocket > > the above tree is only partial of course. but it needn't be complete, > either. less > used protocols, like X25 or ICMP could be constructed directly with the Socket > class, in the old fashion of passing parameters. after all, the suggested class > hierarchy only wraps the existing socket constructor and adds a more python > API to its options. > > here's an example: > s = TCPSocket(AF_INET6) > s.reuse_address = True # this option is inherited from Socket > s.no_delay = True # this is a TCP-level option > s.bind(("", 12345)) > s.listen(1) > s2 = s.accept() > s2.send("hello") > > or > s = UDPSocket() > s.allow_broadcast = True > s.sendto("hello everybody", ("255.255.255.255", 12345)) > > perhaps we should consider adding an "options" namespace, in order to > keep the root level of the instance simpler. for example: > s.options.reuse_address = True > > it clarifies that reuse_address is an option. is it necessary? donno. > > and since we can override bind(), perhaps we should override it to provide > a more specific interface, i.e. > def bind(self, addr, port): > super(self, ...).bind((addr, port)) > > because we *know* it's a tcp socket, so we don't need to *retain support* for > all addressing forms: it's an IP address and a port. > > --- > > i would also want to replace the current BSD semantics for *client sockets*, > of first creating a socket and then connecting it, i.e., > s = socket() > s.connect(("localhost", 80)) > > i would prefer > s = ConnectedSocket(("localhost", 80)) > > because a *connecting the socket* is part of *initiallizing* it, hence > it should > be part of the class' constructor, and not a separate phase of the socket's > life. > > perhaps the syntax should be > s = TCPSocket.connect(("localhost", 80)) > # or s = TCPSocket.connect("localhost", 80) > # if we override connect() > > where .connect would be a classmethod, which returns a > new instance of the class, connected to the server. of course DgramSockets > don't need such a mechanism. > > i would like to suggest the same about connection-oriented server sockets, > but the case with those is a little more complicated, and possibly > asynchronous (select()ing before accept()ing), so i would retain the existing > semantics. > > --- > > another thing i find quite silly is the way sockets behave on shutdown and > in non-blocking mode. > > when the connection breaks, i would expect recv() to raise EOFError, or > some sort of socket.error, instead of returning "". moreover, when i'm using > a non-blocking recv(), and there's no data to return, i would expect "", not a > socket.timeout exception. > > to sum it up: > * no data = "" > * connection breaks = EOFError > > the situation, however, is *exactly the opposite*. which is quite not intuitive > or logical, and i remember having to write this code: > > def recv(s): > try: > data = s.recv(1000) > if not data: # socket closed > raise EOFError > except socket.timeout: > data = "" # timeout > return data > > to accumulate data from non-blocking sockets, in a friendly way. > > so yeah, the libsocket version of recv returns 0 on EOF and -1 with some > errno when there's no data, but the pythonic version shouldn't just *copy* > this behavior -- it should *translate* it to pythonic standards. > > you have to remember that libsocket and the rest where written in the 80's, > and are very platform-dependent. plus, C doesn't allow multiple return values > or exceptions, so they had to do it this way. > > the question that should guide you is, "if you where to write pythonic sockets, > how would they look?" rather than "how do i write a 1:1 wrapper for libsocket?" > > --- > > by the way, a little cleanup: > > * why does accept return a tuple? instead of > newsock, sockname = sock.accept() > > why not do > newsock = sock.accept() > sockname = newsock.getsockname() > > i'm always having strange bugs because i forget accept gives me a tuple rather > than just a socket... and you don't generally need the sockname, especially > since you can get it later with getsockname. > > * the host-to-network functions, are they needed? can't you just use struct.pack > and unpack? why not throw them away? > > what do you say? > > > -tomer > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/guido%40python.org > -- --Guido van Rossum (home page: http://www.python.org/~guido/) From solipsis at pitrou.net Sat Apr 29 01:16:21 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Apr 2006 01:16:21 +0200 Subject: [Python-3000] Add a standard GUI system In-Reply-To: References: <20060428000801.671E.JCARLSON@uci.edu> <06Apr28.101121pdt."58641"@synergy1.parc.xerox.com> Message-ID: <1146266181.5808.6.camel@fsol> Le vendredi 28 avril 2006 ? 15:12 -0400, Terry Reedy a ?crit : > The generic solution is better package fetching, such as apparently done (I > have not used them yet) by setuptools and easy_install. The installing an > app needing wxPython should fetch and install that also and installing an > app needing PyGUI would do the same for PyGUI. I don't see what the problem is with installing wxPython separately from Python. There are installers for Windows, and any decent Linux distro packages it (so you can just type "urpmi wxPython" or whatever). Also, py2exe and py2app handle it gracefully if you want to ship a standalone installer for your application. The remaining question is why Tk is shipped by default with Python. It is a very outdated UI and feels totally awful to a newcomer. So it may give the bad impression that "this is what Python GUIs look like" if it comes by default with Python. Better not ship a "standard GUI system" and let everybody choose, IMHO. Regards Antoine. From bwinton at latte.ca Sat Apr 29 01:29:59 2006 From: bwinton at latte.ca (Blake Winton) Date: Fri, 28 Apr 2006 19:29:59 -0400 Subject: [Python-3000] suggestion for a new socket io stack In-Reply-To: <1d85506f0604281511i3829ce1dq5b2aa7e9e3a96044@mail.gmail.com> References: <1d85506f0604281511i3829ce1dq5b2aa7e9e3a96044@mail.gmail.com> Message-ID: <4452A577.4020600@latte.ca> tomer filiba wrote: > Each protocol would subclass {Stream/Dgram/Raw}Socket > and add its protocol-specific options. > > here's an example for a hierarchy: > Socket > RawSocket > DgramSocket > UDPSocket > StreamSocket > TCPSocket > SSLSocket My one small complaint with that is that SSLSocket should really be an SSL wrapper around a TCPSocket, because I wouldn't want to have two classes for HTTP/HTTPS, or FTP/FTPS/SFTP. Java does something reasonable in this regard, I think, where you can have subclasses of Filter(Input,Output)Stream which can build on one-another, so I can have a base-64 encoded, TEA-encrypted, SSL, TCP socket by writing a line similar to: sock = TcpSocket( "129.97.134.11", 8080 ) myIn = Base64Stream( TeaStream( SslStream( sock.getInputStream() ) ) ) myOut = Base64Stream( TeaStream( SslStream( sock.getOutputStream() ) ) ) The real beauty of that is that I can hook up my Base64Stream to a StringIO-like object to base64-encode stuff in memory, for testing, or for actual use. Or I could Tea-encrypt a file while writing it to disk. On the downside, it's a larger change than the one you're proposing. On the upside, it unifies stream sockets, strings, files, and any other source of stream data you can come up with (random generators? keyboards?). Further reading of your examples suggests to me that what I would really like is a stream-view of the Socket class, and let SSL (, etc...) be a wrapper around the Socket's input/output streams. Other than that small suggestion, I quite like your proposal. Later, Blake. From rasky at develer.com Sat Apr 29 01:40:25 2006 From: rasky at develer.com (Giovanni Bajo) Date: Sat, 29 Apr 2006 01:40:25 +0200 Subject: [Python-3000] Add a standard GUI system References: <20060428000801.671E.JCARLSON@uci.edu><06Apr28.101121pdt."58641"@synergy1.parc.xerox.com> <1146266181.5808.6.camel@fsol> Message-ID: <01c301c66b1d$1f9d6470$8d472597@bagio> Antoine Pitrou wrote: > Better not ship a "standard GUI system" and let everybody choose, > IMHO. In this case, I agree. There is simply no good enough solution for a GUI. There are far too many advanced different libraries with different wrappers. Some are portable, some are not. Some are open-source, some are not. Some are GPL but not BSD, some are not. Some use native controls, some do not. There are legitimate usages for all these cases. It really "feels" like a standard GUI library doesn't help in any way. Also, GUI libraries are too huge to bundle 3 or 4 of them (like we have different XML libraries for different *legitimate* *real-world* usage cases). > The generic solution is better package fetching, such as apparently done (I > have not used them yet) by setuptools and easy_install. Disagree. The mere existence of the stdlib improve consistency among Python programs. Try finding two Perl programs using the same email library so that you don't have to read the man page to know what it's doing. Having a cpan-like system and *not* a standard library is a good way for madness to happen. In fact, the presence of the standard library is even *more* important if Python grows a powerful packaging system. Giovanni Bajo From msoulier at digitaltorque.ca Sat Apr 29 03:29:28 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Fri, 28 Apr 2006 21:29:28 -0400 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <1146266181.5808.6.camel@fsol> References: <20060428000801.671E.JCARLSON@uci.edu> <06Apr28.101121pdt."58641"@synergy1.parc.xerox.com> <1146266181.5808.6.camel@fsol> Message-ID: <4452C178.1090006@digitaltorque.ca> Antoine Pitrou wrote: > I don't see what the problem is with installing wxPython separately from > Python. There are installers for Windows, and any decent Linux distro > packages it (so you can just type "urpmi wxPython" or whatever). Also, > py2exe and py2app handle it gracefully if you want to ship a standalone > installer for your application. As someone who is trying to do cross-platform development with Python, wxPython, and other components like sqlite and pysqlite, I would like to see the versions that I have to synchronize reduced. Linux distros ship a given version of Python, often several revs older because it changes too fast for them to keep up. If they package wxPython at all, it's an older version. Then on Windows, I have to install the same older versions, synchronize on a version of wxGlade so that I have a decent RAD tool, grab an editor from elsewhere... Tough sell to my coworkers who just install the JDK+Netbeans and have it all included, on any platform that I work on. > The remaining question is why Tk is shipped by default with Python. It > is a very outdated UI and feels totally awful to a newcomer. So it may > give the bad impression that "this is what Python GUIs look like" if it > comes by default with Python. > > Better not ship a "standard GUI system" and let everybody choose, IMHO. The API is great, but the widgets need updating. I guess no one has bothered to do it yet. Pmw isn't included either, is it? High-level widgets are certainly needed, Tk doesn't some with enough of them. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-3000/attachments/20060428/dc4851be/attachment.pgp From nnorwitz at gmail.com Sat Apr 29 04:07:36 2006 From: nnorwitz at gmail.com (Neal Norwitz) Date: Fri, 28 Apr 2006 19:07:36 -0700 Subject: [Python-3000] suggestion for a new socket io stack In-Reply-To: <4452A577.4020600@latte.ca> References: <1d85506f0604281511i3829ce1dq5b2aa7e9e3a96044@mail.gmail.com> <4452A577.4020600@latte.ca> Message-ID: If you move forward with this, please review (learn from and steal the best parts of) the ACE APIs. http://cs.wustl.edu/~schmidt/ACE.html They've done a lot of work in this area and have a lot of experience. It would also be good to talk to the twisted guys. We should also support epoll beyond select/poll/kqueue. Also, the scheme should be extensible to other types of sockets that aren't as commonly used. Say something like X.25. :-) n -- On 4/28/06, Blake Winton wrote: > tomer filiba wrote: > > Each protocol would subclass {Stream/Dgram/Raw}Socket > > and add its protocol-specific options. > > > > here's an example for a hierarchy: > > Socket > > RawSocket > > DgramSocket > > UDPSocket > > StreamSocket > > TCPSocket > > SSLSocket > > My one small complaint with that is that SSLSocket should really be an > SSL wrapper around a TCPSocket, because I wouldn't want to have two > classes for HTTP/HTTPS, or FTP/FTPS/SFTP. Java does something > reasonable in this regard, I think, where you can have subclasses of > Filter(Input,Output)Stream which can build on one-another, so I can have > a base-64 encoded, TEA-encrypted, SSL, TCP socket by writing a line > similar to: > sock = TcpSocket( "129.97.134.11", 8080 ) > myIn = Base64Stream( TeaStream( SslStream( sock.getInputStream() ) ) ) > myOut = Base64Stream( TeaStream( SslStream( sock.getOutputStream() ) ) ) > > The real beauty of that is that I can hook up my Base64Stream to a > StringIO-like object to base64-encode stuff in memory, for testing, or > for actual use. Or I could Tea-encrypt a file while writing it to disk. > > On the downside, it's a larger change than the one you're proposing. On > the upside, it unifies stream sockets, strings, files, and any other > source of stream data you can come up with (random generators? keyboards?). > > Further reading of your examples suggests to me that what I would really > like is a stream-view of the Socket class, and let SSL (, etc...) be a > wrapper around the Socket's input/output streams. > > Other than that small suggestion, I quite like your proposal. > > Later, > Blake. > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/nnorwitz%40gmail.com > From martin at v.loewis.de Sat Apr 29 05:07:29 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sat, 29 Apr 2006 05:07:29 +0200 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: References: <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> Message-ID: <4452D871.9020901@v.loewis.de> Guido van Rossum wrote: >> The distinction of letters and digits is also straight-forward: >> a digit is ASCII [0-9]; it's a separate lexical class only >> because it plays a special role in (number) literals. More >> generally, there is the distinction of starter and non-starter >> characters. > > But Unicode has many alternative sets digits for which "isdigit" is true. You mean, the Python isdigit() method? Sure, but the tokenizer uses the C isdigit function, which gives true only for [0-9]. FWIW, POSIX allows 6 alternative characters to be defined as hexdigits for isxdigit, so the tokenizer shouldn't really use isxdigit for hexadecimal literals. So from the implementation point of view, nothing much would have to change: the usage of isalnum in the tokenizer is already wrong, as it already allows to put non-ASCII characters into identifiers, if the locale classifies them as alpha-numeric. I can't see why the Unicode notion of digits should affect the language specification in any way. The notion of digit is only used to define what number literals are, and I don't propose to change the lexical rules for number literals - I propose to change the rules for identifiers. > You can as far a the lexer is concerned because the lexer treats > keywords as "just" identifiers. Only the parser knows which ones are > really keywords. Right. But if the identifier syntax was [:identifier_start:][:identifier_cont:]* then thinks would work out just fine: identifier_start intersected with ASCII would be [A-Za-z_], and identifier_cont intersected with ASCII would be [A-Za-z0-9_]; this would include all keywords. You would still need punctuation between two subsequent "identifiers", and that punctuation would have to be ASCII, as non-ASCII characters would be restricted to comments, string literals, and identifiers. Regards, Martin From ncoghlan at gmail.com Sat Apr 29 05:37:29 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Apr 2006 13:37:29 +1000 Subject: [Python-3000] sets in P3K? In-Reply-To: <445293A2.7020301@ewtllc.com> References: <20060428213207.GA22469@nightcrawler> <445293A2.7020301@ewtllc.com> Message-ID: <4452DF79.8050506@gmail.com> Raymond Hettinger wrote: > The only bump in the road is that the {} notation is so similar to > dictionaries that care must be taken to keep the two as distinct as > possible. Here's a summary of the suggestions we've seen so far for empty sets & dicts Possibilities for an empty set: set() - explicit, but kind of 'second-class' compared to (), [] and {} {} - most obvious option, but currently means dict() {,} - Guido's already said he thinks this is too ugly {/} - suggestive of phi, but considered too unintuitive/ugly by some Possibilities for an empty dict: dict() - explicit, but kind of 'second-class' compared to (), [] and {} {} - status quo, but ambiguous given the addition of set literals {:} - matches non-empty dicts by including ':' to indicate mapping However, I'm now -0 on having a set literal at all, and it's because I believe we can solve this problem in a more general fashion that applies to more functions than just the set() constructor. Currently, [] and () can appear both standalone or following an expression (list literals and comprehensions, and subscripting for [], tuple literals and precedence manipulation, and function calls for ()) {} however, can appear only standalone. It is not permitted for it to follow an expression. If we change that, we can use it to support an alternate invocation syntax for functions that currently expect an iterable as their first argument. "EXPR{ARGS}" would be equivalent to "EXPR((ARGS,))", with the following differences in the parsing of ARGS: x:y would be permitted, and map to a 2-tuple (x, y) x:y:z would be permitted, and map to a 3-tuple (x, y, z) x=y would be permitted, and map to a 2-tuple ('x', y) *x would be permitted, and would extend the passed in tuple with x This could all be done in the front-end parser, with no back-end AST or compiler changes necessary (although we may decide they're desirable for optimisation purposes). Dictionary displays would then be a strict subset of dict{EXPR} and tuple literals would be equivalent to tuple{EXPR} (although the latter wouldn't need the trailing comma for singleton tuples). List displays would generally be equivalent to list{EXPR}, except for list comprehensions, which map to list(EXPR) (as omitting the parentheses around the genexp would otherwise be illegal). A constructor for named tuples, a clean syntax for which is the biggest hurdle named tuples face, could accept an iterable of name-value pairs, with explicit construction looking like: result = named_tuple{x=x, y=y} return result Unlike keyword arguments in a normal call, this would preserve the argument ordering, so we know that result.x is result[0] and result.y is result[1] Explicitly constructed sets would look like: set() == set{} set(x) # No braced equivalent set([x]) == set{x} # Ignoring list comprehensions set((x,)) == set{x} set((a, b, c)) == set{a, b, c} And it would work for frozenset, too: frozenset() == frozenset{} frozenset(x) # No braced equivalent frozenset([x]) == froxenset{x} # Ignoring list comprehensions frozenset((x,)) == frozenset{x} frozenset((a, b, c)) == frozenset{a, b, c} itertools.cycle is another possible beneficiary, as it becomes easy to cycle through a simple sequence: cycle{1, 2, 3} # Infinite cycle through the values 1, 2 and 3 The any and all builtins also become more useful for chained 'or' and 'and' tests: if all{x, y, z}: return if any{x, y, z}: return Joining a known set of strings, or extend a mutable sequence also benefits: ", ".join{x, y, z} seq.extend{x, y, z} The sundry set methods that accept iterables also benefit: s.intersection{1, 2, 3} s.issuperset{1, 2, 3} s.update{1, 2, 3} The question of "accept an iterable or accept *args" goes away - the obvious way to do it is to accept an iterable, and then use a braced call to conveniently invoke the call with a literal sequence instead of a pre-existing iterable. The signatures of dict and dict.update could optionally be simplified back to accepting a single mapping or iterable, with the explicit keyword construction support being moved to the separate {} form. The 'iterable of 2-tuples or mapping' questions can be addressed by making the keys(), values() and items() methods officially part of the definition of a mapping. So dict.update might simplify to: try: items_view = obj.items except AttributeError: items = obj else: items = items_view() for key, value in items: self[key] = value Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From steven.bethard at gmail.com Sat Apr 29 06:10:43 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Fri, 28 Apr 2006 22:10:43 -0600 Subject: [Python-3000] sets in P3K? In-Reply-To: <4452DF79.8050506@gmail.com> References: <20060428213207.GA22469@nightcrawler> <445293A2.7020301@ewtllc.com> <4452DF79.8050506@gmail.com> Message-ID: On 4/28/06, Nick Coghlan wrote: > {} however, can appear only standalone. It is not permitted for it to follow > an expression. If we change that, we can use it to support an alternate > invocation syntax for functions that currently expect an iterable as their > first argument. "EXPR{ARGS}" would be equivalent to "EXPR((ARGS,))", with the > following differences in the parsing of ARGS: > x:y would be permitted, and map to a 2-tuple (x, y) > x:y:z would be permitted, and map to a 3-tuple (x, y, z) > x=y would be permitted, and map to a 2-tuple ('x', y) > *x would be permitted, and would extend the passed in tuple with x [snip] > Explicitly constructed sets would look like: > set() == set{} > set(x) # No braced equivalent > set([x]) == set{x} # Ignoring list comprehensions > set((x,)) == set{x} > set((a, b, c)) == set{a, b, c} > [snip] > Joining a known set of strings, or extend a mutable sequence also benefits: > > ", ".join{x, y, z} > seq.extend{x, y, z} [snip] > The question of "accept an iterable or accept *args" goes away - the obvious > way to do it is to accept an iterable, and then use a braced call to > conveniently invoke the call with a literal sequence instead of a pre-existing > iterable. +1 This is really nice. I know I've written ", ".join([x, y, z]) a number of times and wished there was a nicer way to write it. I've also debated countless times with function APIs whether to give them a *args or iterable argument. For the sake of flexibility, I know I should use a single iterable, but when I know I'm going to use the constructor primarily with list literals, it's hard to resist the convenience of the *args form. I really like that this proposal gives me one obvious way of doing things. =) STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From jcarlson at uci.edu Sat Apr 29 07:49:32 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 28 Apr 2006 22:49:32 -0700 Subject: [Python-3000] suggestion for a new socket io stack In-Reply-To: <1d85506f0604281511i3829ce1dq5b2aa7e9e3a96044@mail.gmail.com> References: <1d85506f0604281511i3829ce1dq5b2aa7e9e3a96044@mail.gmail.com> Message-ID: <20060428220603.6724.JCARLSON@uci.edu> Maybe it's just me, but I've previously implemented much of this with asyncore subclasses. While I didn't go all-out on the properties (I never need to change the underlying options beyond setting it as non-blocking), I did end up with a fairly standard __init__ signature that solved many of my annoyances. class ...: def __init__(self, host, port, sock=None, factory=None, family=AF_INET, type=SOCK_STREAM, proto=0): Listening sockets: have host, port, and factory, but no sock New incoming sockets: have host, port, and sock, but no factory Outgoing sockets: have host and port, but no sock or factory It seems as though much of what Tomer suggests can be implemented with subclassing of sockets. When I first read his email, I started working on a sample implementation of TCPSocket, but stopped when I started using various metaprogramming tricks to ease the mapping of properties to methods (I love using the Property metaclass). While I think that sockets can be made easier to use in their creation and setup (whether by subclass or by a factory function), both recv/send could use a bit of a change, and the exception nonsense that Guido mentioned should certainly be cleaned up; I'm not sure how much the other changes will really ease socket programming, in general. Why? Once I wrote my own send/recv handling in my socket (or asyncore.dispatcher) subclass, I got sane-enough-for-me behavior, and had no issues implementing various line or packet based protocols. I certainly hope that people have been writing sane wrappers/subclasses, but if not, then I guess that such changes would significantly ease socket programming with Python (though even with the suggested changes, still feels like BSD sockets to me). - Josiah "tomer filiba" wrote: > i've seen discussion about improving the IO stack of python, to rely less on > the low-level lib-c implementation. so, i wanted to share my ideas in > that niche. > > i feel today's sockets are way *outdated* and overloaded. python's sockets are > basically a wrapper for the low-level BSD sockets... but IMHO it would be much > nicer to alleviate this dependency: expose a more high-level interface to socket > programming. the *BSD-socket methodology* does not sit well with pythonic > paradigms. > > let's start with {set/get}sockopt... that's one of the ugliest things > in python, i > believe most would agree. it's basically C programming in python. so, indeed, > it's a way to overcome differences between platforms and protocols, but i > believe it's not the way python should handle it. > > my suggestion is nothing "revolutionary". it's basically taking the existing > socket module and extending it for most common use cases. > > there are two types of sockets, streaming and datagram. the underlying > protocols don't matter. and these two types of sockets have different semantics > to them: send/recv vs. sendto/recvfrom. so why not introduce a StreamSocket > and DgramSocket types? and of course RawSocket should be introduced > to completement them. > > you can argue that recvfrom and sendto can be used on streaming sockets > as well, but did anyone ever use it? i never saw such code, and i can't think > why you would want to use it. > > next, all the socket options would become properties or methods (i prefer > properties). each protocol would subclass {Stream/Dgram/Raw}Socket > and add its protocol-specific options. > > here's an example for a hierarchy: > Socket > RawSocket > DgramSocket > UDPSocket > StreamSocket > TCPSocket > SSLSocket > > the above tree is only partial of course. but it needn't be complete, > either. less > used protocols, like X25 or ICMP could be constructed directly with the Socket > class, in the old fashion of passing parameters. after all, the suggested class > hierarchy only wraps the existing socket constructor and adds a more python > API to its options. > > here's an example: > s = TCPSocket(AF_INET6) > s.reuse_address = True # this option is inherited from Socket > s.no_delay = True # this is a TCP-level option > s.bind(("", 12345)) > s.listen(1) > s2 = s.accept() > s2.send("hello") > > or > s = UDPSocket() > s.allow_broadcast = True > s.sendto("hello everybody", ("255.255.255.255", 12345)) > > perhaps we should consider adding an "options" namespace, in order to > keep the root level of the instance simpler. for example: > s.options.reuse_address = True > > it clarifies that reuse_address is an option. is it necessary? donno. > > and since we can override bind(), perhaps we should override it to provide > a more specific interface, i.e. > def bind(self, addr, port): > super(self, ...).bind((addr, port)) > > because we *know* it's a tcp socket, so we don't need to *retain support* for > all addressing forms: it's an IP address and a port. > > --- > > i would also want to replace the current BSD semantics for *client sockets*, > of first creating a socket and then connecting it, i.e., > s = socket() > s.connect(("localhost", 80)) > > i would prefer > s = ConnectedSocket(("localhost", 80)) > > because a *connecting the socket* is part of *initiallizing* it, hence > it should > be part of the class' constructor, and not a separate phase of the socket's > life. > > perhaps the syntax should be > s = TCPSocket.connect(("localhost", 80)) > # or s = TCPSocket.connect("localhost", 80) > # if we override connect() > > where .connect would be a classmethod, which returns a > new instance of the class, connected to the server. of course DgramSockets > don't need such a mechanism. > > i would like to suggest the same about connection-oriented server sockets, > but the case with those is a little more complicated, and possibly > asynchronous (select()ing before accept()ing), so i would retain the existing > semantics. > > --- > > another thing i find quite silly is the way sockets behave on shutdown and > in non-blocking mode. > > when the connection breaks, i would expect recv() to raise EOFError, or > some sort of socket.error, instead of returning "". moreover, when i'm using > a non-blocking recv(), and there's no data to return, i would expect "", not a > socket.timeout exception. > > to sum it up: > * no data = "" > * connection breaks = EOFError > > the situation, however, is *exactly the opposite*. which is quite not intuitive > or logical, and i remember having to write this code: > > def recv(s): > try: > data = s.recv(1000) > if not data: # socket closed > raise EOFError > except socket.timeout: > data = "" # timeout > return data > > to accumulate data from non-blocking sockets, in a friendly way. > > so yeah, the libsocket version of recv returns 0 on EOF and -1 with some > errno when there's no data, but the pythonic version shouldn't just *copy* > this behavior -- it should *translate* it to pythonic standards. > > you have to remember that libsocket and the rest where written in the 80's, > and are very platform-dependent. plus, C doesn't allow multiple return values > or exceptions, so they had to do it this way. > > the question that should guide you is, "if you where to write pythonic sockets, > how would they look?" rather than "how do i write a 1:1 wrapper for libsocket?" > > --- > > by the way, a little cleanup: > > * why does accept return a tuple? instead of > newsock, sockname = sock.accept() > > why not do > newsock = sock.accept() > sockname = newsock.getsockname() > > i'm always having strange bugs because i forget accept gives me a tuple rather > than just a socket... and you don't generally need the sockname, especially > since you can get it later with getsockname. > > * the host-to-network functions, are they needed? can't you just use struct.pack > and unpack? why not throw them away? > > what do you say? > > > -tomer > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/jcarlson%40uci.edu From tim.peters at gmail.com Sat Apr 29 08:07:30 2006 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 29 Apr 2006 02:07:30 -0400 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> <4450FE5B.4010208@ieee.org> <1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com> <1f7befae0604271154w6ba78d3p3b6851e85d4dc68d@mail.gmail.com> Message-ID: <1f7befae0604282307x10335f80x55d75509ff55eedb@mail.gmail.com> [Fredrik] >>> I don't think anyone on this list can take the "but if there's more than >>> one argument, *I* am going to be confused" argument seriously. [Tim] >> Then you don't remember that the order of arguments here _was_ a >> frequent confusion in the old days. [Fredrik] > nope. > > any pointers ? Let's see. Should I watch a movie now, or search pre-string-method archives for quotes nobody really cares about? While I think about that ;-), you could look in _this_ thread for: [Aahz] > I never have gotten mixed up about the order of arguments since switching > to ''.join(l). [Tim H, replying to the Aahz quote] > Me too [Barry W] > But hey, yeah, a join() builtin would be fine if it took the string arg first [Alex M] > I think I would prefer a signature of: join(seq, joiner='') [Barry W, replying to the Alex M quote] > Except that I think it would be a mistake to have that and keep > the .join() method on strings because that would increase the confusion > in the argument order. OK, the movie sucked. Two of my favorites, which I remembered but took damn near forever to find: http://mail.python.org/pipermail/python-list/2001-March/031744.html ... from string import join l = join(';', ['a','b','c']) # thanks, this is clear and obvious Please o' gods of python, do not deprecate the clean and obvious syntax that is found in the string module. Of course the joke is that he got the "clear and obvious" order backwards. My other favorite showed up in a proposed PEP: http://mail.python.org/pipermail/python-list/2001-June/048258.html ... It is *not* suggested that file objects take on this aspect of print behavior, such as by implementing a method print(self, items, print_newline=1, softsep=' ', linesep='\n'), and if not implemented, falling back to stream.write(string.join(' ', map(str, items)) + "\n"). And the arguments are backwards there too. If you don't remember these confusions, I think it should suffice to remind that Perl's join() does take the separator first (which is essentially forced in Perl, given its odd LIST syntax): http://perldoc.perl.org/functions/join.html While I don't know for sure, I always assumed that was the true source of most of the relevant "how come my code doesn't work?!" complaints on comp.lang.python before string methods were introduced. This wasn't as frequent as posts showing confusions over the order of arguments to (say) re.search, because _trying_ to do string.join(a_string, a_list) raised an exception immediately, although it wasn't a particularly clear exception. If you try the old string.join() code under a current Python, the exception message is better than it was then, but would still be baffling to many newbies: TypeError: can only concatenate list (not "str") to list From jcarlson at uci.edu Sat Apr 29 08:14:24 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri, 28 Apr 2006 23:14:24 -0700 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <06Apr28.101121pdt."58641"@synergy1.parc.xerox.com> References: <20060428000801.671E.JCARLSON@uci.edu> <06Apr28.101121pdt."58641"@synergy1.parc.xerox.com> Message-ID: <20060428225107.6727.JCARLSON@uci.edu> Bill Janssen wrote: > > > Python already comes with a standard GUI toolkit; tk. > > With a decidedly unpythonic bad interface. And it's not standard; I > frequently find Python installations where no one has installed Tk. > Pardon me for being blunt: Tkinter is a half-assed "solution" that's > set Python back for years. No argument here; I tried to use Tk, spent a weekend with it, got frustrated, tried wxPython, and was going in about 30 minutes. I've only ever really dealt with Windows GUI programming, so it was certainly my mistake to assume that Tk was always available with Python. > > Also, while I > > generally like Greg Ewing's work (Pyrex has saved me so much brain power > > interfacing with C extensions), PyGUI doesn't have nearly the market > > penetration (into the Python GUI toolkit area) to justify its inclusion > > into the python standard library. > > Doesn't matter. Putting it into the stdlib will solve the penetration > problem. Yes and no. It will only really solve the penetration problem it if it actually superior to the other options available (in widget availability, ease of use, etc.). Note how Tk in Python isn't used nearly as often as wxPython is? > > Also, it would require shipping a Python GTK binding, which would > > basically necessitate shipping GTK+ with Python. > > No, it wouldn't. It runs natively on Mac OS X, and could be made to > do on Windows. I'd suggest using the existing GTK+ binding for Linux, > and only in the interim for Windows. I certainly wouldn't ship GTK+ > with Python, or wxWindows either. As I suggested, I think that it > would be a good idea to do a PyGUI port to a lower-level system, such > as cairo, for more generic use. I wouldn't object to shipping cairo > to those few systems where it won't be pre-installed. Ultimately, PyGUI is essentially a wrapper of native or near-native widgets into a common interface. wxPython is exactly the same thing, only it has a much larger user base, developer support, and (native and emulated) platform support. Similar things can be said about the GTK and Qt bindings for Python (at least in terms of user base, developer and platform support), but I don't have experience with PyGTK or PyQt, so cannot comment on their quality/support/etc. versus wxPython. I should also mention that while it would take Greg and/or others a time to go and wrap the Windows native widgets, it has already been done with wxPython. > > it's market penetration (into the Python GUI toolkit area) > > is growing steadily > > Pointless consideration. Tkinter has a great penetration, but it's a > bad solution. It is not pointless. Even though it is more difficult to install wxPython than Tk (which should have build rules in the Python source, and should be distributed with the Python binaries, if I understand correctly), it is still used far more than the Python Tk bindings. In this case, wxPython's market penetration at least suggests higher-quality. - Josiah From kay.schluehr at gmx.net Sat Apr 29 09:28:44 2006 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Sat, 29 Apr 2006 09:28:44 +0200 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> References: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> Message-ID: <445315AC.9010701@gmx.net> Bill Janssen schrieb: >I mentioned this a couple of weeks ago, and here it is again. > >The biggest wart in Python 2, the one that forces me to use another >language, is the lack of a standard GUI. By standard, I mean that >when you install Python on a system, the GUI toolkit is also >installed. > >I suggest that for Python 3K, we adopt Greg Ewing's PyGUI into the >standard library. It's been around for years, works on every major >platform (using GTK+ for Windows and X11, and Cocoa for Mac OS X), is >Pythonic in flavor, and is aimed at eventual inclusion into the >standard library. > > No one uses GTK+ on Windows. Personally I'd suggest to delay the question of a standard GUI toolkit for Py3K and observe what's going on in the XUL, XPCOM, Firefox... domain. I think that's where the more exciting things are happening right now and I would like to see a statement of the Python community embracing this kind of development instead of quarreling about legacy GUI toolkits. From janssen at parc.com Sat Apr 29 09:40:25 2006 From: janssen at parc.com (Bill Janssen) Date: Sat, 29 Apr 2006 00:40:25 PDT Subject: [Python-3000] Add a standard GUI system In-Reply-To: Your message of "Fri, 28 Apr 2006 12:12:59 PDT." Message-ID: <06Apr29.004032pdt."58641"@synergy1.parc.xerox.com> Terry Reedy writes: > To me, your logic is backwards. wxWidgets/wxPython are popular and have > perhaps the highest usage (not counting IDLE) by voluntary choice. Terry, "popular" isn't the goal of Python 3000. Nor is "backwards compatibility" -- Py3K is about breaking compatibility. "Getting it right" is the goal. wxPython, PyGTK, PyQt, Tkinter, Jython/Swing -- all are fatally non-Pythonic due to an overly literal mapping of their C++/C++/C++/Tcl/Java APIs into the Python API. (Single inheritance has a lot to answer for!) If I'm going to deal with a Py3K, I want the right kind of GUI, and PyGUI seems to have the best combination of features for that. I don't want to program in Java-masquerading-as-Python, C++-masquerading-as-Python, or Tcl-masquerading-as-Python. Leave that to the poor Python 2.x folks. But I think there's also an argument to be made for adopting a relatively little-used toolkit, as long as it's technically adequate. Adopting any toolkit into the standard library will be a big blow to partisans of, and users of, all the other toolkits. Seems only fair to award "advantage" to as few as possible. > So you want to artificially boost the > penetration of your minority favorite by putting it in the standard lib. I don't have a "favorite" here. I don't use any of the common Python toolkits, including PyGUI, because none of them are standard. I write GUI code in Swing (Java) and OpenGL (C++). My analysis is purely theoretical. I'd be happy to support a better solution, if I knew of one. Let's discuss it. Convince me (and everyone else). I also considered Fredrik Lundh's WCK. My assessment is that PyGUI has the better combination of technologies and APIs for "doing it right". Bill ------------------------------------------ Bill Janssen (650) 812-4763 FAX: (650) 812-4258 PARC User Interface Research Area http://www.parc.com/janssen/ From janssen at parc.com Sat Apr 29 09:46:56 2006 From: janssen at parc.com (Bill Janssen) Date: Sat, 29 Apr 2006 00:46:56 PDT Subject: [Python-3000] Add a standard GUI system In-Reply-To: Your message of "Fri, 28 Apr 2006 18:29:28 PDT." <4452C178.1090006@digitaltorque.ca> Message-ID: <06Apr29.004657pdt."58641"@synergy1.parc.xerox.com> > Tough sell to my coworkers who just install the > JDK+Netbeans and have it all included, on any platform that I work on. Yep, that's the problem in a nutshell. Not standardizing a GUI system was an acceptable answer before Java, but it won't wash in the 21st century. Of course, wxPython, PyGTK, etc., will all still be available for those who still want to use them. I'm just suggesting that Greg's system be made official and standard. Think of it as the "reference" GUI. And, I guess, I'm proposing that Py3K drop Tkinter. By the way, let me repeat the URL: http://www.cosc.canterbury.ac.nz/~greg/python_gui/ Download the code and read the test cases. Bill From janssen at parc.com Sat Apr 29 09:53:45 2006 From: janssen at parc.com (Bill Janssen) Date: Sat, 29 Apr 2006 00:53:45 PDT Subject: [Python-3000] Add a standard GUI system In-Reply-To: Your message of "Fri, 28 Apr 2006 23:14:24 PDT." <20060428225107.6727.JCARLSON@uci.edu> Message-ID: <06Apr29.005352pdt."58641"@synergy1.parc.xerox.com> > Ultimately, PyGUI is essentially a wrapper of native or near-native > widgets into a common interface. Have you read the PyGUI code? It's really more of an abstract API that exploits certain bundles of functionality provide by already-implemented toolkits for its implementation. It could certainly be implemented on lower-level platforms, like the cairo system I suggested in my original post. Of course, these days everything's ultimately a wrapper of bitblit :-). Bill ------------------------------------------ Bill Janssen (650) 812-4763 FAX: (650) 812-4258 PARC -- User Interface Research Area http://www.parc.com/janssen/ From p.f.moore at gmail.com Sat Apr 29 12:16:57 2006 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 29 Apr 2006 11:16:57 +0100 Subject: [Python-3000] sets in P3K? In-Reply-To: <4452DF79.8050506@gmail.com> References: <20060428213207.GA22469@nightcrawler> <445293A2.7020301@ewtllc.com> <4452DF79.8050506@gmail.com> Message-ID: <79990c6b0604290316l603a89cxcf736e7e0ef85937@mail.gmail.com> On 4/29/06, Nick Coghlan wrote: > However, I'm now -0 on having a set literal at all, and it's because I believe > we can solve this problem in a more general fashion that applies to more > functions than just the set() constructor. > > Currently, [] and () can appear both standalone or following an expression > (list literals and comprehensions, and subscripting for [], tuple literals and > precedence manipulation, and function calls for ()) > > {} however, can appear only standalone. It is not permitted for it to follow > an expression. If we change that, we can use it to support an alternate > invocation syntax for functions that currently expect an iterable as their > first argument. "EXPR{ARGS}" would be equivalent to "EXPR((ARGS,))", with the > following differences in the parsing of ARGS: > x:y would be permitted, and map to a 2-tuple (x, y) > x:y:z would be permitted, and map to a 3-tuple (x, y, z) > x=y would be permitted, and map to a 2-tuple ('x', y) > *x would be permitted, and would extend the passed in tuple with x +1. This looks like a really nice option. One downside, which I only point out because someone will, is that optimization opportunities will be lost for the usual reason - the optimizer can't be sure that set (or whatever) hasn't been reassigned. But I still see that as something to be addressed independently, if at all. Paul. From solipsis at pitrou.net Sat Apr 29 13:25:27 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Apr 2006 13:25:27 +0200 Subject: [Python-3000] Automatically invoking str() in str.join() In-Reply-To: <1f7befae0604282307x10335f80x55d75509ff55eedb@mail.gmail.com> References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu> <4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com> <1146157955.10752.161.camel@resist.wooz.org> <4450FE5B.4010208@ieee.org> <1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com> <1f7befae0604271154w6ba78d3p3b6851e85d4dc68d@mail.gmail.com> <1f7befae0604282307x10335f80x55d75509ff55eedb@mail.gmail.com> Message-ID: <1146309927.5687.4.camel@fsol> Le samedi 29 avril 2006 ? 02:07 -0400, Tim Peters a ?crit : > If you don't remember these confusions, I think it should suffice to > remind that Perl's join() does take the separator first (which is > essentially forced in Perl, given its odd LIST syntax): PHP's join does too, also some versions have a hack to accept both orders :-O http://fr.php.net/manual/en/function.join.php http://fr.php.net/manual/en/function.implode.php From solipsis at pitrou.net Sat Apr 29 13:40:28 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 29 Apr 2006 13:40:28 +0200 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <20060428225107.6727.JCARLSON@uci.edu> References: <20060428000801.671E.JCARLSON@uci.edu> <06Apr28.101121pdt."58641"@synergy1.parc.xerox.com> <20060428225107.6727.JCARLSON@uci.edu> Message-ID: <1146310828.5687.13.camel@fsol> Le vendredi 28 avril 2006 ? 23:14 -0700, Josiah Carlson a ?crit : > Ultimately, PyGUI is essentially a wrapper of native or near-native > widgets into a common interface. wxPython is exactly the same thing, > only it has a much larger user base, developer support, and (native and > emulated) platform support. Similar things can be said about the GTK > and Qt bindings for Python (at least in terms of user base, developer > and platform support), but I don't have experience with PyGTK or PyQt, > so cannot comment on their quality/support/etc. versus wxPython. FWIW, wxWidgets is far from perfect in abstraction terms. There are lots of small differences among platform implementations which can be quite annoying (e.g. radio buttons send different messages in different orders). wx is a very massive library with lots of functionality, and the "cross-platform/native-look" aspect is great, but care of details is not its strength. From ncoghlan at gmail.com Sat Apr 29 13:55:23 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 29 Apr 2006 21:55:23 +1000 Subject: [Python-3000] sets in P3K? In-Reply-To: <79990c6b0604290316l603a89cxcf736e7e0ef85937@mail.gmail.com> References: <20060428213207.GA22469@nightcrawler> <445293A2.7020301@ewtllc.com> <4452DF79.8050506@gmail.com> <79990c6b0604290316l603a89cxcf736e7e0ef85937@mail.gmail.com> Message-ID: <4453542B.6000503@gmail.com> Paul Moore wrote: > One downside, which I only point out because someone will, is that > optimization opportunities will be lost for the usual reason - the > optimizer can't be sure that set (or whatever) hasn't been reassigned. > But I still see that as something to be addressed independently, if at > all. If we decide to go this route (supporting braced calls), I'd quite like to see {} become a simple abbreviation for dict{}, similar to the way [] would be an abbreviation of list{} and () would be an abbreviation of tuple{}. While I felt the optimisation benefits of a set literal were worth mentioning, they're significantly less important to me than the readability benefits. Maybe {1, 2, 3} should actually result in the old pseudo-set of {1:None, 2:None, 3:None}. It eliminates the type ambiguity as bare braces would always result in a dict, and getting a real set or frozenset is just a matter of putting the relevant name before the first brace. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From msoulier at digitaltorque.ca Sat Apr 29 14:41:45 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Sat, 29 Apr 2006 08:41:45 -0400 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <445315AC.9010701@gmx.net> References: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> <445315AC.9010701@gmx.net> Message-ID: <44535F09.9060407@digitaltorque.ca> Kay Schluehr wrote: > No one uses GTK+ on Windows. Actually, I believe that WingIDE is done with PyGTK, on both Linux and Windows. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-3000/attachments/20060429/80feca36/attachment.pgp From msoulier at digitaltorque.ca Sat Apr 29 14:43:02 2006 From: msoulier at digitaltorque.ca (Michael P. Soulier) Date: Sat, 29 Apr 2006 08:43:02 -0400 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <06Apr29.004032pdt."58641"@synergy1.parc.xerox.com> References: <06Apr29.004032pdt."58641"@synergy1.parc.xerox.com> Message-ID: <44535F56.5020408@digitaltorque.ca> Bill Janssen wrote: > Terry, "popular" isn't the goal of Python 3000. Nor is "backwards > compatibility" Well, I would certainly hope that a certain amount of popularity is a goal, or Python3k could be the best language that no one is using. Mike -- Michael P. Soulier "Any intelligent fool can make things bigger and more complex... It takes a touch of genius - and a lot of courage to move in the opposite direction." --Albert Einstein -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 252 bytes Desc: OpenPGP digital signature Url : http://mail.python.org/pipermail/python-3000/attachments/20060429/dca9f888/attachment.pgp From g.brandl at gmx.net Sat Apr 29 15:12:22 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 29 Apr 2006 15:12:22 +0200 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: <4452916F.1030304@v.loewis.de> References: <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> Message-ID: Martin v. L?wis wrote: > Guido van Rossum wrote: >>> I was hoping to propose a PEP on non-ASCII identifiers some >>> day; that would (of course) include a requirement that the >>> standard library would always be restricted to ASCII-only >>> identifiers as a style-guide. >> >> IMO communication about code becomes much more cumbersome if there are >> non-ASCII letters in identifiers, and the rules about what's a letter, >> what's a digit, and what separates two identifiers become murky. > > It depends on the language you use to communicate. In English, > it is certainly cumbersome to talk about Chinese identifiers. > OTOH, I believe it is cumbersome to communicate about English > identifiers in Chinese, either, because the speakers might > not even know what the natural-language concept behind the > identifiers is, and because they can't pronounce the identifier. They'll still have to type, pronounce and talk about English keywords, English operators and English standard library and 3rd-party module, class and method names, which makes a big percentage of code still Latin. Georg From jdahlin at async.com.br Sat Apr 29 15:49:19 2006 From: jdahlin at async.com.br (Johan Dahlin) Date: Sat, 29 Apr 2006 10:49:19 -0300 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <445315AC.9010701@gmx.net> References: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> <445315AC.9010701@gmx.net> Message-ID: <44536EDF.2080306@async.com.br> Kay Schluehr wrote: > Bill Janssen schrieb: > >> I mentioned this a couple of weeks ago, and here it is again. >> >> The biggest wart in Python 2, the one that forces me to use another >> language, is the lack of a standard GUI. By standard, I mean that >> when you install Python on a system, the GUI toolkit is also >> installed. >> >> I suggest that for Python 3K, we adopt Greg Ewing's PyGUI into the >> standard library. It's been around for years, works on every major >> platform (using GTK+ for Windows and X11, and Cocoa for Mac OS X), is >> Pythonic in flavor, and is aimed at eventual inclusion into the >> standard library. >> >> > No one uses GTK+ on Windows. Quite a few application does. The most popular application is Bittorrent, which has used it for quite some time. And Bittorrent is probably the most used python application. Johan From kay.schluehr at gmx.net Sat Apr 29 17:23:03 2006 From: kay.schluehr at gmx.net (Kay Schluehr) Date: Sat, 29 Apr 2006 17:23:03 +0200 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <44536EDF.2080306@async.com.br> References: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> <445315AC.9010701@gmx.net> <44536EDF.2080306@async.com.br> Message-ID: <445384D7.4000800@gmx.net> Johan Dahlin schrieb: >>No one uses GTK+ on Windows. >> >> > >Quite a few application does. > >The most popular application is Bittorrent, which has used it for quite >some time. And Bittorrent is probably the most used python application. > >Johan > > I don't want to insist on this point. There might be quite a lot of people using PyGTK but I'm not aware that Windows developers are among them. Suggesting to use Python on a server and working through a browser window works also in companies that won't touch Python otherwise. Suggesting to use GTK for GUI development in a Windows environment is a non-starter. Declaring something as "standard" implies for me that people are aware about what they are going to rule out. From phd at mail2.phd.pp.ru Sat Apr 29 17:28:24 2006 From: phd at mail2.phd.pp.ru (Oleg Broytmann) Date: Sat, 29 Apr 2006 19:28:24 +0400 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <445315AC.9010701@gmx.net> References: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> <445315AC.9010701@gmx.net> Message-ID: <20060429152824.GH19491@phd.pp.ru> On Sat, Apr 29, 2006 at 09:28:44AM +0200, Kay Schluehr wrote: > No one uses GTK+ on Windows. GAIM. Oleg. -- Oleg Broytmann http://phd.pp.ru/ phd at phd.pp.ru Programmers don't die, they just GOSUB without RETURN. From qrczak at knm.org.pl Sat Apr 29 18:13:59 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Sat, 29 Apr 2006 18:13:59 +0200 Subject: [Python-3000] sets in P3K? In-Reply-To: <4452DF79.8050506@gmail.com> (Nick Coghlan's message of "Sat, 29 Apr 2006 13:37:29 +1000") References: <20060428213207.GA22469@nightcrawler> <445293A2.7020301@ewtllc.com> <4452DF79.8050506@gmail.com> Message-ID: <87y7xonyuw.fsf@qrnik.zagroda> Nick Coghlan writes: > If we change that, we can use it to support an alternate invocation > syntax for functions that currently expect an iterable as their > first argument. "EXPR{ARGS}" would be equivalent to "EXPR((ARGS,))", > with the following differences in the parsing of ARGS: > x:y would be permitted, and map to a 2-tuple (x, y) > x:y:z would be permitted, and map to a 3-tuple (x, y, z) > x=y would be permitted, and map to a 2-tuple ('x', y) > *x would be permitted, and would extend the passed in tuple with x I like it. A disadvantage is that there are now three families contexts where comma-separated sequences of exprssions appear, with slightly different rules: a) tuples and lists b) function arguments c) EXPR{ARGS} where e.g. x=y has a different meaning in each. Perhaps unifying a) with c) would be better, i.e. allow the above syntactic features in tuples and lists too. -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From qrczak at knm.org.pl Sat Apr 29 17:55:36 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Sat, 29 Apr 2006 17:55:36 +0200 Subject: [Python-3000] sets in P3K? In-Reply-To: <445293A2.7020301@ewtllc.com> (Raymond Hettinger's message of "Fri, 28 Apr 2006 15:13:54 -0700") References: <20060428213207.GA22469@nightcrawler> <445293A2.7020301@ewtllc.com> Message-ID: <874q0cml53.fsf@qrnik.zagroda> Raymond Hettinger writes: > Good question. To see the answer, look at a code tranformation from: > > if file_ext.lower() in set(['html', 'xml', 'xhtml']): > handle(filename) > > into: > > > if file_ext.lower() in {'html', 'xml', 'xhtml'}: > handle(filename) Beware that without optimization this: if file_ext.lower() in ['html', 'xml', 'xhtml']: handle(filename) is probably faster than this: if file_ext.lower() in {'html', 'xml', 'xhtml'}: handle(filename) and with some trivial optimization this: if file_ext.lower() in ('html', 'xml', 'xhtml'): handle(filename) is even faster (the tuple can be allocated statically). So from the efficiency point of view using sets in such cases is a loss. And I'm not sure that optimizing 'x in {a,b,c}' to comparisons would be justified: it gives different results if some of a,b,c are not hashable. -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From g.brandl at gmx.net Sat Apr 29 18:27:09 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sat, 29 Apr 2006 18:27:09 +0200 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <20060429152824.GH19491@phd.pp.ru> References: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> <445315AC.9010701@gmx.net> <20060429152824.GH19491@phd.pp.ru> Message-ID: Oleg Broytmann wrote: > On Sat, Apr 29, 2006 at 09:28:44AM +0200, Kay Schluehr wrote: >> No one uses GTK+ on Windows. > > GAIM. And Gimp, yes, but these only use it because they were originally not made with a Windows port in mind. Georg From guido at python.org Sat Apr 29 18:40:06 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 29 Apr 2006 09:40:06 -0700 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: <4452D871.9020901@v.loewis.de> References: <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> <4452D871.9020901@v.loewis.de> Message-ID: On 4/28/06, "Martin v. L?wis" wrote: > Guido van Rossum wrote: > >> The distinction of letters and digits is also straight-forward: > >> a digit is ASCII [0-9]; it's a separate lexical class only > >> because it plays a special role in (number) literals. More > >> generally, there is the distinction of starter and non-starter > >> characters. > > > > But Unicode has many alternative sets digits for which "isdigit" is true. > > You mean, the Python isdigit() method? Sure, but the tokenizer uses > the C isdigit function, which gives true only for [0-9]. Isn't that because it's only defined on 8-bit characters though? And if we're talking about Unicode, why shouldn't we use the Unicode isdigit()? After all you were talking about the Unicode consortium's rules for which characters can be part of identifiers. > FWIW, POSIX > allows 6 alternative characters to be defined as hexdigits for > isxdigit, so the tokenizer shouldn't really use isxdigit for > hexadecimal literals. I think if we're talking Unicode, POSIX is irrelevant though, right? > So from the implementation point of view, nothing much would have > to change: the usage of isalnum in the tokenizer is already wrong, > as it already allows to put non-ASCII characters into identifiers, > if the locale classifies them as alpha-numeric. But we force the locale to be C, right? I've never heard of someone who managed to type non-ASCII letters into identifiers, and I'm sure it would've been reported as a bug. > I can't see why the Unicode notion of digits should affect the > language specification in any way. The notion of digit is only > used to define what number literals are, and I don't propose > to change the lexical rules for number literals - I propose > to change the rules for identifiers. Well identifiers can contain digits too. > > You can as far a the lexer is concerned because the lexer treats > > keywords as "just" identifiers. Only the parser knows which ones are > > really keywords. > > Right. But if the identifier syntax was > [:identifier_start:][:identifier_cont:]* > then thinks would work out just fine: identifier_start intersected > with ASCII would be [A-Za-z_], and identifier_cont intersected > with ASCII would be [A-Za-z0-9_]; this would include all keywords. > You would still need punctuation between two subsequent > "identifiers", and that punctuation would have to be ASCII, as > non-ASCII characters would be restricted to comments, string > literals, and identifiers. OK, I trust you that it can be made to work. But regardless, I really don't like it. I expect we'd be getting tons of questions on c.l.py about programs with identifiers containing squiggles we can neither read nor type, and for which we may not even have the fonts or the display capabilities (if it's right-to-left script). I do think that *eventually* we'll have to support this. But I don't think Python needs to lead the pack here; I don't think the tools are ready yet. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From jdahlin at async.com.br Sat Apr 29 15:49:19 2006 From: jdahlin at async.com.br (Johan Dahlin) Date: Sat, 29 Apr 2006 10:49:19 -0300 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <445315AC.9010701@gmx.net> References: <06Apr27.215050pdt."58641"@synergy1.parc.xerox.com> <445315AC.9010701@gmx.net> Message-ID: <44536EDF.2080306@async.com.br> Kay Schluehr wrote: > Bill Janssen schrieb: > >> I mentioned this a couple of weeks ago, and here it is again. >> >> The biggest wart in Python 2, the one that forces me to use another >> language, is the lack of a standard GUI. By standard, I mean that >> when you install Python on a system, the GUI toolkit is also >> installed. >> >> I suggest that for Python 3K, we adopt Greg Ewing's PyGUI into the >> standard library. It's been around for years, works on every major >> platform (using GTK+ for Windows and X11, and Cocoa for Mac OS X), is >> Pythonic in flavor, and is aimed at eventual inclusion into the >> standard library. >> >> > No one uses GTK+ on Windows. Quite a few application does. The most popular application is Bittorrent, which has used it for quite some time. And Bittorrent is probably the most used python application. Johan From tjreedy at udel.edu Sat Apr 29 08:47:21 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 29 Apr 2006 02:47:21 -0400 Subject: [Python-3000] Add a standard GUI system References: <20060428000801.671E.JCARLSON@uci.edu><06Apr28.101121pdt."58641"@synergy1.parc.xerox.com><1146266181.5808.6.camel@fsol> <01c301c66b1d$1f9d6470$8d472597@bagio> Message-ID: "Giovanni Bajo" wrote in message news:01c301c66b1d$1f9d6470$8d472597 at bagio... > There is simply no good enough solution for a GUI. There are far too many > advanced different libraries with different wrappers. >> The generic solution is better package fetching, such as apparently done >> (I >> have not used them yet) by setuptools and easy_install. > > Disagree. The mere existence of the stdlib improve consistency among > Python > programs. The next two sentences made clear that the sentence quoted was in respect to GUI libraries and the general problem of distributing an app that requires a particular GUI library not in the standard lib. I in no way suggested getting rid of the standard lib. Please don't distort what I write so badly. Terry Jan Reedy From talin at acm.org Sat Apr 29 21:09:36 2006 From: talin at acm.org (Talin) Date: Sat, 29 Apr 2006 19:09:36 +0000 (UTC) Subject: [Python-3000] Speeding up function calls Message-ID: In the current implementation, a function call starts by gathering 4 collections of arguments: -- The positional arguments, which are in a flat array -- The keyword arguments, which are in a flat array of keyword/value pairs. -- The varargs argument, which is a reference to a sequence. -- The keyword dictionary argument, which is a reference to a mapping. The function calling machinery starts by merging these various parts into two containers: A tuple and a dict. -- The tuple consists of the concatenation of the positional argument array with the varargs argument sequence. -- The dict consists of the union of the keyword argument array and the keyword dictionary array, with the restriction that duplicate keys are an error. Once control is transferred to the actual function, the arguments are unpacked using the specified argument unpacking logic - i.e. assigning argument values to formal parameter slots. My proposal is that we skip the middle portion of this process, and pass the initial 4 collections directly to the called function. Thus, the PyObject_Call function will look something like this: PyObject_Call( PyObject **positionalArgs, int numPositionalArgs, PyObject **keywordArgs, int numKeywordArgs, PyTuple *varargs, PyDict *kwdictargs ); (Alternatively, the various parameters could be passed in a single struct argument.) The calling function, upon recieving these arguments, can do a number of different things: 1) It can go ahead and construct the tuple/dict pair used in the previous implementation. 2) It can attempt to assign the arguments directly to parameter slots. This is fairly straightforward, because the current slot assignment algorithm is a one-pass algorithm that sequentially visits each input argument. For positional arguments, instead of iterating once over a PyTuple, the algorithm would iterate over the positional args array, and the iterate over the varargs argument. Keyword arguments would be handled similarly - iterate over the keyword/value pair array, and then iterate over the dict. 3) It can attempt to use the input arguments directly. For example, a function that accepts only positional arguments would do the following: # To use positional argument 'n': value = n < numPositionalArgs ? positionalArgs[ n ] : varargs[ n - numPositionalArgs ] 4) It can do some combination of the above. The decision as to how to handle the arguments will be made by the compiler as an optimization decision, based on the function signature. For example, if a function contains no formal **kwargs parameter, then part of the slot assignment algorithm can be skipped. On the other hand, if the function only has a **kwargs parameter, and no other keyword parameters, then a different part of the processing can be skipped. In other words, moving the argument packing logic into the caller doesn't by itself speed up function calling; What it does is open up a broad range of optimization possibilities. The simpler the recieving function's signature, the more room the compiler will have to be able to cull out unneeded parts of the argument assignment logic. -- Talin From tomerfiliba at gmail.com Sat Apr 29 21:10:15 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sat, 29 Apr 2006 21:10:15 +0200 Subject: [Python-3000] revamping the io stack, part 2 Message-ID: <1d85506f0604291210qd1493abk2ffb4ff859e3b906@mail.gmail.com> i first thought on focusing on the socket module, because it's the part that bothers me most, but since people have expressed their thoughts on completely revamping the IO stack, perhaps we should be open to adopting new ideas, mainly from the java/.NET world (keeping the momentum from the previous post). there is an inevitable issue of performance here, since it basically splits what used to be "file" or "socket" into many layers... each adding additional overhead, so many parts should be lowered to C. if we look at java/.NET for guidance, they have come up with two concepts: * stream - an arbitrary, usually sequential, byte data source * readers and writers - the way data is encoded into/decoded from the stream. we'll use the term "codec" for these readers and writers in general. so "stream" is the "where" and "codec" is the "how", and the concept of codecs is not limited to ASCII vs UTF-8. it can grow into fully-fledged protocols. - - - - - - - Streams - - - - - - - streams provide an interface to data sources, like memory, files, pipes, or sockets. the basic interface of all of these is class Stream: def close(self) def read(self, count) def readall(self) def write(self, data) and unlike today's files and sockets, when you read from a broken socket or past the end of the file, you get EOFError. read(x) guarantees to return x bytes, or EOFError otherwise (and also restoing the stream position). on the other hand, readall() makes no such guarantee: it reads all the data up to EOF, and if you readall() from EOF, you get "". perhaps readall() should return all *available* data, not necessarily up to EOF. for files, this is equivalent, but for sockets, readall would return all the data that sits in the network stack. this could be a nice way to do non-blocking IO. and if we do that already, perhaps we should introduce async operations as a built-in feature? .NET does (BeginRead, EndRead, etc.) def async_read(self, count, callback) def async_write(self, data, callback) i'm not sure about these two, but it does seem like a good path to follow. ----- another issue is the current class hierarchy: fileno, seek, and readline are meaningless in many situations, yet they are considered the part of the file- protocol (take a look at StringIO implementing isatty!). these methods, which may be meaningless for several types of streams, must not be part of the base Stream class. for example, only FileStream and MemoryStream are seekable, so why have seek as part of the base Stream class? ----- streams that don't rely on an operating-system resource, would derive directly from Stream. as examples for such streams, we can condier class MemoryStream(Stream): # like today's StringIO # allows seeking class RandomStream(Stream): # provider of random data ----- on the other hand, streams that rely on operating-system resources, like files or sockets, would derive from class OSStream(Stream): def isatty(self) def fileno(self) # for select() def dup(self) and there are several examples for this kind: FileStream is the entity that works with files, instead of the file/open class of today. since files provide random-access (seek/tell), this kind of stream is "seekable" and "tellable". class FileStream(OSStream): def __init__(self, filename, mode = "r") def seek(self, pos, offset = None) def tell(self) def set_size(self, size) def get_size(self) although i prefer properties instead position = property(tell, seek) size = property(get_size, set_size) PipeStream represents a stream over a (simplex) pipe: class PipeStream(OSStream): def get_mode(self) # read or write DuplexPipeStream is an abstraction layer that uses two simplex pipes as a full-duplex stream: class DuplexPipeStream(OSStream): def __init__(self, incoming, outgoing): @classmethod def open(cls): incoming, outgoing = os.pipe() return cls(incoming, outgoing) NetworkStreams provide a stream over a socket. unlike files, sockets may get quite complicated (options, accept, bind), so we keep the distinction: * sockets as the underlying "physical resource" * NetworkStreams wrap them with a nice stream interface. for example, while socket.recv(x) may return less than x bytes, networkstream.read(x) returns x bytes. we must keep this distinction because streams are *data sources*, and there's no way to represent things like bind or accept in a data source. only client (connected) sockets would be wrappable by NetworkStream. server sockets don't provide data and hence have nothing to do with streams. class NetworkStream(OSStream): def __init__(self, sock) - - - - - - - - - Special Streams - - - - - - - - - it will also be useful to have a way to duplicate a stream, like the unix tee command does class TeeStream(Stream): def __init__(self, src_stream, dst_stream) f1 = FileStream("c:\\blah") f2 = FileStream("c:\\yaddah") f1 = TeeStream(f1, f2) f1.write("hello") will write "hello" to f2 as well. that's useful for monitoring/debugging, like echoing everything from a NetworkStream to a file, so you could debug it easily. ----- buffering is always *explicit* and implemented at the interpreter level, rather than by libc, so it is consistent between all platforms and streams. all streams, by nature, and *non-buffered* (write the data as soon as possible). buffering wraps an underlying stream, making it explicit class BufferedStream(Stream): def __init__(self, stream, bufsize) def flush(self) (BufferedStream appears in .NET) class LineBufferedStream(BufferedStream): def __init__(self, stream, flush_on = b"\n") f = LineBufferedStream(FileStream("c:\\blah")) where flush_on specifies the byte (or sequence of bytes?) to flush upon writing. by default it would be on newline. - - - - - - - Codecs - - - - - - - as was said earlier, formatting defines how the data (or arbitrary objects) are to be encoded into and decoded from a stream. class StreamCodec: def __init__(self, stream) def write(self, ...) def read(self, ...) for example, in order to serialize binary records into a file, you would use class StructCodec(StreamCodec): def __init__(self, stream, format): Codec.__init__(self, stream) self.format = format def write(self, *args): self.stream.write(struct.pack(self.format, *args)) def read(self): size = struct.calcsize(self.format) data = self.stream.read(size) return struct.unpack(self.format, data) (similar to BinaryReader/BinaryWriter in .NET) and for working with text, you would have class TextCodec(StreamCodec): def __init__(self, stream, textcodec = "utf-8"): Codec.__init__(self, stream) self.textcodec = textcodec def write(self, data): self.stream.write(data.encode(self.textcodec)) def read(self, length): return self.stream.read(length).decode(self.textcodec) def __iter__(self) # iter by lines def readline(self) # read the next line def writeline(self, data) # write a line as you can see, only the TextCodec adds the readline/writeline methods, as they are meaningless to most binary formats. the stream itself has no notion of a line. no more newline issues! the TextCodec will do the translation for you. all newlines are \n in python, and are written to the underlying stream in a way that would please the underlying platform. so the "rb" and "wb" file modes will deminish, and instead you would wrap the FileStream with a TextCodec. it's explicit, so you won't be able to corrupt data accidentally. ----- it's worth to note that in .NET (and perhaps java as well), they splitted TextCodec into two parts, the TextReader and TextWriter classes, which you initialize over a stream: f = new FileStream("c:\\blah"); sr = new StreamReader(f, Encoding.UTF8); sw = new StreamWriter(f, Encoding.UTF8); sw.Write("hello"); f.Position = 0; sr.read(5); but why separate the two? it could only cause problems, as you may initialize them with different encodings, which leads to no good. under the guidelines of this suggestion, it would be implemented this way: f = TextCodec(FileStream("c:\\blah"), "utf-8") which can of course be refactored to a function: def textfile(filename, mode = "r", codec = "utf-8"): return TextCodec(FileStream(filename, mode), codec) for line in textfile("c:\\blah"): print line unlike today's file objects, FileStream objects don't know about lines, so you can't iterate through a file directly. it's quite logical if you think about it, as there's no meaning to iterating over a binary file by lines. it's a feature of text files. ----- many times, especially in network protocols, you need framing for transfering frames/packets/messages over a stream. so a very useful FramingCodec can be introduced: class FramingCodec(Codec): def write(self, data): self.stream.write(struct.pack(" References: <1d85506f0604291210qd1493abk2ffb4ff859e3b906@mail.gmail.com> Message-ID: On 4/29/06, tomer filiba wrote: > i first thought on focusing on the socket module, because it's the part that > bothers me most, but since people have expressed their thoughts on > completely > revamping the IO stack, perhaps we should be open to adopting new ideas, > mainly from the java/.NET world (keeping the momentum from the previous > post). > > there is an inevitable issue of performance here, since it basically splits > what used to be "file" or "socket" into many layers... each adding > additional > overhead, so many parts should be lowered to C. > > if we look at java/.NET for guidance, they have come up with two concepts: I am a little weary of taking too much from Java/.NET since I have always found the I/O system way too heavy for the common case. I can't remember what it takes to get a reader in Java in order to read by lines. In Python, I love that I don't have to think about that; just pass a file object to 'for' and I am done. While I am all for allowing for more powerful I/O through stacking a stream within various readers (which feels rather functional to me, but that must just be because of my latest reading material), I don't want to make the 90% case require hardly any memorizing of what readers I need in what order. > * stream - an arbitrary, usually sequential, byte data source > * readers and writers - the way data is encoded into/decoded from the > stream. > we'll use the term "codec" for these readers and writers in general. > > so "stream" is the "where" and "codec" is the "how", and the concept of > codecs is not limited to ASCII vs UTF-8. it can grow into fully-fledged > protocols. [SNIP - a whole lot of detailed ideas] > ----- > > buffering is always *explicit* and implemented at the interpreter level, > rather than by libc, so it is consistent between all platforms and streams. > all streams, by nature, and *non-buffered* (write the data as soon as > possible). buffering wraps an underlying stream, making it explicit > > class BufferedStream(Stream): > def __init__(self, stream, bufsize) > def flush(self) > > (BufferedStream appears in .NET) > > class LineBufferedStream(BufferedStream): > def __init__(self, stream, flush_on = b"\n") > > f = LineBufferedStream(FileStream("c:\\blah")) > > where flush_on specifies the byte (or sequence of bytes?) to flush upon > writing. by default it would be on newline. > See, this is what I am worried about. I **really** like not having to figure out what I need to do to read by lines from a file. If the FileStream object had an __iter__ that did the proper wrapping with LinedBufferedStream, then great, I'm happy. But if we do not add some reasonable convenience functions or iterators, this is going to feel rather heavy-handed rather quickly. -Brett From qrczak at knm.org.pl Sat Apr 29 22:52:37 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Sat, 29 Apr 2006 22:52:37 +0200 Subject: [Python-3000] genexp syntax / lambda In-Reply-To: <44474404.9070600@canterbury.ac.nz> (Greg Ewing's message of "Thu, 20 Apr 2006 20:19:16 +1200") References: <77f4ed2b0604191306i3b9ebf56s5fd4964c2b326cee@mail.gmail.com> <44474404.9070600@canterbury.ac.nz> Message-ID: <87bqukhzoq.fsf@qrnik.zagroda> greg.ewing at canterbury.ac.nz (Greg Ewing) writes: >> given - much clearer and slightly more concise; or >> ? - slightly clearer and much more concise ? > > That would be replacing something that's familiar to at least some > people (even if they are geeks) with something that's familiar to > nobody. "?" would be familiar for me, because this is what my language Kogut uses :-) (I know that Python won't change here.) Here are some other ideas that Py3k is going to borrow from Kogut: - int/int doesn't truncate - writing to stdout uses a function rather than a builtin syntax - < > <= >= don't work across types by default - floats are not accepted as sequence indices - the distinction between int and long is an implementation detail - all strings are Unicode - the set of keys of a dict is a view - vararg application uses a builtin syntax rather than a function - generic functions are supported - collections are often built by applying the constructor of a concrete type to a generator I'm glad that languages are somewhat converging. Unfortunately they will still disagree about what iteration over a dict yields, which required a tweak in my Kogut<->Python bridge (checking iteritems before PyObject_GetIter to obtain the other semantics). -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From jcarlson at uci.edu Sun Apr 30 00:01:56 2006 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat, 29 Apr 2006 15:01:56 -0700 Subject: [Python-3000] Add a standard GUI system In-Reply-To: <1146310828.5687.13.camel@fsol> References: <20060428225107.6727.JCARLSON@uci.edu> <1146310828.5687.13.camel@fsol> Message-ID: <20060429145635.6735.JCARLSON@uci.edu> Antoine Pitrou wrote: > > Le vendredi 28 avril 2006 ? 23:14 -0700, Josiah Carlson a ?crit : > > Ultimately, PyGUI is essentially a wrapper of native or near-native > > widgets into a common interface. wxPython is exactly the same thing, > > only it has a much larger user base, developer support, and (native and > > emulated) platform support. Similar things can be said about the GTK > > and Qt bindings for Python (at least in terms of user base, developer > > and platform support), but I don't have experience with PyGTK or PyQt, > > so cannot comment on their quality/support/etc. versus wxPython. > > FWIW, wxWidgets is far from perfect in abstraction terms. There are lots > of small differences among platform implementations which can be quite > annoying (e.g. radio buttons send different messages in different > orders). > > wx is a very massive library with lots of functionality, and the > "cross-platform/native-look" aspect is great, but care of details is not > its strength. In my experience, they would like to offer consistancy cross-platform whenever possible. I'm never claimed wxPython's perfection, but I've never found it to be less than high-quality. - Josiah From aahz at pythoncraft.com Sun Apr 30 01:11:03 2006 From: aahz at pythoncraft.com (Aahz) Date: Sat, 29 Apr 2006 16:11:03 -0700 Subject: [Python-3000] Speeding up function calls In-Reply-To: References: Message-ID: <20060429231102.GB458@panix.com> On Sat, Apr 29, 2006, Talin wrote: > > My proposal is that we skip the middle portion of this process, and > pass the initial 4 collections directly to the called function. IIRC, you've already been asked to profile the machinery to find out whether frame creation is still the primary bottleneck for slow function calls. Would you mind doing that before pushing a radical API change? -- Aahz (aahz at pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From martin at v.loewis.de Sun Apr 30 01:51:26 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 Apr 2006 01:51:26 +0200 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: References: <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> <4452D871.9020901@v.loewis.de> Message-ID: <4453FBFE.7060001@v.loewis.de> Guido van Rossum wrote: >> > But Unicode has many alternative sets digits for which "isdigit" is >> true. >> >> You mean, the Python isdigit() method? Sure, but the tokenizer uses >> the C isdigit function, which gives true only for [0-9]. > > Isn't that because it's only defined on 8-bit characters though? No: the C standard requires that isdigit is true if and only if the character is from [0-9]; it also requires that the digits must have subsequent ordinals in the "execution character set", and that they must be represented using a single char (rather than requiring multiple bytes). Currently, the tokenizer operates on UTF-8, which is multi-byte, but still, isdigit works "correctly". > And if we're talking about Unicode, why shouldn't we use the Unicode > isdigit()? After all you were talking about the Unicode consortium's > rules for which characters can be part of identifiers. The tokenizer doesn't use isdigit() to determine what an identifier is; it uses isalnum(). The parser uses isdigit only to determine what a number literal is - I don't propose to change that. The Unicode consortium rules are listed here: http://www.unicode.org/reports/tr31/ This recommendation mentions two classes ID_Start and ID_Continue: ID_Start: Uppercase letters, lowercase letters, titlecase letters, modifier letters, other letters, letter numbers, stability extensions ID_Continue: All of the above, plus nonspacing marks, spacing combining marks, decimal numbers, connector punctuations, stability extensions. These are also known simply as Identifier Characters, since they are a superset of the ID_Start. The set of ID_Start characters minus the ID_Continue characters are known as ID_Only_ Continue characters. In the implementation, a compact table should be used to determine whether a character is ID_Start or ID_Continue, instead of calling some library function. There are some problems with the UAX#31 definitions IIRC, although I forgot the exact details (might be that the underscore is missing, or that the dollar is allowed); the definitions should be adjusted so that they match the current language for ASCII. >> FWIW, POSIX >> allows 6 alternative characters to be defined as hexdigits for >> isxdigit, so the tokenizer shouldn't really use isxdigit for >> hexadecimal literals. > > I think if we're talking Unicode, POSIX is irrelevant though, right? What I'm saying is that the tokenizer currently uses isxdigit; it should stop doing so (whether or not Unicode identifiers become part of the language). As source code would (still) be parsed as UTF-8, isxdigit would continue to "work", but definitely shouldn't be used anymore. > But we force the locale to be C, right? I've never heard of someone > who managed to type non-ASCII letters into identifiers, and I'm sure > it would've been reported as a bug. Python 2.3.5 (#2, Mar 6 2006, 10:12:24) [GCC 4.0.3 20060304 (prerelease) (Debian 4.0.2-10)] on linux2 Type "help", "copyright", "credits" or "license" for more information. py> import locale py> locale.setlocale(locale.LC_ALL, "") 'de_DE at euro' py> l?wis=1 py> print l?wis 1 We don't force the C locale - we just happen to start with it initially. We shouldn't change it later, as that isn't thread-safe. Nobody reported it, because people just don't try to do that, except in interactive mode. >> I can't see why the Unicode notion of digits should affect the >> language specification in any way. The notion of digit is only >> used to define what number literals are, and I don't propose >> to change the lexical rules for number literals - I propose >> to change the rules for identifiers. > > Well identifiers can contain digits too. Sure. But they dont' "count" as digits then, lexically - they are ID_Continue characters (which is a superset of digits). So what we need is to extend the definition of ID_Continue, not the definition of digits. > I do think that *eventually* we'll have to support this. But I don't > think Python needs to lead the pack here; I don't think the tools are > ready yet. Python doesn't really lead here. The C family of languages (C, C++, Java, C#) all have Unicode identifiers, so there is plenty of experience. Primarily, the experience is that the feature isn't used much, because of obstacles I think we can overcome (primarily, that all these languages make the source encoding implementation-defined; we don't, as we put the source encoding into the source file). Regards, Martin From martin at v.loewis.de Sun Apr 30 01:58:05 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 Apr 2006 01:58:05 +0200 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: References: <4450C311.8040603@fas.harvard.edu> <4450CB36.7090504@gmail.com> <44511237.5000609@ewtllc.com> <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> Message-ID: <4453FD8D.5070407@v.loewis.de> Georg Brandl wrote: > They'll still have to type, pronounce and talk about English keywords, English > operators and English standard library and 3rd-party module, class and method > names, which makes a big percentage of code still Latin. They don't have to understand the English meaning of the keywords. For example, I am still uncertain why the keyword to create a new function is "def", but I have accepted that as a fact of life; it could have been "onserber", and I would have used that just as well. For library functions, I believe it is different: you really have to understand the intent for giving a certain API its name, because you won't be able to remember that many "non-sensical" character sequences - just as you wouldn't be able to use the API if it was written in Greek letters, with a Greek meaning. Regards, Martin From rudyrudolph at excite.com Sun Apr 30 02:53:56 2006 From: rudyrudolph at excite.com (Rudy Rudolph) Date: Sat, 29 Apr 2006 20:53:56 -0400 (EDT) Subject: [Python-3000] in-out parameters Message-ID: <20060430005356.3DA9B8B34E@xprdmxin.myway.com> One thing that has always bothered me about Python is the lack of in-out parameters. Python is such a wonderful object-oriented language, but it lacks this feature that nearly every language has. Maybe we can fix this in 3.0. There are two possibilities for how to do this: by reference or by copy-back. For a single-threaded app the effect is the same if the function returns normally. If the function raises an exception, then the actual argument may have a changed value with by-reference semantics but not with copy-back semantics. Either semantics is acceptable to me. Here's what I have in mind. def f(inout param): param += 5 return "this is my result" # call it x = 2 result = f(x) # x is now 7 I don't much care about the syntax. The proposed "inout" reserved word could be "ref" or "&" or "in out". We could even do "param : inout [type]". Whatever. There are two current workarounds for this missing feature. 1) copy-back: def f(param): param += 5 return "this is my result", param # call it x = 2 result,x = f(x) # ugly, but x is now 7 # If we implement copy-back inout, the meaning would be similar to this. 2) pass-by-reference: def f(wrappedParam): wrappedParam[0] += 5 # ugh return "this is my result" # call it x = 2 result = f([x]) # also ugly, but x is now 7 # If we implement by-reference inout, the meaning would be similar to this. # Passing some kind of proxy is also possible. The effect of approaches 1 and 2 is the same unless f raises an exception between incrementing by 5 and returning. With copy-back semantics, x is still 2 when the exception is caught. If x is a global accessed by other threads, the choice of semantics also affects them if an exception is raised. If we do either of these, we should also consider "out". This is not as big of a concern because the workaround has no parameter and the function just returns a pair of results. This isn't as unnatural as mentioning x twice or "listing" it. I propose that if we do "out" it would mean the same as either meaning of inout, but passing None as the argument (for copy-back) or doing "x = None" before the call (for pass-by-reference). This is, of course, subject to debate. Impact : a) Inside the function, inout and out params would be treated differently. If inout implemented as copy-back: return an extra value (make a tuple with the complete nominal return value as the first element, followed by special params in order) If inout implemented as pass-by-reference: treat param as param[0] (if list is passed. If a proxy is used, then something like calling a property setter) If out : same as one of the above b) At the point of call, an extra step or two is necessary for each inout or out param in the function to be called. If inout implemented as copy-back: assign after return (returned value is implicitly a tuple with the nominal result or None at 0, followed by new values for inout arguments) If inout implemented as pass-by-reference: wrap in a list or proxy before call If out implemented as copy-back: pass None for the value of param, then do same as inout via copy-back If out implemented as pass-by-reference: x = None before call, then do same as inout by reference This does not introduce any backward incompatabilities so far as I know, so it could possibly go into 2.6. I consider it very important for 3.0. I suppose this could be accomplished via a decorator that messed with the caller's stack frame, but I think syntactic support is better. I don't remember seeing a PEP for this. Is it worth writing one or is the initial reaction strongly negative? If I write a PEP, what other issues would need to be examined besides the discussion above and interaction with the other proposed changes to the parameter list? (BTW, I'm +1 on keyword-only parameters, keyword arguments before positional ones in the call, and optional type on parameters.) I have never written a patch for the Python core but am willing to try if the people on the dev list are willing to help when I have questions. I'm an experienced C programmer but have never looked at the CPython source code. This is my first time posting to 3000, but I've been a Python user for 9 years and a software engineer for 19 years. Good to meet you, everyone. I'm working on a PhD in computer science, specializing in programming languages. I was a reviewer for the Ada95 changes. #Rudy _______________________________________________ Join Excite! - http://www.excite.com The most personalized portal on the Web! From guido at python.org Sun Apr 30 04:49:50 2006 From: guido at python.org (Guido van Rossum) Date: Sat, 29 Apr 2006 19:49:50 -0700 Subject: [Python-3000] Speeding up function calls In-Reply-To: <20060429231102.GB458@panix.com> References: <20060429231102.GB458@panix.com> Message-ID: On 4/29/06, Aahz wrote: > On Sat, Apr 29, 2006, Talin wrote: > > My proposal is that we skip the middle portion of this process, and > > pass the initial 4 collections directly to the called function. > > IIRC, you've already been asked to profile the machinery to find out > whether frame creation is still the primary bottleneck for slow function > calls. Would you mind doing that before pushing a radical API change? What he said. I'd be very excited if you produced a patch that showed a significant speedup. Until then I don't have the time to think about this much. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From steven.bethard at gmail.com Sun Apr 30 04:53:59 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 29 Apr 2006 20:53:59 -0600 Subject: [Python-3000] in-out parameters In-Reply-To: <20060430005356.3DA9B8B34E@xprdmxin.myway.com> References: <20060430005356.3DA9B8B34E@xprdmxin.myway.com> Message-ID: On 4/29/06, Rudy Rudolph wrote: > One thing that has always bothered me about Python is the lack of in-out > parameters. Python is such a wonderful object-oriented language, but it lacks > this feature that nearly every language has. Maybe we can fix this in 3.0. -1 since I can't figure out what code this would make clearer/easier to read. Can you give a bunch of examples where you think in-out parameters would help? STeVe -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From ncoghlan at gmail.com Sun Apr 30 05:04:14 2006 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 30 Apr 2006 13:04:14 +1000 Subject: [Python-3000] in-out parameters In-Reply-To: <20060430005356.3DA9B8B34E@xprdmxin.myway.com> References: <20060430005356.3DA9B8B34E@xprdmxin.myway.com> Message-ID: <4454292E.6050403@gmail.com> Rudy Rudolph wrote: > I don't much care about the syntax. The proposed "inout" reserved word could > be "ref" or "&" or "in out". We could even do "param : inout [type]". Whatever. A more flexible approach would be to look at weakref.proxy and develop a version that takes a strong reference to the object it contains instead of a weak reference. This approach means the same mechanism will work for closures as well as for parameter passing. Then the usage looks like: >>> def f(param): ... param += 5 ... print "Parameter modified:", param ... >>> x = 2 >>> f(x) Parameter modified: 7 >>> print x 2 >>> x = ref(2) >>> f(x) Parameter modified: 7 >>> print x 7 So it is up to the caller to decide whether they want to allow the function to modify their local namespace. Closures can also selectively enable rebinding in a slightly more intuitive fashion than with boxing into a list. To use the infamous accumulator example: >>> def accum(n): ... n = ref(n) ... def increment(i): ... n += i ... return n.__value__ ... return increment ... # Use 'return lambda i: n.__iadd__(i).__value__' instead of the last ... # 4 lines if you prefer brevity to readability. Or you can go even ... # further and use 'return lambda i, n=ref(n): n.__iadd__(i).__value__' ... # as the sole line in the body of the function, although doing so ... # slightly changes the signature of the returned function ... >>> inc = accum(2) >>> inc(5) 7 >>> inc(3) 10 The one trick with this is that simple assignment to a proxy reference won't work properly. Instead, you need to write something like: >>> def f(param): ... param = ref.bind(param, 5) ... print "Parameter replaced:", param ... >>> x = 2 >>> f(x) Parameter replaced: 5 >>> print x 2 >>> x = ref(2) >>> f(x) Parameter replaced: 5 >>> print x 5 Cheers, Nick. ------------------------------- # The following naive version of ref was used for the above examples # A real proxy type would modify its class dynamically whenever # __value__ was rebound to a different type of object so that it # only provided the methods that the referent exposed. class ref(object): def __init__(self, referent): self.__value__ = referent # Proxying string representation def __str__(self): return str(self.__value__) # Proxying addition def __add__(self, other): return self.__value__ + other __radd__ = __add__ def __iadd__(self, other): self.__value__ += other return self # Proxying assignment @staticmethod def bind(proxy, other): # Static method so it works for non-ref objects try: proxy.__value__ = other except AttributeError: return other else: return proxy -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia --------------------------------------------------------------- http://www.boredomandlaziness.org From steven.bethard at gmail.com Sun Apr 30 05:07:56 2006 From: steven.bethard at gmail.com (Steven Bethard) Date: Sat, 29 Apr 2006 21:07:56 -0600 Subject: [Python-3000] PEP 3002 (Procedure for Backwards-Incompatible Changes) In-Reply-To: References: Message-ID: On 4/27/06, Guido van Rossum wrote: > I just read PEP 3002 and while I like most of what I see (thanks > Steven Bethard!) there's one section that stands out as problematic. [snip] > Rather than a version of Python > 3000, I suggest to add such warning code to Python 2.x, and not as a > branch, but as a permanent feature (that is however only enabled by a > command-line flag). Yes, I think it definitely makes more sense to make the modifications on 2.x. I was hesitant to require integrating it into the main-line Python 2.x code because I was worried about affecting performance. But of course with a command-line flag, that point is moot. So basically python3warn.py could be become "python -SOME_OPTION" where SOME_OPTION would enable all the Python 3.0 checks. So I guess the question then is what the option should enable. Should it try to implement the Python 3.0 behavior? Or should it just warn when the behavior is changed? The former requires much more Python 3.0 code to be back-ported to Python 2.X, but would presumably give much better error reporting. I'd love to require that, but I'm afraid of the work it would take. Steve P.S. I have commit access to the PEPs now, so I can update PEP 3002 with whatever we decide. -- Grammar am for people who can't think for myself. --- Bucky Katt, Get Fuzzy From brett at python.org Sun Apr 30 05:17:11 2006 From: brett at python.org (Brett Cannon) Date: Sat, 29 Apr 2006 20:17:11 -0700 Subject: [Python-3000] PEP 3002 (Procedure for Backwards-Incompatible Changes) In-Reply-To: References: Message-ID: On 4/29/06, Steven Bethard wrote: > On 4/27/06, Guido van Rossum wrote: > > I just read PEP 3002 and while I like most of what I see (thanks > > Steven Bethard!) there's one section that stands out as problematic. > [snip] > > Rather than a version of Python > > 3000, I suggest to add such warning code to Python 2.x, and not as a > > branch, but as a permanent feature (that is however only enabled by a > > command-line flag). > > Yes, I think it definitely makes more sense to make the modifications > on 2.x. I was hesitant to require integrating it into the main-line > Python 2.x code because I was worried about affecting performance. > But of course with a command-line flag, that point is moot. So > basically python3warn.py could be become "python -SOME_OPTION" where > SOME_OPTION would enable all the Python 3.0 checks. > > So I guess the question then is what the option should enable. Should > it try to implement the Python 3.0 behavior? Or should it just warn > when the behavior is changed? The former requires much more Python > 3.0 code to be back-ported to Python 2.X, but would presumably give > much better error reporting. I'd love to require that, but I'm afraid > of the work it would take. > My vote is the latter (i.e., just warnings). Changes whose semantics are desirable for 2.x will be backported. Those that are not won't be. Thus we shouldn't go through the trouble of implementing the semantics if for some reason it was not desirable to have them used by default in the first place. -Brett > Steve > > P.S. I have commit access to the PEPs now, so I can update PEP 3002 > with whatever we decide. > -- > Grammar am for people who can't think for myself. > --- Bucky Katt, Get Fuzzy > _______________________________________________ > Python-3000 mailing list > Python-3000 at python.org > http://mail.python.org/mailman/listinfo/python-3000 > Unsubscribe: http://mail.python.org/mailman/options/python-3000/brett%40python.org > From tjreedy at udel.edu Sun Apr 30 05:42:04 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 29 Apr 2006 23:42:04 -0400 Subject: [Python-3000] revamping the io stack, part 2 References: <1d85506f0604291210qd1493abk2ffb4ff859e3b906@mail.gmail.com> Message-ID: "Brett Cannon" wrote in message news:bbaeab100604291350u6ef6f630je67bf9cc2cc7f465 at mail.gmail.com... > See, this is what I am worried about. I **really** like not having to > figure out what I need to do to read by lines from a file. If the > FileStream object had an __iter__ that did the proper wrapping with > LinedBufferedStream, then great, I'm happy. But if we do not add some > reasonable convenience functions or iterators, this is going to feel > rather heavy-handed rather quickly. Perhaps open() could take a parameter to specify the iteraton chunk: bytes, chars, words, lines, para(graph)s, blocks of size n, splits on string s. I think it somewhat common that an outer line loop contains an inner loop for the true iteration chunk (most often, bytes or the result of split()). The double loop is only there because that is what is easier. Terry Jan Reedy From tjreedy at udel.edu Sun Apr 30 06:01:02 2006 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 30 Apr 2006 00:01:02 -0400 Subject: [Python-3000] in-out parameters References: <20060430005356.3DA9B8B34E@xprdmxin.myway.com> Message-ID: "Rudy Rudolph" wrote in message news:20060430005356.3DA9B8B34E at xprdmxin.myway.com... > > One thing that has always bothered me about Python is the lack of in-out > parameters. Python is such a wonderful object-oriented language, but it > lacks > this feature that nearly every language has. Maybe we can fix this in > 3.0. -1 a. Not needed because Python allows one to return multiple values (forming a tuple) instead of just one. b. Not needed because Python already has them: any mutable arg. Under the CPython covers, the current pass-by-name-binding amounts to pass by reference as some define it. Terry Jan Reedy From fredrik at pythonware.com Sun Apr 30 10:05:59 2006 From: fredrik at pythonware.com (Fredrik Lundh) Date: Sun, 30 Apr 2006 10:05:59 +0200 Subject: [Python-3000] Automatically invoking str() in str.join() References: <4450C80D.50002@iinet.net.au> <4450D8B8.40408@fas.harvard.edu><4450EE32.7000909@ewtllc.com> <20060427170803.GB17595@panix.com><1146157955.10752.161.camel@resist.wooz.org><4450FE5B.4010208@ieee.org><1f7befae0604271046n58484382hf7fa5c66b8a4f570@mail.gmail.com><1f7befae0604271154w6ba78d3p3b6851e85d4dc68d@mail.gmail.com> <1f7befae0604282307x10335f80x55d75509ff55eedb@mail.gmail.com> Message-ID: Tim Peters wrote: > Let's see. Should I watch a movie now, or search pre-string-method > archives for quotes nobody really cares about? While I think about > that ;-), you could look in _this_ thread for: well, designing things based on how old farts think things where back in the old days isn't really optimal. and as we've seen on python-dev, there's nothing that old python farts do better than arguing for keeping the status quo, no matter what that is... > While I don't know for sure, I always assumed that was the true source > of most of the relevant "how come my code doesn't work?!" complaints > on comp.lang.python before string methods were introduced. for the record, this just appeared on comp.lang.python: hello everyone! I can't seem to find a function that combines a list of items into a string with a seperator between the individual elements.. which, based on the law of design based on random sampling of c.l.python, indicates that the current situation is not optimal. From qrczak at knm.org.pl Sun Apr 30 11:01:59 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Sun, 30 Apr 2006 11:01:59 +0200 Subject: [Python-3000] in-out parameters In-Reply-To: <20060430005356.3DA9B8B34E@xprdmxin.myway.com> (Rudy Rudolph's message of "Sat, 29 Apr 2006 20:53:56 -0400 (EDT)") References: <20060430005356.3DA9B8B34E@xprdmxin.myway.com> Message-ID: <87slnvzbaw.fsf@qrnik.zagroda> "Rudy Rudolph" writes: > def f(inout param): > param += 5 > return "this is my result" > > # call it > x = 2 > result = f(x) > # x is now 7 In a dynamically typed language it's not going to work unless both the parameter in the function definition and the actual argument are marked. Then it's easy, well-defined, and can even support other "lvalues", e.g.: def f(ref x): x += 1 f(ref a[g()]) # g() is executed once # reading x does __getitem__ # setting x does __setitem__ I'm not judging whether it's worth having, only how it should work if supported. -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From hyeshik at gmail.com Sun Apr 30 15:46:57 2006 From: hyeshik at gmail.com (Hye-Shik Chang) Date: Sun, 30 Apr 2006 22:46:57 +0900 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: References: <17489.31069.526965.186604@montanaro.dyndns.org> <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> Message-ID: <4f0b69dc0604300646v53567a89x9c6e62777232387b@mail.gmail.com> On 4/29/06, Guido van Rossum wrote: > On 4/28/06, "Martin v. L?wis" wrote: > > Guido van Rossum wrote: > > >> I was hoping to propose a PEP on non-ASCII identifiers some > > >> day; that would (of course) include a requirement that the > > >> standard library would always be restricted to ASCII-only > > >> identifiers as a style-guide. > > > > > > IMO communication about code becomes much more cumbersome if there are > > > non-ASCII letters in identifiers, and the rules about what's a letter, > > > what's a digit, and what separates two identifiers become murky. > > > > It depends on the language you use to communicate. In English, > > it is certainly cumbersome to talk about Chinese identifiers. > > OTOH, I believe it is cumbersome to communicate about English > > identifiers in Chinese, either, because the speakers might > > not even know what the natural-language concept behind the > > identifiers is, and because they can't pronounce the identifier. > > True; but (and I realize we're just pitting our beliefs against each > other here) I believe that Chinese computer users are more likely to > be able (and know how) to type characters in the Latin alphabet than > Western programmers are able to type Chinese. For example, I notice > that baidu.cn (a Chinese search engine) spells its own name (a big > brand in China) using the Latin alphabet. I expect that Chinese users > are used to typing "baidu.cn" in their browser's search bar, rather > than Chinese characters. The example is a quite corner case. CJK users can't use their own languages in URL because MSIE doesn't support IDNA yet. Even the most of ISPs of the countries hijacks DNS queries and forwards to install a ActiveX control that handles their original scripts (hanzi or hangul). And there is an another practical problem in romanized identifiers. Because their romanization method isn't quite inconsistent among users though it's standardized. For example, a Korean word meaning "maintenance" is "unyeong" in standard method. But Korean people writes it as "woonyoung", "unyoung", "oonyeong" or even "unyeong" etc. This would make it NameError-prone and build a big barrier for children to learn a complex standard romanization system first to learn Python. Hye-Shik From tomerfiliba at gmail.com Sun Apr 30 17:18:41 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sun, 30 Apr 2006 17:18:41 +0200 Subject: [Python-3000] revamping the io stack, part 2 In-Reply-To: References: <1d85506f0604291210qd1493abk2ffb4ff859e3b906@mail.gmail.com> Message-ID: <1d85506f0604300818o1372e362kce41a0c52251d644@mail.gmail.com> > I don't > want to make the 90% case require hardly any memorizing of what > readers I need in what order. ... > See, this is what I am worried about. I **really** like not having to > figure out what I need to do to read by lines from a file. If the > FileStream object had an __iter__ that did the proper wrapping with > LinedBufferedStream, then great, I'm happy. But if we do not add some > reasonable convenience functions or iterators yes, i totally agree with that: we do need convenience functions. take a look at this: def file(filename, mode = "r", bufsize = None): # open the file f = FileStream(filename, "r") # add buffering if requested if bufsize is not None: f = BufferedStream(f, bufsize) return f def textfile(filename, *args): # add "text mode" return TextCodec(file(filename)) so today's file() remains in tact, but is accompanied by a textfile() counterpart, that opens in textmode. or we could add a "t" mode to the file's mode list, but that's ugly. and the TextCodec adds __iter__ over lines, so you *can* do for line in textfile("c:\\blah"): pass but not for line in file("c:\\blah"): pass because only text files have the notion of lines. i think you'd agree that it's meaningless to iterate by *lines* over arbitrary streams, like binary files or whatever. so it should be explicit, that you want to treat the data as text. and don't forget line translation could corrupt your binary data, etc. > this is going to feel > rather heavy-handed rather quickly. so i hope it doesn't seem heavy or too complex now. greetings, -tomer On 4/29/06, Brett Cannon wrote: > On 4/29/06, tomer filiba wrote: > > i first thought on focusing on the socket module, because it's the part that > > bothers me most, but since people have expressed their thoughts on > > completely > > revamping the IO stack, perhaps we should be open to adopting new ideas, > > mainly from the java/.NET world (keeping the momentum from the previous > > post). > > > > there is an inevitable issue of performance here, since it basically splits > > what used to be "file" or "socket" into many layers... each adding > > additional > > overhead, so many parts should be lowered to C. > > > > if we look at java/.NET for guidance, they have come up with two concepts: > > I am a little weary of taking too much from Java/.NET since I have > always found the I/O system way too heavy for the common case. I > can't remember what it takes to get a reader in Java in order to read > by lines. In Python, I love that I don't have to think about that; > just pass a file object to 'for' and I am done. > > While I am all for allowing for more powerful I/O through stacking a > stream within various readers (which feels rather functional to me, > but that must just be because of my latest reading material), I don't > want to make the 90% case require hardly any memorizing of what > readers I need in what order. > > > * stream - an arbitrary, usually sequential, byte data source > > * readers and writers - the way data is encoded into/decoded from the > > stream. > > we'll use the term "codec" for these readers and writers in general. > > > > so "stream" is the "where" and "codec" is the "how", and the concept of > > codecs is not limited to ASCII vs UTF-8. it can grow into fully-fledged > > protocols. > [SNIP - a whole lot of detailed ideas] > > ----- > > > > buffering is always *explicit* and implemented at the interpreter level, > > rather than by libc, so it is consistent between all platforms and streams. > > all streams, by nature, and *non-buffered* (write the data as soon as > > possible). buffering wraps an underlying stream, making it explicit > > > > class BufferedStream(Stream): > > def __init__(self, stream, bufsize) > > def flush(self) > > > > (BufferedStream appears in .NET) > > > > class LineBufferedStream(BufferedStream): > > def __init__(self, stream, flush_on = b"\n") > > > > f = LineBufferedStream(FileStream("c:\\blah")) > > > > where flush_on specifies the byte (or sequence of bytes?) to flush upon > > writing. by default it would be on newline. > > > > See, this is what I am worried about. I **really** like not having to > figure out what I need to do to read by lines from a file. If the > FileStream object had an __iter__ that did the proper wrapping with > LinedBufferedStream, then great, I'm happy. But if we do not add some > reasonable convenience functions or iterators, this is going to feel > rather heavy-handed rather quickly. > > -Brett > From tomerfiliba at gmail.com Sun Apr 30 17:33:02 2006 From: tomerfiliba at gmail.com (tomer filiba) Date: Sun, 30 Apr 2006 17:33:02 +0200 Subject: [Python-3000] revamping the io stack, part 2 In-Reply-To: <1d85506f0604300818o1372e362kce41a0c52251d644@mail.gmail.com> References: <1d85506f0604291210qd1493abk2ffb4ff859e3b906@mail.gmail.com> <1d85506f0604300818o1372e362kce41a0c52251d644@mail.gmail.com> Message-ID: <1d85506f0604300833g76cd9791u8f9d8341cd5447d4@mail.gmail.com> and one small thing i forgot to mention -- file.read/write work with the new bytes() type, while textfile.read/write work with strings (depends on the encoding) -tomer On 4/30/06, tomer filiba wrote: > > I don't > > want to make the 90% case require hardly any memorizing of what > > readers I need in what order. > ... > > See, this is what I am worried about. I **really** like not having to > > figure out what I need to do to read by lines from a file. If the > > FileStream object had an __iter__ that did the proper wrapping with > > LinedBufferedStream, then great, I'm happy. But if we do not add some > > reasonable convenience functions or iterators > > yes, i totally agree with that: we do need convenience functions. > take a look at this: > > def file(filename, mode = "r", bufsize = None): > # open the file > f = FileStream(filename, "r") > # add buffering if requested > if bufsize is not None: > f = BufferedStream(f, bufsize) > return f > > def textfile(filename, *args): > # add "text mode" > return TextCodec(file(filename)) > > so today's file() remains in tact, but is accompanied by a textfile() > counterpart, that opens in textmode. or we could add a "t" mode to > the file's mode list, but that's ugly. > > and the TextCodec adds __iter__ over lines, so you *can* do > for line in textfile("c:\\blah"): > pass > > but not > for line in file("c:\\blah"): > pass > > because only text files have the notion of lines. i think you'd agree > that it's meaningless to iterate by *lines* over arbitrary streams, like > binary files or whatever. so it should be explicit, that you want to > treat the data as text. and don't forget line translation could corrupt > your binary data, etc. > > > this is going to feel > > rather heavy-handed rather quickly. > so i hope it doesn't seem heavy or too complex now. > > > greetings, > -tomer > > On 4/29/06, Brett Cannon wrote: > > On 4/29/06, tomer filiba wrote: > > > i first thought on focusing on the socket module, because it's the part that > > > bothers me most, but since people have expressed their thoughts on > > > completely > > > revamping the IO stack, perhaps we should be open to adopting new ideas, > > > mainly from the java/.NET world (keeping the momentum from the previous > > > post). > > > > > > there is an inevitable issue of performance here, since it basically splits > > > what used to be "file" or "socket" into many layers... each adding > > > additional > > > overhead, so many parts should be lowered to C. > > > > > > if we look at java/.NET for guidance, they have come up with two concepts: > > > > I am a little weary of taking too much from Java/.NET since I have > > always found the I/O system way too heavy for the common case. I > > can't remember what it takes to get a reader in Java in order to read > > by lines. In Python, I love that I don't have to think about that; > > just pass a file object to 'for' and I am done. > > > > While I am all for allowing for more powerful I/O through stacking a > > stream within various readers (which feels rather functional to me, > > but that must just be because of my latest reading material), I don't > > want to make the 90% case require hardly any memorizing of what > > readers I need in what order. > > > > > * stream - an arbitrary, usually sequential, byte data source > > > * readers and writers - the way data is encoded into/decoded from the > > > stream. > > > we'll use the term "codec" for these readers and writers in general. > > > > > > so "stream" is the "where" and "codec" is the "how", and the concept of > > > codecs is not limited to ASCII vs UTF-8. it can grow into fully-fledged > > > protocols. > > [SNIP - a whole lot of detailed ideas] > > > ----- > > > > > > buffering is always *explicit* and implemented at the interpreter level, > > > rather than by libc, so it is consistent between all platforms and streams. > > > all streams, by nature, and *non-buffered* (write the data as soon as > > > possible). buffering wraps an underlying stream, making it explicit > > > > > > class BufferedStream(Stream): > > > def __init__(self, stream, bufsize) > > > def flush(self) > > > > > > (BufferedStream appears in .NET) > > > > > > class LineBufferedStream(BufferedStream): > > > def __init__(self, stream, flush_on = b"\n") > > > > > > f = LineBufferedStream(FileStream("c:\\blah")) > > > > > > where flush_on specifies the byte (or sequence of bytes?) to flush upon > > > writing. by default it would be on newline. > > > > > > > See, this is what I am worried about. I **really** like not having to > > figure out what I need to do to read by lines from a file. If the > > FileStream object had an __iter__ that did the proper wrapping with > > LinedBufferedStream, then great, I'm happy. But if we do not add some > > reasonable convenience functions or iterators, this is going to feel > > rather heavy-handed rather quickly. > > > > -Brett > > > From solipsis at pitrou.net Sun Apr 30 17:43:52 2006 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 30 Apr 2006 17:43:52 +0200 Subject: [Python-3000] revamping the io stack, part 2 In-Reply-To: <1d85506f0604291210qd1493abk2ffb4ff859e3b906@mail.gmail.com> References: <1d85506f0604291210qd1493abk2ffb4ff859e3b906@mail.gmail.com> Message-ID: <1146411832.5713.21.camel@fsol> Le samedi 29 avril 2006 ? 21:10 +0200, tomer filiba a ?crit : > and if we do that already, perhaps we should introduce async > operations as a > built-in feature? .NET does (BeginRead, EndRead, etc.) > def async_read(self, count, callback) > def async_write(self, data, callback) I think many people would be happy if the proposal would use the Deferred abstraction which is built in Twisted. :-) > many protocols can also be represented as codecs. textual protocols, > like > HTTP or SMTP, can be easily implemented that way: > > class HttpClientCodec( *TextCodec* ): > def __init__(self, stream): > TextCodec.__init__(self, stream, textcodec = "ascii") HTTP can transport binary (non-text) data as well, while having text headers. How does your proposal address this point? From guido at python.org Sun Apr 30 18:28:30 2006 From: guido at python.org (Guido van Rossum) Date: Sun, 30 Apr 2006 09:28:30 -0700 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: <4453FBFE.7060001@v.loewis.de> References: <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> <4452D871.9020901@v.loewis.de> <4453FBFE.7060001@v.loewis.de> Message-ID: On 4/29/06, "Martin v. L?wis" wrote: > Python 2.3.5 (#2, Mar 6 2006, 10:12:24) > [GCC 4.0.3 20060304 (prerelease) (Debian 4.0.2-10)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > py> import locale > py> locale.setlocale(locale.LC_ALL, "") > 'de_DE at euro' > py> l?wis=1 > py> print l?wis > 1 But a file with "l?wis=1" in it causes a syntax error (even if an encoding is specified). I believe this is a quirk of interactive mode only. Certainly the language spec doesn't intend to allow this. > > I do think that *eventually* we'll have to support this. But I don't > > think Python needs to lead the pack here; I don't think the tools are > > ready yet. > > Python doesn't really lead here. The C family of languages (C, C++, > Java, C#) all have Unicode identifiers, so there is plenty of > experience. Primarily, the experience is that the feature isn't > used much, because of obstacles I think we can overcome (primarily, > that all these languages make the source encoding > implementation-defined; we don't, as we put the source encoding into > the source file). I still think it's premature. In any case, it doesn't strike me as something that needs to be synchronized with Py3k -- it could be introduced earlier or later since it introduces no backwards compatibility. Python can respond much more agile here than most other languages. I can also see this as a Jython or IronPython "language extension" -- after all, if Java and C# support unicode identifiers e.g. for class names, there will be a need to support importing those classes... That way some experience can be gained. -- --Guido van Rossum (home page: http://www.python.org/~guido/) From martin at v.loewis.de Sun Apr 30 20:39:35 2006 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Sun, 30 Apr 2006 20:39:35 +0200 Subject: [Python-3000] Unicode identifiers (Was: sets in P3K?) In-Reply-To: References: <4451AF74.8040702@gmx.net> <445279F7.8080306@v.loewis.de> <4452916F.1030304@v.loewis.de> <4452D871.9020901@v.loewis.de> <4453FBFE.7060001@v.loewis.de> Message-ID: <44550467.20304@v.loewis.de> Guido van Rossum wrote: > But a file with "l?wis=1" in it causes a syntax error (even if an > encoding is specified). That's because it gets converted to UTF-8 first, and then the UTF-8 bytes don't count as Latin-1 letters. > I believe this is a quirk of interactive mode only. Certainly the > language spec doesn't intend to allow this. Only insofar as it doesn't do the to-UTF-8 conversion: UTF-8, by design, has very little overlap with any other encoding, so it is unlikely that the UTF-8 version of some character would satisfy isalnum for all bytes in some encoding. If you are curious, I'll try to construct an example where (certain) non-ASCII characters can be used in source code if the locale is set to the "right" value. So my point is that the tokenizer shouldn't use isalnum to find out what characters are valid in an identifier. > I still think it's premature. In any case, it doesn't strike me as > something that needs to be synchronized with Py3k -- it could be > introduced earlier or later since it introduces no backwards > compatibility. Python can respond much more agile here than most other > languages. Ok. I was only worried about your change in PEP 3099: "Python won't use Unicode characters for anything except string literals or comments." If that is only meant to say "this won't be introduced in Python 3", I'm fine with it. Regards, Martin From janssen at parc.com Sun Apr 30 21:55:31 2006 From: janssen at parc.com (Bill Janssen) Date: Sun, 30 Apr 2006 12:55:31 PDT Subject: [Python-3000] PyGUI in Py3K stdlib, more details Message-ID: <06Apr30.125533pdt."58641"@synergy1.parc.xerox.com> I've looked over the PyGUI code a bit more carefully. It still looks good to me as a Py3K standard portable GUI candidate. In particular, it doesn't look like it would be hard to port to Java Swing (for Jython) and Windows.Forms (for IronPython on .NET and Mono). Might make good summer projects... A few things that need to be done (IMO): 1) I'd add some kind of standard analog value control class, similar to Swing's JSlider or wxWidget's wxSlider or Cocoa's NSSlider. This can be implemented in pure Python (I've done it), but it's such a common widget in underlying toolkits that it should be possible to use that underlying widget when porting to a new widget platform. 2) There needs to be some kind of combobox multiple-value choice widget other than RadioButtons, more like wxChoice or non-editable JComboBox. 3) I'd implement (2) by slightly generalizing the Menu class, which would also allow them to be used for pop-up menus. 4) The widget class hierarchy should probably be re-worked a bit to use more abstract mix-in classes. For instance, the PyGUI Container class should probably be abstract (rather than inheriting from Component), and the View class should probably mix it in along with Component. This would make it easier to implement new components in pure Python. I think I see why the current implementation exists, because the toolkit dates from the pre-grand-type-unification days, but this should be spruced up. 5) There should probably be a new class to permit rendering into a PDF stream, for printing. This could either be a new subclass of Canvas (probably would have to be), or share a common base class with Pixmap, which already exists. Perhaps Pixmap should be a subclass of Canvas. I'd need to look at more use cases. An editable styled-text widget would be interesting, instead of the simpler editable text widget that already exists. It would put more demands on the underlying widget platform, but at least four (GTK+, Swing, Cocoa, wxWidgets) all support this kind of widget. It would also be useful to have pure-Python implementations of all the widgets, to make porting it to a new platform dirt simple. Just port the Canvas and Window (I think), and you already have working UIs. Doesn't have to look great, at least to start out with. By the way, I would not include the OpenGL widgets currently in PyGUI into the standard GUI. At least, not yet -- they are still clearly being experimented with. Down the road, I think it's a good idea -- every platform needs an OpenGL interface, and it makes sense to integrate it with the standard GUI. Bill ------------------------------------------ Bill Janssen (650) 812-4763 FAX: (650) 812-4258 PARC -- User Interface Research Area http://www.parc.com/janssen/ From gvwilson at cs.utoronto.ca Sun Apr 30 22:16:09 2006 From: gvwilson at cs.utoronto.ca (Greg Wilson) Date: Sun, 30 Apr 2006 16:16:09 -0400 (EDT) Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: > Ron Adam: > How about the '!' which is currently used indicate a python long and an > integer. Here it would indicate the difference between a frozen > container and a mutable container. > frozen_set = {1, 2, 3}! > frozen_dict = {1:'a', 2:'b', 3:'c'}! Greg Wilson: I mostly don't like suffix operators --- too easy to miss them when reading code (especially if the thing they're being applied to is more than a couple of symbols long). > Nick Coghlan: > set() == set{} > set(x) # No braced equivalent > set([x]) == set{x} # Ignoring list comprehensions > set((x,)) == set{x} > set((a, b, c)) == set{a, b, c} I would rather stick to what we have than introduce two notations for construction, call, subscripting, etc. --- Perl's [] vs. {} *always* causes headaches for newcomers. Thanks, Greg From g.brandl at gmx.net Sun Apr 30 22:22:32 2006 From: g.brandl at gmx.net (Georg Brandl) Date: Sun, 30 Apr 2006 22:22:32 +0200 Subject: [Python-3000] sets in P3K? In-Reply-To: References: Message-ID: Greg Wilson wrote: >> Ron Adam: >> How about the '!' which is currently used indicate a python long and an >> integer. Here it would indicate the difference between a frozen >> container and a mutable container. >> frozen_set = {1, 2, 3}! >> frozen_dict = {1:'a', 2:'b', 3:'c'}! > > Greg Wilson: > I mostly don't like suffix operators --- too easy to miss them when > reading code (especially if the thing they're being applied to is more > than a couple of symbols long). > >> Nick Coghlan: >> set() == set{} >> set(x) # No braced equivalent >> set([x]) == set{x} # Ignoring list comprehensions >> set((x,)) == set{x} >> set((a, b, c)) == set{a, b, c} > > I would rather stick to what we have than introduce two notations for > construction, call, subscripting, etc. --- Perl's [] vs. {} *always* > causes headaches for newcomers. And last time I looked they were gonna change it for Perl 6. Georg From qrczak at knm.org.pl Sun Apr 30 22:32:53 2006 From: qrczak at knm.org.pl (Marcin 'Qrczak' Kowalczyk) Date: Sun, 30 Apr 2006 22:32:53 +0200 Subject: [Python-3000] Iterables and scalars was: sets in P3K? In-Reply-To: (Guido van Rossum's message of "Wed, 26 Apr 2006 15:22:49 -0700") References: <20060425141630.6qz3ngrjy81wgco4@login.werra.lunarpages.com> <444EDBE8.3020303@gradient.cis.upenn.edu> <444FB6EB.5050407@ewtllc.com> Message-ID: <877j56olca.fsf@qrnik.zagroda> "Guido van Rossum" writes: > In general dict should not be looked at as an example: the dict > constructor signature is already too far overloaded, giving a > different meaning to keyword args, and distinguishing between a > mapping and a sequence is iffy: it requires a "keys" method which is > kind of sub-optimal compared to using "iteritems" if it existed... This would be solved if iteration over a dict produced key/value pairs. -- __("< Marcin Kowalczyk \__/ qrczak at knm.org.pl ^^ http://qrnik.knm.org.pl/~qrczak/ From robinbryce at gmail.com Sun Apr 30 23:37:40 2006 From: robinbryce at gmail.com (Robin Bryce) Date: Sun, 30 Apr 2006 22:37:40 +0100 Subject: [Python-3000] revamping the io stack, part 2 In-Reply-To: <1146411832.5713.21.camel@fsol> References: <1d85506f0604291210qd1493abk2ffb4ff859e3b906@mail.gmail.com> <1146411832.5713.21.camel@fsol> Message-ID: >> def async_read(self, count, callback) >> def async_write(self, data, callback) >I think many people would be happy if the proposal would use the >Deferred abstraction which is built in Twisted. :-) I almost agree. Not sure whether a Defered implementation in python core would be preferable to: def async_read(self, count, callback, errback=None) def async_write(self, data, callback, errback=None) The addition of the errback, which need not be specified, would be sufficient to hook into twisted defereds, or any other implementation of that pattern. If errback is None I would expect the exception to be passed as the result for callback. Alternately, if callback was a generator instance, it could be 'resumed' with the exception and we could forget all about defereds/errbacks. If practical, this would definitely get +1 from me. Robin -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-3000/attachments/20060430/90282fc3/attachment.htm