From arigo at tunes.org Sat May 1 03:08:48 2004 From: arigo at tunes.org (Armin Rigo) Date: Sat May 1 03:11:58 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200404302313.i3UND0a08111@guido.python.org> References: <200404231826.i3NIQxM18955@guido.python.org> <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200404301216.i3UCGXU07211@guido.python.org> <20040430130700.GA28030@vicky.ecs.soton.ac.uk> <200404302313.i3UND0a08111@guido.python.org> Message-ID: <20040501070848.GA23938@vicky.ecs.soton.ac.uk> Hello Guido, On Fri, Apr 30, 2004 at 04:12:59PM -0700, Guido van Rossum wrote: > That's not new, we all know that that's the downside of late binding. > > My counter to that is that *any* use of genexps where the consumer > doesn't consume (or discard) the iterator before the next line is > reached is extremely advanced use. My argument was that late-binding make it even more mysterious, not less suprizing, because in the minority of cases where a listcomp is not used immediately but still need a genexpr for performance, you have to think carefully. Well, I insist a bit because I cannot helping thinking that the arguments for early-binding are "that's useful in a very small number of cases", and the arguments for late-binding are "you don't want to do that anyway so let's pick the semantic of free vars in functions". I don't immediately think of genexps as similar to functions, rather closer to listcomps, even if I know that they are functions under the hood. Armin From dmgass at hotmail.com Sat May 1 15:31:11 2004 From: dmgass at hotmail.com (Dan Gass) Date: Sat May 1 15:31:15 2004 Subject: [Python-Dev] Proposal: A more powerful alternative to ConfigParser Message-ID: Per suggestions posted on this mailing list I created a sourceforge project (config-py), registered it with PyPi and submitted the following announcement to python-announce. I look forward to your feedback. Thanks, Dan Gass Announcement ------------ config-py version 1.0 has been released and is available at SourceForge. This is the initial release of a Python module for reading python based user configuration files to control script execution. (It is a more powerful alternative to Python's ConfigParser module). The long term goal is to refine this module for eventual inclusion in the standard Python distribution. Module Description ------------------ This python module provides a simple, but extremely flexible and powerful method of controlling python script execution by use of configuration settings stored in user configuration files. This module is a more powerful alternative to the ConfigParser module already available in the standard Python distribution. The standard ConfigParser module parses text based user configuration files and provides access to that information. This module reads python based configuration files and provides access to the python objects contained within. In addition, this module supports hierarchical organization of the settings within the configuration files so that settings may be accessed using keys specified by the user from the command line or by other means. By use of python based configuration files, this module provides a simple solution to most configuration problems but also has the flexibility to solve the most complex configuration issues. Other Important Information --------------------------- License: GNU General Public License (GPL) Project Home Page: http://config-py.sourceforge.net/ Download URL: https://sourceforge.net/projects/config-py/ Author/Maintainer: Dan Gass (dmgass@hotmail.com) _________________________________________________________________ Check out the coupons and bargains on MSN Offers! http://youroffers.msn.com From greg at cosc.canterbury.ac.nz Sun May 2 23:03:11 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun May 2 23:07:21 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <20040430104158.GA31871@vicky.ecs.soton.ac.uk> Message-ID: <200405030303.i4333Bxw019255@cosc353.cosc.canterbury.ac.nz> Armin Rigo : > Early binding is closer to the idea that turning a listcomp into a > genexprs should just work if you only iterate once on the result. I'm not sure that's an idea we should be promulgating in the first place. The original motivation for generator expressions was things like total = sum(x**2 for x in stuff) where not only will the sequence be used just once, but it will be used *immediately* before doing anything else. Those are the use cases we need to advertise this as targeting, I think, not "any listcomp that's only used once". Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Sun May 2 23:05:46 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun May 2 23:07:24 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <5.2.1.1.0.20040430142620.030304e8@pop.bluewin.ch> Message-ID: <200405030305.i4335kaL019266@cosc353.cosc.canterbury.ac.nz> Samuele Pedroni : > "There should be one-- and preferably only one --obvious way to do > it." of course this is the case for generator expressions and list > comprehsensions because they are separated by performance > characteristics and somehow by the fact that one directly produces a > list. I suspect that we have to learn to live with the fact that > they are distinguished by (just) that, both a nuisance (sometimes > people don't want to bother to choose) and a feature. I just had a thought. What if generator expressions were allowed *only* as a parameter to a function call? Would that help avoid leading people into the trap of using them where they're not appropriate? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Sun May 2 23:23:23 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun May 2 23:27:11 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <20040430093439.138D.JCARLSON@uci.edu> Message-ID: <200405030323.i433NNKj019302@cosc353.cosc.canterbury.ac.nz> Josiah Carlson : > Semantically, in the first case you are modifying your dictionary, which > has consequences to anything that has a reference to the object. Mutable > modification during runtime should be something we are used to by now. The point is that anyone who's going to be confused by the difference between early and late binding behaviour is likely to be equally confused by this. The issue is really about when expressions are evaluated, not when names are bound. Patching up just one part of it (binding of free variables) doesn't really fix the problem. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Sun May 2 23:27:41 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun May 2 23:30:42 2004 Subject: [Python-Dev] Accessing base class C struct of subclass instance In-Reply-To: <6E9B692A-9AD0-11D8-961D-000393CBDF94@opnet.com> Message-ID: <200405030327.i433RfBS019314@cosc353.cosc.canterbury.ac.nz> Nick Bastin : > If I define a new style class in C, and subclass it in python, how can > I get the base class instance object from a subclass instance in C? What do you mean by "base class instance object"? That's a meaningless combination of words to me. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Sun May 2 23:41:14 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun May 2 23:42:36 2004 Subject: [Python-Dev] Re: PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: Message-ID: <200405030341.i433fERl019326@cosc353.cosc.canterbury.ac.nz> "Russell E. Owen" : > Personally, I'd only like to see generator expressions added to Python > if they can allow list comprehensions to be deprecated -- to avoid > language clutter. That's possible in the sense that a list comprehension can be written as list(generator_expression). But it will still be necessary to decide whether to wrap a list() around it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Mon May 3 00:08:35 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 3 00:08:06 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: Your message of "Mon, 03 May 2004 15:03:11 +1200." <200405030303.i4333Bxw019255@cosc353.cosc.canterbury.ac.nz> References: <200405030303.i4333Bxw019255@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405030408.i4348Zk01989@guido.python.org> > Armin Rigo : > > > Early binding is closer to the idea that turning a listcomp into a > > genexprs should just work if you only iterate once on the result. Greg Ewing: > I'm not sure that's an idea we should be promulgating in the first > place. The original motivation for generator expressions was things > like > > total = sum(x**2 for x in stuff) > > where not only will the sequence be used just once, but it will be > used *immediately* before doing anything else. > > Those are the use cases we need to advertise this as targeting, I > think, not "any listcomp that's only used once". Right -- thanks for saying it so clearly! Early binding seems to be trying to solve a problem with a use for genexprs that's questionable at best. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon May 3 00:09:18 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 3 00:08:55 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: Your message of "Mon, 03 May 2004 15:05:46 +1200." <200405030305.i4335kaL019266@cosc353.cosc.canterbury.ac.nz> References: <200405030305.i4335kaL019266@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405030409.i4349IP02000@guido.python.org> > What if generator expressions were allowed *only* as a > parameter to a function call? > > Would that help avoid leading people into the trap of > using them where they're not appropriate? Nah, people would just use an identity function to work around the restriction. --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Mon May 3 09:47:08 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon May 3 09:41:36 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200405030408.i4348Zk01989@guido.python.org> References: <200405030303.i4333Bxw019255@cosc353.cosc.canterbury.ac.nz> Message-ID: <5.2.1.1.0.20040503154238.027cd650@pop.bluewin.ch> At 21:08 02.05.2004 -0700, Guido van Rossum wrote: > > Armin Rigo : > > > > > Early binding is closer to the idea that turning a listcomp into a > > > genexprs should just work if you only iterate once on the result. > >Greg Ewing: > > I'm not sure that's an idea we should be promulgating in the first > > place. The original motivation for generator expressions was things > > like > > > > total = sum(x**2 for x in stuff) > > > > where not only will the sequence be used just once, but it will be > > used *immediately* before doing anything else. > > yes, but the question is how much easy is to grasp what "used *immediately*" means for a genexp, or IOW how much education is needed. this looks innocent enough: l = list_of_my_strings() for pfx in discard_prefixes: l = ( x for x in l if not x.startswith(pfx) ) # this is not using l *immediately* use_strings(l) > > Those are the use cases we need to advertise this as targeting, I > > think, not "any listcomp that's only used once". > >Right -- thanks for saying it so clearly! Early binding seems to be >trying to solve a problem with a use for genexprs that's questionable >at best. From jim.jewett at eds.com Mon May 3 09:58:01 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Mon May 3 10:01:04 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward Message-ID: Guido van Rossum: > My counter to that is that *any* use of genexps where the consumer > doesn't consume (or discard) the iterator before the next line is > reached is extremely advanced use. This might be worth putting in the PEP and the docs -- Normal usage of a generator expression is to immediately consume the entire sequence, or to discard the remainder. If portions of a generator *are* used later, programmers may need some extra precautions to ensure that all names still have the desired bindings. -jJ From aahz at pythoncraft.com Mon May 3 11:58:12 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon May 3 11:58:15 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <1082749292.9092.40.camel@localhost.localdomain> References: <1082746969.40896859bcf09@mcherm.com> <1082749292.9092.40.camel@localhost.localdomain> Message-ID: <20040503155812.GA9753@panix.com> On Fri, Apr 23, 2004, Jeremy Hylton wrote: > > That leads me to wonder what exactly the rationale for generator > expressions is. The PEP says that "time, clarity, and memory are > conserved by using an generator expression" but I can only see how > memory is conserved. That is, I don't find them any easier to read than > list comprehensions and I don't understand the performance implications > very well. It's not obvious to me that their faster. I've been skimming due to being out of town and catching up, but I haven't seen a direct response to Jeremy's question about the rationale. Jeremy, do you still want an answer? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "I used to have a .sig but I found it impossible to please everyone..." --SFJ From arigo at tunes.org Mon May 3 12:27:50 2004 From: arigo at tunes.org (Armin Rigo) Date: Mon May 3 12:31:18 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200405030303.i4333Bxw019255@cosc353.cosc.canterbury.ac.nz> References: <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200405030303.i4333Bxw019255@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040503162750.GA11346@vicky.ecs.soton.ac.uk> Hello Greg, On Mon, May 03, 2004 at 03:03:11PM +1200, Greg Ewing wrote: > I'm not sure that's an idea we should be promulgating in the first > place. The original motivation for generator expressions was things > like > > total = sum(x**2 for x in stuff) > > where not only will the sequence be used just once, but it will be > used *immediately* before doing anything else. I understand perfecty well that anything else is advanced stuff, and that people shouldn't do advanced stuff if they are not aware of all the issues. I also know that any kind of laziness together with mutating objects creates delicate problems. What I am looking for is a genexpr implementation that would, as a bonus, be useful in advanced cases too instead of being just unusable beyond these 95% of cases where the binding model doesn't matter. The problem I have with late-bound names is that small, semi-advanced, "innocent enough" examples like Samuele's keep popping up that would just give wrong answers (not just crash). Personally when I look at all these examples I think they are elegant ways to express things that I could want to do, and I could easily read and write this kind of code -- but it just doesn't do that I expect it to. The real answer is much more convoluted. The only advantage I see to late binding is that they prevent people from doing any advanced stuff with genexprs, and thus beginners never have to read code containing advanced uses of genexprs (even clean-looking one like Samuele's). Armin From guido at python.org Mon May 3 13:14:04 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 3 13:14:12 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: Your message of "Mon, 03 May 2004 17:27:50 BST." <20040503162750.GA11346@vicky.ecs.soton.ac.uk> References: <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200405030303.i4333Bxw019255@cosc353.cosc.canterbury.ac.nz> <20040503162750.GA11346@vicky.ecs.soton.ac.uk> Message-ID: <200405031714.i43HE4p03162@guido.python.org> > The problem I have with late-bound names is that small, > semi-advanced, "innocent enough" examples like Samuele's keep > popping up that would just give wrong answers (not just crash). Hm. I don't think I've seen a single example that wasn't specifically concocted by early-binding advocates (yours included -- you took random listcomps and mused on what would happen if they'd be genexprs instead). > Personally when I look at all these examples I think they are > elegant ways to express things that I could want to do, and I could > easily read and write this kind of code -- but it just doesn't do > that I expect it to. The real answer is much more convoluted. Remember that your brain is several times larger than that of most Python users. > The only advantage I see to late binding is that they prevent people > from doing any advanced stuff with genexprs, and thus beginners > never have to read code containing advanced uses of genexprs (even > clean-looking one like Samuele's). My problem with early binding is that it intruduces a new set of behind-the-scenes semantics that you cannot always ignore -- while the examples where it doesn't DWYM are even more obscure than the examples that it is intended to save, it means that you still have to think about the new semantics when doing advanced stuff -- and these new semantics are unlike anything seen before in Python: transformations that require a definition of "free variables" are not seen anywhere else (nested scopes use a different concept). --Guido van Rossum (home page: http://www.python.org/~guido/) From ark-mlist at att.net Mon May 3 15:19:39 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Mon May 3 15:19:36 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <20040503155812.GA9753@panix.com> Message-ID: <001501c43143$94a38210$6402a8c0@arkdesktop> > > That leads me to wonder what exactly the rationale for generator > > expressions is. The PEP says that "time, clarity, and memory are > > conserved by using an generator expression" but I can only see how > > memory is conserved. That is, I don't find them any easier to read than > > list comprehensions and I don't understand the performance implications > > very well. It's not obvious to me that their faster. > I've been skimming due to being out of town and catching up, but I > haven't seen a direct response to Jeremy's question about the rationale. > Jeremy, do you still want an answer? I can see two potentially important cases where generator expressions win big over list comprehensions: 1) Where the code that is consuming the sequence yielded by the generator expression terminates before consuming the entire sequence; 2) Where the code that is consuming the sequence is an online algorithm, and there is a potential delay between generating elements of the sequence. It is easier to find an example of the second case than of the first: foo(bar(line) for line in sys.stdin) If foo expects a generator as its input: def foo(x): for line in x: print line then using a generator expression instead of a list comprehension will cause the program to print each line of output after reading the corresponding line of input, rather than consuming all the input and then printing all the output. From arigo at tunes.org Mon May 3 15:17:48 2004 From: arigo at tunes.org (Armin Rigo) Date: Mon May 3 15:21:08 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200405031714.i43HE4p03162@guido.python.org> References: <20040430104158.GA31871@vicky.ecs.soton.ac.uk> <200405030303.i4333Bxw019255@cosc353.cosc.canterbury.ac.nz> <20040503162750.GA11346@vicky.ecs.soton.ac.uk> <200405031714.i43HE4p03162@guido.python.org> Message-ID: <20040503191748.GA12891@vicky.ecs.soton.ac.uk> Hello Guido, On Mon, May 03, 2004 at 10:14:04AM -0700, Guido van Rossum wrote: > (...) these new > semantics are unlike anything seen before in Python: transformations > that require a definition of "free variables" are not seen anywhere > else (nested scopes use a different concept). I think I see your point, although I'm not sure I see how nested scopes work if not with free variables ( = variables used but not bound in the current scope). Armin From tim.one at comcast.net Mon May 3 16:31:48 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon May 3 16:31:54 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200405031714.i43HE4p03162@guido.python.org> Message-ID: [Guido] > Hm. I don't think I've seen a single example that wasn't specifically > concocted by early-binding advocates (yours included -- you took > random listcomps and mused on what would happen if they'd be genexprs > instead). That's a bit of a revisionist stretch: the examples I trotted out last October *made* me an early-binding advocate at the time. For example, http://mail.python.org/pipermail/python-dev/2003-October/039323.html That example doesn't have lists of generators either, which is another thing you don't remember ever seeing . It's still the case that I haven't seen a plausible example where late-binding semantics both make a difference and are desirable, or where early-binding semantics both make a difference and are undesirable -- although the magical implicitness of both schemes leaves me queasy. ... > My problem with early binding is that it intruduces a new set of > behind-the-scenes semantics that you cannot always ignore -- while the > examples where it doesn't DWYM are even more obscure than the examples > that it is intended to save, it means that you still have to think > about the new semantics when doing advanced stuff -- and these new > semantics are unlike anything seen before in Python: transformations > that require a definition of "free variables" are not seen anywhere > else (nested scopes use a different concept). Late-binding in genexps can't be ignored either when doing advanced stuff, and are also unlike anything seen before in Python: transforming what looks like an expression into a delayed function body is also unprecendented (in the absence of a lambda). The fact that it "looks like" an expression is exactly what's so surprising about that "the current values" of p and pipe don't get used in the above example's # add a filter over the current pipe, and call that the new pipe pipe = (e for e in pipe if p(e)) It didn't strike me as an advanced use at the time, just as an example of something that seemed like it should be natural. It sounds like, today, "the current value" of pipe will be used after all, but the evaluation of p will get delayed? If so, that's not notably easy to explain either (or so it seems to me). From tdelaney at avaya.com Mon May 3 17:56:42 2004 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Mon May 3 17:56:50 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE01729514@au3010avexu1.global.avaya.com> Tim Peters wrote: > # add a filter over the current pipe, and call that the new pipe > pipe = (e for e in pipe if p(e)) This is a fairly convincing argument to me - that is something I do all the time with listcomps. Saves having to invent extra names. Tim Delaney From tim.one at comcast.net Mon May 3 22:30:30 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon May 3 22:30:38 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DE01729514@au3010avexu1.global.avaya.com> Message-ID: >> # add a filter over the current pipe, and call that the new pipe >> pipe = (e for e in pipe if p(e)) [Delaney, Timothy C (Timothy)] > This is a fairly convincing argument to me - Except it's not an argument, it's just a six-month old example. I don't have an argument to make here. I've observed repeatedly that in all "reasonable" examples to date where the distinction makes a difference, early binding does better than late binding, but I still haven't dreamt up a "deep reason" for that. The arguments on both sides have been singularly uncompelling (to me, anyway). > that is something I do all the time with listcomps. Saves having to > invent extra names. I may have lost track, but I think the last plan Guido approved of here would actually evaluate the target of the leftmost "for" ("pipe" in the example above) immediately, leaving only "p" to get picked up later. It's impossible for me to guess how that might affect the things you do all the time with listcomps; also impossible for me to guess whether you'd do similar things with genexps. This issue has always been long on specious arguments and short on concrete examples. But that's OK -- we'll get the concrete examples, but after it's too late . From jeremy at alum.mit.edu Mon May 3 22:42:53 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Mon May 3 22:44:54 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: References: Message-ID: <1083638573.17924.9.camel@localhost.localdomain> On Mon, 2004-05-03 at 09:58, Jewett, Jim J wrote: > Guido van Rossum: > > > My counter to that is that *any* use of genexps where the consumer > > doesn't consume (or discard) the iterator before the next line is > > reached is extremely advanced use. > > This might be worth putting in the PEP and the docs -- We are in rather dire need of docs and a revised PEP. > Normal usage of a generator expression is to immediately consume > the entire sequence, or to discard the remainder. > > If portions of a generator *are* used later, programmers may need > some extra precautions to ensure that all names still have the > desired bindings. There may be a general rule that functions that make use of free variables are best programmed in a functional style, where the function doesn't depend on side-effects to those variables. Jeremy From greg at cosc.canterbury.ac.nz Mon May 3 22:53:07 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon May 3 22:53:47 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: Message-ID: <200405040253.i442r7f5021005@cosc353.cosc.canterbury.ac.nz> Tim Peters : > This issue has always been long on specious arguments and short on concrete > examples. But that's OK -- we'll get the concrete examples, but after it's > too late . Maybe for a few releases they should have to be imported from __hypotheticalfuture__, a module whose contents are understood to be subject to merciless revision should they turn out to be a bad idea... Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From aahz at pythoncraft.com Tue May 4 00:02:47 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue May 4 00:02:50 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200405040253.i442r7f5021005@cosc353.cosc.canterbury.ac.nz> References: <200405040253.i442r7f5021005@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040504040247.GA11979@panix.com> On Tue, May 04, 2004, Greg Ewing wrote: > Tim Peters : >> >> This issue has always been long on specious arguments and short on concrete >> examples. But that's OK -- we'll get the concrete examples, but after it's >> too late . > > Maybe for a few releases they should have to be imported from > __hypotheticalfuture__, a module whose contents are understood to be > subject to merciless revision should they turn out to be a bad idea... You're probably joking, but +1 from me. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! From anthony at interlink.com.au Tue May 4 00:54:08 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue May 4 00:54:41 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: <1083638573.17924.9.camel@localhost.localdomain> References: <1083638573.17924.9.camel@localhost.localdomain> Message-ID: <409721F0.1020102@interlink.com.au> Jeremy Hylton wrote: > On Mon, 2004-05-03 at 09:58, Jewett, Jim J wrote: > >>Guido van Rossum: >> >> >>>My counter to that is that *any* use of genexps where the consumer >>>doesn't consume (or discard) the iterator before the next line is >>>reached is extremely advanced use. >> >>This might be worth putting in the PEP and the docs -- > > > We are in rather dire need of docs and a revised PEP. Indeed. The timeframe we came up with at PyCon had the generator expressions being landed "in the first week of May". At the moment the discussion seems to be spinning wheels -- it looks (to me) like there's not even 100% agreement from everyone that they should even go into 2.4 at all. I'm not fussed if we slip a week or two here, but if it slips another 6 weeks, say, then the release time line could be in some trouble. -- Anthony Baxter It's never too late to have a happy childhood. From gward at python.net Tue May 4 08:16:00 2004 From: gward at python.net (Greg Ward) Date: Tue May 4 08:16:04 2004 Subject: [Python-Dev] Re: Small additions to optparser In-Reply-To: <20040429194735.GA7068@burma.localdomain> References: <20040429194735.GA7068@burma.localdomain> Message-ID: <20040504121600.GB1680@cthulhu.gerg.ca> On 29 April 2004, Gustavo Niemeyer said: > I have two small customizations I've used in most projects I've > plugged optparse into. Since they're simple and generic, I'd > like to integrate them back in optparser. These features are: > > - Add a help formatter which uses capitalized headers. Sure, looks simple enough. > - Add a "help" keyword in OptionParser, allowing straightforward > addition of a custom help messages. How is that different from the existing 'description' that David Goodger added for Optik 1.4? > Comments? > > Also, how are we currently handling changes in optparser? Do we > have an "official" maintainer (Greg?). IOW, should I: As it says at the top of optparse.py: Originally distributed as Optik; see http://optik.sourceforge.net/ . If you have problems with this module, please do not file bugs, patches, or feature requests with Python; instead, use Optik's SourceForge project page: http://sourceforge.net/projects/optik For support, use the optik-users@lists.sourceforge.net mailing list (http://lists.sourceforge.net/lists/listinfo/optik-users). So either post your patches to optik-users@lists.sourceforge.net, or file 'em on the Optik tracker at SF. Nothing wrong with cc'ing python-dev at the same time, though. Greg -- Greg Ward http://www.gerg.ca/ All of science is either physics or stamp collecting. From niemeyer at conectiva.com Tue May 4 09:50:43 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue May 4 10:06:59 2004 Subject: [Python-Dev] Re: Small additions to optparser In-Reply-To: <20040504121600.GB1680@cthulhu.gerg.ca> References: <20040429194735.GA7068@burma.localdomain> <20040504121600.GB1680@cthulhu.gerg.ca> Message-ID: <20040504135043.GA3096@burma.localdomain> > > - Add a help formatter which uses capitalized headers. > > Sure, looks simple enough. Great! Thanks! > How is that different from the existing 'description' that David > Goodger added for Optik 1.4? Here is real example. I have the following help text: HELP = """\ Usage: repsys COMMAND [COMMAND ARGUMENTS] Useful commands: co submit create getspec getsrpm changed authoremail Run "repsys COMMAND --help" for more information. """ Using OptionParser(description=HELP) and "repsys --help", I get: """ Usage: repsys [options] Usage: repsys COMMAND [COMMAND ARGUMENTS] Useful commands: co submit create getspec getsrpm changed authoremail Run "repsys COMMAND --help" for more information. Options: --version show program's version number and exit -h, --help show this help message and exit --debug """ While using OptionParser(help=HELP) and "repsys --help", I get: """ Usage: repsys COMMAND [COMMAND ARGUMENTS] Useful commands: co submit create getspec getsrpm changed authoremail Run "repsys COMMAND --help" for more information. """ [...] > So either post your patches to optik-users@lists.sourceforge.net, or > file 'em on the Optik tracker at SF. Nothing wrong with cc'ing > python-dev at the same time, though. Will do that. Thanks! -- Gustavo Niemeyer http://niemeyer.net From ark-mlist at att.net Tue May 4 10:31:38 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Tue May 4 10:31:33 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: <1083638573.17924.9.camel@localhost.localdomain> Message-ID: <003401c431e4$831962c0$6402a8c0@arkdesktop> > There may be a general rule that functions that make use of free > variables are best programmed in a functional style, where the function > doesn't depend on side-effects to those variables. The troublesome cases tend to be ones where the function depends on *the lack* of side effects to free variables. From arigo at tunes.org Tue May 4 10:58:52 2004 From: arigo at tunes.org (Armin Rigo) Date: Tue May 4 11:01:52 2004 Subject: [Python-Dev] negative modular exponentiation Message-ID: <20040504145852.GA27574@vicky.ecs.soton.ac.uk> Hello, SF patch 936813 (fast modular exponentiation) introduces some optimization of exponentiation with longs, including a new feature: in a pow(x,y,z), the exponent y is allowed to be negative when it makes mathematical sense. The result of (x ** -y) % z is a number that is the inverse of (x ** y) % z, i.e. multiplying the two gives 1 modulo z. This is a very convenient feature if you are doing modulo arithmetic. Quoting Trevor Perrin, the patch's author: Libraries like GMP and LibTomMath work the same way. Being able to take inverses mod a number is useful for cryptography (e.g. RSA, DSA, and Elgamal). As this is a new feature it should be approved of. If it is, I'd volunteer to review the patch and make the necessary extension to the plain integer's pow() so that they work the same way. E.g. >>> pow(2,-1,9) 5 because (5*2)%9 == 1. Currently this gives a TypeError (why not a ValueError?) so changing the behaviour seems safe. Armin From guido at python.org Tue May 4 11:33:53 2004 From: guido at python.org (Guido van Rossum) Date: Tue May 4 11:33:59 2004 Subject: [Python-Dev] negative modular exponentiation In-Reply-To: Your message of "Tue, 04 May 2004 15:58:52 BST." <20040504145852.GA27574@vicky.ecs.soton.ac.uk> References: <20040504145852.GA27574@vicky.ecs.soton.ac.uk> Message-ID: <200405041533.i44FXsD04839@guido.python.org> > SF patch 936813 (fast modular exponentiation) introduces some > optimization of exponentiation with longs, including a new feature: > in a pow(x,y,z), the exponent y is allowed to be negative when it > makes mathematical sense. The result of (x ** -y) % z is a number > that is the inverse of (x ** y) % z, i.e. multiplying the two gives > 1 modulo z. This is a very convenient feature if you are doing > modulo arithmetic. Quoting Trevor Perrin, the patch's author: > > Libraries like GMP and LibTomMath work the same way. > Being able to take inverses mod a number is useful for > cryptography (e.g. RSA, DSA, and Elgamal). > > As this is a new feature it should be approved of. If it is, I'd > volunteer to review the patch and make the necessary extension to > the plain integer's pow() so that they work the same way. > > E.g. > > >>> pow(2,-1,9) > 5 > > because (5*2)%9 == 1. Currently this gives a TypeError (why not a > ValueError?) so changing the behaviour seems safe. This seems of little practical value (especially since real crypto wizards only complain that Python's longs are so much slower than gmp), but I can't see it could possibly break something, so a +0 from me. (Hm, it could break the expectation that pow(x, y, z), when defined, equals x**y%z. At the very least the docs need to be updated.) --Guido van Rossum (home page: http://www.python.org/~guido/) From anthony at interlink.com.au Tue May 4 13:16:27 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue May 4 13:17:29 2004 Subject: [Python-Dev] Python 2.3.4 schedule Message-ID: <4097CFEB.1090500@interlink.com.au> My plan is to cut a release candidate for 2.3.4 next Thursday (13th April) Australian time (so Wednesday night for the merkins), and a release the following Thursday (the 20th). Can we say that the 2.3 branch is frozen from Tuesday? As this is likely to be the final 2.3 release (modulo any suprising new bugs), I'd like the time to try and nail this one. As per normal, I'll be on #python-dev on freenode for the next two weeks or so. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From mwh at python.net Tue May 4 13:29:47 2004 From: mwh at python.net (Michael Hudson) Date: Tue May 4 13:29:51 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <4097CFEB.1090500@interlink.com.au> (Anthony Baxter's message of "Wed, 05 May 2004 03:16:27 +1000") References: <4097CFEB.1090500@interlink.com.au> Message-ID: <2my8o8p0c4.fsf@starship.python.net> Anthony Baxter writes: > My plan is to cut a release candidate for 2.3.4 next > Thursday (13th April) Australian time (so Wednesday night > for the merkins), and a release the following Thursday > (the 20th). You mean May, presumably? Cheers, mwh -- In short, just business as usual in the wacky world of floating point . -- Tim Peters, comp.lang.python From tim.one at comcast.net Tue May 4 13:31:20 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 4 13:31:23 2004 Subject: [Python-Dev] negative modular exponentiation In-Reply-To: <20040504145852.GA27574@vicky.ecs.soton.ac.uk> Message-ID: [Armin Rigo] > SF patch 936813 (fast modular exponentiation) introduces some > optimization of exponentiation with longs, including a new feature: > in a pow(x,y,z), the exponent y is allowed to be negative when it > makes mathematical sense. The result of (x ** -y) % z is a number that > is the inverse of (x ** y) % z, i.e. multiplying the two gives 1 > modulo z. ... > As this is a new feature it should be approved of. It's fine by me, and new features are finer by me than adding ever more obscure optimizations in longobject.c. The Python long implementation has a great track record wrt portability and lack of bugs, and gonzo optimizations work against that. For example, did Python really need the Karatsuba multiplication gimmick? I added the patch for it after a lot of reworking, but overall I'm not sure it was a net win; and the current patch in question boosts the smallest integer at which Karatsuba triggers to such a large value that anyone reaching it would probably be much better off using a GMP wrapper anyway. > If it is, I'd volunteer to review the patch One thing I noticed a few times in the patch was neglecting to check C calls for error returns. Even _PyLong_NumBits() can fail! > and make the necessary extension to the plain integer's pow() so that > they work the same way. > > E.g. > > >>> pow(2,-1,9) > 5 > > because (5*2)%9 == 1. Currently this gives a TypeError (why not a > ValueError?) so changing the behaviour seems safe. ValueError would make more sense to me too. I expect it was TypeError for compatibility with older exceptions raised by the float version. From fdrake at acm.org Tue May 4 13:32:46 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue May 4 13:33:00 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <2my8o8p0c4.fsf@starship.python.net> References: <4097CFEB.1090500@interlink.com.au> <2my8o8p0c4.fsf@starship.python.net> Message-ID: <200405041332.46873.fdrake@acm.org> Anthony Baxter writes: > My plan is to cut a release candidate for 2.3.4 next > Thursday (13th April) Australian time (so Wednesday night On Tuesday 04 May 2004 01:29 pm, Michael Hudson wrote: > You mean May, presumably? We just let the Aussies *think* they're a few hours ahead of us, but it looks like the truth is out... -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one at comcast.net Tue May 4 13:47:28 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 4 13:47:32 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <4097CFEB.1090500@interlink.com.au> Message-ID: [Anthony Baxter] > My plan is to cut a release candidate for 2.3.4 next > Thursday (13th April) Australian time (so Wednesday night ^^May in North America > for the merkins), and a release the following Thursday > (the 20th). > > Can we say that the 2.3 branch is frozen from Tuesday? As > this is likely to be the final 2.3 release (modulo any > suprising new bugs), I'd like the time to try and nail > this one. I tried to ask about 2.3.4 plans earlier here today, but because I used a different account to mail to python-dev, it's still sitting in a moderator queue. The same msg can be found here: http://mail.zope.org/pipermail/zope-dev/2004-May/022838.html Short course is that a serious bug with threads on Linux was reported almost a year ago, against Python 2.2.2: http://www.python.org/sf/756924 but wasn't properly diagnosed before last night (by Andrew Langmead; he added his diagnosis to the Python bug report today). I know this affects the OP, and affects Zope on Linux systems using LinuxThreads (but not those using NPTL). I don't know whether anyone is actively pursuing a workaround. If someone is, or if a LinuxHead on python-dev wants to, I think it's important to squash this bug in 2.3.4. I don't have any other 2.3.4 issues in mind. From Tino.Lange at isg.de Tue May 4 17:06:12 2004 From: Tino.Lange at isg.de (Tino Lange) Date: Tue May 4 17:07:15 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <4097CFEB.1090500@interlink.com.au> References: <4097CFEB.1090500@interlink.com.au> Message-ID: <409805C4.1020609@isg.de> Anthony Baxter wrote: Hi Anthony, hi Python-Dev! > My plan is to cut a release candidate for 2.3.4 next > Thursday (13th April) Australian time (so Wednesday night > for the merkins), and a release the following Thursday > (the 20th). > this is likely to be the final 2.3 release Is it possible to include the patch #945642 in 2.3.4? If not, this would mean that *no version Python of 2.3(.x)* was ever able to handle nonblocking SSL socket connections properly (what the former Python-Versions 2.x could). See: https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id=5470 Cheers, Tino From tim.one at comcast.net Tue May 4 17:53:47 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 4 17:53:59 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: <409721F0.1020102@interlink.com.au> Message-ID: [Jeremy] >> We are in rather dire need of docs and a revised PEP. [Anthony] > Indeed. The timeframe we came up with at PyCon had the > generator expressions being landed "in the first week of > May". At the moment the discussion seems to be spinning > wheels -- it looks (to me) like there's not even 100% > agreement from everyone that they should even go into > 2.4 at all. Well, only one opinion really counts, and the discussion has more the feel of whining at a wake about the deceased's bad qualities . Alas, the PEP is indeed out of date, and I can't remember exactly which set of gimmicks Guido favored. The PEP currently says early-binding is the rule, but I *think* Guido favored mostly-late binding -- which is late binding, except that the iterable in the leftmost for-clause is evaluated at once. So ... (e1 for stuff in e2 for morestuff in e3 ...) ... is effectively replaced by def __g(primary): for stuff in primary: for morestuff in e3: ... yield e1 ... __g(e2) ... > I'm not fussed if we slip a week or two here, but if > it slips another 6 weeks, say, then the release time > line could be in some trouble. Is anyone working (or willing to work) on the PEP, implementation, and docs now? I can't make time for it, but if I could I think the semantic issues are settled (in Guido's mind, and possibly in some email of his I can't find now). From pedronis at bluewin.ch Tue May 4 18:03:37 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Tue May 4 18:03:57 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: References: <409721F0.1020102@interlink.com.au> Message-ID: <5.2.1.1.0.20040505000153.0303da68@pop.bluewin.ch> At 17:53 04.05.2004 -0400, Tim Peters wrote: >[Jeremy] > >> We are in rather dire need of docs and a revised PEP. > >[Anthony] > > Indeed. The timeframe we came up with at PyCon had the > > generator expressions being landed "in the first week of > > May". At the moment the discussion seems to be spinning > > wheels -- it looks (to me) like there's not even 100% > > agreement from everyone that they should even go into > > 2.4 at all. > >Well, only one opinion really counts, and the discussion has more the feel >of whining at a wake about the deceased's bad qualities . well, the fact is that this particular deceased is going to stay with us >Is anyone working (or willing to work) on the PEP, implementation, and docs >now? I can't make time for it, but if I could I think the semantic issues >are settled (in Guido's mind, and possibly in some email of his I can't find >now). that's the one (I think) http://mail.python.org/pipermail/python-dev/2004-April/044555.html From jjl at pobox.com Tue May 4 18:16:18 2004 From: jjl at pobox.com (John J Lee) Date: Tue May 4 18:17:02 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <4097CFEB.1090500@interlink.com.au> References: <4097CFEB.1090500@interlink.com.au> Message-ID: On Wed, 5 May 2004, Anthony Baxter wrote: [...] > Can we say that the 2.3 branch is frozen from Tuesday? As > this is likely to be the final 2.3 release (modulo any > suprising new bugs), I'd like the time to try and nail > this one. [...] Could somebody apply this before the 2.3 freeze? http://www.python.org/sf/851736 Trivial fix to an obviously-broken class, has been sitting there, with test, for some months. John From aml at world.std.com Tue May 4 18:45:00 2004 From: aml at world.std.com (Andrew M. Langmead) Date: Tue May 4 18:45:11 2004 Subject: [Python-Dev] signals and linuxthreads (was: Python 2.3.4 schedule) In-Reply-To: References: <4097CFEB.1090500@interlink.com.au> Message-ID: <20040504224500.GA2826220@world.std.com> On Tue, May 04, 2004 at 01:47:28PM -0400, Tim Peters wrote: > http://mail.zope.org/pipermail/zope-dev/2004-May/022838.html > > Short course is that a serious bug with threads on Linux was reported almost > a year ago, against Python 2.2.2: I have a patch that seems to fix things and a c program to test for LinuxThreads behavior that would be suitable for autoconf I just need a bit of time testing the test suite changes for how they run on other platforms, and to integrate the test program into configure.in. It puts all the changes into a REDIRECT_SIGNALS_TO_MAIN_THREAD define, so there should be any effect on other platforms. From guido at python.org Tue May 4 19:44:53 2004 From: guido at python.org (Guido van Rossum) Date: Tue May 4 19:48:53 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: Your message of "Tue, 04 May 2004 17:53:47 EDT." References: Message-ID: <200405042344.i44Nisv05570@guido.python.org> > Alas, the PEP is indeed out of date, and I can't remember exactly > which set of gimmicks Guido favored. The PEP currently says > early-binding is the rule, but I *think* Guido favored mostly-late > binding -- which is late binding, except that the iterable in the > leftmost for-clause is evaluated at once. So > > ... (e1 for stuff in e2 for morestuff in e3 ...) ... > > is effectively replaced by > > def __g(primary): > for stuff in primary: > for morestuff in e3: > ... > yield e1 > > ... __g(e2) ... Exactly. > > I'm not fussed if we slip a week or two here, but if > > it slips another 6 weeks, say, then the release time > > line could be in some trouble. > > Is anyone working (or willing to work) on the PEP, implementation, > and docs now? I can't make time for it, but if I could I think the > semantic issues are settled (in Guido's mind, and possibly in some > email of his I can't find now). How about this one. :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From tim at zope.com Tue May 4 11:03:41 2004 From: tim at zope.com (Tim Peters) Date: Tue May 4 20:14:33 2004 Subject: [Python-Dev] RE: [Zope-dev] Segfault and Deadlock In-Reply-To: Message-ID: As Andrew Langmead has already discovered, the LinuxThreads issue with SIGSEGV was reported on the Python bug tracker almost a year ago (well, reported, but not diagnosed): SIGSEGV causes hung threads (Linux) http://www.python.org/sf/756924 Looks like: can't CNTRL-C when running os.system in a thread http://www.python.org/sf/756940 is related. python-dev'ers, do we have a release manager for 2.3.4 (I didn't see a resolution to the brouhaha at the end of March)? If so, is 2.3.4 still planned for this month? tick-tock-tick-tock-ing-ly y'rs - tim From trevp at trevp.net Tue May 4 22:49:38 2004 From: trevp at trevp.net (Trevor Perrin) Date: Tue May 4 22:49:46 2004 Subject: [Python-Dev] negative modular exponentiation In-Reply-To: References: <20040504145852.GA27574@vicky.ecs.soton.ac.uk> Message-ID: <5.2.0.9.0.20040504151541.02675cb8@pop.sbcglobal.yahoo.com> Hi Armin, Guido, Tim, everyone, Thanks for looking at this patch! I've submitted a few other crypto-related patches too. They're all waiting eagerly for good, loving people to adopt them :-)... They've all got basically the same motivation. If I explain that, it might help people decide whether they're worth it or not: Rationale for some crypto patches in 2.4 ----------------------------------------- Almost all crypto protocols require the same fast or OS-dependent primitives: modular exponentiation, hashing, ciphers, and entropy-harvesting. Everything else can be done in Python. To get these primitives in Python you could install M2Crypto (needing OpenSSL and SWIG) or pycrypto (needing GMP). But that complicates software distribution. So I was hoping the few missing primitives could be added to 2.4, enabling fast and portable pure-Python crypto. I've submitted the below patches. The first 2 aren't that important: #923643, long <-> byte-string: Convenience methods so longs are easier to read/write in network protocols: n = long(someString, 256) # str -> long s = n.tostring() # long -> str #935454, sha256 module: Since SHA-256 is preferred over SHA-1 for certain things these days. #934711, platform-specific entropy: An os.urandom() function providing the platform equivalent of /dev/urandom (CryptGenRandom on Windows, presumably SecureRandom on Java). #936813, faster modular exponentiation... The only thing left is block ciphers (AES, maybe 3DES). Those would be easy (just copy from pycrypto) but I think this is courting enough controversy at the moment.. Anyways, about modular exponentiation: On crypto-sized numbers (1000-8000 bits) on a P4, Python mod-exp is ~6-12x slower than GMP. With this patch it's only ~3-4x slower, which is on par with another C library I tested (LibTomCrypt). Mod-exp pretty much determines how long an SSL or IPsec handshake takes, or signing/decrypting a PGP message. These often take a sizeable fraction of a second, so they're worth speeding up. [Tim] >The Python long implementation has a >great track record wrt portability and lack of bugs, and gonzo optimizations >work against that. Good points. The speedup is worth it to me, but it certainly adds complexity (though these are boring, straightforward optimizations that everyone does, and are described in lots of books. I ain't good enough to do gonzo :-) >For example, did Python really need the Karatsuba >multiplication gimmick? I added the patch for it after a lot of reworking, >but overall I'm not sure it was a net win; It seems a good speedup to me, particularly if the cutoff is increased - on 4000-8000 bit numbers it gives around 10-30% faster multiplication, and on bigger numbers it's wildly faster. > and the current patch in question >boosts the smallest integer at which Karatsuba triggers to such a large >value that anyone reaching it would probably be much better off using a GMP >wrapper anyway. You'd be better off using GMP at pretty much any value. But if you want to widely distribute a tool or product that uses crypto, it's a pain for every user to have to do that. >One thing I noticed a few times in the patch was neglecting to check C calls >for error returns. Even _PyLong_NumBits() can fail! Darn, thanks. The 2 problems I see are in the long <-> byte-string code (which I left in the longobject.c patch), not the mod-exp code, which I went over more carefully. I'll fix and re-submit the long<->byte-string patch.. Trevor From niemeyer at conectiva.com Wed May 5 11:17:03 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Wed May 5 11:17:05 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <4097CFEB.1090500@interlink.com.au> References: <4097CFEB.1090500@interlink.com.au> Message-ID: <20040505151703.GA2958@burma.localdomain> Hello Anthony, > Can we say that the 2.3 branch is frozen from Tuesday? As > this is likely to be the final 2.3 release (modulo any > suprising new bugs), I'd like the time to try and nail > this one. I'd like to include some of the proposed gettext improvements into that candidate, but I'm not sure I'll get the necessary time, since we're in a rush for a new distribution release. Will try to prepare a patch in the weekend. -- Gustavo Niemeyer http://niemeyer.net From guido at python.org Wed May 5 11:28:17 2004 From: guido at python.org (Guido van Rossum) Date: Wed May 5 11:28:27 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: Your message of "Wed, 05 May 2004 12:17:03 -0300." <20040505151703.GA2958@burma.localdomain> References: <4097CFEB.1090500@interlink.com.au> <20040505151703.GA2958@burma.localdomain> Message-ID: <200405051528.i45FSH213712@guido.python.org> > > Can we say that the 2.3 branch is frozen from Tuesday? As > > this is likely to be the final 2.3 release (modulo any > > suprising new bugs), I'd like the time to try and nail > > this one. > > I'd like to include some of the proposed gettext improvements > into that candidate, but I'm not sure I'll get the necessary > time, since we're in a rush for a new distribution release. > Will try to prepare a patch in the weekend. Huh? We're talking about a bugfix release for 2.3 here. Not a good time for improvements, which sounds to me like a euphemism for new features, not for bug fixes. Even if they are backwards compatible, I'm still -1 on adding new features to a bugfix release. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed May 5 11:28:35 2004 From: guido at python.org (Guido van Rossum) Date: Wed May 5 11:29:27 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: Your message of "Tue, 04 May 2004 14:54:08 +1000." <409721F0.1020102@interlink.com.au> References: <1083638573.17924.9.camel@localhost.localdomain> <409721F0.1020102@interlink.com.au> Message-ID: <200405051528.i45FSZh13723@guido.python.org> > Indeed. The timeframe we came up with at PyCon had the > generator expressions being landed "in the first week of > May". At the moment the discussion seems to be spinning > wheels -- it looks (to me) like there's not even 100% > agreement from everyone that they should even go into > 2.4 at all. There doesn't need to be 100% agreement. This thread (which is indeed getting nowhere) started with my assertion that we should do it, using late binding. I haven't changed my mind yet. I'm still waiting for someone to volunteer to review the code once more and check it in. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed May 5 11:28:44 2004 From: guido at python.org (Guido van Rossum) Date: Wed May 5 11:29:39 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: Your message of "Wed, 05 May 2004 03:16:27 +1000." <4097CFEB.1090500@interlink.com.au> References: <4097CFEB.1090500@interlink.com.au> Message-ID: <200405051528.i45FSiN13730@guido.python.org> > My plan is to cut a release candidate for 2.3.4 next > Thursday (13th April) Australian time (so Wednesday night > for the merkins), and a release the following Thursday > (the 20th). > > Can we say that the 2.3 branch is frozen from Tuesday? As > this is likely to be the final 2.3 release (modulo any > suprising new bugs), I'd like the time to try and nail > this one. Either you meant May or that mail spent a really long time in some mail queue... ;-) One thing I'd love to see backported to 2.3.4 is the fix to platform.py to properly recognize Windows XP when win32 is detected. Mind if I backport this one personally? --Guido van Rossum (home page: http://www.python.org/~guido/) From pedronis at bluewin.ch Wed May 5 11:44:54 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed May 5 11:45:00 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: <200405051528.i45FSZh13723@guido.python.org> References: <1083638573.17924.9.camel@localhost.localdomain> <409721F0.1020102@interlink.com.au> Message-ID: <5.2.1.1.0.20040505174120.030398a0@pop.bluewin.ch> At 08:28 05.05.2004 -0700, Guido van Rossum wrote: > > Indeed. The timeframe we came up with at PyCon had the > > generator expressions being landed "in the first week of > > May". At the moment the discussion seems to be spinning > > wheels -- it looks (to me) like there's not even 100% > > agreement from everyone that they should even go into > > 2.4 at all. > >There doesn't need to be 100% agreement. This thread (which is indeed >getting nowhere) one likely aspect here is that someone should write the doc for this feature, and consider whether it should be in the tutorial (at least list comprehensions are introduced there), and see how it reads. >started with my assertion that we should do it, using >late binding. I haven't changed my mind yet. I'm still waiting for >someone to volunteer to review the code once more and check it in. > >--Guido van Rossum (home page: http://www.python.org/~guido/) > >_______________________________________________ >Python-Dev mailing list >Python-Dev@python.org >http://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: >http://mail.python.org/mailman/options/python-dev/pedronis%40bluewin.ch From anthony at interlink.com.au Wed May 5 12:14:14 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed May 5 12:14:27 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <200405051528.i45FSiN13730@guido.python.org> References: <4097CFEB.1090500@interlink.com.au> <200405051528.i45FSiN13730@guido.python.org> Message-ID: <409912D6.6070105@interlink.com.au> Guido van Rossum wrote: >>My plan is to cut a release candidate for 2.3.4 next >>Thursday (13th April) Australian time (so Wednesday night >>for the merkins), and a release the following Thursday >>(the 20th). >> >>Can we say that the 2.3 branch is frozen from Tuesday? As >>this is likely to be the final 2.3 release (modulo any >>suprising new bugs), I'd like the time to try and nail >>this one. > > > Either you meant May or that mail spent a really long time in some > mail queue... ;-) Yes, I meant May. > One thing I'd love to see backported to 2.3.4 is the fix to > platform.py to properly recognize Windows XP when win32 is detected. > Mind if I backport this one personally? Go for it. -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Wed May 5 12:16:34 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed May 5 12:16:50 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: <200405051528.i45FSZh13723@guido.python.org> References: <1083638573.17924.9.camel@localhost.localdomain> <409721F0.1020102@interlink.com.au> <200405051528.i45FSZh13723@guido.python.org> Message-ID: <40991362.7050207@interlink.com.au> Guido van Rossum wrote: > There doesn't need to be 100% agreement. This thread (which is indeed > getting nowhere) started with my assertion that we should do it, using > late binding. I haven't changed my mind yet. I'm still waiting for > someone to volunteer to review the code once more and check it in. Sure. And this was what I was referring to. Lotsa noise, little traction. This isn't meant as a slap at anyone - I am quite aware of time limitations. I guess I was hoping to prod someone into stepping forward and doing the work. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From niemeyer at conectiva.com Wed May 5 12:54:09 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Wed May 5 12:54:04 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <200405051528.i45FSH213712@guido.python.org> References: <4097CFEB.1090500@interlink.com.au> <20040505151703.GA2958@burma.localdomain> <200405051528.i45FSH213712@guido.python.org> Message-ID: <20040505165409.GA3644@burma.localdomain> > Huh? We're talking about a bugfix release for 2.3 here. Not a good > time for improvements, which sounds to me like a euphemism for new > features, not for bug fixes. Adding a gettext compatible lgettext() might be a good thing, as well as bind_textdomain_codeset() (which pygtk users certainly have missed already) but I have to admit this is not a bug fix per se. > Even if they are backwards compatible, I'm still -1 on adding new > features to a bugfix release. Acknowledged. Will prepare a patch for the next major asap. -- Gustavo Niemeyer http://niemeyer.net From mal at egenix.com Wed May 5 14:14:32 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Wed May 5 14:14:41 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <200405051528.i45FSiN13730@guido.python.org> References: <4097CFEB.1090500@interlink.com.au> <200405051528.i45FSiN13730@guido.python.org> Message-ID: <40992F08.6080904@egenix.com> Guido van Rossum wrote: >>My plan is to cut a release candidate for 2.3.4 next >>Thursday (13th April) Australian time (so Wednesday night >>for the merkins), and a release the following Thursday >>(the 20th). >> >>Can we say that the 2.3 branch is frozen from Tuesday? As >>this is likely to be the final 2.3 release (modulo any >>suprising new bugs), I'd like the time to try and nail >>this one. > > > Either you meant May or that mail spent a really long time in some > mail queue... ;-) > > One thing I'd love to see backported to 2.3.4 is the fix to > platform.py to properly recognize Windows XP when win32 is detected. > Mind if I backport this one personally? This should only be a matter of copy the HEAD version to the release branch. Note that the HEAD version is slightly different from the one on the 2.3 branch - I haven't backported the cache changes I made. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 05 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 32 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From bac at OCF.Berkeley.EDU Wed May 5 14:23:16 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed May 5 14:23:30 2004 Subject: [Python-Dev] Passing compile(...,'exec') code to 'eval' Message-ID: <40993114.5060708@ocf.berkeley.edu> In preparations for my coding my thesis I have been trying to figure out how a local variable could be assigned to without me explicitly knowing during compilation. Obviously 'exec' can. But I wasn't sure about 'eval'. Reading the docs, I didn't think it could since 'eval', when taking a string, only evaluates expressions. But what about code objects? The docs say, "The code object must have been compiled passing 'eval' as the kind argument". But I didn't read that when I started testing. This is when I discovered you *can* pass in something using 'compile' with the kind argument of "exec":: >>> x Traceback (most recent call last): File "", line 1, in ? NameError: name 'x' is not defined [16425 refs] >>> eval(compile("x = 1", "", "exec")) [16426 refs] >>> x 1 Is this a bug, or are the docs wrong? I am hoping it is the former since if it is the latter my thesis just got a big caveat pasted into it about how 'eval' can cause problems and invalidate the type inferencing in irreparable ways. -Brett From jim.jewett at eds.com Wed May 5 14:31:01 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Wed May 5 14:31:07 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions Message-ID: Guido van Rossum: >... mostly-late binding -- which is late binding, except that > the iterable in the leftmost for-clause is evaluated at once. Did you really mean leftmost, or all-but-the-rightmost? Special-casing the left-most causes the two-generator version to behave as expected, but it makes the three-generator case even more confusing. Perhaps the entire early/late binding problem comes from trying to treat multiple generators as a special case. For a single-generator expression, late binding makes sense (or why use a free variable), but early binding would also work most of the time. For multiple generators, the current behavior is that the final generator is exhausted by the first loop, so the other generators really only need one value each. The only problem with that is that it isn't usually what people want. If users do want a cross-product, then either the values have to be stored somewhere, or python has to create a whole series of "equivalent" generators. Python can't do this any better than the user, so why not make the storage explicit? (x,y,z for x in e1 for y in e2 for z in e3) <==> for x in e1.__iter__(): for x in e2.__iter__() for z in e3.__iter__() yield (x,y,z) It is the user's responsibility to ensure that e2 and e3 can return an iterator multiple times, perhaps by wrapping them in a list or tuple. -jJ From tim.one at comcast.net Wed May 5 15:12:35 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed May 5 15:12:42 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions In-Reply-To: Message-ID: [Jewett, Jim J] > Guido van Rossum: > >... mostly-late binding -- which is late binding, except that > > the iterable in the leftmost for-clause is evaluated at once. Guido didn't write that -- I did. Guido agreed with it, though. > Did you really mean leftmost, Yes. > or all-but-the-rightmost? No. ... Sorry, I didn't understand the rest of the msg, so I have to skip the rest. Try a concrete example? From jim.jewett at eds.com Wed May 5 16:43:13 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Wed May 5 16:44:08 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions Message-ID: > Try a concrete example? t1 = (1,2,3) t2 = ("a", "b", "c") t3 = ("first", "second", "third") (x,y,z for x in t1 for y in t2 for z in t3) Do What I Mean is the crossproduct, 27 elements long, which you get with a list comprehension. This will happen if at least (all but one) generators are pre-evaluated. It will also happen if at least (all but one) generators are restarted after they raise StopIterator. Doing strictly what happens today gets only (1,'a',"first"), (1,'a',"second"), (1,'a',"third") because t3 has raised StopIteration by the time t1 or t2 advance. Effectively, only the last iterator matters. Keeping the last value of an iterator also adds (1, 'b', "third"), (1, 'c', "third"), (2, 'c', "third"), (3, '3', "third") for seven values -- but perhaps not a well-chosen seven. Pre-evaluating only the leftmost gets all combinations of (1,2,3) and ("first", "second", "third"), but still misses items like (2, 'b', "second"). Since these are in the middle of the series, they are the most likely to be forgotten during testing. Restarting the second (and later) iterators for each value of the first iterator does the right thing, without requiring any magical pre-evaluation. (Unless the iterator can't be restarted -- in which case there is no right thing, and the user should have decided what to cache.) Thus my suggestion of (x,y,z for x in t1 for y in t2 for z in t3) <==> for x in t1.__iter__(): for y in t2.__iter__(): for z in t3.__iter__(): yield x,y,z For the example data, this creates 3 separate iterators on the letters and nine iterators on the words - but produces the expected result, without having to store or pre-evaluate anything. -jJ From dave at boost-consulting.com Wed May 5 17:02:17 2004 From: dave at boost-consulting.com (David Abrahams) Date: Wed May 5 17:10:24 2004 Subject: [Python-Dev] horrible thought Message-ID: Remember the vote on adding conditional expressions? I'm afraid that the imminent proliferation of xxxcomps may indicate the "right" syntax for this feature, if it ever does get added. Something like (a else b if condition) Now I've done my bad deed for the day. Cheers, Dave -- Dave Abrahams Boost Consulting http://www.boost-consulting.com From tim.one at comcast.net Wed May 5 17:24:40 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed May 5 17:38:57 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions In-Reply-To: Message-ID: [Jewett, Jim J] > t1 = (1,2,3) > t2 = ("a", "b", "c") > t3 = ("first", "second", "third") > > (x,y,z for x in t1 for y in t2 for z in t3) > > Do What I Mean is the crossproduct, 27 elements long, > which you get with a list comprehension. Well, that expression by itself will result in a generator-iterator getting created, with no actual results. If you put it in a context that drives generation, then it will deliver 27 results, and regardless of whether early-binding, late-binding, or mostly-late binding is used (although the latter two are sensitive to changes in the bindings for t2 and t3, if any such occur between the time the genexp is created and the time the genexp is driven). For mostly-late binding, it's the same as this Python program today (stuffing the genexp in a list() call to force it to do something): """ t1 = (1,2,3) t2 = ("a", "b", "c") t3 = ("first", "second", "third") def __g(primary): for x in primary: for y in t2: for z in t3: yield x, y, z print list(__g(t1)) """ Try running that, and see what you get. > This will happen if at least (all but one) generators > are pre-evaluated. There are no generators in your example, apart from the one created by the genexp itself. You may be confusing generators with iterable objects. > It will also happen if at least (all but one) generators are > restarted after they raise StopIterator. > > Doing strictly what happens today gets only > > (1,'a',"first"), (1,'a',"second"), (1,'a',"third") No -- run the above. You'll get the 27 you expect. > because t3 has raised StopIteration by the time t1 or t2 > advance. The innermost "for" loop swallows StopIteration each of the 9 times it occurs, and the outer for loops never see it. They keep going. This is how "for" loops always work (nothing is changing wrt that). From guido at python.org Wed May 5 19:20:00 2004 From: guido at python.org (Guido van Rossum) Date: Wed May 5 19:20:20 2004 Subject: [Python-Dev] Passing compile(...,'exec') code to 'eval' In-Reply-To: Your message of "Wed, 05 May 2004 11:23:16 PDT." <40993114.5060708@ocf.berkeley.edu> References: <40993114.5060708@ocf.berkeley.edu> Message-ID: <200405052320.i45NK0d14202@guido.python.org> > This is when I discovered you *can* pass in something using 'compile' > with the kind argument of "exec":: > > >>> x > Traceback (most recent call last): > File "", line 1, in ? > NameError: name 'x' is not defined > [16425 refs] > >>> eval(compile("x = 1", "", "exec")) > [16426 refs] > >>> x > 1 > > Is this a bug, or are the docs wrong? I am hoping it is the former > since if it is the latter my thesis just got a big caveat pasted into it > about how 'eval' can cause problems and invalidate the type inferencing > in irreparable ways. You shouldn't do this, and if you do, you will pay for your sin in a special kind of hell set aside for people who take the implementation as the spec. It's hard to detect such undefined usage without making eval() (and possibly everything) slower though. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed May 5 19:16:45 2004 From: guido at python.org (Guido van Rossum) Date: Wed May 5 19:21:07 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: Your message of "Wed, 05 May 2004 13:54:09 -0300." <20040505165409.GA3644@burma.localdomain> References: <4097CFEB.1090500@interlink.com.au> <20040505151703.GA2958@burma.localdomain> <200405051528.i45FSH213712@guido.python.org> <20040505165409.GA3644@burma.localdomain> Message-ID: <200405052316.i45NGkE14180@guido.python.org> > > Huh? We're talking about a bugfix release for 2.3 here. Not a good > > time for improvements, which sounds to me like a euphemism for new > > features, not for bug fixes. > > Adding a gettext compatible lgettext() might be a good thing, > as well as bind_textdomain_codeset() (which pygtk users certainly > have missed already) but I have to admit this is not a bug fix > per se. Definitely -1 for 2.3.4. > > Even if they are backwards compatible, I'm still -1 on adding new > > features to a bugfix release. > > Acknowledged. Will prepare a patch for the next major asap. Sure, if that's what Barry agreed to. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Wed May 5 20:30:25 2004 From: guido at python.org (Guido van Rossum) Date: Wed May 5 20:30:38 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: Your message of "Wed, 05 May 2004 20:14:32 +0200." <40992F08.6080904@egenix.com> References: <4097CFEB.1090500@interlink.com.au> <200405051528.i45FSiN13730@guido.python.org> <40992F08.6080904@egenix.com> Message-ID: <200405060030.i460UQA14346@guido.python.org> > This should only be a matter of copy the HEAD version to the release > branch. Note that the HEAD version is slightly different from > the one on the 2.3 branch - I haven't backported the cache > changes I made. OK, done. The cache fix is now in 2.3.4 too. Note to Brett C and Raymond H: one of you added a call to reversed() to platform.py; that's a no-no since the source code needs to remain compatible with 1.5.2, per a comment in the source. I've added platform.py to PEP 291. --Guido van Rossum (home page: http://www.python.org/~guido/) From ndbecker2 at verizon.net Wed May 5 20:44:19 2004 From: ndbecker2 at verizon.net (Neal Becker) Date: Wed May 5 20:44:29 2004 Subject: [Python-Dev] distutils problem: /usr/lib64/python2.3 on x86_64 Message-ID: I just tried to build wxPython on x86_64 (athlon64) on Fedora core3 test 3. It didn't work. It looks like python is a 64-bit exec on this platform, and there is /usr/lib64/python2.3/ with what looks like the usual stuff. There is also a /usr/lib/python2.3, with not much in it, but I guess if you install a 32-bit python it would go there. Anyway, distutils/sysconfig.py def get_python_lib(... if os.name == "posix": libpython = os.path.join(prefix, "lib", "python" + get_python_version()) This is wrong, and it doesn't seem to give me anyway to overide the choice. Is there a fix? From pedronis at bluewin.ch Wed May 5 20:59:40 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Wed May 5 20:59:46 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move Forward In-Reply-To: <200404231826.i3NIQxM18955@guido.python.org> Message-ID: <5.2.1.1.0.20040506021127.030398a0@pop.bluewin.ch> At 11:26 23.04.2004 -0700, Guido van Rossum wrote: >My reasons for (still!) preferring late binding are: > >(a) I find it hard to defend automatic variable capture given Python's > general late-binding semantics I have been thinking about reason (a), the point is that a genexp is not simply a new way to introduce a closure, it is not simply sugar over lambda or def. A genexp is sugar for defining on the fly a generator closure and second instatiating it! Now there are two parts to that, it is true that closures in Python have late-bindings semantics but when they are deployed people in most cases use idioms that circuvent and neutralize late-bindings. def make_show_action(msg): def action(evt,msg): show(msg) return action used as for act,msg in msgs: pnl.add(Button(act,action=make_show_action(msg)) or for act,msg in msgs: pnl.add(Button(act,lambda msg=msg: show(msg))) in fact here lambda has late-bindings semantics and we eschew them by not having any free variable at all. Of course here enters (b), mixing for and genexp is advanced use. That's really a gut-feelings call at this point. For most practical cases, for closures, late-bindings sematics seems something to keep under control, it's an implementation detail to allow recursive and mutally recursive nested functions. So to repeat genexp have no simple direct correspondence to a lambda or a def. They are sugar for some more involved code, so I think it is more a call of what is the general idiomatic form for that kind of code, also maybe considering that the genexp is hiding the details of what is really going on (kind of supplying sheep's clothes to a wolf). Maybe the sane thing is late-bindings, maybe early. But I don't think that as suggested by (a) that there is some deep design logic internal to Python at stake here. At most the argument could be that late-bindings make things easier to explain because they are kind of like lambdas. Maybe, maybe not. Then there is (b). >(b) I believe that use cases needing early binding are uncommon and > strained: they all involve creating a list of generator > expressions, which IMO is a pretty unusual thing to do regards From martin at v.loewis.de Wed May 5 21:20:58 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed May 5 21:20:54 2004 Subject: [Python-Dev] distutils problem: /usr/lib64/python2.3 on x86_64 In-Reply-To: References: Message-ID: <409992FA.2090009@v.loewis.de> Neal Becker wrote: > This is wrong, and it doesn't seem to give me anyway to overide the choice. > Is there a fix? Not that I know of. Did you even manage to install Python without python.org/sf/926209 Regards, Martin From ndbecker2 at verizon.net Wed May 5 21:54:31 2004 From: ndbecker2 at verizon.net (Neal Becker) Date: Wed May 5 21:54:38 2004 Subject: [Python-Dev] Re: distutils problem: /usr/lib64/python2.3 on x86_64 References: <409992FA.2090009@v.loewis.de> Message-ID: "Martin v. L?wis" wrote: > Neal Becker wrote: >> This is wrong, and it doesn't seem to give me anyway to overide the >> choice. Is there a fix? > > Not that I know of. Did you even manage to install Python without > python.org/sf/926209 > python-2.3 is already standard on Fedora Core3 Test3, I didn't build it. From greg at cosc.canterbury.ac.nz Thu May 6 02:06:28 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu May 6 02:07:01 2004 Subject: [Python-Dev] PEP 328: import...as with relative import Message-ID: <200405060606.i4666Stx024952@cosc353.cosc.canterbury.ac.nz> There's been some discussion on c.l.py about whether the new relative import system should allow things like import ..abc as def It seems the PEP currently disallows this, but nobody seems to be able to adequately explain why. Is there any technical reason it couldn't be done? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From mal at egenix.com Thu May 6 03:11:53 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Thu May 6 03:11:53 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <200405060030.i460UQA14346@guido.python.org> References: <4097CFEB.1090500@interlink.com.au> <200405051528.i45FSiN13730@guido.python.org> <40992F08.6080904@egenix.com> <200405060030.i460UQA14346@guido.python.org> Message-ID: <4099E539.4090405@egenix.com> Guido van Rossum wrote: >>This should only be a matter of copy the HEAD version to the release >>branch. Note that the HEAD version is slightly different from >>the one on the 2.3 branch - I haven't backported the cache >>changes I made. > > OK, done. The cache fix is now in 2.3.4 too. Thanks. > Note to Brett C and Raymond H: one of you added a call to reversed() > to platform.py; that's a no-no since the source code needs to remain > compatible with 1.5.2, per a comment in the source. I've added > platform.py to PEP 291. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 06 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 31 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From arigo at tunes.org Thu May 6 04:51:16 2004 From: arigo at tunes.org (Armin Rigo) Date: Thu May 6 04:54:36 2004 Subject: [Python-Dev] Passing compile(...,'exec') code to 'eval' In-Reply-To: <40993114.5060708@ocf.berkeley.edu> References: <40993114.5060708@ocf.berkeley.edu> Message-ID: <20040506085116.GA17048@vicky.ecs.soton.ac.uk> Hello Brett, On Wed, May 05, 2004 at 11:23:16AM -0700, Brett C. wrote: > Is this a bug, or are the docs wrong? I am hoping it is the former > since if it is the latter my thesis just got a big caveat pasted into it > about how 'eval' can cause problems and invalidate the type inferencing > in irreparable ways. There is another improbable way to bind variables with eval, with list comprehensions: >>> eval("[x for x in [3]]") [3] >>> x 3 If the goal is not security but type inference, I guess both this and the compile() trick should be considered as outside any reasonable spec. A bientot, Armin. From arigo at tunes.org Thu May 6 05:10:18 2004 From: arigo at tunes.org (Armin Rigo) Date: Thu May 6 05:13:34 2004 Subject: [Python-Dev] PEP 289 - Generator Expressions - Let's Move For ward In-Reply-To: <200405051528.i45FSZh13723@guido.python.org> References: <1083638573.17924.9.camel@localhost.localdomain> <409721F0.1020102@interlink.com.au> <200405051528.i45FSZh13723@guido.python.org> Message-ID: <20040506091018.GB17048@vicky.ecs.soton.ac.uk> Hello, On Wed, May 05, 2004 at 08:28:35AM -0700, Guido van Rossum wrote: > There doesn't need to be 100% agreement. This thread (which is indeed > getting nowhere) I'll leave it alone, then. It can't go anywhere as long as we'll keep opposing a bunch of concrete examples and an intuition. I retract because your intuition is firmly grounded, and I respect that even if I disagree -- I, for example, have the intuition that high-level languages should be faster than low-level ones, despite all the examples :-) Armin From niemeyer at conectiva.com Thu May 6 09:55:37 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Thu May 6 09:55:28 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <200405052316.i45NGkE14180@guido.python.org> References: <4097CFEB.1090500@interlink.com.au> <20040505151703.GA2958@burma.localdomain> <200405051528.i45FSH213712@guido.python.org> <20040505165409.GA3644@burma.localdomain> <200405052316.i45NGkE14180@guido.python.org> Message-ID: <20040506135537.GA3440@burma.localdomain> > > Acknowledged. Will prepare a patch for the next major asap. > > Sure, if that's what Barry agreed to. I said I'm going to prepare a patch. After that, I'll ask for comments on this list. That's what I have always done about things I'm not sure about, and that's what I would do even if you hadn't mentioned the 2.3.4 issue. -- Gustavo Niemeyer http://niemeyer.net From guido at python.org Thu May 6 10:04:33 2004 From: guido at python.org (Guido van Rossum) Date: Thu May 6 10:04:40 2004 Subject: [Python-Dev] PEP 328: import...as with relative import In-Reply-To: Your message of "Thu, 06 May 2004 18:06:28 +1200." <200405060606.i4666Stx024952@cosc353.cosc.canterbury.ac.nz> References: <200405060606.i4666Stx024952@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405061404.i46E4Xg16720@guido.python.org> > There's been some discussion on c.l.py about whether the > new relative import system should allow things like > > import ..abc as def > > It seems the PEP currently disallows this, but nobody > seems to be able to adequately explain why. Is there > any technical reason it couldn't be done? Because "import ..abc" is clearly nonsense, since after "import X" you should be able to the imported entity as "X" in your program, and "..abc" is not a valid expression. This doesn't strictly apply to "import ..abc as xyz" ("def" was a poor choice for "X" :-), but it doesn't seem worth stretching the syntax when you can already do that using "from .. import abc as xyz". --Guido van Rossum (home page: http://www.python.org/~guido/) From barry at python.org Thu May 6 10:08:35 2004 From: barry at python.org (Barry Warsaw) Date: Thu May 6 10:08:43 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <200405052316.i45NGkE14180@guido.python.org> References: <4097CFEB.1090500@interlink.com.au> <20040505151703.GA2958@burma.localdomain> <200405051528.i45FSH213712@guido.python.org> <20040505165409.GA3644@burma.localdomain> <200405052316.i45NGkE14180@guido.python.org> Message-ID: <1083852514.28538.95.camel@anthem.wooz.org> On Wed, 2004-05-05 at 19:16, Guido van Rossum wrote: > > > Even if they are backwards compatible, I'm still -1 on adding new > > > features to a bugfix release. > > > > Acknowledged. Will prepare a patch for the next major asap. > > Sure, if that's what Barry agreed to. Martin and I are on the same page about this: no change to the existing API, but +1 on adding new functions to provide the desired feature. You can assign a patch to me, but be sure it includes documentation and unit tests. -Barry From gtalvola at nameconnector.com Thu May 6 17:03:43 2004 From: gtalvola at nameconnector.com (Geoffrey Talvola) Date: Thu May 6 17:03:52 2004 Subject: [Python-Dev] Python 2.3.4 schedule Message-ID: <61957B071FF421419E567A28A45C7FE59AFB0A@mailbox.nameconnector.com> Tino Lange wrote: > Is it possible to include the patch #945642 in 2.3.4? > > If not, this would mean that *no version Python of 2.3(.x)* was ever > able to handle nonblocking SSL socket connections properly (what the > former Python-Versions 2.x could). > > See: > https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id =5470 I'd like to see this in 2.3.4 too; since I was responsible for the breakage in the first place, it will help assuage my guilt to see this fix get in... - Geoff From jim.jewett at eds.com Thu May 6 17:19:51 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Thu May 6 17:22:26 2004 Subject: [Python-Dev] what level of backward compatibility? how Message-ID: urlparse.py uses lists of strings to indicate which protocols have which properties. For instance: uses_relative = ['ftp', 'http', 'gopher', 'nntp', 'imap', 'wais', 'file', 'https', 'shttp', 'mms', 'prospero', 'rtsp', 'rtspu', ''] Logically, these are sets, rather than lists. CVS change 143 changed the lists to sets. Unfortunately, it breaks client code that did urlparse.uses_relative.append('my_protocol') which was as close to a documented API as existed. (1) Is this an OK breakage with the 2.4 switch? (2) If not, should this change be backed out, or replaced with some set subclass that takes a few of the list methods? -jJ From martin at v.loewis.de Thu May 6 17:34:47 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu May 6 17:34:44 2004 Subject: [Python-Dev] what level of backward compatibility? how In-Reply-To: References: Message-ID: <409AAF77.20900@v.loewis.de> Jewett, Jim J wrote: > Unfortunately, it breaks client code that did > > urlparse.uses_relative.append('my_protocol') > > which was as close to a documented API as existed. Why are you saying that (or: what does that mean)? The uses_relative attribute of urlparse was never documented, AFAICT. > (1) Is this an OK breakage with the 2.4 switch? If it wasn't documented, it is OK to break it, but it should be mentioned in whatsnew24.tex. Regards, Martin From greg at cosc.canterbury.ac.nz Thu May 6 20:39:12 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu May 6 20:39:20 2004 Subject: [Python-Dev] PEP 328: import...as with relative import In-Reply-To: <200405061404.i46E4Xg16720@guido.python.org> Message-ID: <200405070039.i470dChE026277@cosc353.cosc.canterbury.ac.nz> Guido: > This doesn't strictly apply to "import ..abc as xyz", but it doesn't > seem worth stretching the syntax when you can already do that using > "from .. import abc as xyz". I can see your point of view. It just seems a bit strange coming at from the other direction -- if you can say 'import a.b as z' there doesn't seem to be any good reason not to say 'import ..a.b as z'. I can imagine people getting tripped up by this. I guess you're right that it doesn't matter much either way. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Fri May 7 00:24:55 2004 From: guido at python.org (Guido van Rossum) Date: Fri May 7 00:25:06 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: Your message of "Thu, 06 May 2004 17:03:43 EDT." <61957B071FF421419E567A28A45C7FE59AFB0A@mailbox.nameconnector.com> References: <61957B071FF421419E567A28A45C7FE59AFB0A@mailbox.nameconnector.com> Message-ID: <200405070424.i474OtT17542@guido.python.org> > Tino Lange wrote: > > Is it possible to include the patch #945642 in 2.3.4? > > > > If not, this would mean that *no version Python of 2.3(.x)* was ever > > able to handle nonblocking SSL socket connections properly (what the > > former Python-Versions 2.x could). > > > > See: > > > https://sourceforge.net/tracker/?func=detail&atid=305470&aid=945642&group_id > =5470 > > I'd like to see this in 2.3.4 too; since I was responsible for the breakage > in the first place, it will help assuage my guilt to see this fix get in... Haven't seen the patch, but the description of the fix makes me go "+1 please!" :-) --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri May 7 00:29:01 2004 From: guido at python.org (Guido van Rossum) Date: Fri May 7 00:29:13 2004 Subject: [Python-Dev] what level of backward compatibility? how In-Reply-To: Your message of "Thu, 06 May 2004 23:34:47 +0200." <409AAF77.20900@v.loewis.de> References: <409AAF77.20900@v.loewis.de> Message-ID: <200405070429.i474T1t17567@guido.python.org> > Jewett, Jim J wrote: > > Unfortunately, it breaks client code that did > > > > urlparse.uses_relative.append('my_protocol') > > > > which was as close to a documented API as existed. > > Why are you saying that (or: what does that mean)? > > The uses_relative attribute of urlparse was never > documented, AFAICT. > > > (1) Is this an OK breakage with the 2.4 switch? > > If it wasn't documented, it is OK to break it, but > it should be mentioned in whatsnew24.tex. I have a different POV. I don't think there is a compelling reason to change this attribute into a set (I doubt it's so time-critical as to make a difference) and given that the attribute isn't flagged as "private" by having a name starting with underscore, I think the change ought to be reverted. --Guido van Rossum (home page: http://www.python.org/~guido/) From cgipm at unixbeast.com Fri May 7 00:33:16 2004 From: cgipm at unixbeast.com (Dave Hastings) Date: Fri May 7 00:33:21 2004 Subject: [Python-Dev] New Subscriber Message-ID: <20040507043316.GB45563@unixbeast.com> Hello fellow developers. My name is David Hastings and I am a new subscriber to the python-dev list. I'm also a little new to python but I can get around in it. I have been learning how to use the language for about a year now. First, some information about myself. I am a senior attending the University of Minnesota. Today was the last day of classes for the spring semester and also the last day of my college career. I am receiving a Bachelor of Science in Computer Science degree. Man it feels good to be done ! This last semester a friend (Michael Sampson) and I took an independent study class where we sign up with a Professor and decide on a project and work on it until the end of the semester. The project that was decided upon, with the advice of the Professor, was to make a CGIpm python module that reflected CGI.pm for perl. We noticed that there is a current cgi module for python and even a cgipm module. These modules are very useful, but did not include all the functions that CGI.pm for perl provides. It turns out that my Professor really likes python and would have loved to see a CGIpm python module for python that was like the CGI.pm module for perl. (I think he is a bit turned off by perl, more because it is a language that is not that easy to pick up unless you are familiar with unix, namely sed and awk.) We decided to take the oportunity and wrote CGIpm.py. I'm not going to go into a lot of details in this e-mail since it is an introductory e-mail, but the first version of CGIpm.py is complete. I feel that this e-mail is a bit long winded as is, so I think I will end it. I can not wait to start reading about all the new ideas, development, and contributions that this list will provide. Dave From paul at prescod.net Fri May 7 01:15:36 2004 From: paul at prescod.net (Paul Prescod) Date: Fri May 7 01:15:52 2004 Subject: [Python-Dev] New Subscriber In-Reply-To: <20040507043316.GB45563@unixbeast.com> References: <20040507043316.GB45563@unixbeast.com> Message-ID: <409B1B78.9020301@prescod.net> Welcome to the Python world. If you are just new to Python, you will get a much more varied and interesting impression of Python from python-list[1] . Python-dev is where the people evolving the language argue about the precise semantics of generator expressions and what bytecodes will give the best performance. You are of course welcome to stick around but be aware that the discussion is very focused unlike on python-list where everything Pythonic is welcome. Paul Prescod [1] http://mail.python.org/mailman/listinfo/python-list From cgipm at unixbeast.com Fri May 7 01:34:16 2004 From: cgipm at unixbeast.com (David Hastings) Date: Fri May 7 01:40:17 2004 Subject: [Python-Dev] Re: New Subscriber References: <20040507043316.GB45563@unixbeast.com> <409B1B78.9020301@prescod.net> Message-ID: Paul Prescod prescod.net> writes: > > Welcome to the Python world. If you are just new to Python, you will get > a much more varied and interesting impression of Python from > python-list[1] . Python-dev is where the people evolving the language > argue about the precise semantics of generator expressions and what > bytecodes will give the best performance. You are of course welcome to > stick around but be aware that the discussion is very focused unlike on > python-list where everything Pythonic is welcome. > > Paul Prescod > > [1] http://mail.python.org/mailman/listinfo/python-list > > _______________________________________________ > Python-Dev mailing list > Python-Dev python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python- dev%40m.gmane.org > > Oh. I'm sorry. I thought that "...and first drafts of PEPs are often posted here for initial review ..." meant that I was supposed to post here if I had something new that might be welcomed. I got that quote off of the help section in the python page in the section about the python-dev list. Is there a better place where I should post this ? thanks, Dave From paul at prescod.net Fri May 7 01:49:22 2004 From: paul at prescod.net (Paul Prescod) Date: Fri May 7 01:50:57 2004 Subject: [Python-Dev] Re: New Subscriber In-Reply-To: References: <20040507043316.GB45563@unixbeast.com> <409B1B78.9020301@prescod.net> Message-ID: <409B2362.5040108@prescod.net> David Hastings wrote: > ... > > Oh. I'm sorry. I thought that "...and first drafts of PEPs are often posted > here for initial review ..." meant that I was supposed to post here if I had > something new that might be welcomed. I got that quote off of the help > section in the python page in the section about the python-dev list. Is there > a better place where I should post this ? I didn't know that you had a PEP in mind. Even so, it seems that your PEP would be about a new module (which is a tough sell) and a module that replaces an existing module (an even tougher sell). You should refine the idea in python-list before writing up a formal PEP. Paul Prescod From jcarlson at uci.edu Fri May 7 01:53:38 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Fri May 7 01:57:58 2004 Subject: [Python-Dev] Re: New Subscriber In-Reply-To: References: <409B1B78.9020301@prescod.net> Message-ID: <20040506225112.F8C1.JCARLSON@uci.edu> > Oh. I'm sorry. I thought that "...and first drafts of PEPs are often posted > here for initial review ..." meant that I was supposed to post here if I had > something new that might be welcomed. I got that quote off of the help > section in the python page in the section about the python-dev list. Is there > a better place where I should post this ? Having a PEP is different from what you posted. Read a few of the PEPs listed here: http://www.python.org/peps/ to get a feel for how PEPs are written. - Josiah From cgipm at unixbeast.com Fri May 7 02:03:49 2004 From: cgipm at unixbeast.com (David Hastings) Date: Fri May 7 02:03:56 2004 Subject: [Python-Dev] Re: New Subscriber References: <20040507043316.GB45563@unixbeast.com> <409B1B78.9020301@prescod.net> <409B2362.5040108@prescod.net> Message-ID: Paul Prescod prescod.net> writes: > > > I didn't know that you had a PEP in mind. Even so, it seems that your > PEP would be about a new module (which is a tough sell) and a module > that replaces an existing module (an even tougher sell). You should > refine the idea in python-list before writing up a formal PEP. > > Paul Prescod > > > _______________________________________________ > Python-Dev mailing list > Python-Dev python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python- dev%40m.gmane.org > > Once again, I'm sorry. I hope there is a way that these e-mails can be removed from the web page. My intention wasn't to cause clutter. Also, my intention was not to replace any modules. I just wanted to build on it I guess. We did actually use some of the funcionality of the cgi module. I'm obviously in the wrong place. Later all ! Dave From fdrake at acm.org Fri May 7 02:10:20 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Fri May 7 02:10:35 2004 Subject: [Python-Dev] what level of backward compatibility? how In-Reply-To: <200405070429.i474T1t17567@guido.python.org> References: <409AAF77.20900@v.loewis.de> <200405070429.i474T1t17567@guido.python.org> Message-ID: <200405070210.20467.fdrake@acm.org> On Friday 07 May 2004 12:29 am, Guido van Rossum wrote: > I have a different POV. I don't think there is a compelling reason to > change this attribute into a set (I doubt it's so time-critical as to > make a difference) and given that the attribute isn't flagged as > "private" by having a name starting with underscore, I think the > change ought to be reverted. I'd like to suggest that we either document the "right way" to extend the information to support new URL schemes, and possibly add a function as the way to do that. Having to update a whole set of lists from outside the module seems a poor and error-prone way to do this. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From ncoghlan at iinet.net.au Fri May 7 08:07:04 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri May 7 08:07:46 2004 Subject: [Python-Dev] PEP 328: import...as with relative import In-Reply-To: <200405070039.i470dChE026277@cosc353.cosc.canterbury.ac.nz> References: <200405070039.i470dChE026277@cosc353.cosc.canterbury.ac.nz> Message-ID: <409B7BE8.70904@iinet.net.au> Greg Ewing wrote: > I can see your point of view. It just seems a bit strange coming at > from the other direction -- if you can say 'import a.b as z' there > doesn't seem to be any good reason not to say 'import ..a.b as z'. I > can imagine people getting tripped up by this. > > I guess you're right that it doesn't matter much either way. The other point in favour of the current way is that a bare import statement is *always* absolute, which further encourages absolute imports as the standard approach. It's only once someone starts playing with the 'from' syntax for import that they need to know about the difference between absolute and relative import. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Fri May 7 08:18:17 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Fri May 7 08:18:24 2004 Subject: [Python-Dev] horrible thought In-Reply-To: References: Message-ID: <409B7E89.9070402@iinet.net.au> David Abrahams wrote: > Remember the vote on adding conditional expressions? > > I'm afraid that the imminent proliferation of xxxcomps may indicate > the "right" syntax for this feature, if it ever does get added. > > Something like > > (a else b if condition) > > Now I've done my bad deed for the day. That's actually an interesting thought. Is there anything to be learned about conditionals by considering the syntactic sugar for list comps and gen comps? List comp [f(x) for x in original if p(x)]: def _result(f, p, original): lc = [] for x in original: if p(x): lc.append(f(x)) return lc Gen comp (f(x) for x in original if p(x)): def _result(f, p, original): for x in original: if p(x): yield f(x) Conditional (syntactic sugar not defined): def _result(a, b, p): if p: return a else: return b Nope, still suggests (a if p else b) to me. With nesting then being (a if p1 else b if p2 else c if p3 else d). OK, done now. We can just let that discussion go back into its box. . . Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From jhylton at google.com Fri May 7 08:55:45 2004 From: jhylton at google.com (Jeremy Hylton) Date: Fri May 7 08:55:55 2004 Subject: [Python-Dev] what level of backward compatibility? how In-Reply-To: <200405070210.20467.fdrake@acm.org> References: <409AAF77.20900@v.loewis.de> <200405070429.i474T1t17567@guido.python.org> <200405070210.20467.fdrake@acm.org> Message-ID: <1083934545.4530.64.camel@slothrop.corp.google.com> On Fri, 2004-05-07 at 02:10, Fred L. Drake, Jr. wrote: > On Friday 07 May 2004 12:29 am, Guido van Rossum wrote: > > I have a different POV. I don't think there is a compelling reason to > > change this attribute into a set (I doubt it's so time-critical as to > > make a difference) and given that the attribute isn't flagged as > > "private" by having a name starting with underscore, I think the > > change ought to be reverted. > > I'd like to suggest that we either document the "right way" to extend the > information to support new URL schemes, and possibly add a function as the > way to do that. Having to update a whole set of lists from outside the > module seems a poor and error-prone way to do this. It sounds like appending to the list was the right way up until now. That is, extending the list of protocols is a reasonable feature and using a list with a non-underscore name provided one obvious was to do that. Jeremy From guido at python.org Fri May 7 10:31:46 2004 From: guido at python.org (Guido van Rossum) Date: Fri May 7 10:31:51 2004 Subject: [Python-Dev] what level of backward compatibility? how In-Reply-To: Your message of "Fri, 07 May 2004 02:10:20 EDT." <200405070210.20467.fdrake@acm.org> References: <409AAF77.20900@v.loewis.de> <200405070429.i474T1t17567@guido.python.org> <200405070210.20467.fdrake@acm.org> Message-ID: <200405071431.i47EVkw01614@guido.python.org> > > I have a different POV. I don't think there is a compelling reason to > > change this attribute into a set (I doubt it's so time-critical as to > > make a difference) and given that the attribute isn't flagged as > > "private" by having a name starting with underscore, I think the > > change ought to be reverted. [Fred] > I'd like to suggest that we either document the "right way" to extend the > information to support new URL schemes, and possibly add a function as the > way to do that. Having to update a whole set of lists from outside the > module seems a poor and error-prone way to do this. Let's be conservative and not change the code just because the (rarely-needed) API is a bit ugly. Let's just document the existing way and not declare old code that uses it out of date. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Fri May 7 10:38:57 2004 From: guido at python.org (Guido van Rossum) Date: Fri May 7 10:39:03 2004 Subject: [Python-Dev] New Subscriber In-Reply-To: Your message of "Thu, 06 May 2004 23:33:16 CDT." <20040507043316.GB45563@unixbeast.com> References: <20040507043316.GB45563@unixbeast.com> Message-ID: <200405071438.i47EcvT01660@guido.python.org> Seems David Hastings got a rather cold welcome. I would simply have suggested that in general, these days, new modules aren't incorporated into the standard library before they've seen a lot of use "in the field"; and that the appropriate way for David to promote his module would be a webpage (maybe on SourceForge?) and announcements to comp.lang.python.announce. Another suggestion for David is to rename the module -- "CGIpm" seems a rather strange name for a module; the ".pm" in the original stands for "Perl Module" which obviously isn't appropriate for a Python translation. :-) David, success with your new module, and say hi to your Prof! --Guido van Rossum (home page: http://www.python.org/~guido/) From tim.one at comcast.net Fri May 7 11:39:52 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri May 7 11:39:55 2004 Subject: [Python-Dev] Re: New Subscriber In-Reply-To: Message-ID: [David Hastings] > Once again, I'm sorry. Stop apologizing -- it's embarrassing . > I hope there is a way that these e-mails can be removed from the web > page. My intention wasn't to cause clutter. It's not a problem. > Also, my intention was not to replace any modules. I just wanted to > build on it I guess. We did actually use some of the funcionality of > the cgi module. I'm obviously in the wrong place. Not necessarily. While enthusiasm is rare on python-dev some weeks, that doesn't mean we all object to it . My advice is to read a bit first about how Python development gets done: http://www.python.org/dev/ You're already getting the sharp end of what the "Why Work on Python?" link there describes as The most important skill Python can teach is the delicate skill of working in a diverse group. I hope you stick around, and bring your module enhancements to a successful release! From FBatista at uniFON.com.ar Fri May 7 15:03:27 2004 From: FBatista at uniFON.com.ar (Batista, Facundo) Date: Fri May 7 15:06:06 2004 Subject: [Python-Dev] Re: New Subscriber Message-ID: [Tim Peters] #- doesn't mean we all object to it . My advice is to #- read a bit first #- about how Python development gets done: #- #- http://www.python.org/dev/ David, I repost these Tim statements because as a new Python developer I found something amusing. If you follow the rules, you'll have a lot of valuable humans that donate their time to help you to help everybody. With follow the rules I mean to get used to how things get done here (for example: learn how to write a PEP, that is a wonderful tool). Welcome! . Facundo From doko at cs.tu-berlin.de Sun May 9 09:13:31 2004 From: doko at cs.tu-berlin.de (Matthias Klose) Date: Sun May 9 09:13:35 2004 Subject: [Python-Dev] Source of the FAQ? Message-ID: <16542.11899.318040.19846@gargle.gargle.HOWL> Is the source text for the FAQ's available in some form, that could be included in a packaged form for distribution? Thanks, Matthias From bac at OCF.Berkeley.EDU Sun May 9 14:43:40 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sun May 9 14:43:54 2004 Subject: [Python-Dev] Source of the FAQ? In-Reply-To: <16542.11899.318040.19846@gargle.gargle.HOWL> References: <16542.11899.318040.19846@gargle.gargle.HOWL> Message-ID: <409E7BDC.30805@ocf.berkeley.edu> Matthias Klose wrote: > Is the source text for the FAQ's available in some form, that could be > included in a packaged form for distribution? > For basically all pages on the python.org web site, just change the extension from html to ht and you will get the text version that is worked on by the web team. It will most likely either be HTML or reST. Luckily most of the FAQs have been converted to reST for easier editing (thanks to AM Kuchling and anyone else who has helped out on the conversion). -Brett From bac at OCF.Berkeley.EDU Sun May 9 17:04:51 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Sun May 9 17:05:05 2004 Subject: [Python-Dev] python-dev Summary for 2004-04-01 through 2004-04-30 [rough draft] Message-ID: <409E9CF3.5060504@ocf.berkeley.edu> By not getting into any details on method decorators and relative imports I was actually able to finish this summary. =) I would like to get this out Wednesday night, so please send in your corrections by then. Thanks. ------------------------------- ===================== Summary Announcements ===================== Well, as you may have noticed, I wasn't able to keep up the pace of releasing the summaries twice in a month. Joys of school. Hopefully over the summer I will be able to semi-monthly releases since I will only be working and not have homework hanging over my head 24/7. To give everyone a a heads-up, a release candidate for Python 2.3.4 is going to be released May 13/14 (depending where you are on the globe). If you have time, please download it and run the regression test suite (instructions can be found in the module documentation for the 'test' package) and report any issues you have. ========= Summaries ========= -------------------------------------------------------- Method decorators and the discussion that will never end -------------------------------------------------------- Method decorators and `PEP 318`_ were discussed ad nausea this past month. Read the PEP for the updated details. .. _PEP 318: http://www.python.org/peps/pep-0318.html Contributing threads: - `PEP 318 -- a couple use cases `__ - `PEP 318: Decorators last before colon/bakeoff `__ - `PEP 318 bake-off? `__ - `PEP 318: Is decorators the right term for these things? `__ - `Re: PEP 318: Let's propose some useful built-in decorators `__ - `Help with PEP 318 `__ - `PEP 318: Properties `__ - `PEP 318: More examples of decorator use `__ - `PEP 318: How I would implement decorators `__ - `PEP 318 - check for consensus `__ ------------------------------------------------------------ Relative imports: the *other* discussion that will never end ------------------------------------------------------------ `PEP 328`_, which covers relative imports, has been updated to cover its lengthy discussion. It seems the PEP is pretty much finalized. .. _PEP 328: http://www.python.org/peps/pep-0328.html Contributing threads: - `Re: PEP 328 -- relative and multi-line import `__ - ` Some comments on PEP 328 (absolute/relative imports) `__ - `PEP 328: __path__ `__ - `from ...sys import path `__ -------------------- stdlib: Python or C? -------------------- The question of whether stdlib modules should be coded in Python or C came up again. Armin Rigo posted some numbers comparing performance of heapq using the Python version both in a stock Python interpreter and in Psyco_ and then the C version. Armin's quick timings showed that the Python version of heapq was actually after than the C version when used in Psyco. If it is possible to get the Python version to go faster, should it and future modules be coded in Python? The usual arguments over maintenance and readability were brought up. In the end the Python version of heapq was put back into the library with the C version renamed to _heapq. Contributing threads: - `Python is faster than C `__ ----------------------------------- Getting an Interest Group on Artima ----------------------------------- Bill Venners of Artima_ asked if there was anyone who wanted to act as a moderator for an interest group for Python on the site. If you are interested, read the email for details. .. _Artima: http://www.artima.com/ Contributing threads: - `Python Interest Group `__ ------------------------------- Informative reprs for iterators ------------------------------- Raymond Hettinger suggested having the repr of iterators for built-in types list the first three objects returned by the iterator. That way you would actually know what was in the iterator. It was suggested only for iterators where the length of the iterator was known and extracting the first three objects from the iterator to display them would be cheap. In the end, though, it was not added for built-in types. Some itertools iterators, though, did get more informative reprs. Contributing threads: - `More imformative iterator representations `__ - `Proposed iterator representations `__ ------------------------------ Turning globals into constants ------------------------------ Raymond Hettinger came up with some code that could take a code object and make all globals constants, thus speeding up access. It was proposed for the stdlib in `PEP 329`_, but was rejected in the end for being too "hackish". It did bring up the idea of trying to come up with a way to flag built-ins that might be masked. If the compiler knew what built-ins were not going to be masked a direct lookup in the built-in namespace would be all that was needed to access a built-in. As it stands now, though, since an external module could inject a shadowing function into the global namespace of a module (such as having module bar having the code ``import foo; foo.len = lambda x: 42`` to shadow 'len' in the module foo), Python has to check the global namespace first, and then the built-in namespace for all non-local name accesses. That is costly and it could possibly be avoided since the common case is to not shadow a built-in. Guido suggested having programmers have to specify when a built-in could be shadowed. This could be compared to the 'volatile' keyword in C; the programmer has to specify when the compiler should not try to optimize access. This seemed like the best way to go since forcing the programmer to flag when a built-in will *not* be shadowed would be more work for the common case. Obviously this would all break backwards-compatibility and thus would either require a __future__ statement initially or would be like true division and always be a __future__ import until Python 3.0 . But it was all just being thrown around with no concrete ideas on implementation and such. .. _PEP 329: http://www.python.org/peps/pep-0329.html Contributing threads: - `Candidate Function Decorator `__ - `PEP 329: Treating Builtins as Constants... `__ - `peps 329, 266, 267 `__ ----------------------------- Decimal "stuff" was discussed ----------------------------- Two big topics came up about the Decimal package. The first one was over whether there should be an exponent limit. The discussion went back and forth, but in the end it was decided that having a limit is good since it will prevent undetected overflow and underflow. The second topic was about adding extra methods. This was shot down since the point of the Decimal module is to follow the standard as put forth at http://www2.hursley.ibm.com/decimal/decarith.html . After the module has been in the library and seen some wide usage then possible additions beyond the standard could be made. exponent limit repr (passable to eval if possible) sticking to spec initially Contributing threads: - `Decimal data type issues `__ - `Decimal conversion to string `__ - `Decimal - context manipulation `__ ------------------------------------------------- Magic hashing numbers segues into JIT compilation ------------------------------------------------- Raymond Hettinger suggested using a different number for hashing since it could be expressed in shifts and additions. But Tim Peters said it was a bad idea to futz with the number since the current one had proven its usefulness for so long. Since part of the justification for changing to another number was the number of cycles it would take to do the calculations, the discussion shifted to JIT compilation. Clarification for JIT compared to specialized compilation (former does single compiled version of a function while the latter does multiple versions based on the argument types) came up along with wondering how some of the optimizations used in the Self_ language could be applied to Python. .. _Self: http://research.sun.com/self/language.html Contributing threads: - `String hash function multiplier `__ - `Optimization targets - refcount `__ ---------------------------------- Modules that need some documentin' ---------------------------------- Yours truly compiled a list of modules that need documentation that can be found both in the email starting this thread (although I made one mistake on that list of including the imp module) or at http://www.python.org/cgi-bin/moinmoin/ModulesThatNeedDocs where Michael Chermside was nice enough to put it on up on the wiki. Writing docs for modules is not that hard and is a great way to help out. See http://www.python.org/dev/doc/devel/doc/doc.html on how to do documentation using LaTeX. And if that scares you, even just writing the docs in reST_ is helpful since someone else can volunteer to convert it to LaTeX. Contributing threads: - `Possible modules that could use docs `__ -------------------------------------- A change in the name of the CVS server -------------------------------------- SourceForge_ started forcing people to use cvs.sf.net:/cvsroot/python as the location of the CVS server as compared to the previously supported (but not "official") path of cvs.python.sf.net:/cvsroot/python . Barry Warsaw posted a Python script by Greg Ward that would traverse a CVS checkout and change the name of the CVS server. That script can be found at http://mail.python.org/pipermail/python-dev/2004-April/044593.html . Contributing threads: - `SSH problems getting into SourceForge's CVS? `__ --------------------------- A ConfigParser replacement? --------------------------- Dan Gass presented his `config.py`_ to python-dev to see if there was any interest in adding it. While he was given the standard response that no new modules will be accepted for inclusion in the stdlib until it has seen widespread acceptance and use by the community, Barry Warsaw suggested a possible ConfigParser replacement shootout ala the `getopt-sig`_ and how it was decided that optparse (aka Optik_) should be the suggested command-line parser used. This won't happen for 2.4, but if people are interested enough this could be done for 2.5 if someone decides to spear-head this. People also immediately pointed out ZConfig_ (by our very own Fred Drake) as another possibility. .. _config.py: http://config-py.sf.net/ .. _getopt-sig: http://mail.python.org/mailman/listinfo/getopt-sig .. _Optik: http://optik.sourceforge.net/ .. _ZConfig: http://www.zope.org/Members/fdrake/zconfig/ Contributing threads: - `Proposal: A more powerful alternative to ConfigParser `__ - `Proposal: A more powerful alternativeto... `__ ----------------------------------------- Moving forward with generator expressions ----------------------------------------- Guido said he wanted to get generator expressions into CVS. While this is good and all, it still doesn't address the whole argument over whether early or late binding for variables in the genexps should be used (in case you have not been following, early binding would have a genexp save the state of the variables used in the genexp at creation time while late binding will use the state of the variables at the time of each execution of the genexp). Since Guido prefers the late binding for simplicity, he suggested using it for 2.4a1 and a2. If it was found that late binding was bad then for 2.4b1 early binding could be used. No one had issues with that idea. What people did was the whole thing about late bindings period. Basically the issue boils down to whether late binding is too much of a surprise for the cases where they can have issues. But Greg Ewing pointed out that the use case that genexps were being created for were for arguments to functions; a use case where the genexp is used immediately and whose possible abuse of variables would not be an issue. It was realized that genexps are not exactly the same as listcomps and thus not a direct replacement. Genexps are meant for places where memory is going to be an issue; if a listcomp would do just a good of a job, then just go ahead and use a listcomp. Contributing threads: - `PEP 289 - Generator Expressions - Let's Move Forward `__ --------------------------------------------------- How should gettext return strings from an .mo file? --------------------------------------------------- Gustavo Niemeyer noticed that if you called 'gettext.gettext' it would return the exact bytes stored in the .mo file. Apparently `GNU gettext`_ translates the bytes according to the current locale instead of just passing them through untouched. Gustavo wanted to make Python match the GNU implementation. Barry Warsaw and Martin v. Loewis disagreed, though. Since Python's way of doing it has been that way for so long, they didn't want to break backwards-compatibility. It was also stated that code should use 'gettext.ugettext' anyway. In the end it was agreed upon that if Gustavo wanted this functionality he should add another function to 'gettext' (such as 'lgettext') that did what he wanted. .. _GNU gettext: http://www.gnu.org/software/gettext/gettext.html Contributing threads: - `Small issues in gettext support `__ ----------------------------------------------- And how long have you been hacking on the core? ----------------------------------------------- Anthony Baxter, fearless release manager, decided to run the Misc/ACKS file through ``cvs annotate`` which prints out who edited every line and when it was edited and then he sorted it. This was to see when people were added to the ACKS file and thus find our roughly how long people have been hacking on the Python core. The file wasn't started until January 26, 1994 so a lot of the "old-timers" don't have exact dates (of which Anthony is one of them). Yours truly was added on July 19, 2002. Contributing threads: - `today's pointless, yet slightly terrifying, factoid(s) `__ From fumanchu at amor.org Sun May 9 19:23:32 2004 From: fumanchu at amor.org (Robert Brewer) Date: Sun May 9 19:26:24 2004 Subject: [Python-Dev] python-dev Summary for 2004-04-01 through 2004-04-30[rough draft] Message-ID: Brett C wrote Yet Another Great Summary: > Subject: [Python-Dev] python-dev Summary for 2004-04-01 > through 2004-04-30[rough draft] > > I would like to get this out Wednesday night, so please send in your > corrections by then. Thanks. >8 > -------------------- > stdlib: Python or C? > -------------------- > The question of whether stdlib modules should be coded in Python or C > came up again. Armin Rigo posted some numbers comparing > performance of > heapq using the Python version both in a stock Python > interpreter and in > Psyco_ and then the C version. Armin's quick timings showed that the > Python version of heapq was actually after than the C version > when used in Psyco. I think you mean "faster than" instead of "after than". >8 > ----------------------------------------- > Moving forward with generator expressions > ----------------------------------------- >8 > > What people did was the whole thing about late bindings period. I don't understand this sentence. HTH, Robert Brewer MIS Amor Ministries fumanchu@amor.org From greg at cosc.canterbury.ac.nz Sun May 9 21:03:29 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun May 9 21:03:48 2004 Subject: [Python-Dev] PEP 328: import...as with relative import In-Reply-To: <409B7BE8.70904@iinet.net.au> Message-ID: <200405100103.i4A13Tjw031202@cosc353.cosc.canterbury.ac.nz> Nick Coghlan : > The other point in favour of the current way is that a bare import > statement is *always* absolute, which further encourages absolute > imports as the standard approach. I don't understand why absolute imports should be regarded as the "standard approach". Absolute imports are appropriate for some things, relative imports are appropriate for other things. You can't say that one is "standard" and the other isn't. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From bob at redivi.com Sun May 9 21:17:53 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun May 9 21:18:03 2004 Subject: [Python-Dev] PEP 328: import...as with relative import In-Reply-To: <200405100103.i4A13Tjw031202@cosc353.cosc.canterbury.ac.nz> References: <200405100103.i4A13Tjw031202@cosc353.cosc.canterbury.ac.nz> Message-ID: On May 9, 2004, at 9:03 PM, Greg Ewing wrote: > Nick Coghlan : > >> The other point in favour of the current way is that a bare import >> statement is *always* absolute, which further encourages absolute >> imports as the standard approach. > > I don't understand why absolute imports should be regarded > as the "standard approach". Absolute imports are appropriate > for some things, relative imports are appropriate for other > things. You can't say that one is "standard" and the other > isn't. Either way, right now it's a real pain when you have a relative and absolute module with the same name. You simply can't import the absolute one by standard means. It would be at least nice if you could explicitly state that some import should be absolute (given the current relative-first search order). I generally use absolute imports everywhere, because I've found it generally avoids problem -- but it certainly wasn't my first instinct when learning Python. -bob From aahz at pythoncraft.com Sun May 9 21:44:09 2004 From: aahz at pythoncraft.com (Aahz) Date: Sun May 9 21:44:13 2004 Subject: [Python-Dev] PEP 328: import...as with relative import In-Reply-To: <200405100103.i4A13Tjw031202@cosc353.cosc.canterbury.ac.nz> References: <409B7BE8.70904@iinet.net.au> <200405100103.i4A13Tjw031202@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040510014409.GA2603@panix.com> On Mon, May 10, 2004, Greg Ewing wrote: > Nick Coghlan : >> >> The other point in favour of the current way is that a bare import >> statement is *always* absolute, which further encourages absolute >> imports as the standard approach. > > I don't understand why absolute imports should be regarded as the > "standard approach". Absolute imports are appropriate for some things, > relative imports are appropriate for other things. You can't say that > one is "standard" and the other isn't. The way I'd put it is that relative imports are used only by experts: people building chunks of Python code large enough to require packages. Therefore absolute imports make more sense as the default. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! From guido at python.org Sun May 9 23:56:04 2004 From: guido at python.org (Guido van Rossum) Date: Sun May 9 23:56:14 2004 Subject: [Python-Dev] PEP 328: import...as with relative import In-Reply-To: Your message of "Mon, 10 May 2004 13:03:29 +1200." <200405100103.i4A13Tjw031202@cosc353.cosc.canterbury.ac.nz> References: <200405100103.i4A13Tjw031202@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405100356.i4A3u5p13615@guido.python.org> > I don't understand why absolute imports should be regarded > as the "standard approach". Absolute imports are appropriate > for some things, relative imports are appropriate for other > things. You can't say that one is "standard" and the other > isn't. Maybe, but somewhere "underneath" all imports are converted to absolute imports before they are carried out. This is an important normalization since every module ought to have exactly one true name, and that must be its absolute name. --Guido van Rossum (home page: http://www.python.org/~guido/) From ncoghlan at iinet.net.au Mon May 10 03:33:13 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon May 10 03:33:32 2004 Subject: [Python-Dev] Generator expression used in test suite? Message-ID: <409F3039.2070009@iinet.net.au> I just finished setting up a Mandrake 9.2 box for Python development, and got a doctest failure in test_deque (after running ./configure, make, make test). It looks like one of the examples (roundrobin) is using a generator expression to populate the deque. Since this will work once generator expressions hit CVS, is there any point in raising a bug report for it? Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From ncoghlan at iinet.net.au Mon May 10 06:06:32 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon May 10 06:06:43 2004 Subject: [Python-Dev] SF Bug 933068 - Unexpected result from super() call Message-ID: <409F5428.20709@iinet.net.au> This bug report can probably be closed (the reported error is the result of supplying 'self' an an argument to the already bound method returned from the super call). The relevant docs for super looked fairly clear to me, too, so I don't think it is a documentation bug, either. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From barry at python.org Mon May 10 09:50:03 2004 From: barry at python.org (Barry Warsaw) Date: Mon May 10 09:50:11 2004 Subject: [Python-Dev] Generator expression used in test suite? In-Reply-To: <409F3039.2070009@iinet.net.au> References: <409F3039.2070009@iinet.net.au> Message-ID: <1084197002.1735.451.camel@anthem.wooz.org> On Mon, 2004-05-10 at 03:33, Nick Coghlan wrote: > Since this will work once generator expressions hit CVS, is there any > point in raising a bug report for it? Yes. The standard test suite should never have failures. -Barry From ncoghlan at iinet.net.au Mon May 10 10:27:47 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Mon May 10 10:27:53 2004 Subject: [Python-Dev] Generator expression used in test suite? In-Reply-To: <1084197002.1735.451.camel@anthem.wooz.org> References: <409F3039.2070009@iinet.net.au> <1084197002.1735.451.camel@anthem.wooz.org> Message-ID: <409F9163.7020503@iinet.net.au> Barry Warsaw wrote: > On Mon, 2004-05-10 at 03:33, Nick Coghlan wrote: > > >>Since this will work once generator expressions hit CVS, is there any >>point in raising a bug report for it? > > > Yes. The standard test suite should never have failures. Done. Bug ID is 951270. http://sourceforge.net/tracker/?func=detail&aid=951270&group_id=5470&atid=105470 The hard fix is to get generator expressions checked in, the easy fix is to put '[' and ']' around the expression in the roundrobin example. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From tjreedy at udel.edu Mon May 10 12:19:49 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Mon May 10 12:20:00 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested Message-ID: The following was posted in the last 25 hours on c.l.p: "Xaver Hinterhuber" wrote in message news:c7o6al$i1j$1@online.de... > codeObject = new.code( > 0, # argcount > 0, # nlocals > 0, # stacksize > 0, # flags > codeString, # code > (), # consts > (), # names > (), # varnames > 'content', # filename > 'content', # name > 3, # first line number > codeString # lnotab > ) > f = new.function(codeObject, dict, 'f') > f() > > Everything runs fine, until the function is called with f(). > When python tries to execute f(), the core dump happens. > I don't have any clue why python core dumps. > The codeString is nothing complex, its a one-liner. ... but, as too typical of new posters, not posted ;-) My question: I understand that core dumping by the Python interpreter is normally considered a bug -- one that should be reported on SF -- and fixed. But does that principle apply to this sort of use (or probably abuse) of the new module. In other words, does the 'no core dump' contract include a 'byte code compiled by the interpreter' proviso. Or is ceval intended to be robust against random strings masquerading as valid byte code (which would seem require an InvalidByteCode exception). Or in still other words, is there a bug that I should encourage reporting of, or is 'don't do that' the extent of a proper response? Terry J. Reedy From fumanchu at amor.org Mon May 10 12:44:19 2004 From: fumanchu at amor.org (Robert Brewer) Date: Mon May 10 12:47:10 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested Message-ID: Terry Reedy wrote: > > The following was posted in the last 25 hours on c.l.p: > > "Xaver Hinterhuber" wrote in message > news:c7o6al$i1j$1@online.de... > > codeObject = new.code( > > 0, # argcount > > 0, # nlocals > > 0, # stacksize > > 0, # flags > > codeString, # code > > (), # consts > > (), # names > > (), # varnames > > 'content', # filename > > 'content', # name > > 3, # first line number > > codeString # lnotab > > ) > > f = new.function(codeObject, dict, 'f') > > f() > > > > Everything runs fine, until the function is called with f(). > > When python tries to execute f(), the core dump happens. > > I don't have any clue why python core dumps. > > The codeString is nothing complex, its a one-liner. > > ... but, as too typical of new posters, not posted ;-) > > My question: I understand that core dumping by the Python > interpreter is > normally considered a bug -- one that should be reported on SF -- and > fixed. But does that principle apply to this sort of use (or probably > abuse) of the new module. In other words, does the 'no core > dump' contract > include a 'byte code compiled by the interpreter' proviso. > Or is ceval > intended to be robust against random strings masquerading as > valid byte > code (which would seem require an InvalidByteCode exception). > Or in still > other words, is there a bug that I should encourage reporting > of, or is > 'don't do that' the extent of a proper response? There are other problems; once I passed a unicode string in co_names--it took me quite a while to track down THAT mistake. :) Perhaps some more rigorous assertions, returning ValueErrors? If not in C, then in the 'new' module, since that seems to be the vector of disaster for Xaver, above. Perhaps this would also be an argument for keeping the 'new' module around longer, as a 'safer' Pythonic wrapper over bare code/function construction. Robert Brewer MIS Amor Ministries fumanchu@amor.org From python at rcn.com Mon May 10 13:25:00 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon May 10 13:26:35 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: Message-ID: <008801c436b3$b9689020$e841fea9@oemcomputer> > is ceval > intended to be robust against random strings masquerading as valid byte > code (which would seem require an InvalidByteCode exception). I don't think ceval.c can protect itself against invalid bytecoding without tons of additional checks. For example, BUILD_LIST n, can potentially pop past the beginning of the stack where n is a random code. Raymond From altis at semi-retired.com Mon May 10 13:57:08 2004 From: altis at semi-retired.com (Kevin Altis) Date: Mon May 10 13:57:12 2004 Subject: [Python-Dev] Fwd: OSCON discount for core contributors Message-ID: <74B72782-A2AB-11D8-8113-000A9598382A@semi-retired.com> OSCON will be held at the Portland Marriott Downtown in Portland, Oregon, July 26-30, 2004. Gina Blaber at O'Reilly is offering a special 20% discount for 20 core Python developers. Anyone on python-dev qualifies, so we'll handle this as first-come, first-served. Note that if you have attended OSCON in the past you are already qualified for a 20% alumni discount in addition to the 20% early bird discount, so you don't need to use this special developer discount. If you qualify for the academic discounts, then those are better discount deals, and again you don't need to use this developer discount code. Regardless of which discount you use, if you plan to attend OSCON, you'll definitely save on registration by registering before June 18, 2004 for an additional 20% early bird discount. Registration: http://conferences.oreillynet.com/pub/w/29/register.html OSCON site: http://conferences.oreillynet.com/os2004/ Python 12 track at OSCON: http://conferences.oreillynet.com/pub/w/29/track_python.html http://conferences.oreillynet.com/pub/w/29/tutorial_python.html Hope to see you in Portland in July, ka --- Kevin Altis altis@semi-retired.com http://altis.pycs.net/ Python 12 co-chair Begin forwarded message: > From: "Gina Blaber" > Cc: gina@oreilly.com, lholder@oreilly.com > Subject: OSCON discount for core contributors > > Hi Kevin, > > O'Reilly would like to help the core contributors of important open > source software projects to attend OSCON. Therefore, I'd like to > offer a discount of 15% off the registration price to 20 core > contributors to Python. This discount can be combined with the 20% > Early Bird pricing, which ends June 18, 2004. > > The registration discount code for the 20 core contributors to use is > os04py. I've cc'ed Linda Holder on this message; Linda can answer > any registration questions you may have. > > Thanks, and I look forward to seeing you at OSCON! > > -Gina Blaber > -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/enriched Size: 2142 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040510/37623b6e/attachment.bin From python at rcn.com Mon May 10 12:28:00 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon May 10 15:12:17 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: Message-ID: <007001c436ab$c375c7c0$e841fea9@oemcomputer> > My question: I understand that core dumping by the Python interpreter is > normally considered a bug -- one that should be reported on SF -- and > fixed. But does that principle apply to this sort of use (or probably > abuse) of the new module. In other words, does the 'no core dump' > contract > include a 'byte code compiled by the interpreter' proviso. Or is ceval > intended to be robust against random strings masquerading as valid byte > code (which would seem require an InvalidByteCode exception). Or in still > other words, is there a bug that I should encourage reporting of, or is > 'don't do that' the extent of a proper response? > > Terry J. Reedy File the bug report and assign to me. Raymond From tjreedy at udel.edu Mon May 10 16:07:56 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Mon May 10 16:11:05 2004 Subject: [Python-Dev] Re: Is core dump always a bug? Advice requested References: <007001c436ab$c375c7c0$e841fea9@oemcomputer> Message-ID: "Raymond Hettinger" wrote in message news:007001c436ab$c375c7c0$e841fea9@oemcomputer... > File the bug report and assign to me. Done. http://sourceforge.net/tracker/index.php?func=detail&aid=951482&group_id=5470&atid=105470 tjr From Tino.Lange at isg.de Mon May 10 18:53:19 2004 From: Tino.Lange at isg.de (Tino Lange) Date: Mon May 10 18:54:37 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <200405070424.i474OtT17542@guido.python.org> References: <61957B071FF421419E567A28A45C7FE59AFB0A@mailbox.nameconnector.com> <200405070424.i474OtT17542@guido.python.org> Message-ID: <40A007DF.1050809@isg.de> Guido van Rossum wrote: [Patch #945642] >>> (TL) Is it possible to include the patch #945642 in 2.3.4? >> (GT) I'd like to see this in 2.3.4 too [...] > (GvR) Haven't seen the patch, but the description of the fix makes me go "+1 > please!" :-) Hi! Hmmm, the 13th is coming soon and still noone has checked that in ... But I think there was enough agreement that this really needs to be merged and checked in soon? I understand that there's lots of things to do before such an maintenance release ... If you like I can offer to do the changes in CVS by myself - if you don't mind giving me checkin' rights for that. I will change _ssl.c accordingly for CVS HEAD and also backport it for the 'release23-maint' branch. Of course I'll also try to check/test once again that it doesn't break anything. No problem if you don't think that this is OK -- but then I'd like to ask again, that someone of you looks at it soon for 2.3.4 - according to the posted timeline there are only 3 days left... Thanks and best regards! Tino From flaw at rhid.com Mon May 10 21:23:04 2004 From: flaw at rhid.com (James William Pye) Date: Mon May 10 21:23:05 2004 Subject: [Python-Dev] Access to generator type Message-ID: <20040511012304.GE29924@void.ph.cox.net> Greetings, Can access to the generator type(gentype in ceval.c) and associated data/structures be made available in a public interface/header(genobject.h)? I know I can get the type by compiling a little generator and fetching its type, but that seems to be a bit redundant as each project that needs it, needs to make its own, nonstandardized version of access. Regards, James William Pye -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 479 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040510/53cec2e2/attachment.bin From greg at cosc.canterbury.ac.nz Mon May 10 21:57:09 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon May 10 21:58:35 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: Message-ID: <200405110157.i4B1v9OC000490@cosc353.cosc.canterbury.ac.nz> Terry Reedy : > Or is ceval intended to be robust against random strings masquerading > as valid byte code (which would seem require an InvalidByteCode > exception). As long as it's possible to attempt to execute arbitrary strings as bytecode, I'd say ceval should be robust against this. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one at comcast.net Mon May 10 22:27:03 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon May 10 22:27:21 2004 Subject: [Python-Dev] Access to generator type In-Reply-To: <20040511012304.GE29924@void.ph.cox.net> Message-ID: [James William Pye] > Can access to the generator type(gentype in ceval.c) and associated > data/structures be made available in a public > interface/header(genobject.h)? Possibly, if someone wants it enough to make a patch (including doc changes). There are many types that aren't exposed at the C level, BTW. Advertising their internals has an inhibiting effect on evolution. The generator type has been around long enough w/o change that the value in keeping it private has diminished. > I know I can get the type by compiling a little generator and fetching its > type, but that seems to be a bit redundant as each project that needs it, > needs to make its own, nonstandardized version of access. Anyone who wants it can do from types import GeneratorType today (types is a standard Python module). From guido at python.org Mon May 10 23:28:21 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 10 23:32:48 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: Your message of "Tue, 11 May 2004 13:57:09 +1200." <200405110157.i4B1v9OC000490@cosc353.cosc.canterbury.ac.nz> References: <200405110157.i4B1v9OC000490@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405110328.i4B3SLB15406@guido.python.org> > As long as it's possible to attempt to execute arbitrary strings > as bytecode, I'd say ceval should be robust against this. This ought to be a long-term project then: write a bytecode verifier. It's not a trivial task! --Guido van Rossum (home page: http://www.python.org/~guido/) From greg at cosc.canterbury.ac.nz Tue May 11 00:20:56 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue May 11 00:21:09 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405110328.i4B3SLB15406@guido.python.org> Message-ID: <200405110420.i4B4Ku3P000714@cosc353.cosc.canterbury.ac.nz> Guido: > > As long as it's possible to attempt to execute arbitrary strings > > as bytecode, I'd say ceval should be robust against this. > > This ought to be a long-term project then: write a bytecode verifier. > It's not a trivial task! When I wrote that, I was assuming that ceval was already mostly robust in this respect, and that what was being reported was a new hole recently opened up. But it appears I was gravely mistaken, and that ceval has been full of gaping holes from the beginning. I'm disappointed to learn this, because I had always regarded it as an axiom that no Python-level code should be capable of crashing the interpreter, and if it can, this represents a bug. However, it seems this axiom has not been adhered to in the design of ceval. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From bob at redivi.com Tue May 11 00:49:37 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue May 11 00:49:44 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405110420.i4B4Ku3P000714@cosc353.cosc.canterbury.ac.nz> References: <200405110420.i4B4Ku3P000714@cosc353.cosc.canterbury.ac.nz> Message-ID: <9B5AC266-A306-11D8-BCDD-000A95686CD8@redivi.com> On May 11, 2004, at 12:20 AM, Greg Ewing wrote: > Guido: > >>> As long as it's possible to attempt to execute arbitrary strings >>> as bytecode, I'd say ceval should be robust against this. >> >> This ought to be a long-term project then: write a bytecode verifier. >> It's not a trivial task! > > When I wrote that, I was assuming that ceval was already mostly robust > in this respect, and that what was being reported was a new hole > recently opened up. > > But it appears I was gravely mistaken, and that ceval has been full of > gaping holes from the beginning. > > I'm disappointed to learn this, because I had always regarded it as an > axiom that no Python-level code should be capable of crashing the > interpreter, and if it can, this represents a bug. However, it seems > this axiom has not been adhered to in the design of ceval. Well with modules like ctypes and PyObjC becoming more common, it's pretty easy to make the interpreter crash whenever you really want it to :) Python is no Java, there's not even a real attempt to bake security into it.. only good programming practices. Doing extensive checking of bytecode at runtime would make Python's interpreter much slower than it already is. I can't imagine that someone with a real need for mangling bytecode is going to need their hand held here. In most cases such a person is probably already pretty familiar with the VM at the CPython implementation level anyways (since it's not really documented elsewhere, as far as I know). As far as bytecode verification goes, I would imagine that running it through PyPy might be good enough to see if it's sane (assuming PyPy supports the same bytecode in a similar enough way). -bob From greg at cosc.canterbury.ac.nz Tue May 11 01:04:43 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue May 11 01:05:16 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <9B5AC266-A306-11D8-BCDD-000A95686CD8@redivi.com> Message-ID: <200405110504.i4B54hmp000811@cosc353.cosc.canterbury.ac.nz> Bob Ippolito : > Doing extensive checking of bytecode at runtime would make Python's > interpreter much slower than it already is. Another approach would be to make the bytecode a special object that can only be created by the compiler (or other C code). If you wanted custom bytecode, you'd have to build an AST and feed it to the compiler, or write an extension module. No doubt some bytecode hackers would find this too limiting, however. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From anthony at interlink.com.au Tue May 11 03:04:56 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue May 11 03:05:18 2004 Subject: [Python-Dev] Python 2.3.4 schedule In-Reply-To: <40A007DF.1050809@isg.de> References: <61957B071FF421419E567A28A45C7FE59AFB0A@mailbox.nameconnector.com> <200405070424.i474OtT17542@guido.python.org> <40A007DF.1050809@isg.de> Message-ID: <40A07B18.10005@interlink.com.au> Tino Lange wrote: > Guido van Rossum wrote: > > [Patch #945642] > >>>> (TL) Is it possible to include the patch #945642 in 2.3.4? >>> >>> (GT) I'd like to see this in 2.3.4 too [...] >> >> (GvR) Haven't seen the patch, but the description of the fix makes me >> go "+1 >> please!" :-) > > > Hi! > > Hmmm, the 13th is coming soon and still noone has checked that in ... > > But I think there was enough agreement that this really needs to be > merged and checked in soon? > > I understand that there's lots of things to do before such an > maintenance release ... If you like I can offer to do the changes in CVS > by myself - if you don't mind giving me checkin' rights for that. I will > change _ssl.c accordingly for CVS HEAD and also backport it for the > 'release23-maint' branch. Of course I'll also try to check/test once > again that it doesn't break anything. > > No problem if you don't think that this is OK -- but then I'd like to > ask again, that someone of you looks at it soon for 2.3.4 - according to > the posted timeline there are only 3 days left... That's correct. If this is really wanted for 2.3.4, someone needs to step up and get it checked in. Today. -- Anthony Baxter It's never too late to have a happy childhood. From guido at python.org Tue May 11 10:12:00 2004 From: guido at python.org (Guido van Rossum) Date: Tue May 11 10:12:07 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: Your message of "Tue, 11 May 2004 16:20:56 +1200." <200405110420.i4B4Ku3P000714@cosc353.cosc.canterbury.ac.nz> References: <200405110420.i4B4Ku3P000714@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405111412.i4BEC0h16352@guido.python.org> Greg: > > > As long as it's possible to attempt to execute arbitrary strings > > > as bytecode, I'd say ceval should be robust against this. > > Guido: > > This ought to be a long-term project then: write a bytecode verifier. > > It's not a trivial task! Greg: > When I wrote that, I was assuming that ceval was already mostly robust > in this respect, and that what was being reported was a new hole > recently opened up. > > But it appears I was gravely mistaken, and that ceval has been full of > gaping holes from the beginning. > > I'm disappointed to learn this, because I had always regarded it as an > axiom that no Python-level code should be capable of crashing the > interpreter, and if it can, this represents a bug. However, it seems > this axiom has not been adhered to in the design of ceval. No, the holes aren't in ceval. The hole is the ability to create new code objects from arbitrary arguments, which wasn't planned at all when ceval was designed. I was always reluctant to add this capability, but finally allowed it, consiciously opening up the hole, under pressure from various people who wanted to do bytecode level hacking. The responsibility to validate bytecode should really be in new.code() (and probably in new.function, which requires the code to behave in certain ways). Note that while I like the general principle that Python code shouldn't be able to cause segfaults, I don't think we should see that property as a safety *guarantee*. It's more a code-of-honor of the Python implementation. There are many violations of the principle, mostly in platform-specific extensions (it is easy to crash Python with certain win32 primitives, for example, and it seems unreasonable to expect otherwise). --Guido van Rossum (home page: http://www.python.org/~guido/) From dmgass at hotmail.com Tue May 11 10:22:19 2004 From: dmgass at hotmail.com (Dan Gass) Date: Tue May 11 10:22:23 2004 Subject: [Python-Dev] Applying the PSF license to the config-py project Message-ID: I've gotten negative feedback on my use of the GPL on this project, especially since I hope to get this module into the python core. I am going to make another release soon with some minor (backward compatible) improvements and would like to change the license to the PSF. I cannot find good definitive documentation on how to use this license in this situation. What text do I need to place or reference in my code? If someone could direct me to that documentation (if it exists) I would very much appreciate it. BTW, feedback on this project has been positive but slow. There have been 55 downloads in the 8 days it has been out there. http://config-py.sourceforge.net/. For better/worse I'm hoping to hear from the Zope/ZConfig guys :-) Thanks, Dan Gass _________________________________________________________________ Express yourself with the new version of MSN Messenger! Download today - it's FREE! http://messenger.msn.com/go/onm00200471ave/direct/01/ From aahz at pythoncraft.com Tue May 11 10:27:34 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue May 11 10:27:58 2004 Subject: [Python-Dev] Applying the PSF license to the config-py project In-Reply-To: References: Message-ID: <20040511142734.GA15795@panix.com> On Tue, May 11, 2004, Dan Gass wrote: > > I am going to make another release soon with some minor (backward > compatible) improvements and would like to change the license to the > PSF. I cannot find good definitive documentation on how to use this > license in this situation. What text do I need to place or reference > in my code? If someone could direct me to that documentation (if it > exists) I would very much appreciate it. Just say that it uses the PSF License 1.0. That's technically inaccurate (because there's not license with that precise name yet), but everyone will know what you mean. ;-) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! From bob at redivi.com Tue May 11 10:59:13 2004 From: bob at redivi.com (Bob Ippolito) Date: Tue May 11 10:59:16 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405110504.i4B54hmp000811@cosc353.cosc.canterbury.ac.nz> References: <200405110504.i4B54hmp000811@cosc353.cosc.canterbury.ac.nz> Message-ID: On May 11, 2004, at 1:04 AM, Greg Ewing wrote: > Bob Ippolito : > >> Doing extensive checking of bytecode at runtime would make Python's >> interpreter much slower than it already is. > > Another approach would be to make the bytecode a special object > that can only be created by the compiler (or other C code). If > you wanted custom bytecode, you'd have to build an AST and feed > it to the compiler, or write an extension module. > > No doubt some bytecode hackers would find this too limiting, > however. Then you should give up .pyc files, because I'd imagine they have the problems as ceval does if someone decided to change some stuff by hand. -bob From fdrake at acm.org Tue May 11 11:19:37 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue May 11 11:19:47 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: References: <200405110504.i4B54hmp000811@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405111119.37843.fdrake@acm.org> On Tuesday 11 May 2004 10:59 am, Bob Ippolito wrote: > Then you should give up .pyc files, because I'd imagine they have the > problems as ceval does if someone decided to change some stuff by hand. No; if the bytecode verification is faster (I'd expect so), then you just need to verify the bytecode loaded from .pyc/.pyo files the same way new.code() and new.function() should. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From fumanchu at amor.org Tue May 11 11:44:43 2004 From: fumanchu at amor.org (Robert Brewer) Date: Tue May 11 11:47:43 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested Message-ID: Greg: > As long as it's possible to attempt to execute arbitrary strings > as bytecode, I'd say ceval should be robust against this. > > But it appears ... that ceval has been full of gaping holes > from the beginning. Guido: > No, the holes aren't in ceval. The hole is the ability to create new > code objects from arbitrary arguments, which wasn't planned at all > when ceval was designed. I was always reluctant to add this > capability, but finally allowed it, consiciously opening up the hole, > under pressure from various people who wanted to do bytecode level > hacking. > > The responsibility to validate bytecode should really be in new.code() > (and probably in new.function, which requires the code to behave in > certain ways). My turn for the time machine :) I wrote: > Perhaps some more rigorous assertions, returning ValueErrors? > If not in C, then in the 'new' module, since that seems to be > the vector of disaster for Xaver, above. Perhaps this would also > be an argument for keeping the 'new' module around longer, > as a 'safer' Pythonic wrapper over bare code/function construction. A good starting point would be some simple boundary and type checking on arguments to new.code(). In this way, we could keep ceval fast and still get the 'safety' benefit. If you like, I can produce some assertions gleaned from my failures the last time I went bytecode hacking. ;) Robert Brewer MIS Amor Ministries fumanchu@amor.org From skip at pobox.com Tue May 11 15:33:41 2004 From: skip at pobox.com (Skip Montanaro) Date: Tue May 11 15:33:16 2004 Subject: [Python-Dev] Applying the PSF license to the config-py project In-Reply-To: <20040511142734.GA15795@panix.com> References: <20040511142734.GA15795@panix.com> Message-ID: <16545.10901.905836.115366@montanaro.dyndns.org> >> I cannot find good definitive documentation on how to use this >> license in this situation. What text do I need to place or reference >> in my code? If someone could direct me to that documentation (if it >> exists) I would very much appreciate it. aahz> Just say that it uses the PSF License 1.0. As in: This package is licensed under the terms of the PSF License, version 1.0. ? I had exactly the same question a few days ago (for the watch project on SF) but got distracted before I remembered to ask. Maybe this is a question worth answering on the PSF section of python.org? Skip From DavidA at ActiveState.com Tue May 11 15:43:22 2004 From: DavidA at ActiveState.com (David Ascher) Date: Tue May 11 15:44:23 2004 Subject: [Python-Dev] Applying the PSF license to the config-py project In-Reply-To: <16545.10901.905836.115366@montanaro.dyndns.org> References: <20040511142734.GA15795@panix.com> <16545.10901.905836.115366@montanaro.dyndns.org> Message-ID: <40A12CDA.2030000@ActiveState.com> Skip Montanaro wrote: > >> I cannot find good definitive documentation on how to use this > >> license in this situation. What text do I need to place or reference > >> in my code? If someone could direct me to that documentation (if it > >> exists) I would very much appreciate it. > > aahz> Just say that it uses the PSF License 1.0. > > As in: > > This package is licensed under the terms of the PSF License, version > 1.0. > > ? I had exactly the same question a few days ago (for the watch project on > SF) but got distracted before I remembered to ask. Maybe this is a question > worth answering on the PSF section of python.org? Alternatively use the BSD or MIT license. There's nothing magical about the PSF license as far as I'm concerned, and its particular wording implies ownership by the PSF, so it's not 'generic', and it refers to other licenses, so another "liberal" license like BSD/MIT works (GPL isn't the same) in terms of the code being 'adoptable' by the PSF. --david From pedronis at bluewin.ch Tue May 11 15:45:00 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Tue May 11 15:44:58 2004 Subject: [Python-Dev] PEP 328 and __import__ contract Message-ID: <5.2.1.1.0.20040511212342.02726fc8@pop.bluewin.ch> PEP 328 seems at the moment to only describe the syntactic, surface and as-seen-by-the-user aspects of the new behavior for absolute/relative imports. What is going to happen to __import__ hook? At the moment relative imports as the default are implemented at that level, through get_parent and the first load_next in import_module_ex which is exported as __import__. Will imports still be implemented in terms of/on top of __import__. What will be the contract for __import__ then? I suppose it could be kept the same, relaxing over time the obligation to implement relative imports. At the moment a __import__ hook can be forced in absolute import mode by passing an empty globals dict. I think the PEP should cover some of this. regards. From barry at python.org Tue May 11 15:51:18 2004 From: barry at python.org (Barry Warsaw) Date: Tue May 11 15:51:29 2004 Subject: [Python-Dev] Applying the PSF license to the config-py project In-Reply-To: <40A12CDA.2030000@ActiveState.com> References: <20040511142734.GA15795@panix.com> <16545.10901.905836.115366@montanaro.dyndns.org> <40A12CDA.2030000@ActiveState.com> Message-ID: <1084305077.28228.324.camel@anthem.wooz.org> On Tue, 2004-05-11 at 15:43, David Ascher wrote: > Alternatively use the BSD or MIT license. There's nothing magical about > the PSF license as far as I'm concerned, and its particular wording > implies ownership by the PSF, so it's not 'generic', and it refers to > other licenses, so another "liberal" license like BSD/MIT works (GPL > isn't the same) in terms of the code being 'adoptable' by the PSF. Of course, if you own the code, you can release it under any (and multiple) license. If/when the time comes, you will also be able to donate it to the PSF at which point they/we will put it under the PSF license. -Barry From tjreedy at udel.edu Tue May 11 15:55:33 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Tue May 11 15:55:41 2004 Subject: [Python-Dev] Re: Mozpython help References: <20040511185624.25266.qmail@web51007.mail.yahoo.com> Message-ID: The pydev list is for discussing the development of future versions of Python. Your usage questions would be better directed to comp.lang.py or, probably better, a mozpython specific group or list, assuming that there is one. tjr From arigo at tunes.org Tue May 11 16:56:02 2004 From: arigo at tunes.org (Armin Rigo) Date: Tue May 11 16:59:12 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: References: Message-ID: <20040511205602.GA1465@vicky.ecs.soton.ac.uk> Hello, As discussed already some time ago, on Linux you can just write random data into /proc/self/mem. There is no way the interpreter can be guaranteed not to crash. Moreover, writing a bytecode verifier is a difficult task which goes against the purpose of new.code(), which was to create strange bytecodes that the compiler would never produce itself. It is possible to write a bytecode verifier that accepts all bytecodes produced by the compiler and then some more, but it is not possible to write any algorithm that can tell for sure whether some bytecode is going to crash or can be proved safe (not even to mention that there exist bytecodes which are safe but that no mathematician -- not just no machine -- will ever be able to prove so). More concretely, what about issuing a warning the first time new.code() is called? People who really use new.code() in their application can silence it with the warnings module. A bientot, Armin. From michel at dialnetwork.com Tue May 11 15:14:42 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Tue May 11 17:24:35 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: References: Message-ID: <200405111414.42559.michel@dialnetwork.com> > Message: 6 > Date: Tue, 11 May 2004 07:12:00 -0700 > From: Guido van Rossum > Subject: Re: [Python-Dev] Is core dump always a bug? Advice requested > To: Greg Ewing > Cc: python-dev@python.org > No, the holes aren't in ceval. The hole is the ability to create new > code objects from arbitrary arguments, which wasn't planned at all > when ceval was designed. I was always reluctant to add this > capability, but finally allowed it, consiciously opening up the hole, > under pressure from various people who wanted to do bytecode level > hacking. Please excuse me if I'm wrong, but wasn't the hole always there, you just made it wider? It seems it has always been possible to handcraft an evil .pyc file and cause its import, it's just more laborious than using the new module. > The responsibility to validate bytecode should really be in new.code() > (and probably in new.function, which requires the code to behave in > certain ways). This seems to only cover the part of the hole you widened, not the whole hole. For example, Java verifies bytecode when a .class file is loaded (See section 4.9 of the Java VM Specification). At this time it does quite a bit of verification, the spec is very interesting and could be a possible starting point for a Python bytecode verifier. It would seem a similar verification should happen any time the Python VM attempts to execute a code object for the first time, not just when the new module is used. > Note that while I like the general principle that Python code > shouldn't be able to cause segfaults, I don't think we should see that > property as a safety *guarantee*. It's more a code-of-honor of the > Python implementation. There are many violations of the principle, > mostly in platform-specific extensions (it is easy to crash Python > with certain win32 primitives, for example, and it seems unreasonable > to expect otherwise). While it's reasonable that C extensions can cause seg-faults, the problem to me seems to be the way in which bytecode can be used to exploit the interpreter without the involvement of extensions and not necessarily causing hard failures like seg faults. -Michel From greg at cosc.canterbury.ac.nz Tue May 11 22:06:29 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue May 11 22:08:51 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405111119.37843.fdrake@acm.org> Message-ID: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> "Fred L. Drake, Jr." : > No; if the bytecode verification is faster (I'd expect so), then you just need > to verify the bytecode loaded from .pyc/.pyo files the same way new.code() > and new.function() should. Just a thought, but is statically verifying the bytecode even possible in principle? Seems to me it could be equivalent to the halting problem. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From fdrake at acm.org Tue May 11 22:14:40 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue May 11 22:14:51 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405112214.40228.fdrake@acm.org> On Tuesday 11 May 2004 10:06 pm, Greg Ewing wrote: > Just a thought, but is statically verifying the bytecode even > possible in principle? Seems to me it could be equivalent to > the halting problem. I don't see any reason to think them equivalent; we don't need to determine that the code will execute to completion, only that the bytecodes can be decoded without error. Not trivial by any means, but I think it's a more constrained problem. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From pje at telecommunity.com Tue May 11 23:33:53 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Tue May 11 23:33:16 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405112214.40228.fdrake@acm.org> References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> Message-ID: <5.1.1.6.0.20040511232902.03a78810@mail.telecommunity.com> At 10:14 PM 5/11/04 -0400, Fred L. Drake, Jr. wrote: >On Tuesday 11 May 2004 10:06 pm, Greg Ewing wrote: > > Just a thought, but is statically verifying the bytecode even > > possible in principle? Seems to me it could be equivalent to > > the halting problem. > >I don't see any reason to think them equivalent; we don't need to determine >that the code will execute to completion, only that the bytecodes can be >decoded without error. Not trivial by any means, but I think it's a more >constrained problem. Right; it should be possible, for example, to verify stack depth used, that stack growth isn't unlimited, that there is a return at the end, no invalid references to locals or co_names, and so on. Basic "sanity" stuff, not in-depth analysis. But it isn't clear to me that this is really necessary for new.code(), let alone .pyc/.pyo files. And anybody that's crazy enough to send bytecode "over the wire" as part of an agent system or whatever darn well better have written their own bytecode verifier already, and maybe a sandbox or two as well. From mhammond at skippinet.com.au Tue May 11 22:23:33 2004 From: mhammond at skippinet.com.au (Mark Hammond) Date: Tue May 11 23:33:30 2004 Subject: [Python-Dev] os.popen and exit codes on windows Message-ID: <0ff401c437c8$212ab390$0200a8c0@eden> I'm looking at bug 602245 - http://www.python.org/sf/602245 - and would like some guidance. The issue is the return-code when closing the last handle from an os.popen() function. In a nutshell: On Linux, the result of closing the last handle is the result from the wait() function. On Windows, the result is the integer return code, exactly as specified by the child program - however, on Windows, when the result is -1, an error is raised (and hence the bug referenced above) On Linux, the value from the wait() system call is split into three parts: mask 0xff00 is the exit status mask 0x007f is the signal causing the exit mask 0x0080 is the cord dumped flag in combination with a signal As part of fixing the bug for Windows, I have 2 basic choices: 1) Fix it by masking off the high bits, thereby returning an "unsigned" result code - preventing -1, but still returning the result exactly as specified by the child process. 2) Fix it by changing the result to resemble the Linux behaviour. This would mean programs using popen() are more portable. My problem is that (2) will change the behaviour for existing programs. Most programs will only check for non-zero, but we can't know that for sure. One option is to do (1) for Python 2.3, and (2) for Python 2.4. That doesn't sound quite right either. If (2) is best, we could probably forget Python 2.3 all together, as the bug is quite old. Any thoughts? Comments directly in the bug are probably best. Thanks, Mark. From tim.one at comcast.net Tue May 11 23:39:57 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 11 23:40:04 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> Message-ID: [Greg Ewing] > Just a thought, but is statically verifying the bytecode even > possible in principle? Seems to me it could be equivalent to > the halting problem. Verifying what, precisely? Some things can clearly be checked. For examples, that all opcodes are defined, that no LOAD_CONST tries to index beyond the actual length of co_consts, and that no jump tries to branch into the middle of some multi-byte opcode sequence Violations of such simple-to-check kinds of things are what cause segfaults most often when handing the PVM nonsense bytes. Harder to do is flow-sensitive eval stack simulation, to ensure that no path through the code can push more on the eval stack than was allocated for it, and that there's enough stuff on the stack at each point to satisfy each opcode that requires accessing the eval stack. A subset of what the Java bytecode verifier does is quite doable: http://java.sun.com/sfaq/verifier.html#1 Java has more things that need to be checked, because many of the JVM instructions have typed arguments (and it has registers as well as a stack). Most PVM arguments are PyObject*, and when specific types are required there's already code in ceval.c to check that at runtime. From greg at cosc.canterbury.ac.nz Tue May 11 23:27:29 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue May 11 23:45:32 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405112214.40228.fdrake@acm.org> Message-ID: <200405120327.i4C3RTNH002597@cosc353.cosc.canterbury.ac.nz> "Fred L. Drake, Jr." : > I don't see any reason to think them equivalent; we don't need to determine > that the code will execute to completion, only that the bytecodes can be > decoded without error. If arbitrary branching patterns are allowed, then in order to ensure absence of stack overflow/underflow, etc., it will be necessary to do things like proving that certain loops can't execute more than a certain number of times. This smells halting-problem-complete to me. It would be possible to prove that certain classes of bytecode sequence are valid, and reject all others. But I suspect that would reject a large number of valid-but-hard-to-prove sequences that bytecode hackers would find "interesting". Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From bob at redivi.com Wed May 12 00:05:49 2004 From: bob at redivi.com (Bob Ippolito) Date: Wed May 12 00:05:57 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <5.1.1.6.0.20040511232902.03a78810@mail.telecommunity.com> References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> <5.1.1.6.0.20040511232902.03a78810@mail.telecommunity.com> Message-ID: On May 11, 2004, at 11:33 PM, Phillip J. Eby wrote: > At 10:14 PM 5/11/04 -0400, Fred L. Drake, Jr. wrote: >> On Tuesday 11 May 2004 10:06 pm, Greg Ewing wrote: >> > Just a thought, but is statically verifying the bytecode even >> > possible in principle? Seems to me it could be equivalent to >> > the halting problem. >> >> I don't see any reason to think them equivalent; we don't need to >> determine >> that the code will execute to completion, only that the bytecodes can >> be >> decoded without error. Not trivial by any means, but I think it's a >> more >> constrained problem. > > Right; it should be possible, for example, to verify stack depth used, > that stack growth isn't unlimited, that there is a return at the end, > no invalid references to locals or co_names, and so on. Basic > "sanity" stuff, not in-depth analysis. > > But it isn't clear to me that this is really necessary for new.code(), > let alone .pyc/.pyo files. And anybody that's crazy enough to send > bytecode "over the wire" as part of an agent system or whatever darn > well better have written their own bytecode verifier already, and > maybe a sandbox or two as well. Exactly, why bother when there's plenty of valid bytecode that crashes? Even without third party extension modules, there are usually platform specific files you can open and play with that'll cause a crash. os._exit and os.abort are essentially crashes too, and then there's the signal module, os.kill, etc. The only example I can think of is if you were experimenting with genetic algorithms or something and you decided to randomly generate and mutate bytecode and see if it produces interesting results.. but in that case you probably want a much smaller and simpler VM anyways. -bob From fdrake at acm.org Wed May 12 00:28:52 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Wed May 12 00:29:05 2004 Subject: [Python-Dev] Python 2.3.x doc tree closed Message-ID: <200405120028.53006.fdrake@acm.org> I'm cutting the docs for Python 2.3.4; please make no changes on the the release23-maint branch. Thanks for everyone's help in getting this ready! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From anthony at interlink.com.au Wed May 12 01:33:37 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed May 12 01:34:28 2004 Subject: [Python-Dev] release23-maint tree closed for 2.3.4rc1 Message-ID: <40A1B731.1050504@interlink.com.au> Unless you're involved in the 2.3.4 release process (so one of Anthony, Thomas, Fred or Jack(?)), please consider the release23-maint branch frozen until after Thursday the 20th of June. If you have something that you want to see on the branch before then, please email me first. Thanks, Anthony -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Wed May 12 01:35:28 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed May 12 01:36:17 2004 Subject: [Python-Dev] Re: release23-maint tree closed for 2.3.4rc1 In-Reply-To: <40A1B731.1050504@interlink.com.au> References: <40A1B731.1050504@interlink.com.au> Message-ID: <40A1B7A0.2000606@interlink.com.au> Anthony Baxter wrote: > Unless you're involved in the 2.3.4 release process > (so one of Anthony, Thomas, Fred or Jack(?)), please > consider the release23-maint branch frozen until after > Thursday the 20th of June. Argh. What is with my internal brain calendar this year?? Of course, I meant Thursday the 20th of _MAY_. May. It's May. I know this. Really. From mwh at python.net Wed May 12 05:45:02 2004 From: mwh at python.net (Michael Hudson) Date: Wed May 12 05:45:07 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: (Bob Ippolito's message of "Wed, 12 May 2004 00:05:49 -0400") References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> <5.1.1.6.0.20040511232902.03a78810@mail.telecommunity.com> Message-ID: <2misf2m129.fsf@starship.python.net> Bob Ippolito writes: > The only example I can think of is if you were experimenting with > genetic algorithms or something and you decided to randomly generate > and mutate bytecode and see if it produces interesting results.. but > in that case you probably want a much smaller and simpler VM anyways. As well as psychiatric help :-) Cheers, mwh -- (Unfortunately, while you get Tom Baker saying "then we were attacked by monsters", he doesn't flash and make "neeeeooww-sploot" noises.) -- Gareth Marlow, ucam.chat, from Owen Dunn's review of the year From arigo at tunes.org Wed May 12 07:01:39 2004 From: arigo at tunes.org (Armin Rigo) Date: Wed May 12 07:04:43 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040512110139.GA14120@vicky.ecs.soton.ac.uk> Hello Tim, On Tue, May 11, 2004 at 11:39:57PM -0400, Tim Peters wrote: > Verifying what, precisely? (...) and that no jump tries to branch into > the middle of some multi-byte opcode sequence Why not? :-) (mwh can surely recommend me a psy) > Harder to do is flow-sensitive eval stack > simulation, to ensure that no path through the code can push more on the > eval stack than was allocated for it, and that there's enough stuff on the > stack at each point to satisfy each opcode that requires accessing the eval > stack. There are a few opcodes whose effect on the stack isn't self-contained, like MAKE_FUNCTION which will pop N cell variables off the stack, for N loaded from the code object which itself comes from the stack. These ones are fun to analyse. Even without these, checking if a bytecode could possibly over/underflow the stack is indeed equivalent to the halting problem; a silly example: POP_TOP This underflows the stack if and only if the algorithm stops. Armin From casey at zope.com Wed May 12 09:34:14 2004 From: casey at zope.com (Casey Duncan) Date: Wed May 12 09:32:59 2004 Subject: [Python-Dev] Re: Is core dump always a bug? Advice requested References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040512093414.15e73078.casey@zope.com> On Tue, 11 May 2004 23:39:57 -0400 "Tim Peters" wrote: > [Greg Ewing] > > Just a thought, but is statically verifying the bytecode even > > possible in principle? Seems to me it could be equivalent to > > the halting problem. > > Verifying what, precisely? Some things can clearly be checked. For > examples, that all opcodes are defined, that no LOAD_CONST tries to > index beyond the actual length of co_consts, and that no jump tries to > branch into the middle of some multi-byte opcode sequence. Damn, there goes my plans for phiX174.py befunging-ly y'rs, -Casey From guido at python.org Wed May 12 09:55:04 2004 From: guido at python.org (Guido van Rossum) Date: Wed May 12 10:01:11 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: Your message of "Wed, 12 May 2004 12:01:39 BST." <20040512110139.GA14120@vicky.ecs.soton.ac.uk> References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> <20040512110139.GA14120@vicky.ecs.soton.ac.uk> Message-ID: <200405121355.i4CDt4n18585@guido.python.org> > There are a few opcodes whose effect on the stack isn't > self-contained, like MAKE_FUNCTION which will pop N cell variables > off the stack, for N loaded from the code object which itself comes > from the stack. These ones are fun to analyse. We're planning to get rid of that one (note that it was a late addition; the original design was intended to be easy to analyze, but others didn't see the constraints). Any others that pose such problems? > Even without these, checking if a bytecode could possibly over/underflow the > stack is indeed equivalent to the halting problem; a silly example: > > > POP_TOP > > This underflows the stack if and only if the algorithm stops. Uninteresting though, since no compiler will emit such code, and we only need to accept a reasonable subset. The same arguments could prove that Java's bytecode verification is "impossible", but nevertheless it's done. --Guido van Rossum (home page: http://www.python.org/~guido/) From tjreedy at udel.edu Wed May 12 11:37:24 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Wed May 12 11:37:32 2004 Subject: [Python-Dev] Re: Is core dump always a bug? Advice requested References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> Message-ID: "Tim Peters" wrote in message news:E1BNkbA-0003fY-8g@mail.python.org... > Verifying what, precisely? Some things can clearly be checked. For > examples, that all opcodes are defined, that no LOAD_CONST tries to index > beyond the actual length of co_consts, and that no jump tries to branch into > the middle of some multi-byte opcode sequence Violations of such > simple-to-check kinds of things are what cause segfaults most often when > handing the PVM nonsense bytes. Harder to do is flow-sensitive eval stack > simulation, to ensure that no path through the code can push more on the > eval stack than was allocated for it, and that there's enough stuff on the > stack at each point to satisfy each opcode that requires accessing the eval > stack. > > A subset of what the Java bytecode verifier does is quite doable: Since I opened this somewhat hidden hornet's nest, the (newby) author of the code that caused the offending crash has admitted on clp that the code he fed to new.code was standard ascii Python code, not bytecode. In other words, something that almost certainly looked like gibberish to ceval. The limited goal of usually rejecting such 'random' byte strings, if adopted, *should* be doable since such should have many errors and we only need to catch one, not all, to reject. Protecting against maliciously or otherwise modified compiler output is a whole different game. Too bad there is no system-independent way to 'catch' crashes and exit more gracefully than usually useless core dumps, blue screens of death, or whatever. Terry J. Reedy From jeremy at alum.mit.edu Wed May 12 11:56:49 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Wed May 12 11:57:03 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405120327.i4C3RTNH002597@cosc353.cosc.canterbury.ac.nz> Message-ID: <200405121556.i4CFusNw025611@mta7.pltn13.pbi.net> > If arbitrary branching patterns are allowed, then in order to ensure > absence of stack overflow/underflow, etc., it will be necessary to do > things like proving that certain loops can't execute more than a > certain number of times. This smells halting-problem-complete to me. I think it would only need to check that any path through the loop has a no net effect on stack depth -- before branching back to the beginning of the loop, it must pop everything it pushed. > It would be possible to prove that certain classes of bytecode > sequence are valid, and reject all others. But I suspect that would > reject a large number of valid-but-hard-to-prove sequences that > bytecode hackers would find "interesting". I find it hard to make a strong conjecture either way. I think the only way to find out would be to examine whatever it is that bytecode hackers do. There could be an unsafe switch that turned off checking. Jeremy From mat at elandem.org Wed May 12 13:07:06 2004 From: mat at elandem.org (Mat Ford) Date: Wed May 12 13:17:55 2004 Subject: [Python-Dev] IPv6 support in Win binary again Message-ID: There was a thread back in October 2003 on the subject of including IPv6 support in the Windows binary: http://mail.python.org/pipermail/python-dev/2003-October/038623.html Can anyone give me a status update on this? Is support there now? If not, is there a predictable timeframe in which it can be expected to appear? Thanks in advance, Mat --- Outgoing mail is certified Virus Free. Checked by AVG anti-virus system (http://www.grisoft.com). Version: 6.0.678 / Virus Database: 440 - Release Date: 06/05/2004 From brian at sweetapp.com Wed May 12 13:42:55 2004 From: brian at sweetapp.com (Brian Quinlan) Date: Wed May 12 13:38:57 2004 Subject: [Python-Dev] factory functions in the datetime module (& others) In-Reply-To: <20040512093414.15e73078.casey@zope.com> References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> <20040512093414.15e73078.casey@zope.com> Message-ID: <40A2621F.8070205@sweetapp.com> It seems to me that the utility of subclassing certain library classes would be increased if the user could set a factory that the base class would use when creating new instances. That way, from the user's point of view, functionality would not "disappear" after operations e.g. class MyDate(datetime.date): def foo(self): ... nextday = MyDate(...) + datetime.timedelta(days=1) nextday.foo() # exception Storing the factory information in the base class has the disadvantage of causing a global behavior change but can probably be made speed efficient. Storing the factory in the subclass would require that the super class check for a class variable when creating instances. It would probably provide the most obvious semantics from the programmer's POV but would be organizationally odd i.e. each class would have to provide a factory for every overriden type that it's superclass creates. Aside from datime.* this feature seems like it would be useful for other types where there is potential for making useful subclasses (unlike int) and where many operations result in new instances. sets come to mind. Any thoughts? Cheers, Brian From martin at v.loewis.de Wed May 12 15:38:34 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed May 12 15:38:46 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> References: <200405120206.i4C26TMm002483@cosc353.cosc.canterbury.ac.nz> Message-ID: <40A27D3A.6000603@v.loewis.de> Greg Ewing wrote: > Just a thought, but is statically verifying the bytecode even > possible in principle? Seems to me it could be equivalent to > the halting problem. Reliably accepting all correct byte code, and rejecting all incorrect one, would indeed be equivalent to the halting problem. Therefore, byte code verification usually puts additional constraints on byte code, rejecting some correct code as unverifiable. Then, in addition to the language spec, you also need to specify what the verification rules are that an implementation needs to follow. The typical assumption is that you have "consistent" stack usage. IOW, you need to be able to clearly identify start offsets of byte code instructions, and associate a fixed stack depth with each instruction. In typed languages, you also need to associate a type with the TOS, and require that type to be consistent across all possible execution paths. For example, in .NET code verification, there is a requirement that the value stack is empty at every branch target, and the start of every exception handler. IOW, it is only necessary that any byte code that the Python byte code generator will ever emit verifies as correct. Accepting additional byte codes is optional. Regards, Martin From martin at v.loewis.de Wed May 12 15:43:43 2004 From: martin at v.loewis.de (=?windows-1252?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Wed May 12 15:43:50 2004 Subject: [Python-Dev] IPv6 support in Win binary again In-Reply-To: References: Message-ID: <40A27E6F.70806@v.loewis.de> Mat Ford wrote: > There was a thread back in October 2003 on the subject of including IPv6 > support in the Windows binary: > > http://mail.python.org/pipermail/python-dev/2003-October/038623.html > > Can anyone give me a status update on this? Is support there now? No. > If not, is > there a predictable timeframe in which it can be expected to appear? With the release of Python 2.4, which, at latest, should happen before January, 2005. In case you wonder what the problem is: if you compile with VC6, you cannot provide IPv6 support unless you also have the platform SDK installed. That is not part of the build process of Python 2.3, so the official binaries of Python 2.3 will never ever support IPv6. Using VC.NET is out of the question, since it would break the ABI across minor releases, which is out of the question. Installing the SDK for the build process might be an option, but the code base has no way of detecting presence of the SDK, so this is essentially also out of question. Regards, Martin From tim.one at comcast.net Wed May 12 16:17:58 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed May 12 16:18:06 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405121355.i4CDt4n18585@guido.python.org> Message-ID: [Armin] >> Even without these, checking if a bytecode could possibly over/underflow >> the stack is indeed equivalent to the halting problem; a silly example: >> >> >> POP_TOP >> >> This underflows the stack if and only if the algorithm stops. [Guido] > Uninteresting though, since no compiler will emit such code, and we > only need to accept a reasonable subset. The same arguments could > prove that Java's bytecode verification is "impossible", but > nevertheless it's done. That's right. Anyone who intends to "do something" here (as opposed to just talking about it) is strongly encouraged to read the Java docs I posted a link to last time. Their answer to my original question ("verify what, precisely?") is "what the algorithm described here checks". As also with Java's compile-time guarantee of "no use of uninitialized variables", there's no attempt to solve impossible problems. Instead they give a reasonable, conservative algorithm, and define "it's OK" as "what this algorithm allows". The difference between that and theoretical perfection isn't an issue in practice. From michel at dialnetwork.com Wed May 12 16:13:27 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Wed May 12 18:24:39 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested Message-ID: <200405121513.27827.michel@dialnetwork.com> > > If arbitrary branching patterns are allowed, then in order to ensure > > absence of stack overflow/underflow, etc., it will be necessary to do > > things like proving that certain loops can't execute more than a > > certain number of times. This smells halting-problem-complete to me. > > I think it would only need to check that any path through the loop has a no > net effect on stack depth -- before branching back to the beginning of the > loop, it must pop everything it pushed. The ASM Java bytecode generation API has a similar flow control algorithm to determine the maximum stack size for java bytecode. http://asm.objectweb.org/ It uses the visitor pattern to visit classes, methods and instructions during bytecode generation that construct a control flow graph that is used at class generation time to calculate the maximum local variables and stack depth for methods (see Label.java and CodeWriter.java in their WebCVS). his information is embedded into the class, and is used at JVM verification time when the bytecode is reanalyzed to ensure that the numbers match the actual behavior of the bytecode. I've seen a lot of concern about bytecode hacking crashing the interpreter, but the Java VM Spec goes further to state that there is a risk of exploitation. I stared at ceval for a while to see if I could pick up such a hole. It hurt. I hear Phillip Eby when he says that anyone sending bytecode over the wire agent-fasion "darn well better" write their own bytecode verifier. But in a world of massive paralleism and dynamic computing, that might not seem like such a bad feature to offer stock. Would there be an interest in at least a PEP to consider a bytecode verifier? I'd be willing to take it on, but my experience is limited to JVM bytecode and verification. -Michel From jjl at pobox.com Wed May 12 18:40:12 2004 From: jjl at pobox.com (John J Lee) Date: Wed May 12 18:40:19 2004 Subject: [Python-Dev] cookielib module in 2.4? Message-ID: I have a module that I'd like to get included in Python 2.4. It adds transparent HTTP cookie handling to urllib2 (and urllib too, I guess, if somebody wants to write the little bit of code I imagine is required). I'm prepared to maintain the thing, of course. It has been distributed for 2 years in a 1.5.2-compatible form, as the ClientCookie package, at wwwsearch.sf.net. I've just made a branch in which I have removed 1.5.2 (and various other) cruft, along with some superfluous features, and turned it into a single-file module (plus a small patch to urllib2). ATM it's in a subversion branch here: http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.4_candidate/ I've renamed it 'cookielib'. That's partly to fit in with the Python standard library's naming convention for network client modules, partly so that it's clear to people whether they're importing the old 1.5.2-compatible version or the 2.4 standard library version. Actually, the most frequent use case does not involve importing the module: import urllib2 opener = urllib2.build_opener(urllib2.HTTPCookieProcessor()) response = opener.open("http://example.com/") So, assuming the module is likely to be accepted (given the answers to my post here last year), I'd be happy if anybody here could 1. indicate if I'm roughly on the right track (especially welcome if you've had a look at the subversion branch above), 2. give me a rough idea how long I have to sort out the remaining issues and submit a patch The major remaining issue is documentation, which is not yet in LaTeX format (nor in Python std. lib. style) -- see my next post. There are also two or three niggles which I'd like to discuss with Jeremy Hylton, Moshe Zadka, or anybody else who knows urllib2. Finally, there's the fact that the module contains untested thread-synchronisation code (I'm pretty thread-ignorant). Perhaps that's an academic issue ATM, though: I failed recently to eliminate deadlocks in a fairly simple bit of code using urllib2 + threading, even when I verified there were no locks allocated by any module other than threading itself... TIA John From michel at dialnetwork.com Wed May 12 16:31:43 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Wed May 12 18:42:41 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested Message-ID: <200405121531.43532.michel@dialnetwork.com> Armin said: > > Even without these, checking if a bytecode could possibly over/underflowthe > > stack is indeed equivalent to the halting problem; a silly example: > > > > > > POP_TOP > > > > This underflows the stack if and only if the algorithm stops. Guido said: > The same arguments could > prove that Java's bytecode verification is "impossible", but > nevertheless it's done. But I belive Armin's argument that it equals the halting problem is incorrect. The net stack effect of bytecode can be verified regardless of the algorithm. If it is seen that two bytecodes produce an element on the stack and one bytecode consumes two, then the net effect is zero. As Jeremy pointed out, iterations (of well formed code) have no net effect. Although Armin's example could certainly fail due to exhausting the stack of return frames (which is a runtime issue), or memory, or anything else, no well formed bytecode can exhaust the bytecode stack. The way to know if it's well formed is to verify it using something like a control flow analysis similar to the way the Java ASM bytecode generation API does it. The secondary benefit of this is that you are assured at runtime that the stack size the compiler computes for the bytecode is genuine. -Michel From jjl at pobox.com Wed May 12 18:43:42 2004 From: jjl at pobox.com (John J Lee) Date: Wed May 12 18:43:50 2004 Subject: [Python-Dev] Build a single file module-foo.html from Python LaTeX source? Message-ID: Is there some magic make command that generates the module-foo.html for just a single libfoo.tex file? Doesn't necessarily have to generate 100%-identical output to the full run. I want to write Python-LaTeX-format docs for a module module I hope to get into Python 2.4, and document urllib2's handlers better. It's nice to be able to see the HTML output as you edit, without having to process the whole std. lib. manual every time. John From aahz at pythoncraft.com Wed May 12 20:01:35 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed May 12 20:03:59 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405121513.27827.michel@dialnetwork.com> References: <200405121513.27827.michel@dialnetwork.com> Message-ID: <20040513000135.GB11008@panix.com> On Wed, May 12, 2004, Michel Pelletier wrote: > > Would there be an interest in at least a PEP to consider a bytecode > verifier? I'd be willing to take it on, but my experience is limited > to JVM bytecode and verification. PEPs for complex issues are always welcome -- they're a valuable resource for the future. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! From bac at OCF.Berkeley.EDU Wed May 12 20:52:30 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed May 12 20:53:36 2004 Subject: [Python-Dev] cookielib module in 2.4? In-Reply-To: References: Message-ID: <40A2C6CE.9060007@ocf.berkeley.edu> John J Lee wrote: [snip - cookielib explanation] > > So, assuming the module is likely to be accepted (given the answers to my > post here last year), I'd be happy if anybody here could > > 1. indicate if I'm roughly on the right track (especially welcome if > you've had a look at the subversion branch above), > Code, docs, tests. But in order for the module to be accepted it needs to show that its usage is truly needed by having it be widely used in the Python community. I don't do web programming so others will have to vouch for the amount of usage. All explained in the "Intro to Development" doc found at http://www.python.org/dev/ . > 2. give me a rough idea how long I have to sort out the remaining issues > and submit a patch How ever long it takes. Not sure if I really understand what you are wondering about. -Brett From greg at cosc.canterbury.ac.nz Wed May 12 22:15:35 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Wed May 12 22:15:58 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405121556.i4CFusNw025611@mta7.pltn13.pbi.net> Message-ID: <200405130215.i4D2FZ2S004169@cosc353.cosc.canterbury.ac.nz> Jeremy Hylton : > I think it would only need to check that any path through the loop has a no > net effect on stack depth -- before branching back to the beginning of the > loop, it must pop everything it pushed. If the branching is sufficiently spaghettoid, it might have trouble identifying any coherent loops or paths. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From skip at pobox.com Wed May 12 22:47:14 2004 From: skip at pobox.com (Skip Montanaro) Date: Wed May 12 22:46:24 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405121531.43532.michel@dialnetwork.com> References: <200405121531.43532.michel@dialnetwork.com> Message-ID: <16546.57778.575514.924484@montanaro.dyndns.org> Michel> The way to know if it's well formed is to verify it using Michel> something like a control flow analysis similar to the way the Michel> Java ASM bytecode generation API does it. This is more-or-less what the Python byte code compiler already does to determine the high water mark for the stack used by a chunk of code it's compiling. Look for c_stacklevel, c_maxstacklevel and the com_push() and com_pop() functions in Python/compile.c. Skip From pje at telecommunity.com Wed May 12 23:13:15 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed May 12 23:12:52 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405130215.i4D2FZ2S004169@cosc353.cosc.canterbury.ac.nz> References: <200405121556.i4CFusNw025611@mta7.pltn13.pbi.net> Message-ID: <5.1.1.6.0.20040512223230.0391f860@mail.telecommunity.com> At 02:15 PM 5/13/04 +1200, Greg Ewing wrote: >Jeremy Hylton : > > > I think it would only need to check that any path through the loop has a no > > net effect on stack depth -- before branching back to the beginning of the > > loop, it must pop everything it pushed. > >If the branching is sufficiently spaghettoid, it might have >trouble identifying any coherent loops or paths. As I think Tim has already mentioned, all you do is track the stack depth for each instruction, and ensure it's consistent. If there is an inconsistency in the predicted stack level for an instruction, as compared with any of the instructions that jump to it (no matter where they jump from), you have a problem. Or, to look at it another way: maxdepth = 0 startpoints = [(0,0)] stackdepth = [None] * len(bytecode) while startpoints: location,depth = startpoints.pop() while True: if location >= len(bytecode): raise VerificationError("Unexpected end of bytecode at",location) if stackdepth[location] is None: stackdepth[location] = depth else: if stackdepth[location]<>depth: raise VerificationError("Inconsistent stack depth at",location) else: break # we've already processed this part of the code instruction = fetch(bytecode,location) instruction.verify_for(code_object,location) # Check operand validity depth += instruction.effectOnStack() maxdepth = max(depth,maxdepth) if depth<0: raise VerificationError("Stack underflow at",location) if instruction.isterminal: (e.g. return, raise) break # Block is over, handle the next one if any elif instruction.isconditional: startpoints.append((instruction.alt_location(location),depth)) location = instruction.next_location(location) This algorithm should readily handle all forward and backward jumps, while ensuring that no stack growth or shrinkage can take place, because every instruction must have a single known stack depth, no matter how many forward or backward jumps it takes to get there. Notice that the algorithm will "follow" every branch, but will not retrace the code at the target location once the stack depth is seen to be consistent. It also tackles underflow and "falling off the end" of the bytecode, while simultaneously computing the maximum stack depth used. Each instruction is fetched and processed for its depth effects and other characteristics exactly once, so the timing is O(n). And, it has a nice place to put opcode-specific operand validation for opcodes that use locals, freevars, names, or constants. Anyway, I think we can safely halt discussion of the halting problem now. :) And, since I've now reduced this from an interesting theoretical idea to talk about, into something resembling work (God forbid), anybody who wants to actually *implement* this crazy idea is now free to do so, without even having to read the paper Tim linked to. :) (Disclaimer: I didn't read the paper either, so there may be other interesting/applicable verifications there that I didn't sketch above. This is just a spur-of-the-moment off-the-top-of-my-head draft thingy.) From tim.one at comcast.net Wed May 12 23:30:24 2004 From: tim.one at comcast.net (Tim Peters) Date: Wed May 12 23:30:33 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405121513.27827.michel@dialnetwork.com> Message-ID: [Michel Pelletier] > ... > Would there be an interest in at least a PEP to consider a bytecode > verifier? I think so, yes. > I'd be willing to take it on, but my experience is limited to JVM bytecode > and verification. Python should be easier, in large part because you have to give up sooner. For example, Python has a single catch-all BINARY_ADD opcode for infix "+". There aren't distinct opcodes for "adding ints" and "adding floats" and "adding strings". So, in Python, there's no need to even try to check the types of BINARY_ADD's stack operands. They're always pointer-to-PyObject, and since pointer-to-PyObject is the only kind of thing that *can* be pushed on the PVM stack, there's nothing for a modestly ambitious static verifier to look for there. Checks for type correctness in the PVM are done at runtime instead. I think it's fair to say that a bytecode verifier is overwhelmingly "just an optimization": if bytecode properties can be verified from static code analysis, runtime code isn't needed to verify them dynamically, or, in the absence of such runtime checks, static analysis plugs ways to provoke segfaults. Examples of things that aren't checked at all now in the PVM were given before, like the PVM doesn't check that it allocated enough stack space, or check that C-level indexing into the co_consts vector is in bounds. Well, in the latter case, that is checked (at runtime, on every co_consts access) in a debug build, but not in a release build. From tim.one at comcast.net Thu May 13 00:05:46 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu May 13 00:05:55 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <5.1.1.6.0.20040512223230.0391f860@mail.telecommunity.com> Message-ID: [Phillip J. Eby] ... > (Disclaimer: I didn't read the paper either, so there may be other > interesting/applicable verifications there that I didn't sketch > above. This is just a spur-of-the-moment off-the-top-of-my-head draft > thingy.) It's important to read what people who have actually done it have to say about subtleties. For example, your algorithm considered 'return' to be 'a terminal instruction'. But what if it appeared in the 'try' clause of a try/finally construct? In Python's PVM, most *possible* control flow is implicit (virtually any opcode can raise an exception, and from there "magically jump" to the code at an enclosing 'except' or 'finally' clause); and even an unexceptional bare 'return' can magically jump to an enclosing 'finally' block. The Java reference discusses these things, and anyone intending to do something here is still strongly urged to read it. From michel at dialnetwork.com Wed May 12 22:11:23 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Thu May 13 00:09:50 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405121531.43532.michel@dialnetwork.com> References: <200405121531.43532.michel@dialnetwork.com> Message-ID: <200405122111.23306.michel@dialnetwork.com> On Wednesday 12 May 2004 15:31, Michel Pelletier wrote: > The secondary benefit of this is that you are assured at runtime that the > stack size the compiler computes for the bytecode is genuine. Thinking aloud, maybe this is how it can be an exploit: create a code object with a falsified maximum stack size, and bytecode that over/underflows the incorrectly allocated value stack so that the right bit pattern is executed to give execution control to malicious C-level code. One way to inject the exploit would be a file permission weakness alowing a bogus, but commonly used Python module in .pyc form to be placed into the system interpreter path. This would allow an exploiter to execute any machine instructions for any use of that module, system wide. A trojan distutils product could also taint a system interpreter. This would not otherwise be a big deal, because a file permission weakness would allow you to place malicious bytecode anyway, but a stack over/underflow would allow not just PVM instructions, but "real" processor machine instructions, thus subverting any protections the PVM or some other tool (rexec, Zope style security) could provide even at the bytecode level. Verifying that bytecode is well formed would prevent this. All of this could be, as the saying goes, a "bag full of fart". I'm examining the source now to see if I can come up with a proof of concept which I will not share publicly. -Michel From judson at mcs.anl.gov Thu May 13 00:33:11 2004 From: judson at mcs.anl.gov (Ivan R. Judson) Date: Thu May 13 00:34:07 2004 Subject: [Python-Dev] IPv6 support in Win binary again In-Reply-To: <40A27E6F.70806@v.loewis.de> Message-ID: <200405130433.i4D4WxU35612@mcs.anl.gov> What is this implying about the 2.4 build assumptions? Vs.net 2008 :)? Platform sdk? Something else? --Ivan > -----Original Message----- > From: python-dev-bounces+judson=mcs.anl.gov@python.org > [mailto:python-dev-bounces+judson=mcs.anl.gov@python.org] On > Behalf Of Martin v. L?wis > Sent: Wednesday, May 12, 2004 2:44 PM > To: mat@elandem.org > Cc: python-dev@python.org > Subject: Re: [Python-Dev] IPv6 support in Win binary again > > Mat Ford wrote: > > There was a thread back in October 2003 on the subject of including > > IPv6 support in the Windows binary: > > > > > http://mail.python.org/pipermail/python-dev/2003-October/038623.html > > > > Can anyone give me a status update on this? Is support there now? > > No. > > > If not, is > > there a predictable timeframe in which it can be expected to appear? > > With the release of Python 2.4, which, at latest, should > happen before January, 2005. > > In case you wonder what the problem is: if you compile with > VC6, you cannot provide IPv6 support unless you also have the > platform SDK installed. That is not part of the build process > of Python 2.3, so the official binaries of Python 2.3 will > never ever support IPv6. > Using VC.NET is out of the question, since it would break the > ABI across minor releases, which is out of the question. > Installing the SDK for the build process might be an option, > but the code base has no way of detecting presence of the > SDK, so this is essentially also out of question. > > Regards, > Martin > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/judson%40mcs.anl.gov > > From anthony at interlink.com.au Thu May 13 00:45:13 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu May 13 00:46:14 2004 Subject: [Python-Dev] cookielib module in 2.4? In-Reply-To: <40A2C6CE.9060007@ocf.berkeley.edu> References: <40A2C6CE.9060007@ocf.berkeley.edu> Message-ID: <40A2FD59.9040906@interlink.com.au> Brett C. wrote: > John J Lee wrote: >> 2. give me a rough idea how long I have to sort out the remaining issues >> and submit a patch > > > How ever long it takes. Not sure if I really understand what you are > wondering about. If the question is "how long has he got before 2.4 alpha 1", the answer is "about 6 weeks". I'm still aiming for a first alpha in the first week of July or thereabouts. -- Anthony Baxter It's never too late to have a happy childhood. From arigo at tunes.org Thu May 13 04:54:01 2004 From: arigo at tunes.org (Armin Rigo) Date: Thu May 13 04:57:15 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405121531.43532.michel@dialnetwork.com> References: <200405121531.43532.michel@dialnetwork.com> Message-ID: <20040513085401.GA28305@vicky.ecs.soton.ac.uk> Hello Michel, On Wed, May 12, 2004 at 03:31:43PM -0500, Michel Pelletier wrote: > But I belive Armin's argument that it equals the halting problem is > incorrect. You are missing my point. Martin summarized what I intended to say: what is equivalent to the halting problem is the question "can you say if the following bytecode is going to execute some illegal opcode or not", which is a purely theoretical question, as I am well aware. The useful and checkable question that interests us here is "does this bytecode follow the following set of sane rules", for a well-specified set of rules. > The net stack effect of bytecode can be verified regardless of the algorithm. Interestingly, it is not the case currently that for any bytecode position the stack depth is always the same. Here is a short review. * 'finally' blocks are executed with one, two or three values pushed on the stack; END_FINALLY will accordingly pop one, two or three values (depending on what the first popped value is). * POP_BLOCK will empty the value stack up to the position stored in the block stack's popped block, for no reason that I am aware of. I can't think of a compiler-emitted construct that uses this feature. Maybe it was introduced to implement 'break' as a jump to the POP_BLOCK, but nowadays 'break' is more complicated anyway, using the general stack-unwinding and exception-handler-dispatching loop after the switch in ceval.c. * MAKE_CLOSURE pops some cells to use for free variables, the number of which is found in the code object on the top of the stack. I believe that all the other instructions have a stack effect that is either constant or computable from their oparg. If some long-term goal is to fix this, then MAKE_CLOSURE could be preceded by a BUILD_TUPLE to collect all free vars into a tuple (which is what MAKE_CLOSURE does anyway). Fixing the 'finally' blocks isn't too complicated -- always push three values on the stack, possibly None's. A better 'fix' would be that the block stack could, as we discussed at PyCon, be replaced by a purely static structure in the code object. Armin From mwh at python.net Thu May 13 05:40:26 2004 From: mwh at python.net (Michael Hudson) Date: Thu May 13 05:40:36 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: (Tim Peters's message of "Wed, 12 May 2004 23:30:24 -0400") References: Message-ID: <2mpt98ll6d.fsf@starship.python.net> "Tim Peters" writes: > Examples of things that aren't checked at all now in the PVM were given > before, like the PVM doesn't check that it allocated enough stack space, or > check that C-level indexing into the co_consts vector is in bounds. Well, > in the latter case, that is checked (at runtime, on every co_consts access) > in a debug build, but not in a release build. I think stack over/underflow is checked in a debug build too. So here's a way to plug these holes: run a debug build the whole time :-) It's only, what, a factor of 3 performance hit. If people are seriously concerned about preventing Python coring in all circumstances, we should look at the various ways of overflowing the stack, too. The recursion counter is a hack, and IMHO can't be made to work sensibly. For instance: list_sort() has quite a large structure on the stack, so if you arrange to call it recursively it falls off the end of the stack much much sooner than a simple Python recursion. I don't know what we can do about this. Armin suggested another hack: stick the address of a stack variable in Py_Main() in a global and compare the address of another stack variable when you want to see how much stack you have left. Obvious problems include knowing what's safe and which direction the stack is growing in... Even more scarily, what SBCL (a Common Lisp implementation) does is mprotect() a VM page and the end of the stack and deal with overflow in a SIGSEGV handler. It's hard to see what else could be really safe (have I mentioned that I hate C recently?). Option 3, I guess, is integrate stackless :-) Cheers, mwh -- 8. A programming language is low level when its programs require attention to the irrelevant. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From tismer at stackless.com Thu May 13 07:08:08 2004 From: tismer at stackless.com (Christian Tismer) Date: Thu May 13 07:06:54 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405110420.i4B4Ku3P000714@cosc353.cosc.canterbury.ac.nz> References: <200405110420.i4B4Ku3P000714@cosc353.cosc.canterbury.ac.nz> Message-ID: <40A35718.8020700@stackless.com> Greg Ewing wrote: > Guido: > > >>>As long as it's possible to attempt to execute arbitrary strings >>>as bytecode, I'd say ceval should be robust against this. >> >>This ought to be a long-term project then: write a bytecode verifier. >>It's not a trivial task! > > > When I wrote that, I was assuming that ceval was already mostly robust > in this respect, and that what was being reported was a new hole > recently opened up. > > But it appears I was gravely mistaken, and that ceval has been full of > gaping holes from the beginning. > > I'm disappointed to learn this, because I had always regarded it as an > axiom that no Python-level code should be capable of crashing the > interpreter, and if it can, this represents a bug. However, it seems > this axiom has not been adhered to in the design of ceval. I don't think so. ceval appears to be absolutely robust against byte code that the bytecode compiler has generated. I think the design error was to expose code object construction without safety belts. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Thu May 13 08:58:30 2004 From: tismer at stackless.com (Christian Tismer) Date: Thu May 13 08:57:09 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <2mpt98ll6d.fsf@starship.python.net> References: <2mpt98ll6d.fsf@starship.python.net> Message-ID: <40A370F6.4020501@stackless.com> Michael Hudson wrote: ... > I don't know what we can do about this. Armin suggested another hack: > stick the address of a stack variable in Py_Main() in a global and > compare the address of another stack variable when you want to see how > much stack you have left. Obvious problems include knowing what's > safe and which direction the stack is growing in... Even more scarily, > what SBCL (a Common Lisp implementation) does is mprotect() a VM page > and the end of the stack and deal with overflow in a SIGSEGV handler. > It's hard to see what else could be really safe (have I mentioned that > I hate C recently?). > > Option 3, I guess, is integrate stackless :-) Just as a note: Even Stackless can get into deep recursions and then uses a hack similar to Armin's suggestion. As a side effect, this made it quite simple to convince cPickle to pickle very deeply nested structures. On verification: I think I'm all against writing a bytecode verifier, because everything needed is already there. Hartmut Goebel has written a nice Python decompyler, based upon John Aycock's spark and prior work. It produces output that often looks better than the original source. And they verify the decompiled bytecode by compiling it again. The drawback is that this appears to be no longer an open source project, see http://www.crazy-compilers.com/decompyle/ Maybe we should talk to Hartmut... On sending code over the network: Bytecode verification is fine, but you don't want to execute foreign bytecode, even if it is valid. This is like executing any binary program, which might do anything, being Python or not. Since people are going to send programs over the network, we will need some way to exchange compiled code in a trusted manner, and I believe this take more considerations which is a matter for the crypto people. Anyway, here is My Proposal (TM) ================ Let's assume that you don't trust any bytecode that has not yet been verified for your machine. Further, you generate a private key for your machine, or yourself. For .pyc archives which you create yourself, your private key is used to initialize a sha digest, and then the .pyc is run through the digest, and the digest added to the .pyc. When loading any .pyc, your private key is used again to compute the digest, and the result verified against the digest which is in the pyc (at its end I guess). For code objects which come over a network, I suggest a similar check, given that you have gotten the other side's key and you can use it to verify foreign code. This works in trusted networks, only. In public networks you would need public/private key pairs... much harder. Now, any bytecode that is not yet verified can be run through decompyle just once, and the result is compiled again, signed with your key, and stored. This is more than requested, since it doesn't allow for any bytecode sequence, but just something that has an existing possible source code. Well, actually I think we want this, because bytecode is an optimization artifact. Summarizing, I think to add SHA keys to bytecode and .pyc files, at least as an option, makes some sense and is fast to check on every external access to bytecode. Verification of healthy code on the VM level could be made easy with decompyle. cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From barry at python.org Thu May 13 08:59:34 2004 From: barry at python.org (Barry Warsaw) Date: Thu May 13 08:59:43 2004 Subject: [Python-Dev] cookielib module in 2.4? In-Reply-To: References: Message-ID: <1084453174.28228.656.camel@anthem.wooz.org> On Wed, 2004-05-12 at 18:40, John J Lee wrote: > I have a module that I'd like to get included in Python 2.4. It adds > transparent HTTP cookie handling to urllib2 Although I haven't looked at your code, I think the functionality would be a good addition to Python. Brett's right though, it needs (Python style) documentation and unit tests. I wonder if it makes sense to combine the functionality of Cookie.py and cookielib.py? Perhaps not, but at the very least I'd like to see something like this integrated into the eventual output of the web-sig (probably not for 2.4). -Barry From anthony at python.org Thu May 13 09:48:03 2004 From: anthony at python.org (Anthony Baxter) Date: Thu May 13 10:19:04 2004 Subject: [Python-Dev] RELEASED Python 2.3.4, release candidate 1 Message-ID: <40A37C93.1090305@python.org> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.4 (release candidate 1). Python 2.3.4 is a bug-fix release. See the release notes at the website (also available as Misc/NEWS in the source distribution) for details of the bugs squished in this release. Assuming no major problems crop up, a final release of Python 2.3.4 will follow late next week. For more information on Python 2.3.4, including download links for various platforms, release notes, and known issues, please see: ~ http://www.python.org/2.3.4 Highlights of this new release include: ~ - Bug fixes. According to the release notes, more than 20 bugs ~ have been fixed, including a couple of bugs that could cause ~ Python to crash. These were discovered by Zope3. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at ~ http://www.python.org/2.3/highlights.html Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python Release Manager (on behalf of the entire python-dev team) -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFAo3yTDt3F8mpFyBYRAicdAJwOngKitELJ25u3oCW+iQcc581wbQCgh1Ah f/Ci2omZVG7p63xY9cfZiSw= =e3Rv -----END PGP SIGNATURE----- From nas at arctrix.com Thu May 13 10:24:53 2004 From: nas at arctrix.com (Neil Schemenauer) Date: Thu May 13 10:24:58 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <20040513085401.GA28305@vicky.ecs.soton.ac.uk> References: <200405121531.43532.michel@dialnetwork.com> <20040513085401.GA28305@vicky.ecs.soton.ac.uk> Message-ID: <20040513142453.GA11925@mems-exchange.org> On Thu, May 13, 2004 at 09:54:01AM +0100, Armin Rigo wrote: > If some long-term goal is to fix this, then MAKE_CLOSURE could be > preceded by a BUILD_TUPLE to collect all free vars into a tuple > (which is what MAKE_CLOSURE does anyway). I've done this in the ast branch. Neil From jim.jewett at eds.com Thu May 13 10:23:55 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Thu May 13 10:25:06 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested Message-ID: Martin v. L?wis: > Therefore, byte code verification usually puts additional > constraints on byte code, rejecting some correct code as > unverifiable. An alternative would be to accept unverifiable code. You won't stop all problems, but you can still catch the most common mistakes. For instance, the example that started this thread was apparently an ASCII string of uncomiled source code. Just checking that all instructions are valid bytecodes might have caught this. Also checking that arguments are possible (no jumping beyond the code) would probably have caught it. Verifying stack depth and such might be overkill for a first pass sanity check. -jJ From tim.one at comcast.net Thu May 13 10:57:58 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu May 13 10:57:58 2004 Subject: [Python-Dev] IPv6 support in Win binary again In-Reply-To: <200405130433.i4D4WxU35612@mcs.anl.gov> Message-ID: [Ivan Judson] > What is this implying about the 2.4 build assumptions? > > Vs.net 2008 :)? Platform sdk? Something else? The Windows Python 2.4 distributed by the PSF will be built with VC 7.1 (aka VC.NET 2003), assuming things go as planned. When Martin said: >> Using VC.NET is out of the question, since it would break the ABI >> across minor releases, which is out of the question. he should perhaps have said "micro" releases, under the usual major.minor.micro release-numbering scheme. In the Python world, a boost in the minor (second) number is more what other projects consider to be a major release. IOW, the Python numbering scheme is more like new_language.major.bugfix From guido at python.org Thu May 13 12:51:54 2004 From: guido at python.org (Guido van Rossum) Date: Thu May 13 12:52:00 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: Your message of "Thu, 13 May 2004 09:54:01 BST." <20040513085401.GA28305@vicky.ecs.soton.ac.uk> References: <200405121531.43532.michel@dialnetwork.com> <20040513085401.GA28305@vicky.ecs.soton.ac.uk> Message-ID: <200405131651.i4DGpsl21234@guido.python.org> > Interestingly, it is not the case currently that for any bytecode > position the stack depth is always the same. Here is a short > review. Thanks for this overview! I'd lost track of details like this over the years. > * 'finally' blocks are executed with one, two or three values pushed > on the stack; END_FINALLY will accordingly pop one, two or three > values (depending on what the first popped value is). This is easily fixed, if necessary -- but is it necessary? > * POP_BLOCK will empty the value stack up to the position stored in > the block stack's popped block, for no reason that I am aware of. > I can't think of a compiler-emitted construct that uses this > feature. Maybe it was introduced to implement 'break' as a jump > to the POP_BLOCK, but nowadays 'break' is more complicated anyway, > using the general stack-unwinding and > exception-handler-dispatching loop after the switch in ceval.c. I can't remember what my intention was; I probably wasn't so sure that the stack level at the end of executing arbitrary code was always the same. > * MAKE_CLOSURE pops some cells to use for free variables, the number > of which is found in the code object on the top of the stack. This is a recent addition, and should have been done differently. It's not hard to fix, as you suggest (and as has been done in the AST branch). > I believe that all the other instructions have a stack effect that > is either constant or computable from their oparg. Me too. > If some long-term goal is to fix this, then MAKE_CLOSURE could be > preceded by a BUILD_TUPLE to collect all free vars into a tuple > (which is what MAKE_CLOSURE does anyway). Fixing the 'finally' > blocks isn't too complicated -- always push three values on the > stack, possibly None's. A better 'fix' would be that the block > stack could, as we discussed at PyCon, be replaced by a purely > static structure in the code object. Correct, I would like to see this happen. There's dynamic stuff here just to make the implementation easy rather than to support necessary dynamism. (The same could be said of some other parts of Python, but there we've come to love the dynamism -- for the most part anyway. ;-) --Guido van Rossum (home page: http://www.python.org/~guido/) From jeremy at alum.mit.edu Thu May 13 13:08:40 2004 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Thu May 13 13:08:45 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <2mpt98ll6d.fsf@starship.python.net> Message-ID: > I think stack over/underflow is checked in a debug build too. > So here's a way to plug these holes: run a debug build the whole time > :-) It's only, what, a factor of 3 performance hit. That's what I do. It's only a small constant factor. Jeremy From jjl at pobox.com Thu May 13 14:36:56 2004 From: jjl at pobox.com (John J Lee) Date: Thu May 13 14:37:03 2004 Subject: [Python-Dev] cookielib module in 2.4? In-Reply-To: <1084453174.28228.656.camel@anthem.wooz.org> References: <1084453174.28228.656.camel@anthem.wooz.org> Message-ID: On Thu, 13 May 2004, Barry Warsaw wrote: > On Wed, 2004-05-12 at 18:40, John J Lee wrote: > > I have a module that I'd like to get included in Python 2.4. It adds > > transparent HTTP cookie handling to urllib2 > > Although I haven't looked at your code, I think the functionality would > be a good addition to Python. Brett's right though, it needs (Python > style) documentation and unit tests. There are unit tests aplenty, and LaTeX docs are on the way. > I wonder if it makes sense to combine the functionality of Cookie.py and > cookielib.py? Perhaps not, but at the very least I'd like to see Not. :-) There is little overlap between the two modules, and it would probably complicate cookielib to combine it with Cookie (and might possibly make my brain explode). > something like this integrated into the eventual output of the web-sig > (probably not for 2.4). The web-sig list has been dead for some time. I haven't seen any client-side output from web-sig, other than my own work on a Browser class. Haven't seen any server-side output about which there's any consensus, either, come to that. John From jjl at pobox.com Thu May 13 14:42:35 2004 From: jjl at pobox.com (John J Lee) Date: Thu May 13 14:42:43 2004 Subject: [Python-Dev] cookielib module in 2.4? In-Reply-To: <40A2FD59.9040906@interlink.com.au> References: <40A2C6CE.9060007@ocf.berkeley.edu> <40A2FD59.9040906@interlink.com.au> Message-ID: On Thu, 13 May 2004, Anthony Baxter wrote: > Brett C. wrote: > > John J Lee wrote: > >> 2. give me a rough idea how long I have to sort out the remaining issues > >> and submit a patch > > > > > > How ever long it takes. Not sure if I really understand what you are > > wondering about. > > If the question is "how long has he got before 2.4 alpha 1", the answer > is "about 6 weeks". I'm still aiming for a first alpha in the first week > of July or thereabouts. Question is : "how long before it's too late to get into 2.4?" I take it from your answer that there's not a long freeze period planned before first alpha, so I'll submit a patch in a week or two. Thanks all John From michel at dialnetwork.com Thu May 13 13:07:55 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Thu May 13 15:09:34 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: References: Message-ID: <200405131207.55026.michel@dialnetwork.com> On Thursday 13 May 2004 04:43, python-dev-request@python.org wrote: > From: "Phillip J. Eby" > As I think Tim has already mentioned, all you do is track the stack depth > for each instruction, and ensure it's consistent. If there is an > inconsistency in the predicted stack level for an instruction, as compared > with any of the instructions that jump to it (no matter where they jump > from), you have a problem. > > Or, to look at it another way: Thanks Phillip, this code is a great starting point. I was trying to reproduce the algorithm given in the JVM Spec section 4.9, but it is seriously complicated by strict type issue which have no meaning here. Also, as Armin pointed out, there are a couple instances of instructions that would be difficult to analize without some changes to their design (END_FINALLY and MAKE_CLOSURE). But he suggests relatively simple solutions to both. I am also working on a short list of static and structural constraints for bytecode, this is what I have so far (the ones i'm unsure about are questions): Static Constraints on Bytecode Instructions 1. The bytecode string must not be empty (len(co_code) > 0). 2. Is there maximum co_code length? 3. The first instruction in the bytecode string begins at index 0. 4. Only valid bytecodes with correct number of operands can be in the bytecode string. 5. The index of the next instruction equals the index of the current instruction plus the length of the current instruction and all its operands. 6. The last byte of the last instruction must be len(co_code) - 1. 7. The last instruction must be RETURN_VALUE? 8. Exception handlers verification? Maybe the compiler could generate a table of handlers, their handled types, and their instruction ranges for verification. The run-time benefit would be that an exception handler could be quickly dispatched instead of falling through each except block in sequence looking for the matching handler as it looks to work now (not that it's a big performance bottleneck...) Static Constraints on Bytecode Instruction Operands 1. The target of a jump instruction must be within the code boundaries and must not fall between an instruction and its operands. 2. The operand of a LOAD_CONST instruction must be an valid index into the co_consts. 3. Can a similar bounds check be made at verfication time for LOAD_FAST? 4. Can index into co_names, co_varnames, co_freevars, co_cellvars be verfied? Structural Constraints between Bytecode Instructions 1. Each instruction must only be executed with the appropriate number of arguments in the operand stack, regardless of the execution path that leads to its invocation. 2. If an instruction can be executed along several different execution paths, the value stack must have the same depth prior to the execution of the instruction, regardless of the path taken. 3. At no point during execution can the value stack grow to a depth greater than that implied by co_stacksize. > This algorithm should readily handle all forward and backward jumps, while > ensuring that no stack growth or shrinkage can take place, because every > instruction must have a single known stack depth, no matter how many > forward or backward jumps it takes to get there. > > Notice that the algorithm will "follow" every branch, but will not retrace > the code at the target location once the stack depth is seen to be > consistent. It also tackles underflow and "falling off the end" of the > bytecode, while simultaneously computing the maximum stack depth > used. Each instruction is fetched and processed for its depth effects and > other characteristics exactly once, so the timing is O(n). And, it has a > nice place to put opcode-specific operand validation for opcodes that use > locals, freevars, names, or constants. Yea! As Tim points out below there are some subtleties, but this is great. > Anyway, I think we can safely halt discussion of the halting problem > now. :) And, since I've now reduced this from an interesting theoretical > idea to talk about, into something resembling work (God forbid), anybody > who wants to actually *implement* this crazy idea is now free to do so, > without even having to read the paper Tim linked to. :) Well I did read it, and I suggest anyone else also read chapter 4 of the Java VM Spec. -Michel From michel at dialnetwork.com Thu May 13 13:04:36 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Thu May 13 15:09:47 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: References: Message-ID: <200405131204.36176.michel@dialnetwork.com> On Thursday 13 May 2004 04:43, python-dev-request@python.org wrote: > It's important to read what people who have actually done it have to say > about subtleties. For example, your algorithm considered 'return' to be 'a > terminal instruction'. But what if it appeared in the 'try' clause of a > try/finally construct? In Python's PVM, most *possible* control flow is > implicit (virtually any opcode can raise an exception, and from there > "magically jump" to the code at an enclosing 'except' or 'finally' clause); > and even an unexceptional bare 'return' can magically jump to an enclosing > 'finally' block. The Java reference discusses these things, and anyone > intending to do something here is still strongly urged to read it. But just looking at some code here, is RETURN_VALUE ever inside a SETUP_EXCEPT block? The end of the SETUP_EXCEPT block apears to ABSOLUTE_JUMP past all the exception handlers to finally, and then to the RETURN_VALUE, which doesn't seem to fall under the protection of a try/except. I'm not saying there are no subtleties as you describe, I'm just trying to quantify some of them and wondering if this is one. -Michel From martin at v.loewis.de Thu May 13 15:23:35 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Thu May 13 15:23:41 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405131204.36176.michel@dialnetwork.com> References: <200405131204.36176.michel@dialnetwork.com> Message-ID: <40A3CB37.4090001@v.loewis.de> Michel Pelletier wrote: > But just looking at some code here, is RETURN_VALUE ever inside a SETUP_EXCEPT > block? What about this code: >>> def foo(): ... try: ... try: ... return 7 ... except: ... pass ... finally: ... print "Hello" ... >>> dis.dis(foo) 2 0 SETUP_FINALLY 22 (to 25) 3 3 SETUP_EXCEPT 8 (to 14) 4 6 LOAD_CONST 1 (7) 9 RETURN_VALUE 10 POP_BLOCK 11 JUMP_FORWARD 7 (to 21) 5 >> 14 POP_TOP 15 POP_TOP 16 POP_TOP 6 17 JUMP_FORWARD 1 (to 21) 20 END_FINALLY >> 21 POP_BLOCK 22 LOAD_CONST 0 (None) 8 >> 25 LOAD_CONST 2 ('Hello') 28 PRINT_ITEM 29 PRINT_NEWLINE 30 END_FINALLY 31 LOAD_CONST 0 (None) 34 RETURN_VALUE From barry at python.org Thu May 13 15:23:48 2004 From: barry at python.org (Barry Warsaw) Date: Thu May 13 15:23:57 2004 Subject: [Python-Dev] cookielib module in 2.4? In-Reply-To: References: <1084453174.28228.656.camel@anthem.wooz.org> Message-ID: <1084476227.28228.753.camel@anthem.wooz.org> On Thu, 2004-05-13 at 14:36, John J Lee wrote: > There are unit tests aplenty, and LaTeX docs are on the way. Cool. > > I wonder if it makes sense to combine the functionality of Cookie.py and > > cookielib.py? Perhaps not, but at the very least I'd like to see > > Not. :-) > > There is little overlap between the two modules, and it would probably > complicate cookielib to combine it with Cookie (and might possibly make my > brain explode). > > > > something like this integrated into the eventual output of the web-sig > > (probably not for 2.4). > > The web-sig list has been dead for some time. I haven't seen any > client-side output from web-sig, other than my own work on a Browser > class. Haven't seen any server-side output about which there's any > consensus, either, come to that. Sigh. Well, don't wait for them. ;) -Barry From tim.one at comcast.net Thu May 13 15:32:30 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu May 13 15:32:35 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405131204.36176.michel@dialnetwork.com> Message-ID: [Michel Pelletier] > But just looking at some code here, is RETURN_VALUE ever inside a > SETUP_EXCEPT block? The end of the SETUP_EXCEPT block apears to > ABSOLUTE_JUMP past all the exception handlers to finally, and then to > the RETURN_VALUE, which doesn't seem to fall under the protection of a > try/except. I don't think I can understand what you're saying without a concrete example. Here's one: """ def f(): try: return 42 finally: print 'yup' from dis import dis dis(f) """ That displays: """ 2 0 SETUP_FINALLY 8 (to 11) 3 3 LOAD_CONST 1 (42) 6 RETURN_VALUE 7 POP_BLOCK 8 LOAD_CONST 0 (None) 4 >> 11 LOAD_CONST 2 ('yup') 5 14 PRINT_ITEM 15 PRINT_NEWLINE 16 END_FINALLY 17 LOAD_CONST 0 (None) 20 RETURN_VALUE """ The RETURN_VALUE at 6 doesn't return right away, although that's not obvious from the byte code. The code at 11, 14, 15, 16 executes first, and that can't be guessed from staring at 6 in isolation. I think the code at 7, 8, 17 and 20 is actually unreachable -- and that's so not obvious I had to say "I think" . From skip at pobox.com Thu May 13 15:35:44 2004 From: skip at pobox.com (Skip Montanaro) Date: Thu May 13 15:34:27 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405131207.55026.michel@dialnetwork.com> References: <200405131207.55026.michel@dialnetwork.com> Message-ID: <16547.52752.985771.539493@montanaro.dyndns.org> Michel> 7. The last instruction must be RETURN_VALUE? Nope. Relatively simple analysis of the basic blocks in the bytecode can determine if the trailing push none/return value pair is dead code and remove it if it is. Michel> 2. The operand of a LOAD_CONST instruction must be an valid Michel> index into the co_consts. Michel> 3. Can a similar bounds check be made at verfication time for Michel> LOAD_FAST? Should be possible, yes. Michel> 4. Can index into co_names, co_varnames, co_freevars, Michel> co_cellvars be verfied? Also yes. These are all fixed-length arrays (tuples actually) whose lengths are determined at compile-time. Skip From fumanchu at amor.org Thu May 13 15:46:32 2004 From: fumanchu at amor.org (Robert Brewer) Date: Thu May 13 15:49:30 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested Message-ID: Michel Pelletier wrote: > I am also working on a short list of static and structural > constraints for > bytecode, this is what I have so far (the ones i'm unsure about are > questions): A couple gotchas when useing new.code(): 1. co_names, co_varnames, co_freevars, and co_cellvars take tuples of strings; passing unicode strings crashes the interpreter. co_consts, however, is unicode-safe. 2. Some co_flags should be checked to be consistent with other args; for example, shouldn't CO_NOFREE be set if co_freevars is empty? Robert Brewer MIS Amor Ministries fumanchu@amor.org From tim.one at comcast.net Thu May 13 19:31:07 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu May 13 19:31:10 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib/email Parser.py, 1.20, 1.20.12.1 In-Reply-To: Message-ID: [Barry] > Log Message: > _parsebody(): Do not create subparts unless the container has a main > type of 'multipart' and the boundary is defined. This fixes SF bug # > 846938, and several recent email-sig bugs ... Yadda yadda yadda. At this point, the more obvious solution is to ignore all that MIME crap, and just guess a good structure for the email. The spambayes Outlook addin does that all the time . From barry at python.org Thu May 13 19:45:44 2004 From: barry at python.org (Barry Warsaw) Date: Thu May 13 19:46:08 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib/email Parser.py, 1.20, 1.20.12.1 In-Reply-To: References: Message-ID: <1084491944.28228.854.camel@anthem.wooz.org> On Thu, 2004-05-13 at 19:31, Tim Peters wrote: > Yadda yadda yadda. At this point, the more obvious solution is to ignore > all that MIME crap, and just guess a good structure for the email. The > spambayes Outlook addin does that all the time . Structure's for monkeyboys and pointy-bracket lovers, not for studly w1nd0z 1nst4ll3r d00dz and 3m41l l33t h4x0rz. From tdelaney at avaya.com Thu May 13 19:51:09 2004 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Thu May 13 19:51:19 2004 Subject: [Python-Dev] .pth files in current or script directory Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A3E5@au3010avexu1.global.avaya.com> A discussion (last week?) on c.l.py prompted a thought about .pth files. Feel free to shoot it down ;) The discussion in question concerned someone using a version control system, and needing to change PYTHONPATH depending on which branch they were in. My immediately two thoughts were: 1. Use a build system (e.g. make) to set PYTHONPATH appropriately. This is they type of thing we do. 2. Use a .pth file in the directory the code is run from (or possibly, the same directory as the script). Before posting the second, I thought I'd better try it out - fortunately for me :) Putting a .pth in the current or script directory did nothing. I'm wondering if anyone sees utility in being able to put a .pth file in those locations and have them used in the same way as if they were in the python executable directory. For example, if you have a package 'p', with a 'test' subpackage (containing scripts), it would be very simple to stick a 'test.pth' file into the 'p.test' package, with the sole contents of 'test.pth' being '..'. This would add the (absolute path of the) parent directory to the root paths for module import. I feel there's some kind of link here with PEP 328, but I'm not quite sure where it fits. Tim Delaney From greg at cosc.canterbury.ac.nz Thu May 13 20:28:28 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu May 13 20:28:51 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <2mpt98ll6d.fsf@starship.python.net> Message-ID: <200405140028.i4E0SSTe005757@cosc353.cosc.canterbury.ac.nz> Michael Hudson : > Even more scarily, what SBCL (a Common Lisp implementation) does is > mprotect() a VM page and the end of the stack and deal with overflow > in a SIGSEGV handler. Even that isn't necessarily reliable. What if a C call allocates such a big stack frame that it jumps right over the protected page? Eventually it will try to access the protected area, but other chaos could occur in the meantime. The only completely safe thing I can think of is to eliminate C-level recursion altogether, a la Stackless. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Thu May 13 20:35:59 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu May 13 20:36:09 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <40A370F6.4020501@stackless.com> Message-ID: <200405140035.i4E0ZxUd005767@cosc353.cosc.canterbury.ac.nz> Christian Tismer : > Now, any bytecode that is not yet verified can be run through > decompyle just once, and the result is compiled again, signed > with your key, and stored. If you're going to all that bother, why not just send suitably-signed source code in the first place? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From guido at python.org Thu May 13 20:25:53 2004 From: guido at python.org (Guido van Rossum) Date: Thu May 13 21:28:58 2004 Subject: [Python-Dev] .pth files in current or script directory In-Reply-To: Your message of "Fri, 14 May 2004 09:51:09 +1000." <338366A6D2E2CA4C9DAEAE652E12A1DE0179A3E5@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A3E5@au3010avexu1.global.avaya.com> Message-ID: <200405140025.i4E0Pro21903@guido.python.org> > The discussion in question concerned someone using a version control > system, and needing to change PYTHONPATH depending on which branch they > were in. Can't they manipulate sys.path *inside* the code in question? That's invariably the solution I end up using -- I never play with PYTHONPATH any more. --Guido van Rossum (home page: http://www.python.org/~guido/) From jepler at unpythonic.net Thu May 13 21:44:13 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Thu May 13 21:44:29 2004 Subject: [Python-Dev] .pth files in current or script directory In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A3E5@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A3E5@au3010avexu1.global.avaya.com> Message-ID: <20040514014413.GB26047@unpythonic.net> I'm -NFS against reading .pth files from the current directory. When a directory is large and is on a remote filesystem, enumerating it at startup can really kill time -- heck, just putting an NFS directory with a handful of entries in PYTHONPATH can slow Python's startup time considerably, now imagine that you "cd /nfs/directory_with_1000_files" and "time python -c 'pass'" .. blah! Jeff From tismer at stackless.com Thu May 13 21:49:44 2004 From: tismer at stackless.com (Christian Tismer) Date: Thu May 13 21:48:22 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405140035.i4E0ZxUd005767@cosc353.cosc.canterbury.ac.nz> References: <200405140035.i4E0ZxUd005767@cosc353.cosc.canterbury.ac.nz> Message-ID: <40A425B8.7080203@stackless.com> Greg Ewing wrote: > Christian Tismer : > > >>Now, any bytecode that is not yet verified can be run through >>decompyle just once, and the result is compiled again, signed >>with your key, and stored. > > > If you're going to all that bother, why not just send > suitably-signed source code in the first place? Because I prefer suitably-signed bytecode, which would usually only need the key verification, but no compile step. There is also no redundancy due to formatting. A program with a specific key always *is* that program. I always thought that with a little addition, Python source code could be completely replaced by bytecode. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From greg at cosc.canterbury.ac.nz Thu May 13 23:46:21 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Thu May 13 23:46:47 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <40A425B8.7080203@stackless.com> Message-ID: <200405140346.i4E3kLoL006025@cosc353.cosc.canterbury.ac.nz> Christian Tismer : > Because I prefer suitably-signed bytecode, which would > usually only need the key verification, but no compile > step. But you've just suggested decompiling and recompiling the bytecode in order to verify it. This can't possibly be faster than just compiling it. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tismer at stackless.com Fri May 14 11:02:37 2004 From: tismer at stackless.com (Christian Tismer) Date: Fri May 14 11:01:23 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405140346.i4E3kLoL006025@cosc353.cosc.canterbury.ac.nz> References: <200405140346.i4E3kLoL006025@cosc353.cosc.canterbury.ac.nz> Message-ID: <40A4DF8D.7050708@stackless.com> Greg Ewing wrote: > Christian Tismer : > > >>Because I prefer suitably-signed bytecode, which would >>usually only need the key verification, but no compile >>step. > > > But you've just suggested decompiling and recompiling the bytecode in > order to verify it. This can't possibly be faster than just compiling > it. This is not what I said and also irrelevant, since it will only happen once on untrusted bytecode. Afterwards it is trusted and stored together with a keyed digest for later verification. -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From theller at python.net Fri May 14 15:44:45 2004 From: theller at python.net (Thomas Heller) Date: Fri May 14 15:44:52 2004 Subject: [Python-Dev] SF CVS question Message-ID: When doing 'cvs -n update', I get a warning like this: cvs server: WARNING: global `-l? option ignored. Does anyone else get the same? Is there something wrong with my configuration? It also occurrs on other SF cvs repositories, not only with python. Thomas From tim.one at comcast.net Fri May 14 15:52:04 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri May 14 15:52:04 2004 Subject: [Python-Dev] SF CVS question In-Reply-To: Message-ID: [Thomas Heller] > When doing 'cvs -n update', I get a warning like this: > cvs server: WARNING: global `-l? option ignored. > > Does anyone else get the same? Is there something wrong with my > configuration? It also occurrs on other SF cvs repositories, not only > with python. LOL! I never tried cvs -n before. Here was my result: C:\Code\python>cvs -n up CVS.EXE update: inflate: unknown compression method CVS.EXE [update aborted]: reading from server: Input/output error Write failed flushing stdout buffer. write stdout: Broken pipe A regular "cvs up" (no -n) still works fine. From michel at dialnetwork.com Fri May 14 13:39:44 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Fri May 14 15:55:08 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405131204.36176.michel@dialnetwork.com> References: <200405131204.36176.michel@dialnetwork.com> Message-ID: <200405141239.44886.michel@dialnetwork.com> On Thursday 13 May 2004 12:04, Michel Pelletier wrote: > But just looking at some code here, is RETURN_VALUE ever inside a > SETUP_EXCEPT block? The end of the SETUP_EXCEPT block apears to > ABSOLUTE_JUMP past all the exception handlers to finally, and then to the > RETURN_VALUE, which doesn't seem to fall under the protection of a > try/except. Never mind, I proved my self wrong. RETURN_VALUE can often happen inside SETUP_EXCEPT and _FINALLY. Expect the occasionally stupid question (I know, there are no stupid questions....) -Michel From tim.one at comcast.net Fri May 14 15:58:47 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri May 14 15:58:46 2004 Subject: [Python-Dev] SF CVS question In-Reply-To: Message-ID: > LOL! I never tried cvs -n before. Here was my result: > > C:\Code\python>cvs -n up > CVS.EXE update: inflate: unknown compression method > CVS.EXE [update aborted]: reading from server: Input/output error ... I should note that I have a .cvsrc file containing the line cvs -q -z6 If I rename .cvsrc so that CVS can't find it, then "cvs -n up" works fine for me (at least in SF's Python repository). From theller at python.net Fri May 14 16:01:42 2004 From: theller at python.net (Thomas Heller) Date: Fri May 14 16:01:49 2004 Subject: [Python-Dev] SF CVS question In-Reply-To: (Tim Peters's message of "Fri, 14 May 2004 15:52:04 -0400") References: Message-ID: <8yfu7p7d.fsf@python.net> "Tim Peters" writes: > [Thomas Heller] >> When doing 'cvs -n update', I get a warning like this: >> cvs server: WARNING: global `-l? option ignored. >> >> Does anyone else get the same? Is there something wrong with my >> configuration? It also occurrs on other SF cvs repositories, not only >> with python. > > LOL! I never tried cvs -n before. Here was my result: > > C:\Code\python>cvs -n up > CVS.EXE update: inflate: unknown compression method > CVS.EXE [update aborted]: reading from server: Input/output error > Write failed flushing stdout buffer. > write stdout: Broken pipe That looks more like the more or less transient error I get from SF from time to time. cvs -n up is useful to display the state of the files in the sandbox. And pcl-cvs (from inside xemacs) uses this when you do a M-x cvs-examine. > > A regular "cvs up" (no -n) still works fine. Yes. Sometimes ;-) Thomas From gregory.lielens at fft.be Fri May 14 14:52:40 2004 From: gregory.lielens at fft.be (Gregory Lielens) Date: Fri May 14 16:16:09 2004 Subject: [Python-Dev] modifying input behavior with PyOS_ReadlineFunctionPointer Message-ID: <001701c439e4$a24034a0$1500a8c0@chikubi> Hello all, I have posted this to the general comp.lang.python list, But as it is really about an implementation detail, it is probably more a topic for the development list I apologize in advance if this is not the correct place to post this.... We are currently developing python bindings to make a MPI parallel application (Finite element solver) script driven. I have written a main interpreter, able to execute scripts or interactive commands, in // , based on the following techniques: *file execution: all python interpreter processes run the file using PyRun_SimpleFile (file must be available to all processes) *file execution using broadcast: process 0 read input file and store it in a file, broadcast it to all processes and the script is run using PyRun_SimpleString (file must be available to process 0 only) *interactive execution: All processes listen to user input using a PyRun_InteractiveLoop. Process 0 read stdin and broadcast the line to all process, the other process listen to this broadcast. This is implemented using the (very useful in this case :-) ) PyOS_ReadlineFunctionPointer , reassigning it to an input function which wrap the default PyOS_ReadlineFunctionPointer (to be able to reuse nice readline functionalities) and add the broadcasting mechanism... All of this work perfectly, but I had to modify the python sources, Parser/myreadline.c to be precise... Indeed, the behavior of PyOS_Readline is to use the user-modifiable PyOS_ReadlineFunctionPointer as input mechanism, except when input is non-interactive in which case it fall back to the non-user-modifiable PyOS_StdioReadline...As my processes are non-interactive except for proc 0 (launched in background), proc 0 broadcast but the other ones do not listen. I thus had to remove this test, so that the user-modifiable function is called in all cases. /*Python code snipped from Parser/myreadline.c*/ char * PyOS_Readline(FILE *sys_stdin, FILE *sys_stdout, char *prompt) { char *rv; if (PyOS_ReadlineFunctionPointer == NULL) { #ifdef __VMS PyOS_ReadlineFunctionPointer = vms__StdioReadline; #else PyOS_ReadlineFunctionPointer = PyOS_StdioReadline; #endif } } Py_BEGIN_ALLOW_THREADS /* This is needed to handle the unlikely case that the * interpreter is in interactive mode *and* stdin/out are not * a tty. This can happen, for example if python is run like * this: python -i < test1.py */ /* my modif: comment out the use of PyOS_StdioReadline for non-interactive input...*/ /*if (!isatty (fileno (sys_stdin)) || !isatty (fileno(sys_stdout))) rv = PyOS_StdioReadline (sys_stdin, sys_stdout, prompt); else*/ rv = (*PyOS_ReadlineFunctionPointer)(sys_stdin, sys_stdout, prompt); Py_END_ALLOW_THREADS return rv; } } /*end of snipped code*/ This is a small modif, but annoying because we would prefer to use the plain python distribution, and anyway I wonder if the current behavior of python is desirable, because basically it removes part of the functionality of PyOS_ReadlineFunctionPointer: flexibility disappear for non-interactive input. In addition, this seems not so robust, it seems to me that it is not correct for vms, where PyOS_StdioReadline will be used in non-interactive case while vms__StdioReadline will be used for interactive one....maybe this is the intended behavior, but then the function naming is strange :-) So basically, shouldn't it be the responsibility of the code which change PyOS_ReadlineFunctionPointer to check if the function given is adapted to interactiveness of input? this seems the more flexible approach, and probably involve only minor modif of the readline module. In addition, wouldn't it be nice to initialize PyOS_ReadlineFunctionPointer to a default value (suitable reading function) at declaration, instead of defining it to NULL and let PyOS_Readline do the initialization when needed? This way, user code can get back a meaningful reading function storing the old value of PyOS_ReadlineFunctionPointer, use it to write an extended input function, and reassign it to PyOS_ReadlineFunctionPointer... This seems to me like the most flexible way to add user-tunable input method... So, to all experienced python developers, having used this PyOS_ReadlineFunctionPointer function or implemented this code, what do you think of these possible slight modifications? Or is there a better way to use it than my current method? Sorry for the long (and possibly strangely written, English is not my native language :-) ) post, Best regards, Greg. From vdbergh at luc.ac.be Sat May 15 10:27:15 2004 From: vdbergh at luc.ac.be (Michel Van den Bergh) Date: Sat May 15 10:24:59 2004 Subject: [Python-Dev] Re: Welcome to the "Python-Dev" mailing list References: Message-ID: <40A628C3.1050409@luc.ac.be> Hello, I apologize for posting so quickly on this list but I am having a problem. I am the author of a game written in python (PyTraffic, it is on the vault) and I am preparing a new version which will be much nicer in many ways. It also includes some help in the form of a html file. Right now I am using my own methods to display html files. However I am also checking out the webbrowser module. Unfortunately this module has some obvious problems (tested on CVS version under Linux with mozilla) (*) If you open an url (with the option "new window") and the browser is not already running then it opens *two* windows: the home page and your url. This is different for example from the default behaviour on Windows and also quite ugly. (**) There is another related problem: if you open an url and the browser is not already running then the code starts a new browser, does a timed sleep and then issues a "-remote" command. Using a timed sleep is very bad programming and *extremely fragile*. Indeed I had failures because the delay (PROCESS_CREATION_DELAY) was not long enough. At the very least the -remote command should be tried a couple of times. BUT: there is no reason *at all* to issue a -remote command in this case. All browsers I know accept a url as argument. So one can just start a new browser with the url as argument. These problems are easy to fix but it is not clear from which version of webbrowser.py to start. There are already two patches waiting (not fixing the above problems), with no indication if they will ever be applied. Any comments? Best regards, Michel From mal at egenix.com Sat May 15 10:30:15 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Sat May 15 10:30:20 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib/email Parser.py, 1.20, 1.20.12.1 In-Reply-To: <1084491944.28228.854.camel@anthem.wooz.org> References: <1084491944.28228.854.camel@anthem.wooz.org> Message-ID: <40A62977.4050606@egenix.com> Barry Warsaw wrote: > On Thu, 2004-05-13 at 19:31, Tim Peters wrote: > > >>Yadda yadda yadda. At this point, the more obvious solution is to ignore >>all that MIME crap, and just guess a good structure for the email. The >>spambayes Outlook addin does that all the time . > > > Structure's for monkeyboys and pointy-bracket lovers, not for studly > w1nd0z 1nst4ll3r d00dz and 3m41l l33t h4x0rz. Hmm, I wonder if we should add another fun codec to the encodings package ?! Anyone have a reference ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 15 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 22 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From paul at prescod.net Sat May 15 13:40:33 2004 From: paul at prescod.net (Paul Prescod) Date: Sat May 15 13:40:54 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib/email Parser.py, 1.20, 1.20.12.1 In-Reply-To: <40A62977.4050606@egenix.com> References: <1084491944.28228.854.camel@anthem.wooz.org> <40A62977.4050606@egenix.com> Message-ID: <40A65611.6090103@prescod.net> M.-A. Lemburg wrote: > Hmm, I wonder if we should add another fun codec to the encodings > package ?! Anyone have a reference ? Well Google turns these up: http://www.bbc.co.uk/dna/h2g2/A787917 http://en.wikipedia.org/wiki/Leet http://www.ninjalane.com/leet.aspx http://home.cwru.edu/~jar20/l33tpaper.htm http://www.ryanross.net/leet/ I wish I could find the URL where a guy talks to a wanna-be hacker with increasingly random syntax and the hacker is amazed at how l33t he is. "You are so l33t that I can't even understand what you are saying!" Paul Prescod From jcarlson at uci.edu Sat May 15 13:39:30 2004 From: jcarlson at uci.edu (Josiah Carlson) Date: Sat May 15 13:44:17 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib/email Parser.py, 1.20, 1.20.12.1 In-Reply-To: <40A62977.4050606@egenix.com> References: <1084491944.28228.854.camel@anthem.wooz.org> <40A62977.4050606@egenix.com> Message-ID: <20040515102327.0A07.JCARLSON@uci.edu> > >>Yadda yadda yadda. At this point, the more obvious solution is to ignore > >>all that MIME crap, and just guess a good structure for the email. The > >>spambayes Outlook addin does that all the time . > > > > > > Structure's for monkeyboys and pointy-bracket lovers, not for studly > > w1nd0z 1nst4ll3r d00dz and 3m41l l33t h4x0rz. > > Hmm, I wonder if we should add another fun codec to the encodings > package ?! Anyone have a reference ? The only problem is that the mapping from English to 1337, or 1337 to English on a per-character basis is not one-to-one: s == s.encode('leet').decode('leet') ... would be false. Generating the encoder would be relatively easy, but would only really produce interesting results if, for example, i -> (l, 1, !, |, ;, :), and the resulting mapping was taken in-context to other choices made in the current word. The decoder would be an exercise in frustration, assuming we don't care about getting something exact, but something 'close enough'. Assuming purely English text, one would need a "possible decoding" of every word, all of which could then be fed into a phonetic spell checker, which would need to decide which of the possible decodings for this word was likely the correct one. Sounds like a fun project, though as: http://cockeyed.com/lessons/viagra/viagra.html ... has shown, there may not be a workable solution to the problem. Pity though, it would be fun. - Josiah From tjreedy at udel.edu Sat May 15 13:56:27 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Sat May 15 13:56:34 2004 Subject: [Python-Dev] Re: Welcome to the "Python-Dev" mailing list References: <40A628C3.1050409@luc.ac.be> Message-ID: "Michel Van den Bergh" wrote in message news:40A628C3.1050409@luc.ac.be... > the webbrowser > module. > > Unfortunately this module has some obvious problems (tested on CVS > version under Linux with mozilla) ... > Any comments? Submit a bug or patch item to SF. Add comments to other patches re desire for coordination of fixes. TJR From fdrake at acm.org Sat May 15 14:01:00 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Sat May 15 14:01:12 2004 Subject: [Python-Dev] Re: Welcome to the "Python-Dev" mailing list In-Reply-To: References: <40A628C3.1050409@luc.ac.be> Message-ID: <200405151401.00897.fdrake@acm.org> On Saturday 15 May 2004 01:56 pm, Terry Reedy wrote: > Submit a bug or patch item to SF. Add comments to other patches re desire > for coordination of fixes. Also make sure they're all assigned to the same person. I think the others are currently assigned to me (since I'm to blame for the original version of the module). Feel free to assign the new tracker item to me if the others are indeed assigned to me. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From tim.one at comcast.net Sat May 15 14:06:37 2004 From: tim.one at comcast.net (Tim Peters) Date: Sat May 15 14:06:47 2004 Subject: [Python-Dev] Berkeley test errors in 2.3.4c1 Message-ID: I tried 2.3.4rc1 on Win XP SP1, and got 30 errors in test_bsddb3. Opened a bug report: http://www.python.org/sf/954530 How's test_bsddb3 working on other 2.3.4c1 installations (I encourage adding info to the bug report instead of via email)? From greg at electricrain.com Sat May 15 17:41:11 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Sat May 15 17:41:16 2004 Subject: [Python-Dev] Berkeley test errors in 2.3.4c1 In-Reply-To: References: Message-ID: <20040515214111.GB28971@zot.electricrain.com> On Sat, May 15, 2004 at 02:06:37PM -0400, Tim Peters wrote: > I tried 2.3.4rc1 on Win XP SP1, and got 30 errors in test_bsddb3. Opened a > bug report: > > http://www.python.org/sf/954530 > > How's test_bsddb3 working on other 2.3.4c1 installations (I encourage adding > info to the bug report instead of via email)? fixed. it was broken test case due to a different default behaviour on 2.3 than 2.4. sorry about that. i took the easy route and just cauterized the broken test rather than writing a proper one for the 2.3 branch. From michel.vandenbergh at luc.ac.be Sat May 15 18:11:13 2004 From: michel.vandenbergh at luc.ac.be (michel.vandenbergh@luc.ac.be) Date: Sat May 15 18:45:01 2004 Subject: [Python-Dev] Re: Welcome to the "Python-Dev" mailing list In-Reply-To: <200405151401.00897.fdrake@acm.org> References: <40A628C3.1050409@luc.ac.be> <200405151401.00897.fdrake@acm.org> Message-ID: <1084659073.40a6958115068@webmail.luc.ac.be> Ok, I submitted a patch. Best regard, Michel Quoting "Fred L. Drake, Jr." : > On Saturday 15 May 2004 01:56 pm, Terry Reedy wrote: > > Submit a bug or patch item to SF. Add comments to other patches re > desire > > for coordination of fixes. > > Also make sure they're all assigned to the same person. I think the others > > are currently assigned to me (since I'm to blame for the original version of > > the module). Feel free to assign the new tracker item to me if the others > are indeed assigned to me. > > > -Fred > > -- > Fred L. Drake, Jr. > PythonLabs at Zope Corporation > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/vdbergh%40luc.ac.be > From tim.one at comcast.net Sat May 15 20:25:01 2004 From: tim.one at comcast.net (Tim Peters) Date: Sat May 15 20:25:01 2004 Subject: [Python-Dev] Berkeley test errors in 2.3.4c1 In-Reply-To: <20040515214111.GB28971@zot.electricrain.com> Message-ID: [Gregory P. Smith, on http://www.python.org/sf/954530 ] > fixed. it was broken test case due to a different default behaviour > on 2.3 than 2.4. sorry about that. i took the easy route and just > cauterized the broken test rather than writing a proper one for the > 2.3 branch. Thanks! I ran the original on WinXP from the released 2.3.4c1, but am not able to build my own 2.3 Python on that box (well, not without more setup time than I can give to it now). Going back to a Win98SE box and building from current release23-maint CVS, test_bsddb3 failed with 4 instances of a different error (see attached). But, IIRC, the test has always failed on Win98SE in this way. All the errors I reported above went away. -------------- next part -------------- ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvBTreeShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvHashShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvThreadBTreeShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') ====================================================================== ERROR: test01_basics (bsddb.test.test_dbshelve.EnvThreadHashShelveTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics self.do_open() File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 238, in do_open self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') ---------------------------------------------------------------------- Ran 216 tests in 264.300s FAILED (errors=4) From greg at electricrain.com Sat May 15 20:28:13 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Sat May 15 20:28:17 2004 Subject: [Python-Dev] Berkeley test errors in 2.3.4c1 In-Reply-To: References: <20040515214111.GB28971@zot.electricrain.com> Message-ID: <20040516002813.GA10087@zot.electricrain.com> On Sat, May 15, 2004 at 08:25:01PM -0400, Tim Peters wrote: > [Gregory P. Smith, on > http://www.python.org/sf/954530 > ] > > > fixed. it was broken test case due to a different default behaviour > > on 2.3 than 2.4. sorry about that. i took the easy route and just > > cauterized the broken test rather than writing a proper one for the > > 2.3 branch. > > Thanks! I ran the original on WinXP from the released 2.3.4c1, but am not > able to build my own 2.3 Python on that box (well, not without more setup > time than I can give to it now). Going back to a Win98SE box and building > from current release23-maint CVS, test_bsddb3 failed with 4 instances of a > different error (see attached). But, IIRC, the test has always failed on > Win98SE in this way. All the errors I reported above went away. > ====================================================================== > ERROR: test01_basics (bsddb.test.test_dbshelve.EnvBTreeShelveTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 75, in test01_basics > self.do_open() > File "C:\CODE\23\lib\bsddb\test\test_dbshelve.py", line 238, in do_open > self.env.open(homeDir, self.envflags | db.DB_INIT_MPOOL | db.DB_CREATE) > DBAgainError: (11, 'Resource temporarily unavailable -- unable to join the environment') Yeah I'd chalk that one up to Win98. -g From tim.one at comcast.net Sun May 16 01:47:03 2004 From: tim.one at comcast.net (Tim Peters) Date: Sun May 16 01:47:07 2004 Subject: [Python-Dev] Berkeley test errors in 2.3.4c1 In-Reply-To: <20040516002813.GA10087@zot.electricrain.com> Message-ID: [Gregory P. Smith, on test_bsddb3 Win98SE failures in test01_basics] > Yeah I'd chalk that one up to Win98. Me too . Back over on WinXP and *current* (HEAD) CVS, there's one failure, which I believe is real: ERROR: test04_n_flag (bsddb.test.test_compat.CompatibilityTestCase) ---------------------------------------------------------------------- Traceback (most recent call last): File "C:\Code\python\lib\bsddb\test\test_compat.py", line 86, in test04_n_flag f = hashopen(self.filename, 'n') File "C:\Code\python\lib\bsddb\__init__.py", line 293, in hashopen d.open(file, db.DB_HASH, flags, mode) DBInvalidArgError: (22, 'Invalid argument -- DB_TRUNCATE illegal with locking specified') I suspect Sleepycat screwed us there, changing the rules in midstream. Someone on c.l.py appeared to track down the same thing here, but in an app instead of in our test suite: http://mail.python.org/pipermail/python-list/2004-May/220168.html The change log of Berkeley DB 4.2.52 says "9. Fix a bug to now disallow DB_TRUNCATE on opens in locking environments, since we cannot prevent race conditions ..." From mal at egenix.com Sun May 16 08:31:58 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Sun May 16 08:32:06 2004 Subject: [Python-Dev] RE: [Python-checkins] python/dist/src/Lib/email Parser.py, 1.20, 1.20.12.1 In-Reply-To: <40A65611.6090103@prescod.net> References: <1084491944.28228.854.camel@anthem.wooz.org> <40A62977.4050606@egenix.com> <40A65611.6090103@prescod.net> Message-ID: <40A75F3E.4010708@egenix.com> Paul Prescod wrote: > M.-A. Lemburg wrote: > >> Hmm, I wonder if we should add another fun codec to the encodings >> package ?! Anyone have a reference ? > > > Well Google turns these up: > > http://www.bbc.co.uk/dna/h2g2/A787917 > http://en.wikipedia.org/wiki/Leet > http://www.ninjalane.com/leet.aspx > http://home.cwru.edu/~jar20/l33tpaper.htm > http://www.ryanross.net/leet/ Wow, nice collection of links, thanks. > I wish I could find the URL where a guy talks to a wanna-be hacker with > increasingly random syntax and the hacker is amazed at how l33t he is. > "You are so l33t that I can't even understand what you are saying!" ... I wonder how these guys deal with the problem of expressing numbers in their small little language. I suppose they use the l33t version of roman literals ;-) (c) |\/||\/|1\/, |?`/+|-|()|\| 5()F7\/\/4R3 F()|_||\||)471()|\| -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 16 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 21 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From tdelaney at avaya.com Sun May 16 20:11:23 2004 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Sun May 16 20:11:33 2004 Subject: [Python-Dev] .pth files in current or script directory Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A652@au3010avexu1.global.avaya.com> Guido van Rossum wrote: > Can't they manipulate sys.path *inside* the code in question? That's > invariably the solution I end up using -- I never play with > PYTHONPATH any more. I do this too, but I always feel that I shouldn't have to, especially when I'm not using `sys` for anything else. It also messes up my desire to keep all the imports together - the `sys` magic needs to go first. Jeff Epler wrote: > I'm -NFS against reading .pth files from the current directory. > When a directory is large and is on a remote filesystem, enumerating > it at startup can really kill time -- heck, just putting an NFS > directory with a handful of entries in PYTHONPATH can slow Python's > startup time considerably, now imagine that you "cd > /nfs/directory_with_1000_files" and "time python -c 'pass'" .. blah! I don't use NFS so I've no experience with the performance problems involved. Of course, Windows shares would have similar issues. Hmm - would this objection go away if a specific filename were mandated for these .pth files - e.g. pythonpath.pth? Tim Delaney From jepler at unpythonic.net Sun May 16 21:40:10 2004 From: jepler at unpythonic.net (Jeff Epler) Date: Sun May 16 22:17:32 2004 Subject: [Python-Dev] .pth files in current or script directory In-Reply-To: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A652@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A652@au3010avexu1.global.avaya.com> Message-ID: <20040517014009.GC23803@unpythonic.net> On Mon, May 17, 2004 at 10:11:23AM +1000, Delaney, Timothy C (Timothy) wrote: > I don't use NFS so I've no experience with the performance problems > involved. Of course, Windows shares would have similar issues. > > Hmm - would this objection go away if a specific filename were mandated > for these .pth files - e.g. pythonpath.pth? Yes, searching for a single named file is a much faster operation than listing a directory, and would probably be a fairly small impact on Python startup, much smaller than listing a directory with many files. It's one round-trip to the server vs many round trips. .. but that makes the feature somewhat different than .pth files in site-packages (any name vs a single name), which may be a bad thing too. Jeff From tdelaney at avaya.com Sun May 16 21:45:16 2004 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Sun May 16 22:17:43 2004 Subject: [Python-Dev] .pth files in current or script directory Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A6BB@au3010avexu1.global.avaya.com> Jeff Epler wrote: > On Mon, May 17, 2004 at 10:11:23AM +1000, Delaney, Timothy C > (Timothy) wrote: >> I don't use NFS so I've no experience with the performance problems >> involved. Of course, Windows shares would have similar issues. >> >> Hmm - would this objection go away if a specific filename were >> mandated for these .pth files - e.g. pythonpath.pth? > > Yes, searching for a single named file is a much faster operation than > listing a directory, and would probably be a fairly small impact on > Python startup, much smaller than listing a directory with many files. > It's one round-trip to the server vs many round trips. > > .. but that makes the feature somewhat different than .pth files in > site-packages (any name vs a single name), which may be a bad thing > too. Yep - I thought about this. OTOH, the use case for .pth files for site-packages is multiple libraries packaged separately. The use case for this feature is a single code base i.e. there would be no need for multiple .pth files since everything is under control of the developer. Please don't get the idea I'm all gung-ho about this - I'm just trying to see if there's enough reason to write a PEP for it (either to be accepted or rejected). Tim Delaney From greg at electricrain.com Sun May 16 15:22:56 2004 From: greg at electricrain.com (Gregory P. Smith) Date: Sun May 16 23:18:25 2004 Subject: [Python-Dev] Berkeley test errors in 2.3.4c1 Message-ID: <20040516192256.GA20831@zot.electricrain.com> On Sun, May 16, 2004 at 01:47:03AM -0400, Tim Peters wrote: > [Gregory P. Smith, on test_bsddb3 Win98SE failures in test01_basics] > > Yeah I'd chalk that one up to Win98. > > Me too . > > Back over on WinXP and *current* (HEAD) CVS, there's one failure, which I > believe is real: > > ERROR: test04_n_flag (bsddb.test.test_compat.CompatibilityTestCase) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "C:\Code\python\lib\bsddb\test\test_compat.py", line 86, in > test04_n_flag > f = hashopen(self.filename, 'n') > File "C:\Code\python\lib\bsddb\__init__.py", line 293, in hashopen > d.open(file, db.DB_HASH, flags, mode) > DBInvalidArgError: (22, 'Invalid argument -- DB_TRUNCATE illegal with > locking specified') > > > I suspect Sleepycat screwed us there, changing the rules in midstream. > Someone on c.l.py appeared to track down the same thing here, but in an app > instead of in our test suite: > > http://mail.python.org/pipermail/python-list/2004-May/220168.html > > The change log of Berkeley DB 4.2.52 says "9. Fix a bug to now > disallow DB_TRUNCATE on opens in locking environments, since we > cannot prevent race conditions ..." heh. nice. yeah i saw that bugreport for pybsddb but hadn't looked into it yet. Binary packages of python 2.3.4 should be compiled using BerkeleyDB 4.1 not 4.2 (as all previous python 2.3s shipped with 4.1). Anyone who compiles it on their own against 4.2 will have to live with this error in the old bsddb compatibility interface. Taking a closer look at this to see if there is a workaround is on my TODO list. -g From guido at python.org Sun May 16 23:23:39 2004 From: guido at python.org (Guido van Rossum) Date: Sun May 16 23:23:50 2004 Subject: [Python-Dev] .pth files in current or script directory In-Reply-To: Your message of "Mon, 17 May 2004 10:11:23 +1000." <338366A6D2E2CA4C9DAEAE652E12A1DE0179A652@au3010avexu1.global.avaya.com> References: <338366A6D2E2CA4C9DAEAE652E12A1DE0179A652@au3010avexu1.global.avaya.com> Message-ID: <200405170323.i4H3NdH02255@guido.python.org> > > Can't they manipulate sys.path *inside* the code in question? That's > > invariably the solution I end up using -- I never play with > > PYTHONPATH any more. > > I do this too, but I always feel that I shouldn't have to, especially > when I'm not using `sys` for anything else. It also messes up my desire > to keep all the imports together - the `sys` magic needs to go first. Yet, it beats depending on environment variables, so I always end up doing it this way. --Guido van Rossum (home page: http://www.python.org/~guido/) From eric at enthought.com Mon May 17 00:34:59 2004 From: eric at enthought.com (eric jones) Date: Mon May 17 00:35:09 2004 Subject: [Python-Dev] ANN: Job Openings at Enthought Message-ID: <40A840F3.2050608@enthought.com> Hello, This is a bit of a status report I guess for SciPy (www.scipy.org) and Enthought (www.enthought.com), but, really, it is a long winded job posting. Among other things, Enthought develops scientific software for clients. Python is central to to our strategy in this arena. We have long supported SciPy and believe strongly in Open Source Software. The purpose is to give people a feeling about what we do, the technical challenges we face, and the skills needed to meet them. SciPy ----- There has been a lot of work on SciPy lately by the Travis Oliphant, Travis Vaught, Pearu Peterson, Joe Cooper, and on the website by Jon-Eric Steinbomer and Janet Swisher. The site is now upgraded to Plone and we have a SciPy 0.3 package out. If you're interested in seeing the activity level at scipy.org, please visit. http://www.scipy.org/map?rmurl=http%3A//scipy.net/scipyusage/ It looks like we're averaging about a 100 downloads a day if you add up all source and binary distributions. When SciPy 0.1 was released a couple a years ago, I think we averaged about 10. Its extremely difficult to extrapolate a user base from such information, the obvious growth is good news. Other Stuff ----------- In addition to SciPy we have multiple projects going on right now that either already plug into or will be plugins to a Python-based scientific application framework called Envisage (sorta like an IDE for science apps) that we're working on. The idea is: (1) To lower the bar as far as work required to get a GUI put on the front end of a scientific algorithm. (2) Develop scientific plugins that can play well together so that, for instance, an electromagnetics simulator built by one person can be used in conjunction with an optimization plugin built by another person and results can be visualized with a 3D plugin. This is a hard, long path that requires much work and though. We have started the process and actually have built an app on a very early version of the Envisage framework. The app works great, but it's usage of the framework is a little mixed at this point. There were a lot of compromises we ended up making on the framework side in order to ship on time. Also, I am not sure whether "easy-to-program" and "flexible architecture for compatible plugins" are not mutually exclusive. I always have in my mind that I want a smart scientist to be able to build the GUI in a short amount of time after the algorithm (which is the "hard part") is finished. I still harbor this wish, but the lessons I've had over the last couple of years suggest that the GUI is actually the most expensive part of development for large apps. This is partially due to the fact that commercial apps are rarely built around "research" algorithms -- meaning that the algorithm code usually already exists in some form. Still, UIs take tons of time. Testing them is hard. Flexibility requires more complexity (factories, adapters, etc.) than seems necessary at first. Further, building *good* UI's is really hard. Scientist rarely have the patience or perspective for it -- and yet it is very often difficult to build a good UI for science apps without a scientist's understanding of the underlying problem. We have a number of tools along with SciPy that we're building that are pieces of this puzzle. 1. Traits -- They at the heart of everything we do. They provide some explicit typing, default GUI's, an observer event model for anything that uses them. They can also require some getting used to. 2. Kiva/Enable -- This is the generic drawing system for Python. 3. Chaco -- Interactive 2D Plotting 4. PyFace -- MVC layer on top of wxPython. Supports trees, menus, and a few other things right now. This will grow. 5. Envisage -- Plugin based GUI framework for scientific applications. Beyond the basic GUI that comes from Envisage, we already have a couple plugins for profiling code using hotshot and also searching for memory leaks. David Morrill wrote these and they are called gotcha and enroll. Martin Chilvers wrote a simple scintilla based plugin for editing scripts, and a PyCrust shell is available by default in the app (not a plugin at the moment). We would love to see a number of other plugins: a. Debugger. b. IPython-like PyCrust going. c. A more full featured script editing plugin. (there is a huge list of features here). d. Mayavi e. etc. These are all used actively in commercial products that we deliver. However, they are also in various stages of development and will continue to evolve and grow. Some are still pretty green. Portions of these are openly available now. All five of the listed tool sets will be open at some point this summer when they are cleaned up a bit. Jobs ---- All of this leads to the fact that we are hiring and looking for smart, pleasant, communicative people with integrity that are excited about building this sort of stuff. There is a lot of software architecture to be done that needs talented designers. There is UI design/development that needs scientists that care about UIs or UI developers/designers that are really quick at grasping scientific problems. We also need really strong scientists/scientific developers to do some of the backend coding for commercial products and also SciPy development. If you have a background in geophysics, electromagnetics (especially multi-pole methods) that is a big plus. For this, a PhD is helpful, but not required. Parallel computing skills wouldn't hurt. 3D visualization a la VTK/Python is a major need. So is knowledge about 2D rendering to help finish out Kiva and the Enable toolset. Strong Python skills, Python extension writing knowledge and strong C/C++ skills are a big benefit. Design/development with or of Java NetBeans or Eclipse-like architecture is great -- or any other solid GUI architecture. Dedication to writing clean/clear code meant for other humans to read is a requirement. We have 3 or 4 scientific/python positions to fill over the coming months, and we're looking for candidates that have the best mix of the above skills. You'll join our existing team of scientist/engineers, computer scientists, technical writers, and an HCI specialist. We like this mix and feel it is the best way to build this sort of software. If you are interested in working at Enthought, please send us your resume at jobs@enthought.com. If not, please send this posting to the smartest people you know that fit some part of the above description (Python experience not explicitly required). Salaries are competitive. Candidates must be willing to relocate to Austin, Tx. thanks, eric jones PS: There are additional positions listed at http://www.enthought.com/careers.htm for those interested in business application development (not necessarily with Python). From doko at cs.tu-berlin.de Mon May 17 02:03:19 2004 From: doko at cs.tu-berlin.de (Matthias Klose) Date: Mon May 17 02:03:29 2004 Subject: [Python-Dev] windist.exe status and license In-Reply-To: <1084541142.696332.6706.nullmailer@calvin.fayauffre.org> References: <1084541142.696332.6706.nullmailer@calvin.fayauffre.org> Message-ID: <16552.21927.581799.337029@gargle.gargle.HOWL> [CC to python-dev] The windist.exe version shipped with current python releasese is outdated and does not have any license information about windist included. Alexandre Fayolle writes: > Package: python2.3-dev > Version: 2.3.3-7 > Severity: normal > > Hi, > > I noticed the file /usr/lib/python2.3/distutils/command/wininst.exe is > missing from my currently installed version of the python2.3-dev > package. I'm pretty sure I used to be able to create windows installers > for the modules I develop from my Debian machine. > > Is this removal caused by an omission, or is there some licensing issue ? > I could not find anything on that topic in the changelog, but maybe I > didn't look hard enough. I cannot find anything about the licensing. The executable only shows $Info: This file is packed with the UPX executable packer http://upx.tsx.org $ $Id: UPX 1.01 Copyright (C) 1996-2000 the UPX Team. All Rights Reserved. $ However I found http://upx.sourceforge.net/. - Maybe the binary can be updated to the SF version? - Add the upx license to the Lib/distutils/command directory? [Debian only: The UPX license itself is discussed at http://lists.debian.org/debian-legal/2001/01/msg00025.html upx-nrv is currently in section non-free/utils, so it's better not to distribute it in the Debian package. ] From skip at pobox.com Mon May 17 09:21:32 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon May 17 09:21:21 2004 Subject: [Python-Dev] Python 2.3.4, release candidate 1 - BUG FIX In-Reply-To: <20040517113327.GA25004@unpythonic.net> References: <67B3A7DA6591BE439001F2736233351202B479BA@xch-nw-28.nw.nos.boeing.com> <20040517113327.GA25004@unpythonic.net> Message-ID: <16552.48220.302165.584705@montanaro.dyndns.org> (Rerouting to python-dev...) Jeff> When a tar, gzipped tar, or bzip2'd tar contains a .spec file, you're Jeff> supposed to be able to build it into a binary RPM with Jeff> rpmbuild -ta Jeff> Unfortunately, with the spec file in this tarball, I get an error: Jeff> $ rpmbuild -ta Python-2.3.4c1.tar.bz2 Jeff> error: %changelog not in descending chronological order I checked in the "obvious correct fix" to the spec file on CVS HEAD. I can't test it though, not having convenient access to a RedHat-ish system these days. I'll leave it for Anthony or one of the other 2.3.4 release folks to backport it for the 2.3.4 release since that branch is closed. (The first date should have a year of 2004.) Skip From mal at egenix.com Mon May 17 09:42:10 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon May 17 09:42:16 2004 Subject: [Python-Dev] Relative vs. absolute imports Message-ID: <40A8C132.3090705@egenix.com> I have a few questions about the PEP 328 which I'd like discussed: * Why do we absolutely need to change the current scheme of 'local, then absolute' for name resolution ? In the past I have often had the case where I wanted to move Python software written as top-level modules into their own package. The current logic makes this painless; the proposed move to 'absolute only' introduces serious editing which is bound to be error-prone. I don't object absolute imports at all, but I like the possibilitiy to package'ify existing Python module/module sets by simply copying them into a newly created package directory. Note that Python's own name lookup behaviour is very similar to the existing import scheme (nested scopes left aside). * How will the proposed 'absolute only' strategy affect the __import__() API ? This is of special concern since quite a few applications out there rely on __import__() having the existing semantics and a change would cause these applications to fail. One such instance is the encodings package search function, which relies on first looking in the package for a suitable codec and then scans for a top-level installation. * Will the DeprecationWarning only be raised for uses of 'import xyz' and 'from xyz import a' or also for __import__() ? * How do these semantics changes affect importers that hook into __import__ ? (imputil.py, ihooks.py, etc.) * What's the affect on zip imports or tools like py2exe, McMillan's installer, etc. ? BTW, is any of this already implemented to play with ? Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 20 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pje at telecommunity.com Mon May 17 10:20:37 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon May 17 10:19:24 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <40A8C132.3090705@egenix.com> Message-ID: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> At 03:42 PM 5/17/04 +0200, M.-A. Lemburg wrote: >I have a few questions about the PEP 328 which I'd like discussed: > >* Why do we absolutely need to change the current scheme > of 'local, then absolute' for name resolution ? Because there's no way to override the current scheme without calling __import__ directly. >* How will the proposed 'absolute only' strategy affect > the __import__() API ? It doesn't change the API, only what arguments it gets called with. Passing a different globals dictionary is all that's required to effect an absolute import. >* Will the DeprecationWarning only be raised for uses of > 'import xyz' and 'from xyz import a' or also for __import__() ? It should not be for __import__(). >* How do these semantics changes affect importers that hook > into __import__ ? (imputil.py, ihooks.py, etc.) > >* What's the affect on zip imports or tools like py2exe, > McMillan's installer, etc. ? Again, since the __import__ API isn't changing, it shouldn't affect any of this. From bh at intevation.de Mon May 17 10:26:06 2004 From: bh at intevation.de (Bernhard Herzog) Date: Mon May 17 10:26:13 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> (Phillip J. Eby's message of "Mon, 17 May 2004 10:20:37 -0400") References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> Message-ID: "Phillip J. Eby" writes: > At 03:42 PM 5/17/04 +0200, M.-A. Lemburg wrote: >>I have a few questions about the PEP 328 which I'd like discussed: >> >>* Why do we absolutely need to change the current scheme >> of 'local, then absolute' for name resolution ? > > Because there's no way to override the current scheme without calling > __import__ directly. Why does it have to be fixed in an incompatible way? Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://sketch.sourceforge.net/ Thuban http://thuban.intevation.org/ From guido at python.org Mon May 17 10:52:30 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 17 10:53:09 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: Your message of "Mon, 17 May 2004 16:26:06 +0200." References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> Message-ID: <200405171452.i4HEqVp03065@guido.python.org> > Why does it have to be fixed in an incompatible way? Because in the current situation the meaning of an import is ambiguous. --Guido van Rossum (home page: http://www.python.org/~guido/) From mal at egenix.com Mon May 17 10:55:30 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon May 17 10:55:58 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> Message-ID: <40A8D262.7060505@egenix.com> Phillip J. Eby wrote: > At 03:42 PM 5/17/04 +0200, M.-A. Lemburg wrote: > >> I have a few questions about the PEP 328 which I'd like discussed: >> >> * Why do we absolutely need to change the current scheme >> of 'local, then absolute' for name resolution ? > > Because there's no way to override the current scheme without calling > __import__ directly. Are you saying that you are not going to change the default __import__() implementation, only the way it is called ? (I wonder how you'll enforce the 'absolute only' strategy then) >> * How will the proposed 'absolute only' strategy affect >> the __import__() API ? > > It doesn't change the API, only what arguments it gets called with. > Passing a different globals dictionary is all that's required to effect > an absolute import. > >> * Will the DeprecationWarning only be raised for uses of >> 'import xyz' and 'from xyz import a' or also for __import__() ? > > It should not be for __import__(). > > >> * How do these semantics changes affect importers that hook >> into __import__ ? (imputil.py, ihooks.py, etc.) >> >> * What's the affect on zip imports or tools like py2exe, >> McMillan's installer, etc. ? > > > Again, since the __import__ API isn't changing, it shouldn't affect any > of this. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 20 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pje at telecommunity.com Mon May 17 11:04:15 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon May 17 11:03:04 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <40A8D262.7060505@egenix.com> References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040517110205.029c9c20@mail.telecommunity.com> At 04:55 PM 5/17/04 +0200, M.-A. Lemburg wrote: >Phillip J. Eby wrote: >>At 03:42 PM 5/17/04 +0200, M.-A. Lemburg wrote: >> >>>I have a few questions about the PEP 328 which I'd like discussed: >>> >>>* Why do we absolutely need to change the current scheme >>> of 'local, then absolute' for name resolution ? >>Because there's no way to override the current scheme without calling >>__import__ directly. > >Are you saying that you are not going to change the default __import__() >implementation, only the way it is called ? (I wonder how you'll >enforce the 'absolute only' strategy then) Huh? You lost me. What 'absolute only'? There's "from .x import y" still, and that's relative. As far as I know, nothing about __import__ is changing, only the spelling of import statements. From edcjones at erols.com Mon May 17 11:00:11 2004 From: edcjones at erols.com (Edward C. Jones) Date: Mon May 17 11:06:58 2004 Subject: [Python-Dev] __str__ does not default to __repr__ if a base class has __str__ Message-ID: <40A8D37B.7030807@erols.com> # In the code below, "a.__str__()" does not default to "a.__repr__() as # might be expected. It finds the "__str__" in the base class X. The # Reference Manual, section 3.3.1, says: # If a class defines __repr__() but not __str__(), then __repr__() is # also used when an 'informal' string representation of instances of # that class is required. # If the reader is in lawyer mode, the documentation is correct. # Is the current behavior what was intended? What should the behavior be? class X(object): def __str__(self): return 'str for X' def __repr__(self): return 'repr for X' class A(X): def __repr__(self): return 'repr for A' # __str__ = __repr__ x = X() a = A() print x.__str__() print x.__repr__() print a.__str__() print a.__repr__() From pedronis at bluewin.ch Mon May 17 11:14:04 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon May 17 11:14:14 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <40A8D262.7060505@egenix.com> References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> Message-ID: <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> At 16:55 17.05.2004 +0200, M.-A. Lemburg wrote: >Phillip J. Eby wrote: >>At 03:42 PM 5/17/04 +0200, M.-A. Lemburg wrote: >> >>>I have a few questions about the PEP 328 which I'd like discussed: >>> >>>* Why do we absolutely need to change the current scheme >>> of 'local, then absolute' for name resolution ? >>Because there's no way to override the current scheme without calling >>__import__ directly. > >Are you saying that you are not going to change the default __import__() >implementation, only the way it is called ? (I wonder how you'll >enforce the 'absolute only' strategy then) if you pass globals = {} you get absolute imports from __import__, It is probably true that in the long run, given the new semantics, moving responsabilities away from/changing __import__ contract could make sense. From mal at egenix.com Mon May 17 11:15:24 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon May 17 11:15:30 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <5.1.1.6.0.20040517110205.029c9c20@mail.telecommunity.com> References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517110205.029c9c20@mail.telecommunity.com> Message-ID: <40A8D70C.5060309@egenix.com> Phillip J. Eby wrote: >>>> I have a few questions about the PEP 328 which I'd like discussed: >>>> >>>> * Why do we absolutely need to change the current scheme >>>> of 'local, then absolute' for name resolution ? >>> >>> Because there's no way to override the current scheme without calling >>> __import__ directly. >> >> >> Are you saying that you are not going to change the default __import__() >> implementation, only the way it is called ? (I wonder how you'll >> enforce the 'absolute only' strategy then) > > Huh? You lost me. What 'absolute only'? There's "from .x import y" > still, and that's relative. > > As far as I know, nothing about __import__ is changing, only the > spelling of import statements. If "import os" is supposed to map to an absolute import you will have to change the semantics of __import__ because simply passing "os" to that API is going to implement the standard 'local, then absolute' import scheme. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 20 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From pje at telecommunity.com Mon May 17 11:38:40 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon May 17 11:37:31 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <40A8D70C.5060309@egenix.com> References: <5.1.1.6.0.20040517110205.029c9c20@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517110205.029c9c20@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040517113704.02d329b0@mail.telecommunity.com> At 05:15 PM 5/17/04 +0200, M.-A. Lemburg wrote: >Phillip J. Eby wrote: >>>>>I have a few questions about the PEP 328 which I'd like discussed: >>>>> >>>>>* Why do we absolutely need to change the current scheme >>>>> of 'local, then absolute' for name resolution ? >>>> >>>>Because there's no way to override the current scheme without calling >>>>__import__ directly. >>> >>> >>>Are you saying that you are not going to change the default __import__() >>>implementation, only the way it is called ? (I wonder how you'll >>>enforce the 'absolute only' strategy then) >>Huh? You lost me. What 'absolute only'? There's "from .x import y" >>still, and that's relative. >>As far as I know, nothing about __import__ is changing, only the spelling >>of import statements. > >If "import os" is supposed to map to an absolute import >you will have to change the semantics of __import__ because >simply passing "os" to that API is going to implement the >standard 'local, then absolute' import scheme. As I believe I mentioned before, the relevant import opcodes will simply pass a different set of parameters to __import__. Supplying a different "globals" dictionary suffices to implement an absolute import. From mal at egenix.com Mon May 17 11:37:31 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon May 17 11:37:41 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> Message-ID: <40A8DC3B.1070107@egenix.com> Samuele Pedroni wrote: > At 16:55 17.05.2004 +0200, M.-A. Lemburg wrote: > >> Phillip J. Eby wrote: >> >>> At 03:42 PM 5/17/04 +0200, M.-A. Lemburg wrote: >>> >>>> I have a few questions about the PEP 328 which I'd like discussed: >>>> >>>> * Why do we absolutely need to change the current scheme >>>> of 'local, then absolute' for name resolution ? >>> >>> Because there's no way to override the current scheme without calling >>> __import__ directly. >> >> >> Are you saying that you are not going to change the default __import__() >> implementation, only the way it is called ? (I wonder how you'll >> enforce the 'absolute only' strategy then) > > if you pass globals = {} you get absolute imports from __import__, Ahh, a hidden feature :-) I didn't know that one yet. > It is probably true that in the long run, given the new semantics, > moving responsabilities away from/changing __import__ contract > could make sense. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 17 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 20 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From bh at intevation.de Mon May 17 11:39:42 2004 From: bh at intevation.de (Bernhard Herzog) Date: Mon May 17 11:39:49 2004 Subject: [Python-Dev] Relative vs. absolute imports References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <200405171452.i4HEqVp03065@guido.python.org> Message-ID: Guido van Rossum writes: >> Why does it have to be fixed in an incompatible way? > > Because in the current situation the meaning of an import is > ambiguous. What exactly is ambiguous about it? There are clear rules how the lookup is performed and determining whether the import refers to another module in the same package or a global module is just a simple directory listing in most cases. The only real problem with the current scheme is that sibling modules can shadow global ones. While that should be fixed it hardly justifies an incompatible change, as it doesn't happen very often, and it would be easy to fix without breaking compatibility. Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://sketch.sourceforge.net/ Thuban http://thuban.intevation.org/ From pje at telecommunity.com Mon May 17 11:50:22 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon May 17 11:49:13 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <40A8DC3B.1070107@egenix.com> References: <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> Message-ID: <5.1.1.6.0.20040517114707.02716660@mail.telecommunity.com> At 05:37 PM 5/17/04 +0200, M.-A. Lemburg wrote: >Samuele Pedroni wrote: >>At 16:55 17.05.2004 +0200, M.-A. Lemburg wrote: >>>Are you saying that you are not going to change the default __import__() >>>implementation, only the way it is called ? (I wonder how you'll >>>enforce the 'absolute only' strategy then) >>if you pass globals = {} you get absolute imports from __import__, > >Ahh, a hidden feature :-) I didn't know that one yet. More of an emergent property. Keep in mind that the current relative mechanism is always relative *to* something, and that "something" is defined by the globals['__name__'], if any. If there is no '__name__', then the import has to be absolute. For example, you would expect this: exec "import foo" in {},{} to be an absolute import, would you not? From tjreedy at udel.edu Mon May 17 11:56:08 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Mon May 17 11:56:16 2004 Subject: [Python-Dev] Re: __str__ does not default to __repr__ if a base classhas __str__ References: <40A8D37B.7030807@erols.com> Message-ID: "Edward C. Jones" wrote in message news:40A8D37B.7030807@erols.com... > # In the code below, "a.__str__()" does not default to "a.__repr__() as > # might be expected. It finds the "__str__" in the base class X. The > # Reference Manual, section 3.3.1, says: > # If a class defines __repr__() but not __str__(), then __repr__() is > # also used when an 'informal' string representation of instances of > # that class is required. > # If the reader is in lawyer mode, the documentation is correct. > # Is the current behavior what was intended? What should the behavior be? > > class X(object): > def __str__(self): > return 'str for X' > > def __repr__(self): > return 'repr for X' > > class A(X): > def __repr__(self): > return 'repr for A' > > # __str__ = __repr__ > > x = X() > a = A() > print x.__str__() > print x.__repr__() > print a.__str__() > print a.__repr__() > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > From pedronis at bluewin.ch Mon May 17 12:04:12 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Mon May 17 12:04:16 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <5.1.1.6.0.20040517114707.02716660@mail.telecommunity.com> References: <40A8DC3B.1070107@egenix.com> <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> Message-ID: <5.2.1.1.0.20040517180028.02ff4198@pop.bluewin.ch> At 11:50 17.05.2004 -0400, Phillip J. Eby wrote: >At 05:37 PM 5/17/04 +0200, M.-A. Lemburg wrote: >>Samuele Pedroni wrote: >>>At 16:55 17.05.2004 +0200, M.-A. Lemburg wrote: >>>>Are you saying that you are not going to change the default __import__() >>>>implementation, only the way it is called ? (I wonder how you'll >>>>enforce the 'absolute only' strategy then) >>>if you pass globals = {} you get absolute imports from __import__, >> >>Ahh, a hidden feature :-) I didn't know that one yet. > >More of an emergent property. Keep in mind that the current relative >mechanism is always relative *to* something, and that "something" is >defined by the globals['__name__'], if any. If there is no '__name__', >then the import has to be absolute. For example, you would expect this: > > exec "import foo" in {},{} > >to be an absolute import, would you not? yes, and __import__('foo') triggers also an absolute import because the default for globals is a conceptually empty namespace, not the caller globals. Respectively this can trigger a relative import: __import__('foo',globals()) and the relevant bit is indeed __name__ . From tjreedy at udel.edu Mon May 17 12:15:12 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Mon May 17 12:15:27 2004 Subject: [Python-Dev] Re: __str__ does not default to __repr__ if a base classhas __str__ References: <40A8D37B.7030807@erols.com> Message-ID: "Edward C. Jones" wrote in message news:40A8D37B.7030807@erols.com... > # In the code below, "a.__str__()" does not default to "a.__repr__() as > # might be expected. It finds the "__str__" in the base class X. The > # Reference Manual, section 3.3.1, says: > # If a class defines __repr__() but not __str__(), then __repr__() is > # also used when an 'informal' string representation of instances of > # that class is required. > # If the reader is in lawyer mode, the documentation is correct. I don't think one has to be in lawyer mode to think that 'attribute x is defined by class C' == 'C.x returns an object without exception', whether the production is by direct lookup, inheritance, or calculation. It is possible that 'defines' could be defined better. The current behavior seems to be try: f = i.__str__ except AttributeError: try: f = i.__repr__ except AttributeError f = ' form > s = f(i) Your implied and underdefined alternative would require special casing of __str__ lookup to not use inheritance. > # Is the current behavior what was intended? What should the behavior be? Since the current behavior is hard to see as a bug, I think it should stay as is, if it has been consistently so forever, even if something else might have been once preferred. Terry J. Reedy From alloydflanagan at comcast.net Mon May 17 15:16:48 2004 From: alloydflanagan at comcast.net (alloydflanagan@comcast.net) Date: Mon May 17 15:16:55 2004 Subject: [Python-Dev] Python-Dev] Re: __str__ does not default to __repr__ if a base classhas __str__ Message-ID: <051720041916.3893.40A90FA00007E8A000000F352200751150020E090E020E04000B970104040E@comcast.net> FWIW, I think the current behavior is correct. If a __str__() method exists, it exists; we shouldn't treat an inherited method differently from one defined directly in the class. So I would be very surprised if the inherited __str__() wasn't used. From anon at spam-trap.richardcooper.net Sun May 16 07:39:08 2004 From: anon at spam-trap.richardcooper.net (Richard Cooper) Date: Mon May 17 15:37:29 2004 Subject: [Python-Dev] Re: .pth files in current or script directory In-Reply-To: <0AA408F4E902C84C8F6BE093CCE9D7A88C2468@server1.artsalliance.co.uk> Message-ID: Tim Delaney wrote: > A discussion (last week?) on c.l.py prompted a thought about .pth > files. > Feel free to shoot it down ;) > > The discussion in question concerned someone using a version control > system, and needing to change PYTHONPATH depending on which branch they > were in. That was me, I think. > My immediately two thoughts were: > > 1. Use a build system (e.g. make) to set PYTHONPATH appropriately. This > is they type of thing we do. I'm on windows and am quite attached to coding and running inside SciTE. I think I could knock up a script to do the right thing and configure SciTE to run it rather than python. However since I sometimes run from the command line and my colleagues use different editors that's a whole other can of worms. > 2. Use a .pth file in the directory the code is run from (or possibly, > the same directory as the script). Guido van Rossum wrote: > Can't they manipulate sys.path *inside* the code in question? That's > invariably the solution I end up using -- I never play with > PYTHONPATH any more. I suppose I could, but adding a bit of boilerplate to the top of every source file to support something that should be completely orthogonal to the the language (using source control branches) seems a bit hacky- workaround-tastic even compared to my current solution (a post-it saying 'check PYTHONPATH' in big letters stuck to my monitor) Anyway, enough discussion of my situation, which is probably more suited to python-list. My real question is: Is there any desire to address this problem in some future python? When I see 2 python-dev'ers say "PYTHONPATH isn't up to it, here's what I do" I smell something that should be fixed once rather that worked around everywhere. My initial idea was something along the lines of: When a module is imported, its import search path includes everything it does now PLUS the directory that is the top of the module's enclosing package structure. That way you wouldn't have to add entries to PYTHONPATH just so that modules could import (absolutely) from their own package. What do you think? Possible? Sensible? Any other ideas? Rich From edcjones at erols.com Mon May 17 10:58:54 2004 From: edcjones at erols.com (Edward C. Jones) Date: Mon May 17 18:07:38 2004 Subject: [Python-Dev] __str__ does not default to __repr__ if a base class has __str__ Message-ID: <40A8D32E.6050102@erols.com> # In the code below, "a.__str__()" does not default to "a.__repr__() as # might be expected. It finds the "__str__" in the base class X. The # Reference Manual, section 3.3.1, says: # If a class defines __repr__() but not __str__(), then __repr__() is # also used when an 'informal' string representation of instances of # that class is required. # If the reader is in lawyer mode, the documentation is correct. # Is the current behavior what was intended? What should the behavior be? class X(object): def __str__(self): return 'str for X' def __repr__(self): return 'repr for X' class A(X): def __repr__(self): return 'repr for A' # __str__ = __repr__ x = X() a = A() print x.__str__() print x.__repr__() print a.__str__() print a.__repr__() From edcjones at erols.com Mon May 17 10:57:57 2004 From: edcjones at erols.com (Edward C. Jones) Date: Mon May 17 18:14:44 2004 Subject: [Python-Dev] __str__ does not default to __repr__ if a base class has __str__ Message-ID: <40A8D2F5.6000302@erols.com> # In the code below, "a.__str__()" does not default to "a.__repr__() as # might be expected. It finds the "__str__" in the base class X. The # Reference Manual, section 3.3.1, says: # If a class defines __repr__() but not __str__(), then __repr__() is # also used when an 'informal' string representation of instances of # that class is required. # If the reader is in lawyer mode, the documentation is correct. # Is the current behavior what was intended? What should the behavior be? class X(object): def __str__(self): return 'str for X' def __repr__(self): return 'repr for X' class A(X): def __repr__(self): return 'repr for A' # __str__ = __repr__ x = X() a = A() print x.__str__() print x.__repr__() print a.__str__() print a.__repr__() From mal at egenix.com Mon May 17 18:36:29 2004 From: mal at egenix.com (M.-A. Lemburg) Date: Mon May 17 18:36:33 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <5.1.1.6.0.20040517114707.02716660@mail.telecommunity.com> References: <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517114707.02716660@mail.telecommunity.com> Message-ID: <40A93E6D.8010605@egenix.com> Phillip J. Eby wrote: > At 05:37 PM 5/17/04 +0200, M.-A. Lemburg wrote: > >> Samuele Pedroni wrote: >> >>> At 16:55 17.05.2004 +0200, M.-A. Lemburg wrote: >>> >>>> Are you saying that you are not going to change the default >>>> __import__() >>>> implementation, only the way it is called ? (I wonder how you'll >>>> enforce the 'absolute only' strategy then) >>> >>> if you pass globals = {} you get absolute imports from __import__, >> >> >> Ahh, a hidden feature :-) I didn't know that one yet. > > More of an emergent property. Keep in mind that the current relative > mechanism is always relative *to* something, and that "something" is > defined by the globals['__name__'], if any. If there is no '__name__', > then the import has to be absolute. Right. Point taken on that account. Perhaps a note should go into the PEP that the internal mechanism is not going to change and that instead only the syntax will get enhanced. I still don't believe that we need to break code in order to introduce relative imports, though, certainly not before Py3k. I also think that prepending "from ." to all package local imports in order to overcome this looks like silly (the dot is hardly visible in an editor), but that may be just me. BTW, what are you going to do about "from . import *" and "from .. import *" ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, May 18 2004) >>> Python/Zope Consulting and Support ... http://www.egenix.com/ >>> mxODBC.Zope.Database.Adapter ... http://zope.egenix.com/ >>> mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/ ________________________________________________________________________ EuroPython 2004, G?teborg, Sweden 19 days left ::: Try mxODBC.Zope.DA for Windows,Linux,Solaris,FreeBSD for free ! :::: From bac at OCF.Berkeley.EDU Mon May 17 19:09:22 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon May 17 19:09:40 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 Message-ID: <40A94622.6040606@ocf.berkeley.edu> Back in February, I tried to fix an issue of threads executed using 'threading' that did not shut themselves down before the interpreter started tearing down globals, the thread triggering an exception, and then a new exception being raised while trying to report the exception raised in the thread by accessing globals (see http://mail.python.org/pipermail/python-dev/2004-February/042457.html for the previous discussion). An initial fix was devised, but since there was no way to trigger the issue every time I was not sure if the fix was thorough enough. Well, someone finally has an app that causes the error consistently and the previous fix did not go far enough. But with the help of the app (mnet, specifically) I was finally able to get rid of all global accesses while reporting exceptions so that the original exception caught in the thread can get printed. The patch is at http://python.org/sf/954922 while the bug report is at http://python.org/sf/754449 . Trouble is that I had to basically fake traceback.format_exc() in the patch to a reasonable level since the 'traceback' module bounces around between its various functions too much to just store a local copy of the function and have it still work after globals are set to None. So I need someone to just look over the code to make sure that the code is not doing something stupid when this code is called (default way is still used when 'sys' is defined and thus the interpreter is not being torn down). And the reason I bring this up now is just in case there is any desire to put this in 2.3.4 . I obviously know that I missed rc1, but I didn't get the details of triggering the problem soon enough and people running 2.3.x are the ones bringing this up since this apparently did not happen under 2.2 for some reason. I doubt Anthony wants to risk mucking with 'threading' this late (although the patch is fairly self-contained but would require applying the previous patch to attempt to fix this as well), but I thought I would at least put it out there. -Brett From barry at python.org Mon May 17 21:36:31 2004 From: barry at python.org (Barry Warsaw) Date: Mon May 17 21:36:48 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <40A93E6D.8010605@egenix.com> References: <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517114707.02716660@mail.telecommunity.com> <40A93E6D.8010605@egenix.com> Message-ID: <1084844191.19299.312.camel@anthem.wooz.org> On Mon, 2004-05-17 at 18:36, M.-A. Lemburg wrote: > I still don't believe that we need to break code in order > to introduce relative imports, though, certainly not > before Py3k. One to to "fix" this in a backward compatible way is to instead introduce a spelling for "really, import this absolutely", e.g. from __python__ import re or import __python__.re and leave the current import semantics alone, for Python 2.x. I know Guido's not in favor of this approach though. -Barry From greg at cosc.canterbury.ac.nz Mon May 17 22:28:47 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon May 17 22:29:13 2004 Subject: [Python-Dev] Re: .pth files in current or script directory In-Reply-To: Message-ID: <200405180228.i4I2Slel012600@cosc353.cosc.canterbury.ac.nz> Richard Cooper : > When a module is imported, its import search path includes everything it > does now PLUS the directory that is the top of the module's enclosing > package structure. I don't see how that changes anything, since in order to get imported in the first place, the module's top-level package has to already be on the module search path somehow. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Mon May 17 22:33:42 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon May 17 22:33:57 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <40A94622.6040606@ocf.berkeley.edu> Message-ID: <200405180233.i4I2XgqL012607@cosc353.cosc.canterbury.ac.nz> "Brett C." : > Back in February, I tried to fix an issue of threads executed using > 'threading' that did not shut themselves down before the interpreter > started tearing down globals, the thread triggering an exception, > and then a new exception being raised while trying to report the > exception raised in the thread by accessing globals Seems to me the way to fix this sort of thing properly is to get rid of the need for explicitly clearing globals. Now that we have a garbage collector, is it really necessary any more? What would happen if we simply set sys.modules to None and then invoked the collector? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From kbk at shore.net Mon May 17 23:03:39 2004 From: kbk at shore.net (Kurt B. Kaiser) Date: Mon May 17 23:03:46 2004 Subject: [Python-Dev] Weekly Python Bug/Patch Summary Message-ID: <200405180303.i4I33doq020408@hydra.localdomain> Patch / Bug Summary ___________________ Patches : 275 open ( +5) / 2380 closed (+14) / 2655 total (+19) Bugs : 753 open ( -1) / 4047 closed (+54) / 4800 total (+53) RFE : 133 open ( +0) / 126 closed ( +2) / 259 total ( +2) New / Reopened Patches ______________________ CodeContext - an extension to show you where you are (2004-04-16) http://python.org/sf/936169 reopened by noamr Explain 'in' keyword as it is first used (2004-04-22) CLOSED http://python.org/sf/940026 opened by Gerrit Holl Replace if/elif chain with dispatch pattern in sre_compile (2004-04-23) CLOSED http://python.org/sf/941071 opened by Raymond Hettinger Allow any encodings other than latin-1 in interactive interp (2004-04-24) http://python.org/sf/941229 opened by Hye-Shik Chang Minimal fix for bug 940578 (glob.glob on broken symlinks) (2004-04-24) http://python.org/sf/941486 opened by Cherniavsky Beni PEP309 Partial implementation (2004-04-26) http://python.org/sf/941881 opened by Hye-Shik Chang CodeContext - an extension to show you where you are (2004-04-16) http://python.org/sf/936169 reopened by noamr Convert glob.glob to generator-based DFS (2004-04-27) http://python.org/sf/943206 opened by Cherniavsky Beni A simple 3-4% speed-up for PCs (2004-04-28) http://python.org/sf/943898 opened by Armin Rigo Add maketrans to string object (2004-04-28) http://python.org/sf/943953 opened by Gyro Funch urllib2 authentication mishandles empty pass bugfix 944082 (2004-04-29) CLOSED http://python.org/sf/944110 opened by sc0rp Bugfix for dircheck() in Objects/fileobject.c (2004-04-30) http://python.org/sf/944928 opened by Ulrich Berning nonblocking i/o with ssl socket not working at all (2004-05-01) http://python.org/sf/945642 opened by Tino Lange askstring => grab fail (2004-05-01) http://python.org/sf/946153 opened by wes Non-blocking Socket Server (2004-05-01) http://python.org/sf/946207 opened by Jesper ´JJ´ Jurcenoks do not add directory of sys.argv[0] into sys.path (2004-05-02) http://python.org/sf/946373 opened by wrobell AF_PACKET Hardware address support in socket module (2004-05-03) http://python.org/sf/947352 opened by Jason Andryuk Fix for #947380 - sys.path is wrong on windows sometimes (2004-05-04) http://python.org/sf/947386 opened by Ori Berger Apply freelist technique to empty lists (2004-05-04) CLOSED http://python.org/sf/947476 opened by Raymond Hettinger Linux signals during threads (2004-05-05) http://python.org/sf/948614 opened by Andrew Langmead synchronous signals blocked in pthreads (2004-05-06) http://python.org/sf/949332 opened by Andrew Langmead fix bug in StringIO.truncate - length not changed (2004-05-11) http://python.org/sf/951915 opened by David Fraser fix typos (2004-05-12) CLOSED http://python.org/sf/952047 opened by George Yoshida Fix os.stat handling of UNC roots (2004-05-14) http://python.org/sf/954115 opened by Greg Chapman Replacement for webbrowser.py which has problems. (2004-05-15) http://python.org/sf/954628 opened by Michel Van den Bergh Patches Closed ______________ CodeContext - an extension to show you where you are (2004-04-16) http://python.org/sf/936169 closed by kbk Py_INCREF/DECREF available as macros only (2004-04-20) http://python.org/sf/938302 closed by theller CodeContext - an extension to show you where you are (2004-04-16) http://python.org/sf/936169 closed by kbk itertools roundrobin() (2003-06-17) http://python.org/sf/756253 closed by rhettinger ConfigParser: fixes mixed-case interpolations (bug 857881) (2003-12-24) http://python.org/sf/865455 closed by fdrake Replace if/elif chain with dispatch pattern in sre_compile (2004-04-23) http://python.org/sf/941071 closed by rhettinger Explain 'in' keyword as it is first used (2004-04-22) http://python.org/sf/940026 closed by loewis Apply freelist technique to empty lists (2004-05-04) http://python.org/sf/947476 closed by rhettinger urllib2 CacheFTPHandler doesn't work on multiple dirs (2003-12-01) http://python.org/sf/851736 closed by mhammond urllib2 authentication mishandles empty pass bugfix 944082 (2004-04-29) http://python.org/sf/944110 closed by loewis fix typos (2004-05-11) http://python.org/sf/952047 closed by fdrake Error caused by patch #852334 (2004-04-10) http://python.org/sf/932796 closed by fdrake API Ref: PyErr_SetInterrupt not Obsolete (2004-03-19) http://python.org/sf/919299 closed by fdrake ConfigParser should raise an error for duplicate options (2003-12-04) http://python.org/sf/854484 closed by fdrake Fix docstring for posixpath.getctime (2004-01-04) http://python.org/sf/870287 closed by fdrake Footnote on bug in Mailbox with Windows text-mode files (2003-11-09) http://python.org/sf/838910 closed by bwarsaw New / Reopened Bugs ___________________ message.as_string hoses headers (2004-04-20) CLOSED http://python.org/sf/938890 opened by Carl Karsten cPickle.Pickler: in "list" mode, no way to set protocol (2004-04-22) http://python.org/sf/939395 opened by John Belmonte Building python 2.3.3 RPM on Mandrake 9.2. fails (2004-04-22) http://python.org/sf/939699 opened by fabien wahl Logging failes for rotating file log (during rollOver) (2004-04-22) CLOSED http://python.org/sf/939836 opened by smeyers2002 xml parser bug (2004-04-22) CLOSED http://python.org/sf/940263 opened by David Tefft pydoc.Helper.help() ignores input/output init parameters (2004-04-22) http://python.org/sf/940286 opened by Maxim Krikun string.lstrip problem with '_' string (2004-04-22) CLOSED http://python.org/sf/940316 opened by Arnold Python crashes with bus error on sparc64. (2004-04-22) CLOSED http://python.org/sf/940354 opened by Generic Player misbehaviour with map / lambda / isdigit() (2004-04-23) CLOSED http://python.org/sf/940522 opened by dreary glob.glob inconsistent about broken symlinks (2004-04-23) http://python.org/sf/940578 opened by Cherniavsky Beni section 5.10: 'not' returns boolean, not int (2004-04-23) CLOSED http://python.org/sf/940579 opened by Jon Ashley List with Canvas.create_line Option arrow=LAST Broke (2004-04-24) http://python.org/sf/941262 opened by Brian Brooks AIX shared library fix (2004-04-24) http://python.org/sf/941346 opened by Dick Dunbar Python crash on __init__/__getattr__/__setattr__ interaction (2004-04-26) http://python.org/sf/942706 opened by has Weakness in tuple hash (2004-04-27) http://python.org/sf/942952 opened by Steve Tregidgo urllib2 authentication mishandles empty password (2004-04-29) CLOSED http://python.org/sf/944082 opened by sc0rp Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty (2004-04-28) http://python.org/sf/944119 opened by Steve Levis Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty (2004-04-28) http://python.org/sf/944119 reopened by levis501 No examples or usage docs for urllib2 (2004-04-29) http://python.org/sf/944394 opened by Chris Withers urllib2 doesn't handle username/password in url (2004-04-29) http://python.org/sf/944396 opened by Chris Withers No documentation for urllib2 exception classes (2004-04-29) http://python.org/sf/944407 opened by Chris Withers csv writer bug on windows (2004-04-29) http://python.org/sf/944890 opened by Brian Kelley cgitb has backwards logic on log file creation (2004-04-30) CLOSED http://python.org/sf/945063 opened by Danny Yoo platform.system() Windows inconsistency (2004-04-30) http://python.org/sf/945665 opened by Guido van Rossum dir(object) does not list __name__ (2004-05-01) http://python.org/sf/945861 opened by Thomas Philips xmlrpclib omits charset in Content-Type HTTP header (2004-05-01) CLOSED http://python.org/sf/946130 opened by Christian Schmidt CGI QUERY_STRING (2004-05-01) http://python.org/sf/946247 opened by Larry sys.path is wrong in some cases (2004-05-04) http://python.org/sf/947380 opened by Ori Berger os.utime() raises bad exception for unicode filenames (2004-05-04) CLOSED http://python.org/sf/947405 opened by Mark Hammond urlib.urlopen() fails to raise exception (2004-05-04) http://python.org/sf/947571 opened by M.-A. Lemburg calendar.weekheader() undocumented (2004-05-04) http://python.org/sf/947894 opened by Leonardo Rochael Almeida calendar.weekheader(n): n should mean chars not bytes (2004-05-04) http://python.org/sf/947906 opened by Leonardo Rochael Almeida LaTeX not required (2004-05-05) CLOSED http://python.org/sf/948517 opened by Aahz texi2html.py not working (2004-05-06) http://python.org/sf/948881 opened by George Yoshida PyExc_* not in index (2004-05-06) http://python.org/sf/948970 opened by Thomas Heller winzip cannot extrat tarfile created tars (2004-05-06) http://python.org/sf/949052 opened by Simon Eisenmann refresh() missing curses reference (2004-05-06) CLOSED http://python.org/sf/949100 opened by Alexander Neumann Memory "read" instruction failed (2004-05-06) CLOSED http://python.org/sf/949301 opened by AnToine Memory "read" instruction failed (2004-05-06) http://python.org/sf/949301 reopened by alexion stringprep.py: sets.Set() --> set() (2004-05-06) http://python.org/sf/949329 opened by Raymond Hettinger UnicodeEncodeError message has syntax error (2004-05-07) CLOSED http://python.org/sf/949605 opened by Tom Lynn file write() method and non-blocking mode. (2004-05-07) http://python.org/sf/949667 opened by Donovan Baarda Invalid page in module PYTHON23.DLL (2004-05-07) CLOSED http://python.org/sf/949706 opened by AnToine Problem w/6.27.2.2 GNUTranslations ungettext() example code (2004-05-07) http://python.org/sf/949832 opened by Martin Miller Is cmdqueue attribute used? (2004-05-07) CLOSED http://python.org/sf/950055 opened by Raymond Hettinger itertools.chain doesn't "process" exceptions as they occur (2004-05-07) CLOSED http://python.org/sf/950057 opened by Tristan Seligmann buffer() can point to freed memory (2004-05-07) CLOSED http://python.org/sf/950274 opened by L. Peter Deutsch -fconstant-cfstrings required for Xcode 1.2 (2004-05-08) http://python.org/sf/950482 opened by Bob Ippolito email: mishandles Content-Transfer-Enc. for text/* messages (2004-05-09) http://python.org/sf/950747 opened by Matthias Klose str(email) breaks message by adding newline (2004-05-10) CLOSED http://python.org/sf/951088 opened by Gerrit Holl strptime doesn't account for %p in output (2004-05-10) CLOSED http://python.org/sf/951235 opened by Skip Montanaro Generator expression used in doc example (2004-05-11) http://python.org/sf/951270 opened by Nick Coghlan ihooks chokes on BioPython package (2004-05-10) http://python.org/sf/951275 opened by mpalmer New module: doc versus action (2004-05-10) http://python.org/sf/951482 opened by Terry J. Reedy Crash when reading "import table" of certain windows DLLs (2004-05-11) http://python.org/sf/951851 opened by Mark Hammond __name__ should be in 2.3.10 Special Attributes (2004-05-11) CLOSED http://python.org/sf/952212 opened by Robert Brewer Funny keywords in library index (2004-05-12) http://python.org/sf/952737 opened by Thomas Heller segfault in subclassing datetime.date & pickling (2004-05-12) http://python.org/sf/952807 opened by Thomas Wouters readline broke between 2.2.2 and 2.2.3 (2004-05-12) CLOSED http://python.org/sf/952852 opened by Jon Nelson "can't multiply sequence *by* non-int" (2004-05-12) CLOSED http://python.org/sf/952866 opened by Tom Lynn execve rejects empty argument list (2004-05-13) http://python.org/sf/952953 opened by newsham cgi module documentation could mention getlist (2004-05-13) http://python.org/sf/953177 opened by Richard Jones asyncore misses socket closes when poll is used (2004-05-13) http://python.org/sf/953599 opened by Shane Kerr simple typo in pprint module (2004-05-14) CLOSED http://python.org/sf/954030 opened by Michal Pasternak New email pkg FeedParser croaks on some spam (2004-05-14) CLOSED http://python.org/sf/954320 opened by Skip Montanaro 2.3 inspect.getframeinfo wrong tb line numbers (2004-05-15) http://python.org/sf/954364 opened by Robin Becker bsddb3 test errors (2004-05-15) CLOSED http://python.org/sf/954530 opened by Tim Peters Error in urllib2 Examples (2004-05-16) http://python.org/sf/954981 opened by Phillip CFLAGS, CPPFLAGS, LDFLAGS are NOT used when building modules (2004-05-17) http://python.org/sf/955197 opened by Niki W. Waibel Bugs Closed ___________ Random.choice doesn't work with sets. (2004-04-17) http://python.org/sf/936988 closed by rhettinger minidom.py writes TEXT_NODE wrong (2004-04-18) http://python.org/sf/937282 closed by loewis re.sub on u'' gives NON-unicode '' (2004-03-31) http://python.org/sf/926075 closed by perky message.as_string hoses headers (2004-04-20) http://python.org/sf/938890 closed by carlk message.as_string hoses headers (2004-04-20) http://python.org/sf/938890 closed by tim_one Shelve slow after 7/8000 key (2004-01-21) http://python.org/sf/881522 closed by tim_one Slicing changes item-attributes of an instance of a UserList (2004-04-19) http://python.org/sf/937700 closed by hflori xml parser bug (2004-04-22) http://python.org/sf/940263 closed by loewis string.lstrip problem with '_' string (2004-04-22) http://python.org/sf/940316 closed by rhettinger misbehaviour with map / lambda / isdigit() (2004-04-23) http://python.org/sf/940522 closed by tim_one section 5.10: 'not' returns boolean, not int (2004-04-23) http://python.org/sf/940579 closed by rhettinger Python crashes with bus error on sparc64. (2004-04-22) http://python.org/sf/940354 closed by genericplayer Unbounded recursion in modulefinder (2004-01-13) http://python.org/sf/876278 closed by theller Signal reception problem. FreeBSD 4.8 and 4.9 after forkpty (2004-04-28) http://python.org/sf/944119 closed by levis501 Logging failes for rotating file log (during rollOver) (2004-04-22) http://python.org/sf/939836 closed by smeyers2002 cgitb has backwards logic on log file creation (2004-04-30) http://python.org/sf/945063 closed by akuchling os.tmpfile seek(0) fails on cygwin (2004-01-01) http://python.org/sf/868743 closed by jlt63 xmlrpclib omits charset in Content-Type HTTP header (2004-05-02) http://python.org/sf/946130 closed by loewis os.utime() raises bad exception for unicode filenames (2004-05-04) http://python.org/sf/947405 closed by mhammond 5454 - documentation wrong for ossaudiodev mixer device (2003-09-29) http://python.org/sf/814606 closed by gward Build fails in ossaudiodev.c with missing macros (2003-10-29) http://python.org/sf/832236 closed by gward PyBool_* not documented in C API (2004-02-02) http://python.org/sf/889154 closed by fdrake socket.gethostbyname raises gaierror, not herror (2003-11-07) http://python.org/sf/837929 closed by fdrake pickle: __getinitargs__ not used for new-style objects (2004-01-08) http://python.org/sf/873246 closed by fdrake urllib2 CacheFTPHandler doesn't work on multiple dirs (2003-05-17) http://python.org/sf/738973 closed by mhammond os.chmod/os.utime/shutil do not work with unicode filenames (2003-11-21) http://python.org/sf/846133 closed by mhammond long list in Pythonwin -> weird text (2002-09-04) http://python.org/sf/604387 closed by mhammond LaTeX not required (2004-05-05) http://python.org/sf/948517 closed by fdrake shutil.copyfile fails when dst exists read-only (2003-09-22) http://python.org/sf/810879 closed by akuchling unable to specify another compiler (2003-07-04) http://python.org/sf/765819 closed by akuchling urllib2 authentication mishandles empty password (2004-04-29) http://python.org/sf/944082 closed by loewis pyexpat LexicalHandler swaps system_id and public_id (2003-07-30) http://python.org/sf/780300 closed by loewis XMLFilterBase has bogus startElementNS (2004-04-17) http://python.org/sf/936637 closed by loewis XMLGenerator ignores encoding in output (2004-04-19) http://python.org/sf/938076 closed by loewis xml.sax.handler.ContentHandler missing argument (2004-01-21) http://python.org/sf/881707 closed by fdrake file built-in object is after file() function (2004-03-25) http://python.org/sf/922964 closed by fdrake refresh() missing curses reference (2004-05-06) http://python.org/sf/949100 closed by akuchling Doc bug in PEP-0278 (2003-09-04) http://python.org/sf/800828 closed by fdrake Need "new style" note regarding descriptors (2003-10-04) http://python.org/sf/817742 closed by fdrake need doc for new trace module (2003-05-14) http://python.org/sf/737734 closed by fdrake Memory "read" instruction failed (2004-05-06) http://python.org/sf/949301 closed by theller Memory "read" instruction failed (2004-05-06) http://python.org/sf/949301 closed by theller UnicodeEncodeError message has syntax error (2004-05-07) http://python.org/sf/949605 closed by doerwalter re seems broken on 64-bit machines/linux (2004-04-08) http://python.org/sf/931848 closed by loewis Invalid page in module PYTHON23.DLL (2004-05-07) http://python.org/sf/949706 closed by theller Is cmdqueue attribute used? (2004-05-07) http://python.org/sf/950055 closed by gvanrossum itertools.chain doesn't "process" exceptions as they occur (2004-05-07) http://python.org/sf/950057 closed by rhettinger Python 2.3.3: socketmodule doesn't build on UnixWare 7.1.3 (2004-04-05) http://python.org/sf/929689 closed by loewis buffer() can point to freed memory (2004-05-08) http://python.org/sf/950274 closed by nascheme email library Parser can't parse some messages (2004-02-12) http://python.org/sf/895510 closed by bwarsaw MIMEMultipart _subparts constructor unusable (2004-01-25) http://python.org/sf/884030 closed by bwarsaw str(email) breaks message by adding newline (2004-05-10) http://python.org/sf/951088 closed by bwarsaw strptime doesn't account for %p in output (2004-05-10) http://python.org/sf/951235 closed by bcannon add support for cjkcodecs to Python email (2003-12-01) http://python.org/sf/852347 closed by bwarsaw email.Utils.formataddr doesn't handle whitespace too well (2004-01-12) http://python.org/sf/875646 closed by bwarsaw email package does not work with mailbox (2002-07-26) http://python.org/sf/586899 closed by bwarsaw Memory error on AIX in email.Utils._qdecode (2003-10-16) http://python.org/sf/824977 closed by bwarsaw __name__ should be in 2.3.10 Special Attributes (2004-05-11) http://python.org/sf/952212 closed by fdrake readline broke between 2.2.2 and 2.2.3 (2004-05-12) http://python.org/sf/952852 closed by jnelson "can't multiply sequence *by* non-int" (2004-05-12) http://python.org/sf/952866 closed by rhettinger email.Parser.Parser doesn't check for valid Content-Type (2003-11-21) http://python.org/sf/846938 closed by bwarsaw simple typo in pprint module (2004-05-14) http://python.org/sf/954030 closed by montanaro New email pkg FeedParser croaks on some spam (2004-05-15) http://python.org/sf/954320 closed by bwarsaw bsddb3 test errors (2004-05-15) http://python.org/sf/954530 closed by greg segmentation fault in test_re (2004-01-03) http://python.org/sf/870120 closed by hoel ConfigParser bug (2003-12-10) http://python.org/sf/857881 closed by fdrake New / Reopened RFE __________________ Allow any lvalue for function definitions (2004-05-09) http://python.org/sf/950644 opened by David Albert Torpey RFE Closed __________ optparse "append" action should always make the em (2003-10-07) http://python.org/sf/819178 closed by gward http://doc.python.org (2004-02-25) http://python.org/sf/904770 closed by etrepum From guido at python.org Mon May 17 23:48:25 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 17 23:48:33 2004 Subject: [Python-Dev] __str__ does not default to __repr__ if a base class has __str__ In-Reply-To: Your message of "Mon, 17 May 2004 11:00:11 EDT." <40A8D37B.7030807@erols.com> References: <40A8D37B.7030807@erols.com> Message-ID: <200405180348.i4I3mPP03950@guido.python.org> > # In the code below, "a.__str__()" does not default to "a.__repr__() as > # might be expected. It finds the "__str__" in the base class X. The > # Reference Manual, section 3.3.1, says: > # If a class defines __repr__() but not __str__(), then __repr__() is > # also used when an 'informal' string representation of instances of > # that class is required. > # If the reader is in lawyer mode, the documentation is correct. > # Is the current behavior what was intended? What should the behavior be? > > class X(object): > def __str__(self): > return 'str for X' > > def __repr__(self): > return 'repr for X' > > class A(X): > def __repr__(self): > return 'repr for A' > > # __str__ = __repr__ > > x = X() > a = A() > print x.__str__() > print x.__repr__() > print a.__str__() > print a.__repr__() This has never been different and the current outcome is exactly what was intended. The only case where __repr__ and __str__ act as each other's default is when there's no base class. And in any case, you're not supposed to call __str__ or __repr__ directly -- you should call str(x) or repr(x), etc. -- not that it makes a difference here. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon May 17 23:50:11 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 17 23:50:19 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: Your message of "Mon, 17 May 2004 17:39:42 +0200." References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <200405171452.i4HEqVp03065@guido.python.org> Message-ID: <200405180350.i4I3oBD03973@guido.python.org> > > Because in the current situation the meaning of an import is > > ambiguous. > > What exactly is ambiguous about it? Looking at "import foo" you can't tell whether there's a module foo expected in the current package or in the top-level module namespace. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon May 17 23:58:00 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 17 23:58:12 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: Your message of "Mon, 17 May 2004 21:36:31 EDT." <1084844191.19299.312.camel@anthem.wooz.org> References: <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517114707.02716660@mail.telecommunity.com> <40A93E6D.8010605@egenix.com> <1084844191.19299.312.camel@anthem.wooz.org> Message-ID: <200405180358.i4I3w0f04074@guido.python.org> > I know Guido's not in favor of this approach though. We've been through this before. Let's just please move forward. --Guido van Rossum (home page: http://www.python.org/~guido/) From michel at dialnetwork.com Mon May 17 22:07:09 2004 From: michel at dialnetwork.com (Michel Pelletier) Date: Tue May 18 00:10:00 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405130325.i4D3PDTs013072@salk.daca.net> References: <200405130325.i4D3PDTs013072@salk.daca.net> Message-ID: <200405172107.09197.michel@dialnetwork.com> On Wednesday 12 May 2004 22:30, Tim Peters wrote: > [Michel Pelletier] > > > ... > > Would there be an interest in at least a PEP to consider a bytecode > > verifier? > > I think so, yes. I'm working on a rough draft now. Initially I would like to start small and easy: a verifier written in python that provides a single function along the lines of the code provided by Phillip Eby that accepts the same arguments as dis(), a class, method, function, or code object. It throws an exception and explanation for invalid bytecode and it will be up to programmers to call this method explicitly and choose to execute the code object or not. This will probably cover almost all uses. > Python should be easier, in large part because you have to give up sooner. > > Checks for type correctness in the PVM are done at runtime instead. I > think it's fair to say that a bytecode verifier is overwhelmingly "just an > optimization": if bytecode properties can be verified from static code > analysis, runtime code isn't needed to verify them dynamically, or, in the > absence of such runtime checks, static analysis plugs ways to provoke > segfaults. I think it's likely that both run-time check elimination and segfault plugging can come out ot this. I'm not sure if it's possible for us to know what they all are up front, so I think a good requirement for a verification package will be a way to register new or project-specific verifications. do you think there is a risk of exploitation? for example, STORE_FAST, which does a direct set into PyObject **fastlocals, could be used to overwrite beyond the bounds of the array. Can this or a stack over/underflow be used to execute arbitrary machine code? > or > check that C-level indexing into the co_consts vector is in bounds. > Well, > in the latter case, that is checked (at runtime, on every co_consts access) > in a debug build, but not in a release build. I've investigated that a bit, and STORE_CONST and LOAD_CONST use GETITEM, which uses PyTuple_GetItem, which does a bounds check at run time even in a production build. I think. LOAD_FAST and STORE_FAST seem to do no runtime bounds check. -Michel From arigo at tunes.org Tue May 18 04:57:33 2004 From: arigo at tunes.org (Armin Rigo) Date: Tue May 18 05:01:02 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> Hello, A minor semantic change that creeped in some time ago was an implicit assumption that any object x should "reasonably" be expected to compare equal to itself. The arguments are summarized below (should this be documented, inserted in NEWS, turned in a mini-PEP a posteriori ... ?). The point is that comparisons now behave differently if they are issued by C or Python code. The expression 'x == y' will always call x.__eq__(y), because in some settings (e.g. Numeric) the result is not just True or False but an arbitrary object (e.g. a Numeric array of zeroes and ones). You cannot just say that 'x == x' should return True, because of that. So the behavior of PyObject_RichCompare() didn't change. On the other hand, when C code does a comparison it uses PyObject_RichCompareBool(), meaning it is only interested in a 1 or 0 answer. So PyObject_RichCompareBool() is where the shortcut about comparing an object with itself has been inserted. The result is the following: say x has a method __cmp__() that always returns -1. >>> x == x # x.__cmp__() is called False >>> x < x # x.__cmp__() is called True >>> cmp(x,x) # x.__cmp__() is NOT called 0 >>> [x] == [x] # x.__cmp__() is NOT called True The only way to explain the semantic is that the expression 'x == x' always call the special methods, but built-in functions are allowed to assume that any object compares equal to itself. In other words, C code usually checks that objects are "identical or equal", which is equalivalent to the Python expression 'x is y or x == y'. For example, the equality of lists now works as follows: def __eq__(lst1, lst2): if len(lst1) != len(lst2): return False for x, y in zip(lst1, lst2): if not (x is y or x == y): return False else: return True Should any of this be documented? An alternative behavior would have been to leave PyObject_RichCompareBool() alone and only insert the short-cut on specific object types' comparison methods on a case-by-case basis. For example, identical lists would just compare equal, without going through the loop comparing each element. This would remove the surprize of x.__cmp__(x) being not always called. The semantics would be easier to explain too: two lists are equal if they are the same list or if elements compare pairwise equal. We would have (with x as above): >>> cmp(x,x) # x.__cmp__(x) is called -1 >>> [x] == [x] False # because the lists contain a non-equal element >>> lst = [x]; lst == lst True # because it is the same list Finally, whatever the final semantic is, we should make sure that existing built-in objects behave in a consistent way. Of course I'm thinking about floats: on my Linux box, >>> f = float('nan') >>> cmp(f,f) 0 # because f is f >>> f == f False # because float.__eq__() is called Note that as discussed below the following behavior is *expected* and in accordance with standards: >>> float('nan') is float('nan') False >>> float('nan') == float('nan') False # not the same object Unless there are serious objections I suggest to (i.e. I plan to) remove the short-cut in PyObject_RichCompareBool() -- performance is probably not an issue here -- and then review all built-in comparison methods and make sure that they return "equal" for identical objects. -+- summary of arguments that lead to the original change (in cvs head). The argument in favor of the change is to remove the complex and not useful code trying to compare self-recursive structures: for example, in some setting comparing __builtin__.__dict__ with itself would trigger recursive comparison of __builtin__.__dict__ with itself in an endless loop. The complex algorithm was able to spot that. The new semantics immediately assume __builtin__.__dict__ to be equal to itself. Removing the complex algorithm means that you will now get an endless loop when comparing two *non-identical* self-recursive structures with the same shape, which is most probably not a problem in practice (on the contrary this implicit algorithm did hide a bug in one of my program). The argument _against_ used to be that e.g. it makes sense that the float value 'nan' should be different from itself, as various standards require. This argument does not apply in Python: these standards are about comparing *values*, not objects, so it makes perfect sense to say that even if x is the result of a computation that yielded an unknown answer 'nan', this answer is still equal to itself; what it is probably not equal to is *another* 'nan' which was obtained differently. In other words *two* float objects both containing 'nan' should be different, but *one* 'nan' object is still equal to itself. This is sane as long as no code considers 'nan' as a singleton, or tries to reuse 'nan' objects for different 'nan' values. -+- Armin From arigo at tunes.org Tue May 18 05:05:04 2004 From: arigo at tunes.org (Armin Rigo) Date: Tue May 18 05:08:43 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <200405180233.i4I2XgqL012607@cosc353.cosc.canterbury.ac.nz> References: <40A94622.6040606@ocf.berkeley.edu> <200405180233.i4I2XgqL012607@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040518090504.GB14842@vicky.ecs.soton.ac.uk> Hello Greg, On Tue, May 18, 2004 at 02:33:42PM +1200, Greg Ewing wrote: > Now that we have a garbage collector, is it really necessary > any more? What would happen if we simply set sys.modules to > None and then invoked the collector? There is a patch on SF that does this. Brett, can you see if this patch helps in your case? http://www.python.org/sf/812369 The problem I can see with my patch is that it's not clear to me that it eventually causes the destruction of enough objects. For that matter it's not clear why, or how many, objects we should try to destroy at shut-down time. I bet the goal is that if Python is embedded, and initialized and shut down repeatedly, it should not leak memory. Someone with such an application should give my patch a serious try before I can start to believe that no object is missed. There are moreover a few other pending issues raised on the patch's message board. A bientot, Armin. From mwh at python.net Tue May 18 07:04:46 2004 From: mwh at python.net (Michael Hudson) Date: Tue May 18 07:04:51 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405172107.09197.michel@dialnetwork.com> (Michel Pelletier's message of "Mon, 17 May 2004 21:07:09 -0500") References: <200405130325.i4D3PDTs013072@salk.daca.net> <200405172107.09197.michel@dialnetwork.com> Message-ID: <2mlljqhu7l.fsf@starship.python.net> Michel Pelletier writes: > do you think there is a risk of exploitation? for example, STORE_FAST, which > does a direct set into PyObject **fastlocals, could be used to overwrite > beyond the bounds of the array. Can this or a stack over/underflow be used > to execute arbitrary machine code? If you're loading arbitrary bytecode, you will presumably at some point be executing it, and that seems a much greater risk to me. Cheers, mwh -- We've had a lot of problems going from glibc 2.0 to glibc 2.1. People claim binary compatibility. Except for functions they don't like. -- Peter Van Eynde, comp.lang.lisp From mwh at python.net Tue May 18 07:34:13 2004 From: mwh at python.net (Michael Hudson) Date: Tue May 18 07:34:16 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> (Armin Rigo's message of "Tue, 18 May 2004 09:57:33 +0100") References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> Message-ID: <2md652hsui.fsf@starship.python.net> Armin Rigo writes: > Hello, > > A minor semantic change that creeped in some time ago was an implicit > assumption that any object x should "reasonably" be expected to compare equal > to itself. The arguments are summarized below (should this be documented, > inserted in NEWS, turned in a mini-PEP a posteriori ... ?). I have to admit that I haven't done my research, but I think you have your history backwards: the recent change AIUI is that it's possible that 'x == x' might be something *other* than Py_True... Cheers, mwh -- The ability to quote is a serviceable substitute for wit. -- W. Somerset Maugham From arigo at tunes.org Tue May 18 11:36:07 2004 From: arigo at tunes.org (Armin Rigo) Date: Tue May 18 11:39:08 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <2md652hsui.fsf@starship.python.net> References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> <2md652hsui.fsf@starship.python.net> Message-ID: <20040518153607.GA10557@vicky.ecs.soton.ac.uk> Hello Michael, On Tue, May 18, 2004 at 12:34:13PM +0100, Michael Hudson wrote: > > A minor semantic change that creeped in some time ago was an implicit > > assumption that any object x should "reasonably" be expected to compare equal > > to itself. > > I have to admit that I haven't done my research, but I think you have > your history backwards: the recent change AIUI is that it's possible > that 'x == x' might be something *other* than Py_True... Uh? You're probably referring to a much older change. It was possible from at least Python 2.1, and I don't know if it has ever been impossible. I am speaking about these two changes in object.c: ---------------------------- revision 2.215 date: 2004/03/21 17:01:44; author: rhettinger; state: Exp; lines: +11 -1 Add identity shortcut to PyObject_RichCompareBool. ---------------------------- and: ---------------------------- revision 2.211 date: 2003/10/28 12:05:47; author: arigo; state: Exp; lines: +8 -180 Deleting cyclic object comparison. SF patch 825639 http://mail.python.org/pipermail/python-dev/2003-October/039445.html ---------------------------- Armin From arigo at tunes.org Tue May 18 12:10:22 2004 From: arigo at tunes.org (Armin Rigo) Date: Tue May 18 12:13:20 2004 Subject: [Python-Dev] Is core dump always a bug? Advice requested In-Reply-To: <200405131651.i4DGpsl21234@guido.python.org> References: <200405121531.43532.michel@dialnetwork.com> <20040513085401.GA28305@vicky.ecs.soton.ac.uk> <200405131651.i4DGpsl21234@guido.python.org> Message-ID: <20040518161022.GB17637@vicky.ecs.soton.ac.uk> Hello Guido, On Thu, May 13, 2004 at 09:51:54AM -0700, Guido van Rossum wrote: > > * 'finally' blocks are executed with one, two or three values pushed > > on the stack; END_FINALLY will accordingly pop one, two or three > > values (depending on what the first popped value is). > > This is easily fixed, if necessary -- but is it necessary? It is a minor inconvenience for analysis tools. They more or less have to consider that within a finally block the structure of the stack is initially unknown (but contains at most N values, for some N) and that no opcode is allowed to peek or modify this unknown portion of the stack until END_FINALLY is reached. Psyco is an example of tool which assumes that the stack depth is only dependent on the bytecode position, and has a couple of hacks to deal with finally blocks. > > * POP_BLOCK will empty the value stack up to the position stored in > > the block stack's popped block, for no reason that I am aware of. > > I can't remember what my intention was; I probably wasn't so sure that > the stack level at the end of executing arbitrary code was always > the same. I guess it should be reviewed and removed. > > ... A better 'fix' would be that the block > > stack could, as we discussed at PyCon, be replaced by a purely > > static structure in the code object. > > Correct, I would like to see this happen. There's dynamic stuff here > just to make the implementation easy rather than to support necessary > dynamism. (The same could be said of some other parts of Python, but > there we've come to love the dynamism -- for the most part anyway. ;-) I'd like to give it a try, but I'd like to see if someone has any idea about the kind of structure that we are looking for here. I was roughly thinking about a binary tree, but we need something more compact than a naive tree. Armin From niemeyer at conectiva.com Tue May 18 13:18:43 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue May 18 13:21:19 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> Message-ID: <20040518171843.GA12816@burma.localdomain> > >>> cmp(x,x) # x.__cmp__() is NOT called > 0 This is odd. We'll have problems trying to implemnt a NaN following the convention of being not-equal to itself, for example. > >>> [x] == [x] # x.__cmp__() is NOT called > True This is correct, IMO. -- Gustavo Niemeyer http://niemeyer.net From jim.jewett at eds.com Tue May 18 13:24:56 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Tue May 18 13:26:06 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: Armin Rigo: say x has a method __cmp__() that always returns -1. >>> x == x # x.__cmp__() is called False >>> x < x # x.__cmp__() is called True >>> cmp(x,x) # x.__cmp__() is NOT called 0 >>> [x] == [x] # x.__cmp__() is NOT called True Note that the last (and only the last) is a change from 2.3 In 2.3, the (identity does not imply equality) trait rolls up to containers. >>> l=[x] >>> l == l False The only way to explain the semantic is that the expression 'x == x' always call the special methods, but built-in functions are allowed to assume that any object compares equal to itself. Note that "allowed" is as strong as you can get; is implies ==, but does not (currently) imply <= or preclude <. Should any of this be documented? I think so; it is a problem only for pathological cases, but who would bother to look up rich comparison in the first place unless they were considering at least borderline strangeness. An alternative behavior would have been to leave PyObject_RichCompareBool() alone and only insert the short-cut on specific object types' comparison methods I think it is reasonable for the default to be that identity implies equality. Either an explicitly defined __cmp__ should always be called, or the notes should say that containers can treat identity as good enough. This is sane as long as no code considers 'nan' as a singleton, or tries to reuse 'nan' objects for different 'nan' values. Note that float('nan') raises on ValueError on at least the default windows 2.3 build; if a NaN is hard to get, I would expect it to be reused. -jJ From aahz at pythoncraft.com Tue May 18 13:36:41 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue May 18 13:36:48 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <40A93E6D.8010605@egenix.com> References: <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <5.2.1.1.0.20040517171026.02ff4198@pop.bluewin.ch> <5.1.1.6.0.20040517114707.02716660@mail.telecommunity.com> <40A93E6D.8010605@egenix.com> Message-ID: <20040518173641.GA28095@panix.com> On Tue, May 18, 2004, M.-A. Lemburg wrote: > > BTW, what are you going to do about "from . import *" and > "from .. import *" ? Those work exactly the same way they currently do: find the appropriate namespace and import all available names. It will raise some kind of ``ImportError`` if it goes outside the package, though (note added to PEP). I'll be adding some verbage about ``__import__()``; thanks for raising the issue. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! From aahz at pythoncraft.com Tue May 18 14:01:53 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue May 18 14:02:34 2004 Subject: [Python-Dev] Re: .pth files in current or script directory In-Reply-To: References: <0AA408F4E902C84C8F6BE093CCE9D7A88C2468@server1.artsalliance.co.uk> Message-ID: <20040518180153.GA2877@panix.com> On Sun, May 16, 2004, Richard Cooper wrote: > > My initial idea was something along the lines of: > > When a module is imported, its import search path includes everything it > does now PLUS the directory that is the top of the module's enclosing > package structure. That way you wouldn't have to add entries to > PYTHONPATH just so that modules could import (absolutely) from their own > package. What's wrong with PEP 328? -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! From tim.one at comcast.net Tue May 18 14:37:31 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 18 14:37:39 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518171843.GA12816@burma.localdomain> Message-ID: > >>> cmp(x,x) # x.__cmp__() is NOT called > 0 [Gustavo Niemeyer] > This is odd. We'll have problems trying to implemnt a NaN > following the convention of being not-equal to itself, > for example. That's what rich comparisons are for. There's no problem getting >>> x = some_nan >>> x = x False >>> in Python 2.4 (the result of that is *currently* a platform-dependent crapshoot, though). cmp(x, y) doesn't make sense for objects having only a partial ordering, so I don't care cmp() does with a NaN. From bac at OCF.Berkeley.EDU Tue May 18 14:43:47 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Tue May 18 14:44:03 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <20040518090504.GB14842@vicky.ecs.soton.ac.uk> References: <40A94622.6040606@ocf.berkeley.edu> <200405180233.i4I2XgqL012607@cosc353.cosc.canterbury.ac.nz> <20040518090504.GB14842@vicky.ecs.soton.ac.uk> Message-ID: <40AA5963.80500@ocf.berkeley.edu> Armin Rigo wrote: > Hello Greg, > > On Tue, May 18, 2004 at 02:33:42PM +1200, Greg Ewing wrote: > >>Now that we have a garbage collector, is it really necessary >>any more? What would happen if we simply set sys.modules to >>None and then invoked the collector? > > > There is a patch on SF that does this. Brett, can you see if this patch helps > in your case? > > http://www.python.org/sf/812369 > It segfaulted when it normally spits out a bunch of tracebacks about sys.stderr being None from the interpreter tearing itself down. -Brett From bac at OCF.Berkeley.EDU Tue May 18 14:48:16 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Tue May 18 14:48:38 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <40AA17DF.5050403@interlink.com.au> References: <40A94622.6040606@ocf.berkeley.edu> <40AA17DF.5050403@interlink.com.au> Message-ID: <40AA5A70.8040404@ocf.berkeley.edu> Anthony Baxter wrote: > Brett C. wrote: > >> And the reason I bring this up now is just in case there is any desire >> to put this in 2.3.4 . I obviously know that I missed rc1, but I >> didn't get the details of triggering the problem soon enough and >> people running 2.3.x are the ones bringing this up since this >> apparently did not happen under 2.2 for some reason. I doubt Anthony >> wants to risk mucking with 'threading' this late (although the patch >> is fairly self-contained but would require applying the previous patch >> to attempt to fix this as well), but I thought I would at least put it >> out there. > > > Not going to be there for 2.3.4. > Figured as much. > My approach to the bug fix releases is that, in general, "there's > always a next release". That is, the bug fix releases are relatively > frequent (5-6 months) and they're "what's in the branch at the moment". > OK. How long do you plan to do this for the 2.3 branch? > Starting with the next (2.3.5/2.4.1) releases, I plan on sending out > a 1 month warning to python-list/python-dev, so that people that care > can make sure their pet bug fix is checked in and done before the > release. I'm really not interested in trying to slot in last minute > changes that may, or may not, break the release. > Makes sense to me. > In case it wasn't clear, I'm trying to get Python into a nice regular > release cycle for the bugfix releases. While there will be the > occasional "oh. my. god." type bug that drive a bug fix release > now now now, I'd prefer that people can get used to a regular > cycle for the bug fix releases. Part of this, of course, is being > harsh on the no-new-features rule. I want people to feel confident > that a bug fix release won't break their installed code base. Getting into a regular groove for releases sounds great to me, especially if we can state confidently that a bugfix release will be along every 6 months (+/- a month). -Brett From arigo at tunes.org Tue May 18 16:04:51 2004 From: arigo at tunes.org (Armin Rigo) Date: Tue May 18 16:07:58 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518171843.GA12816@burma.localdomain> References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> <20040518171843.GA12816@burma.localdomain> Message-ID: <20040518200451.GA4807@vicky.ecs.soton.ac.uk> Hello Gustavo, On Tue, May 18, 2004 at 02:18:43PM -0300, Gustavo Niemeyer wrote: > This is odd. We'll have problems trying to implemnt a NaN > following the convention of being not-equal to itself, > for example. I know the issue is a bit obfuscated, but please do read my mail entierely first: this is exactly my example :-) Armin From aahz at pythoncraft.com Tue May 18 16:08:29 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue May 18 16:08:33 2004 Subject: [Python-Dev] Bugfix cycle In-Reply-To: <40AA5A70.8040404@ocf.berkeley.edu> References: <40A94622.6040606@ocf.berkeley.edu> <40AA17DF.5050403@interlink.com.au> <40AA5A70.8040404@ocf.berkeley.edu> Message-ID: <20040518200828.GA14340@panix.com> > Anthony Baxter wrote: >> >>Starting with the next (2.3.5/2.4.1) releases, I plan on sending out >>a 1 month warning to python-list/python-dev, so that people that care >>can make sure their pet bug fix is checked in and done before the >>release. I'm really not interested in trying to slot in last minute >>changes that may, or may not, break the release. Cool! That'll make some people happier. I'm tempted to suggest adding c.l.py.announce, but on second thought, anyone who's not paying reasonably close attention to at least one of python-list/python-dev probably doesn't have enough attention to properly champion a patch for a bugfix release. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! From niemeyer at conectiva.com Tue May 18 19:32:00 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue May 18 19:31:32 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518200451.GA4807@vicky.ecs.soton.ac.uk> References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> Message-ID: <20040518233200.GA15700@burma.localdomain> > > This is odd. We'll have problems trying to implemnt a NaN > > following the convention of being not-equal to itself, > > for example. > > I know the issue is a bit obfuscated, but please do read my mail > entierely first: this is exactly my example :-) Sorry.. I read your mail in a rush, and the main issue looked to be related to these examples. Here we go again. [...] > >>> f = float('nan') > >>> cmp(f,f) > 0 # because f is f > >>> f == f > False # because float.__eq__() is called I don't get the same results: >>> f = float('nan') >>> cmp(f,f) 0 >>> f == f True Also on a Linux box. > Note that as discussed below the following behavior is *expected* and in > accordance with standards: > > >>> float('nan') is float('nan') > False I belive that testing identity on NaN might return True in cases like: >>> f = float('nan') >>> f is f True > >>> float('nan') == float('nan') > False # not the same object This is expected in an IEEE754 compliant implementation. Something we currently are not (I get different results, as shown). > Unless there are serious objections I suggest to (i.e. I plan to) remove the > short-cut in PyObject_RichCompareBool() -- performance is probably not an > issue here -- and then review all built-in comparison methods and make sure > that they return "equal" for identical objects. +1 -- Gustavo Niemeyer http://niemeyer.net From greg at cosc.canterbury.ac.nz Tue May 18 21:39:04 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue May 18 21:39:40 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> Message-ID: <200405190139.i4J1d40t014215@cosc353.cosc.canterbury.ac.nz> Armin Rigo : > Unless there are serious objections I suggest to (i.e. I plan to) remove the > short-cut in PyObject_RichCompareBool() -- performance is probably not an > issue here -- and then review all built-in comparison methods and make sure > that they return "equal" for identical objects. Sounds right to me. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From greg at cosc.canterbury.ac.nz Tue May 18 22:38:02 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Tue May 18 22:38:32 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <20040518090504.GB14842@vicky.ecs.soton.ac.uk> Message-ID: <200405190238.i4J2c2aO014302@cosc353.cosc.canterbury.ac.nz> Armin Rigo : > I bet the goal is that if Python is embedded, and initialized and shut down > repeatedly, it should not leak memory. I think that's going to be impossible to achieve as long as there is no protocol for shutting down extension modules, which may be holding references to things in C globals. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From tim.one at comcast.net Tue May 18 22:44:05 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 18 22:44:12 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518153607.GA10557@vicky.ecs.soton.ac.uk> Message-ID: [Armin Rigo] >>> A minor semantic change that creeped in some time ago was an implicit >>> assumption that any object x should "reasonably" be expected to >>> compare equal to itself. [Michael Hudson] >> I have to admit that I haven't done my research, but I think you have >> your history backwards: the recent change AIUI is that it's possible >> that 'x == x' might be something *other* than Py_True... [Armin] > Uh? You're probably referring to a much older change. It was possible > from at least Python 2.1, and I don't know if it has ever been > impossible. So-called "rich comparisons" were new in 2.1. Before then, all comparisons, however spelled, ended up invoking a spelling of cmp(), and PyObject_Compare() has had its current if (v == w) return 0; special case ever since revison 1.1 of object.c (that was about 14(!) years ago now). So, yes, before Python 2.1, it wasn't possible for x == x (or x <= x or x >= x) to return false, or for x < x, x > x, or x != x to return true, or for cmp(x, x) to return anything other than 0. Rich comparisons changed all that excepting the last. At first in 2.1, and through all released Pythons since then, the "v==w implies equal" special case has remained in PyObject_Compare, but was left out of implementations of the 6 "new", true/false relational operators. > I am speaking about these two changes in object.c: > ---------------------------- > revision 2.215 > date: 2004/03/21 17:01:44; author: rhettinger; state: Exp; lines: +11 - > 1 > Add identity shortcut to PyObject_RichCompareBool. > ---------------------------- Raymond added that after you, Guido and I agreed to it at the PyCon sprints in March. It was specifically to ease your troubles with comparing recursive dictionaries, which started after the hideous cyclic-compare hacks were removed. As CVS records, removing them was also your fault : > and: > > ---------------------------- > revision 2.211 > date: 2003/10/28 12:05:47; author: arigo; state: Exp; lines: +8 -180 > Deleting cyclic object comparison. > SF patch 825639 > http://mail.python.org/pipermail/python-dev/2003-October/039445.html From tim.one at comcast.net Tue May 18 23:24:50 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 18 23:24:51 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> Message-ID: [Armin Rigo] > ... > Note that as discussed below the following behavior is *expected* and in > accordance with standards: > > >>> float('nan') is float('nan') > False That result simply isn't defined, even on platforms where float('nan') doesn't blow up: floats are immutable objects, and outside of interning strings, Python doesn't define anything about the identity of immutable objects returned from computation. > >>> float('nan') == float('nan') > False # not the same object The one is a platform-dependent crapshoot, and again even on platforms where float('nan') doesn't blow up. In current CVS HEAD, Michael made it much more *likely* to return False than in any Python released to date, but there's still no guarantee. For example, even with current CVS, it returns True if compiled with MSVC 6, but False if compiled with MSVC 7.1. Python simply doesn't define anything about behavior in the presence of NaNs, infinities or signed zeroes (neither as floats nor as string representations (like 'nan')). > Unless there are serious objections I suggest to (i.e. I plan to) remove > the short-cut in PyObject_RichCompareBool() -- performance is probably > not an issue here -- and then review all built-in comparison methods and > make sure that they return "equal" for identical objects. If your recursive dict compares work OK then, fine by me. The exact details of comparison semantics have always been muddy, and I expect always will be (the combination of backward compatibility hacks and new features makes comparison reality as complicated as the code implementing it -- that's some of the most complicated stuff in the entire codebase). From anthony at interlink.com.au Tue May 18 10:04:15 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed May 19 04:02:01 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <40A94622.6040606@ocf.berkeley.edu> References: <40A94622.6040606@ocf.berkeley.edu> Message-ID: <40AA17DF.5050403@interlink.com.au> Brett C. wrote: > And the reason I bring this up now is just in case there is any desire > to put this in 2.3.4 . I obviously know that I missed rc1, but I didn't > get the details of triggering the problem soon enough and people running > 2.3.x are the ones bringing this up since this apparently did not happen > under 2.2 for some reason. I doubt Anthony wants to risk mucking with > 'threading' this late (although the patch is fairly self-contained but > would require applying the previous patch to attempt to fix this as > well), but I thought I would at least put it out there. Not going to be there for 2.3.4. My approach to the bug fix releases is that, in general, "there's always a next release". That is, the bug fix releases are relatively frequent (5-6 months) and they're "what's in the branch at the moment". Starting with the next (2.3.5/2.4.1) releases, I plan on sending out a 1 month warning to python-list/python-dev, so that people that care can make sure their pet bug fix is checked in and done before the release. I'm really not interested in trying to slot in last minute changes that may, or may not, break the release. In case it wasn't clear, I'm trying to get Python into a nice regular release cycle for the bugfix releases. While there will be the occasional "oh. my. god." type bug that drive a bug fix release now now now, I'd prefer that people can get used to a regular cycle for the bug fix releases. Part of this, of course, is being harsh on the no-new-features rule. I want people to feel confident that a bug fix release won't break their installed code base. In addition, I never ever want to see try: True, False except NameError: True, False = 1, 0 or it's like ever again. I've seen it far far too often in code that wants to be "2.2" compatible, but the new True/False in the bugfix release of 2.2 made life unpleasant. I'm open to suggestions for this - although emails about "but I really really want SF bug #XYZ fixed for 2.3.4" is not what I'm after. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From theller at python.net Wed May 19 04:44:04 2004 From: theller at python.net (Thomas Heller) Date: Wed May 19 04:44:21 2004 Subject: [Python-Dev] windist.exe status and license In-Reply-To: <16552.21927.581799.337029@gargle.gargle.HOWL> (Matthias Klose's message of "Mon, 17 May 2004 08:03:19 +0200") References: <1084541142.696332.6706.nullmailer@calvin.fayauffre.org> <16552.21927.581799.337029@gargle.gargle.HOWL> Message-ID: Matthias Klose writes: > [CC to python-dev] > > The windist.exe version shipped with current python releasese is > outdated and does not have any license information about windist > included. > > Alexandre Fayolle writes: >> Package: python2.3-dev >> Version: 2.3.3-7 >> Severity: normal >> >> Hi, >> >> I noticed the file /usr/lib/python2.3/distutils/command/wininst.exe is >> missing from my currently installed version of the python2.3-dev >> package. I'm pretty sure I used to be able to create windows installers >> for the modules I develop from my Debian machine. >> >> Is this removal caused by an omission, or is there some licensing issue ? >> I could not find anything on that topic in the changelog, but maybe I >> didn't look hard enough. > > I cannot find anything about the licensing. The executable only shows > > $Info: This file is packed with the UPX executable packer http://upx.tsx.org $ > $Id: UPX 1.01 Copyright (C) 1996-2000 the UPX Team. All Rights Reserved. $ > > However I found http://upx.sourceforge.net/. > > - Maybe the binary can be updated to the SF version? > - Add the upx license to the Lib/distutils/command directory? > > > [Debian only: > > The UPX license itself is discussed at > http://lists.debian.org/debian-legal/2001/01/msg00025.html > > upx-nrv is currently in section non-free/utils, so it's better > not to distribute it in the Debian package. > ] Does this/should this affect the official 2.3.4 release, or is this only for debian packaging? upx is licensed GPL, with this exception which applies (imo) to the wininst.exe binary: SPECIAL EXCEPTION FOR COMPRESSED EXECUTABLES ============================================ The stub which is imbedded in each UPX compressed program is part of UPX and UCL, and contains code that is under our copyright. The terms of the GNU General Public License still apply as compressing a program is a special form of linking with our stub. Hereby Markus F.X.J. Oberhumer and Laszlo Molnar grant you special permission to freely use and distribute all UPX compressed programs (including commercial ones), subject to the following restrictions: 1. You must compress your program with a completely unmodified UPX version; either with our precompiled version, or (at your option) with a self compiled version of the unmodified UPX sources as distributed by us. 2. This also implies that the UPX stub must be completely unmodfied, i.e. the stub imbedded in your compressed program must be byte-identical to the stub that is produced by the official unmodified UPX version. 3. The decompressor and any other code from the stub must exclusively get used by the unmodified UPX stub for decompressing your program at program startup. No portion of the stub may get read, copied, called or otherwise get used or accessed by your program. Especially 3. puts restrictions on the binary. The full upx license is here: Hm, maybe it would be better to *not* compress wininst.exe by upx any longer. The uncompressed exe is 57344 bytes, the compressed on 21504 bytes. Any bdist_wininst installer would be 36 kB larger, which isn't a large deal. Thomas From theller at python.net Wed May 19 05:04:27 2004 From: theller at python.net (Thomas Heller) Date: Wed May 19 05:04:41 2004 Subject: [Python-Dev] windist.exe status and license In-Reply-To: (Thomas Heller's message of "Wed, 19 May 2004 10:44:04 +0200") References: <1084541142.696332.6706.nullmailer@calvin.fayauffre.org> <16552.21927.581799.337029@gargle.gargle.HOWL> Message-ID: (followup to myself) > Hm, maybe it would be better to *not* compress wininst.exe by upx any > longer. The uncompressed exe is 57344 bytes, the compressed on 21504 > bytes. Any bdist_wininst installer would be 36 kB larger, which isn't a > large deal. After 5 minutes, it became crystal clear to me that this is the obvious solution. Who cares about 36 kB, and that's probably less than the size of the GPL. I'll change this for the 2.3.4 release. Thomas From mwh at python.net Wed May 19 05:11:25 2004 From: mwh at python.net (Michael Hudson) Date: Wed May 19 05:11:33 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040518233200.GA15700@burma.localdomain> (Gustavo Niemeyer's message of "Tue, 18 May 2004 20:32:00 -0300") References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> <20040518233200.GA15700@burma.localdomain> Message-ID: <2mwu38hjcy.fsf@starship.python.net> Gustavo Niemeyer writes: > This is expected in an IEEE754 compliant implementation. What's one of them? > Something we currently are not (I get different results, as shown). Are you using 2.3 or 2.4 here? Cheers, mwh -- surely, somewhere, somehow, in the history of computing, at least one manual has been written that you could at least remotely attempt to consider possibly glancing at. -- Adam Rixey From anthony at interlink.com.au Wed May 19 05:59:55 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed May 19 06:00:14 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <40AA5A70.8040404@ocf.berkeley.edu> References: <40A94622.6040606@ocf.berkeley.edu> <40AA17DF.5050403@interlink.com.au> <40AA5A70.8040404@ocf.berkeley.edu> Message-ID: <40AB301B.7000304@interlink.com.au> Brett C. wrote: > Anthony Baxter wrote: >> My approach to the bug fix releases is that, in general, "there's >> always a next release". That is, the bug fix releases are relatively >> frequent (5-6 months) and they're "what's in the branch at the moment". >> > > OK. How long do you plan to do this for the 2.3 branch? I'm anticipating doing this until 2.4.1. That is, the upcoming release train will be: 2.3.4 2.4a{1,2,...} 2.4b{1,2,...} 2.4rc{1,2,...} 2.4 2.3.5 2.4.1 Unless there's a pressing need for it, I don't see the point to cutting a 2.3.6 or later, once 2.4 is onto the release24-maint branch. Obviously, if there's one of the oh-my-gods type bugs in 2.3, this will probably require a 2.3.6. Hopefully, by the time I finish the 2.4 cycle, there'll be some sane release automation tools finished, too. Once we've forked release24-maint, release23-maint can start to die. There's no point backporting patches to two separate -maint branches. > Getting into a regular groove for releases sounds great to me, > especially if we can state confidently that a bugfix release will be > along every 6 months (+/- a month). Yep. I plan to update the bug fix release PEP and post it out to c.l.py and python-dev for feedback. As ever, I'm open to feedback on any of the above. Except for the "can we please have this whizzo feature in the next bug fix release" feedback. You know the answer to that one ;) -- Anthony Baxter It's never too late to have a happy childhood. From arigo at tunes.org Wed May 19 06:28:20 2004 From: arigo at tunes.org (Armin Rigo) Date: Wed May 19 06:31:19 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040519032538.5EB062F33B@bespin.org> References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> <20040519032538.5EB062F33B@bespin.org> Message-ID: <20040519102820.GA7677@vicky.ecs.soton.ac.uk> Hello Tim, On Tue, May 18, 2004 at 11:24:50PM -0400, Tim Peters wrote: > > >>> float('nan') is float('nan') > > False > > That result simply isn't defined, even on platforms where float('nan') > doesn't blow up: floats are immutable objects, and outside of interning > strings, Python doesn't define anything about the identity of immutable > objects returned from computation. Ok, I won't go into that argument again then. Raymond told me that the PyObject_RichCompareBool() short-cut is essential for performance, as a very large number of internal comparisons end up comparing identical objects (e.g. with dicts). I don't really care about what occurs with float('nan'), but it seems marginally strange that the outcome could, with some compilers, depend on the way you compare them: even though x == x and cmp(x,x) are known to do different things, as rich comparison goes, you wouldn't expect this to show up with floats. But anyway that's fine by me (and could be easily fixed by patching float_compare and float_richcompare to check for identity). A slightly stranger irregularity (though not a problem in general) is that comparing two objects of user-defined classes may invoke neither the __cmp__ nor the __eq__ special method at all. It should at least be documented somewhere, in my opinion. The language reference currently explicitely states that x==y calls x.__eq__(y) and falls back to y.__eq__(x). Armin From niemeyer at conectiva.com Wed May 19 09:38:12 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Wed May 19 09:37:42 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <2mwu38hjcy.fsf@starship.python.net> References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> <20040518233200.GA15700@burma.localdomain> <2mwu38hjcy.fsf@starship.python.net> Message-ID: <20040519133812.GA12100@burma.localdomain> > > This is expected in an IEEE754 compliant implementation. > > What's one of them? I can tell you.. or I'd have to kill you. ;-) > > Something we currently are not (I get different results, as shown). > > Are you using 2.3 or 2.4 here? 2.3.. I wasn't aware about your recent change. Are you planning to include C99 stuff as well? -- Gustavo Niemeyer http://niemeyer.net From mwh at python.net Wed May 19 09:44:08 2004 From: mwh at python.net (Michael Hudson) Date: Wed May 19 09:44:13 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040519133812.GA12100@burma.localdomain> (Gustavo Niemeyer's message of "Wed, 19 May 2004 10:38:12 -0300") References: <20040518085733.GA14842@vicky.ecs.soton.ac.uk> <20040518233200.GA15700@burma.localdomain> <2mwu38hjcy.fsf@starship.python.net> <20040519133812.GA12100@burma.localdomain> Message-ID: <2mpt90h6qf.fsf@starship.python.net> Gustavo Niemeyer writes: >> > This is expected in an IEEE754 compliant implementation. >> >> What's one of them? > > I can tell you.. or I'd have to kill you. ;-) > >> > Something we currently are not (I get different results, as shown). >> >> Are you using 2.3 or 2.4 here? > > 2.3.. I wasn't aware about your recent change. Are you planning > to include C99 stuff as well? No. My recent change was mainly motivated by the wart that in Python 2.3 on linux float('nan') == x came out True for any float x at all! I'm not enough of a numerics guy to know what people who would like real 754 support would actually want. Cheers, mwh -- Finding a needle in a haystack is a lot easier if you burn down the haystack and scan the ashes with a metal detector. -- the Silicon Valley Tarot (another one nicked from David Rush) From bac at OCF.Berkeley.EDU Wed May 19 14:17:41 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Wed May 19 14:18:01 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <40AB301B.7000304@interlink.com.au> References: <40A94622.6040606@ocf.berkeley.edu> <40AA17DF.5050403@interlink.com.au> <40AA5A70.8040404@ocf.berkeley.edu> <40AB301B.7000304@interlink.com.au> Message-ID: <40ABA4C5.6070300@ocf.berkeley.edu> Anthony Baxter wrote: > Brett C. wrote: > >> Anthony Baxter wrote: >> >>> My approach to the bug fix releases is that, in general, "there's >>> always a next release". That is, the bug fix releases are relatively >>> frequent (5-6 months) and they're "what's in the branch at the moment". >>> >> >> OK. How long do you plan to do this for the 2.3 branch? > > > I'm anticipating doing this until 2.4.1. That is, the upcoming > release train will be: > > 2.3.4 > 2.4a{1,2,...} > 2.4b{1,2,...} > 2.4rc{1,2,...} > 2.4 > 2.3.5 > 2.4.1 > > Unless there's a pressing need for it, I don't see the point to > cutting a 2.3.6 or later, once 2.4 is onto the release24-maint > branch. Obviously, if there's one of the oh-my-gods type bugs in > 2.3, this will probably require a 2.3.6. Hopefully, by the time > I finish the 2.4 cycle, there'll be some sane release automation > tools finished, too. > Anything us normal folk can help you with in terms of the aforementioned tools? Maybe this could encompass a tool to run the testing suite, re-run the failed tests with verbose output, and then email them to a Mailman list with system info so we have a partially automated testing framework for alphas, betas, and release candidates (Christopher Blunck proposed this idea back in March; email at http://mail.python.org/pipermail/python-dev/2004-March/043492.html) since we currently don't have a testing server farm running regularly anymore? -Brett From bh at intevation.de Wed May 19 14:53:21 2004 From: bh at intevation.de (Bernhard Herzog) Date: Wed May 19 14:53:26 2004 Subject: [Python-Dev] Relative vs. absolute imports References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <200405171452.i4HEqVp03065@guido.python.org> <200405180350.i4I3oBD03973@guido.python.org> Message-ID: Guido van Rossum writes: >> > Because in the current situation the meaning of an import is >> > ambiguous. >> >> What exactly is ambiguous about it? > > Looking at "import foo" you can't tell whether there's a module foo > expected in the current package or in the top-level module namespace. Well, looking at "print foo" I can't tell whether its a local variable, a global variable or a builtin either. Nobody's going to suggest we should introduce special syntax to distinguish them, I hope. Module import isn't much different and it's only a little harder to figure out whether its a relative import or not, and you only have to distinguish them when the importer is a module in a package. I've been using packages in the current form ever since they've been introduced in Python 1.5. In all that time, relative imports have never been a problem for me, IIRC, and I use them for all sibling imports. Absolute imports have their own difficulties, too, btw, and relative imports don't add much complexity on top of that. I don't think the ambiguity, if you want to call it that, is enough of a problem to justify the incompatibility. Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://sketch.sourceforge.net/ Thuban http://thuban.intevation.org/ From pje at telecommunity.com Wed May 19 15:21:32 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Wed May 19 15:21:25 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <200405171452.i4HEqVp03065@guido.python.org> <200405180350.i4I3oBD03973@guido.python.org> Message-ID: <5.1.1.6.0.20040519151756.037f6b50@mail.telecommunity.com> At 08:53 PM 5/19/04 +0200, Bernhard Herzog wrote: >Guido van Rossum writes: > > >> > Because in the current situation the meaning of an import is > >> > ambiguous. > >> > >> What exactly is ambiguous about it? > > > > Looking at "import foo" you can't tell whether there's a module foo > > expected in the current package or in the top-level module namespace. > >Well, looking at "print foo" I can't tell whether its a local variable, >a global variable or a builtin either. Nobody's going to suggest we >should introduce special syntax to distinguish them, I hope. Module >import isn't much different Yes it is, actually. You can answer the "print foo" question simply by inspecting the code of the function that contains it. (And if it's not in a function, you know it's a global, unless it's never assigned, in which case it's a builtin.) But the "import foo" question cannot be answered by referring to the source of the module in which it occurs. >Absolute imports have their own difficulties, too, btw, and relative >imports don't add much complexity on top of that. I don't think the >ambiguity, if you want to call it that, is enough of a problem to >justify the incompatibility. IIRC, nobody has suggested eliminating them prior to 3.0, which will have lots of other backward-incompatible changes anyway. From aahz at pythoncraft.com Wed May 19 15:32:05 2004 From: aahz at pythoncraft.com (Aahz) Date: Wed May 19 15:33:26 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <5.1.1.6.0.20040519151756.037f6b50@mail.telecommunity.com> References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <200405171452.i4HEqVp03065@guido.python.org> <200405180350.i4I3oBD03973@guido.python.org> <5.1.1.6.0.20040519151756.037f6b50@mail.telecommunity.com> Message-ID: <20040519193204.GA7731@panix.com> On Wed, May 19, 2004, Phillip J. Eby wrote: > At 08:53 PM 5/19/04 +0200, Bernhard Herzog wrote: >> >>Absolute imports have their own difficulties, too, btw, and relative >>imports don't add much complexity on top of that. I don't think the >>ambiguity, if you want to call it that, is enough of a problem to >>justify the incompatibility. > > IIRC, nobody has suggested eliminating them prior to 3.0, which will have > lots of other backward-incompatible changes anyway. Actually, the current plan is to eliminate relative imports that don't use the new syntax in Python 2.6. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ Adopt A Process -- stop killing all your children! From bh at intevation.de Wed May 19 15:34:49 2004 From: bh at intevation.de (Bernhard Herzog) Date: Wed May 19 15:34:55 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <5.1.1.6.0.20040519151756.037f6b50@mail.telecommunity.com> (Phillip J. Eby's message of "Wed, 19 May 2004 15:21:32 -0400") References: <5.1.1.6.0.20040517101458.0635d450@mail.telecommunity.com> <200405171452.i4HEqVp03065@guido.python.org> <200405180350.i4I3oBD03973@guido.python.org> <5.1.1.6.0.20040519151756.037f6b50@mail.telecommunity.com> Message-ID: "Phillip J. Eby" writes: > Yes it is, actually. You can answer the "print foo" question simply by > inspecting the code of the function that contains it. (And if it's not > in a function, you know it's a global, unless it's never assigned, in > which case it's a builtin.) > > But the "import foo" question cannot be answered by referring to the > source of the module in which it occurs. Of course it's different, but it's not all that much different. To determine whether an import is relative or not you look into the directory containing the module with the import statement. Shouldn't be more than a few keystrokes in a decent editor. > IIRC, nobody has suggested eliminating them prior to 3.0, which will > have lots of other backward-incompatible changes anyway. The new import scheme will be the only one available in Python 2.6 and later according to the PEP and the current style of relative imports will produce DeprecationWarnings in 2.5. Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://sketch.sourceforge.net/ Thuban http://thuban.intevation.org/ From ncoghlan at iinet.net.au Wed May 19 20:22:46 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed May 19 20:22:56 2004 Subject: [Python-Dev] LC_NUMERIC, gtk and Patch 774665 Message-ID: <40ABFA56.9050805@iinet.net.au> A problem was recently reported on wxPython-dev that smells strongly of this issue with GTK changing the LC_NUMERIC locale to something other than 'C' causing subsequent severe problems in the handling of floats by the Python interpreter. I went looking for Christian's proposed PEP (to make Python more locale agnostic when it comes to LC_NUMERIC), and didn't find it listed in PEP 0. The last activity on Gustavo's patch was in December 2003, and the last comment in September. What's the current state of this? Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From mcherm at mcherm.com Wed May 19 20:38:43 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Wed May 19 20:38:47 2004 Subject: [Python-Dev] Relative vs. absolute imports Message-ID: <1085013523.40abfe13c8346@mcherm.com> Phillip Eby: > Yes it is, actually. You can answer the "print foo" question simply by > inspecting the code of the function that contains it. (And if it's not > in a function, you know it's a global, unless it's never assigned, in > which case it's a builtin.) > > But the "import foo" question cannot be answered by referring to the > source of the module in which it occurs. Bernhard Herzog: > Of course it's different, but it's not all that much different. To > determine whether an import is relative or not you look into the > directory containing the module with the import statement. Shouldn't be > more than a few keystrokes in a decent editor. On the contrary, I find a big difference between something which can be learned by inspecting the source and something which requires access to the filesystem the source is running on. Many times that filesystem isn't even available: "Susan emailed me this module she uses to fnob large datasets, and I can't understand why it isn't working for me!" -- Michael Chermside From ncoghlan at iinet.net.au Wed May 19 21:48:33 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Wed May 19 21:48:43 2004 Subject: [Python-Dev] __str__ does not default to __repr__ if a base class has __str__ In-Reply-To: <200405180348.i4I3mPP03950@guido.python.org> References: <40A8D37B.7030807@erols.com> <200405180348.i4I3mPP03950@guido.python.org> Message-ID: <40AC0E71.8080003@iinet.net.au> Guido van Rossum wrote: >>class X(object): >> def __str__(self): >> return 'str for X' >> >> def __repr__(self): >> return 'repr for X' >> >>class A(X): >> def __repr__(self): >> return 'repr for A' >> >># __str__ = __repr__ > > This has never been different and the current outcome is exactly what > was intended. The only case where __repr__ and __str__ act as each > other's default is when there's no base class. The OP's example also shows how easy it is to create a class which returns the same thing for str(foo) and repr(foo). Simply write one of either __repr__ or __str__ and then assign the same function to the other one. Regards, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From anthony at interlink.com.au Wed May 19 21:54:20 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed May 19 22:19:55 2004 Subject: [Python-Dev] python 2.3.4 delayed for a week Message-ID: <40AC0FCC.2040705@interlink.com.au> Due to circumstances beyond our control, Thomas is unable to cut the windows installer for 2.3.4 for a week - we'll delay the release until he's back on deck. -- Anthony Baxter It's never too late to have a happy childhood. From flaw at rhid.com Thu May 20 12:42:46 2004 From: flaw at rhid.com (James William Pye) Date: Thu May 20 12:42:52 2004 Subject: [Python-Dev] Access to generator type In-Reply-To: <20040511022708.598B0720E9E@rhid.com> References: <20040511012304.GE29924@void.ph.cox.net> <20040511022708.598B0720E9E@rhid.com> Message-ID: <20040520164246.GO62439@void.ph.cox.net> On 05/10/04:19/1, Tim Peters wrote: > Possibly, if someone wants it enough to make a patch (including doc > changes). There are many types that aren't exposed at the C level, BTW. > Advertising their internals has an inhibiting effect on evolution. The > generator type has been around long enough w/o change that the value in > keeping it private has diminished. Okay, here's a preliminary patch with genobject.[ch]: http://sourceforge.net/tracker/index.php?func=detail&aid=957398&group_id=5470&atid=305470 I assumed(yay) that py-dev would want it inline with how other core objects are managed(*object.[ch]), so I spun it out into its own files in /Object/genobject.c and /Include/genobject.h.. Of course, genobject.c references eval_frame(in ceval.c), which raises a problem; currently it is resolved in my tree by making a wrapper in ceval.c(PyEval_EvaluateFrame(){eval_frame()}). This is unlikely to be an acceptable solution, and I'm not sure how py-dev wants this to be handled. Keep it in its own genobject.[ch] and get eval_frame how? Or keep it in ceval.[ch]? The latter seems to be the right choice to me, but less pertty. Regards, James William Pye -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 479 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040520/0898db7c/attachment.bin From gjc at inescporto.pt Thu May 20 12:52:47 2004 From: gjc at inescporto.pt (Gustavo J. A. M. Carneiro) Date: Thu May 20 12:53:08 2004 Subject: [Python-Dev] LC_NUMERIC, gtk and Patch 774665 In-Reply-To: <40ABFA56.9050805@iinet.net.au> References: <40ABFA56.9050805@iinet.net.au> Message-ID: <1085071967.1037.7.camel@spectrum.inescn.pt> A Qui, 2004-05-20 ?s 01:22, Nick Coghlan escreveu: > A problem was recently reported on wxPython-dev that smells strongly of > this issue with GTK changing the LC_NUMERIC locale to something other > than 'C' causing subsequent severe problems in the handling of floats by > the Python interpreter. > > I went looking for Christian's proposed PEP (to make Python more locale > agnostic when it comes to LC_NUMERIC), and didn't find it listed in PEP > 0. The last activity on Gustavo's patch was in December 2003, and the > last comment in September. > > What's the current state of this? AFAICT, there was interest in including this patch even without a full PEP. Christian is currently on vacation. There is a unoficial PEP draft at http://www.async.com.br/~kiko/pep-numeric.txt I stopped working on the patch because there was nothing more to do. But I'm willing to do some more effort as necessary to include this in Python 2.4. Best regards. -- Gustavo Jo?o Alves Marques Carneiro The universe is always one step beyond logic. From bh at intevation.de Thu May 20 16:15:01 2004 From: bh at intevation.de (Bernhard Herzog) Date: Thu May 20 16:15:29 2004 Subject: [Python-Dev] Relative vs. absolute imports References: <1085013523.40abfe13c8346@mcherm.com> Message-ID: Michael Chermside writes: > Bernhard Herzog: >> Of course it's different, but it's not all that much different. To >> determine whether an import is relative or not you look into the >> directory containing the module with the import statement. Shouldn't be >> more than a few keystrokes in a decent editor. > > On the contrary, I find a big difference between something which can > be learned by inspecting the source and something which requires > access to the filesystem the source is running on. Me too, but, in the case of imports, regardless of whether they're absolute or relative you will always have to refer to information ouside of the module you're looking at. In most cases you'll need to look at the filesystem among other things. However, telling relative and absolute imports apart is easy compared to the difficulty of dealing with imports in general, just like telling local and global variables apart is easy compared to the difficulty of dealing with variables in general. That's what I'm comparing. > Many times that > filesystem isn't even available: "Susan emailed me this module she > uses to fnob large datasets, and I can't understand why it isn't > working for me!" Sure, but errors are always hard to find when you look at a piece of code without access to the relevant context. PEP 328 won't have much if any impact on this problem. Bernhard -- Intevation GmbH http://intevation.de/ Skencil http://sketch.sourceforge.net/ Thuban http://thuban.intevation.org/ From ncoghlan at iinet.net.au Thu May 20 19:02:46 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Thu May 20 19:03:07 2004 Subject: [Python-Dev] Relative vs. absolute imports In-Reply-To: <40A8C132.3090705@egenix.com> References: <40A8C132.3090705@egenix.com> Message-ID: <40AD3916.6070004@iinet.net.au> M.-A. Lemburg wrote: > I don't object absolute imports at all, but I like the > possibilitiy to package'ify existing Python module/module > sets by simply copying them into a newly created package > directory. Doesn't PEP 328 actually make it *easier* to 'packagise' an existing module? If it wasn't in a package before, all of its imports *must* be absolute. With PEP 328, you can copy it into a package, and it will continue to work as before. With the current scheme, I understand that the newly packagised module may break if any if its new sibling modules shadow something in the standard library, site-packages or sys.path. Cheers, Nick. Neologisms ahoy! -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From Mark.Lake at aigfpc.com Thu May 20 13:31:09 2004 From: Mark.Lake at aigfpc.com (Mark.Lake@aigfpc.com) Date: Fri May 21 10:19:27 2004 Subject: [Python-Dev] sets, scanf Message-ID: <75ABDD2D2D569A439D18FFB90DC14945AC46CC@USEVS1.aigfpc.com> Just a couple of modest suggestions for new features. Nothing earth-shattering here but I think these would give a lot of bang for the buck. 1. sscanf functionality: I know this has been discussed for years, but it's time to bite the bullet and do it. 2. Sets are a useful addition to the language, but sorted sets (and dictionaries) would be more useful. C++/Java/C# all provide these types. In particular I'd like to see: A. Dictionaries support union, intersection, and symmetric difference operations as sets do. B. (Sorted) Dictionaries and sets support slicing operations: E.g. dict['a':'z'] should return a sub-dictionary with all the keys between 'a' and 'z'. C. Binary searching for items in sorted sequence and mapping types. D. Ability to handle multiple getitems at once in sequence and mapping types (Akin to what can be done in numarray). Certainly there is no good reason why lists/tuples shouldn't support list[iterable] (not to be confused with list(iterable)) which returns or dictionaries support something like dict.getm(iterable). Mark Lake -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040520/cb63f0a8/attachment.html From rledwith at cas.org Wed May 19 10:06:16 2004 From: rledwith at cas.org (Robert Ledwith) Date: Fri May 21 10:20:13 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: <200405191406.i4JE6GIR020652@mailhost.cas.org> > Armin Rigo wrote: > ... > Unless there are serious objections I suggest to (i.e. I plan to) remove the > short-cut in PyObject_RichCompareBool() -- performance is probably not an > issue here -- and then review all built-in comparison methods and make sure > that they return "equal" for identical objects. I have a serious objection to this because it is a major performance problem. In my Python-2.3.3 code I had implemented an equivalent comparison test because of a performance bottleneck. The bottleneck involved looking up tuples in huge dictionaries of tuples (millions of tuples per dictionary). Putting in a simple equality test reduced the run time of programs by as much as 50%; since each program took days to run, this was a substantial improvement for me. I was considering posting this as a bugfix when I discovered that it had already been incorporated into the CVS. For the sake of people such as myself who use Python to perform large-scale symbolic processing, please consider other approaches to solving the NaN situation before removing this code. :s Robert H. Ledwith INET: rledwith@cas.org :s Unless explicitly stated to the contrary, nothing in this posting should :s be construed as representing my employer's opinions. From seba_woj at interia.pl Fri May 21 10:36:41 2004 From: seba_woj at interia.pl (Sebastian) Date: Fri May 21 10:36:47 2004 Subject: [Python-Dev] SystemError: com_backpatch: offset too large Message-ID: <20040521143641.A721316341@front.interia.pl> I've got a problem with python script. My script return exceptinon: SystemError: com_backpatch: offset too large I'm beginer in python programming, and don't known what do with this message. From andrew-pythondev at puzzling.org Fri May 21 11:02:52 2004 From: andrew-pythondev at puzzling.org (Andrew Bennetts) Date: Fri May 21 11:03:13 2004 Subject: [Python-Dev] sets, scanf In-Reply-To: <75ABDD2D2D569A439D18FFB90DC14945AC46CC@USEVS1.aigfpc.com> References: <75ABDD2D2D569A439D18FFB90DC14945AC46CC@USEVS1.aigfpc.com> Message-ID: <20040521150252.GK22686@frobozz> On Thu, May 20, 2004 at 01:31:09PM -0400, Mark.Lake@aigfpc.com wrote: [...] > > 2. Sets are a useful addition to the language, but sorted sets (and > dictionaries) would be more useful. C++/Java/C# all provide these types. > In particular I'd like to see: > > A. Dictionaries support union, intersection, and symmetric difference > operations as sets do. What's the intersection of {'a': 1} and {'a': 2}? {'a': (1, 2)}? What about of {1.0: 'one point oh'} and {1: 'one'}? Also, this would place a significant burden on people trying to implement their own classes that present a dictionary-like interface, although perhaps UserDict.DictMixin could help there... A PEP for this that's more than a sentence long would probably be a good idea :) [...] > C. Binary searching for items in sorted sequence and mapping types. See the bisect module. I'm not sure what binary searching of a mapping type would mean, though (I guess you mean for your proposed sorted dictionary type?). > D. Ability to handle multiple getitems at once in sequence and mapping > types (Akin to what can be done in numarray). Certainly there is no good > reason why lists/tuples shouldn't support list[iterable] (not to be > confused with list(iterable)) which returns or dictionaries support > something like dict.getm(iterable). Well, one reason is that it places a greater burden on people implementing __getitem__, and it doesn't really offer much improvement over: [l[x] for x in iterable] Or for dictionaries: dict(d[x] for x in iterable) # Assuming 2.4's generator expressions. -Andrew. From john at neggie.net Fri May 21 11:07:28 2004 From: john at neggie.net (John Belmonte) Date: Fri May 21 11:07:39 2004 Subject: [Python-Dev] hash comments in interactive interpreter Message-ID: <40AE1B30.4000806@neggie.net> Hello, I'm wondering what the rationale is for hash comments yielding a continuation prompt (PS2) in the interactive interpreter. I'd like to put such comments in interactive examples, but the output is hard to follow: >>> a = 5 >>> # and now for some multiplication ... >>> a * 10 50 What I would prefer can, ironically, be shown by the interactive interpreter emulator: >>> import code >>> code.interact() Python 2.3.3 (#2, May 1 2004, 06:12:12) [GCC 3.3.3 (Debian 20040401)] on linux2 Type "help", "copyright", "credits" or "license" for more information. (InteractiveConsole) >>> a = 5 >>> # and now for some multiplication >>> a * 10 50 Another apparent inconsistency is that end of line comments don't behave this way: >>> a = 5 >>> a * 10 # end of line comment 50 Is there something preventing this anomaly in the interactive interpreter from being corrected? -John -- http:// ift ile.org/ From mcherm at mcherm.com Fri May 21 11:07:53 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Fri May 21 11:07:56 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: <1085152072.40ae1b490002e@mcherm.com> Armin Rigo writes: > Unless there are serious objections I suggest to (i.e. I plan to) remove the > short-cut in PyObject_RichCompareBool() -- performance is probably not an > issue here -- and then review all built-in comparison methods and make sure > that they return "equal" for identical objects. Robert H. Ledwith writes: > I have a serious objection to this because it is a major performance > problem. Performance is certainly a legitimate concern. But wouldn't your performance problems be fixed just as well if the tuple class implemented "identical-objects-are-immediately-equal" instead of putting it in the general comparison logic for all objects? Because if I understand correctly, that's what Armin is suggesting. Tuples would still be compared just as rapidly. -- Michael Chermside From mwh at python.net Fri May 21 11:11:46 2004 From: mwh at python.net (Michael Hudson) Date: Fri May 21 11:18:53 2004 Subject: [Python-Dev] hash comments in interactive interpreter In-Reply-To: <40AE1B30.4000806@neggie.net> (John Belmonte's message of "Fri, 21 May 2004 11:07:28 -0400") References: <40AE1B30.4000806@neggie.net> Message-ID: <2m65apu85p.fsf@starship.python.net> John Belmonte writes: > Hello, > > I'm wondering what the rationale is for hash comments yielding a > continuation prompt (PS2) in the interactive interpreter. I'd like to > put such comments in interactive examples, but the output is hard to > follow: [...] > Is there something preventing this anomaly in the interactive > interpreter from being corrected? Lack of time and effort? The parser is fairly hairy. Cheers, mwh -- 40. There are two ways to write error-free programs; only the third one works. -- Alan Perlis, http://www.cs.yale.edu/homes/perlis-alan/quotes.html From arigo at tunes.org Fri May 21 11:23:49 2004 From: arigo at tunes.org (Armin Rigo) Date: Fri May 21 11:26:56 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <1085152072.40ae1b490002e@mcherm.com> References: <1085152072.40ae1b490002e@mcherm.com> Message-ID: <20040521152349.GA6340@vicky.ecs.soton.ac.uk> Hello Michael, On Fri, May 21, 2004 at 08:07:53AM -0700, Michael Chermside wrote: > Performance is certainly a legitimate concern. Ok, then the current situation looks like the best one. I suggest we just drop a note in the documentation saying that if there are objects for which x.__eq__(x) isn't necessarily true, you shouldn't expect them to be handled in a fully consistent way. > But wouldn't your > performance problems be fixed just as well if the tuple class implemented > "identical-objects-are-immediately-equal" instead of putting it in the > general comparison logic for all objects? Because if I understand > correctly, that's what Armin is suggesting. Tuples would still be > compared just as rapidly. There is an extra costly indirection until the execution reaches that point. Armin From niemeyer at conectiva.com Fri May 21 13:36:54 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Fri May 21 13:36:39 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040521152349.GA6340@vicky.ecs.soton.ac.uk> References: <1085152072.40ae1b490002e@mcherm.com> <20040521152349.GA6340@vicky.ecs.soton.ac.uk> Message-ID: <20040521173654.GA1170@burma.localdomain> Hello again Armin, > > Performance is certainly a legitimate concern. > > Ok, then the current situation looks like the best one. I suggest we > just drop a note in the documentation saying that if there are objects > for which x.__eq__(x) isn't necessarily true, you shouldn't expect > them to be handled in a fully consistent way. Is it true that the current list comparison algorithm goes trough every element doing identity comparison, and if every element in the first list *is* the element at the same position in the second list, and the list has the same size, then lists are considered equal? If that's true, then I belive there's no reason for not comparing if the first list *is* the second list at the top of list_richcompare(), right? -- Gustavo Niemeyer http://niemeyer.net From tim.one at comcast.net Fri May 21 15:21:09 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri May 21 15:21:22 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040521152349.GA6340@vicky.ecs.soton.ac.uk> Message-ID: [Armin Rigo] > Ok, then the current situation looks like the best one. I suggest we > just drop a note in the documentation saying that if there are objects > for which x.__eq__(x) isn't necessarily true, you shouldn't expect them > to be handled in a fully consistent way. If that's *all* we say, it's not really helpful, is it? Besides, __eq__() isn't even required to return a truth value. If we believe the CPython implementation is correct, then the truth is more that the implementation is allowed to return True for x == y if x is y is true, regardless of what x.__eq__(y), y.__eq__(x), x.__cmp__(y) or y.__cmp__(x) would do if called directly. If that's what we really mean, then better to say that. It then sounds exactly like the cheap-ass micro optimization it is . [Michael Chermside] >> But wouldn't your> performance problems be fixed just as well if the >> tuple class implemented "identical-objects-are-immediately-equal" >> instead of putting it in the general comparison logic for all objects? >> Because if I understand correctly, that's what Armin is suggesting. >> Tuples would still be compared just as rapidly. > There is an extra costly indirection until the execution reaches that > point. Well, gasoline is costly, and so is a Mercedes-Benz. Michael was replying to a fellow worried about factor-of-2 slowdowns. The indirection here isn't in that ballpark of expense. Still, it's not worth any expense unless it's useful. NaN comparison is a red herring not only because Python doesn't have a sane story in any area involving NaNs, but also because the 32 possible 754 comparison predicates can't be mapped onto 6 operator symbols. Note that in C99, the infix C relational operators raise an exception (754's "invalid operation") when applied to a NaN, and C99 adds a pile of new macros for non-exceptional NaN comparison (my favorite is the fetchingly named "islessgreater(x, y)" -- which "is similar to (x) < (y) || (x) > (y); however, islessgreater(x,y) does not raise the invalid exception when x and y are unordered (nor does it evaluate x and y twice)"). In a sane design for handling NaNs, it's important that the "legacy" comparison operators raise an exception in their presence, because code written under the assumption of trichotomy is a bug minefield when unordered NaNs become possible comparands. In short, if Python ever wants "correct" NaN comparison, it needs to add new operators or functions too. From guido at python.org Fri May 21 15:29:51 2004 From: guido at python.org (Guido van Rossum) Date: Fri May 21 15:30:02 2004 Subject: [Python-Dev] hash comments in interactive interpreter In-Reply-To: Your message of "Fri, 21 May 2004 11:07:28 EDT." <40AE1B30.4000806@neggie.net> References: <40AE1B30.4000806@neggie.net> Message-ID: <200405211929.i4LJTpJ22082@guido.python.org> > I'm wondering what the rationale is for hash comments yielding a > continuation prompt (PS2) in the interactive interpreter. I'd like to > put such comments in interactive examples, but the output is hard to follow: > > >>> a = 5 > >>> # and now for some multiplication > ... > >>> a * 10 > 50 It's a bug -- but not easy to fix or it would have been fixed long ago. Probably the lexter could be hacked to fix this, but be careful that a change doesn't botch all the other special cases! --Guido van Rossum (home page: http://www.python.org/~guido/) From casey at zope.com Fri May 21 15:37:08 2004 From: casey at zope.com (Casey Duncan) Date: Fri May 21 15:36:48 2004 Subject: [Python-Dev] Re: cmp(x,x) References: <20040521152349.GA6340@vicky.ecs.soton.ac.uk> Message-ID: <20040521153708.02f1fa1d.casey@zope.com> On Fri, 21 May 2004 15:21:09 -0400 "Tim Peters" wrote: > [Armin Rigo] [..] > NaN comparison is a red herring not only because Python doesn't have a > sane story in any area involving NaNs, but also because the 32 > possible 754 comparison predicates can't be mapped onto 6 operator > symbols. Note that in C99, the infix C relational operators raise an > exception (754's "invalid operation") when applied to a NaN, and C99 > adds a pile of new macros for non-exceptional NaN comparison (my > favorite is the fetchingly named"islessgreater(x, y)" -- which "is > similar to (x) < (y) || (x) > (y); however, islessgreater(x,y) does > not raise the invalid exception when x and y are unordered (nor does > it evaluate x and y twice)"). Finally, a use for <>. Barry will be so happy ;^) -Casey From barry at python.org Fri May 21 15:46:45 2004 From: barry at python.org (Barry Warsaw) Date: Fri May 21 16:01:31 2004 Subject: [Python-Dev] Re: cmp(x,x) In-Reply-To: <20040521153708.02f1fa1d.casey@zope.com> References: <20040521152349.GA6340@vicky.ecs.soton.ac.uk> <20040521153708.02f1fa1d.casey@zope.com> Message-ID: <1085168805.7927.57.camel@localhost> On Fri, 2004-05-21 at 15:37, Casey Duncan wrote: > Finally, a use for <>. Barry will be so happy ;^) And Just, and Fred. We constitute the Pointy Bracket Troika with Fred as our King. all-things-being-unequal-ly y'rs, -Barry From barry at python.org Fri May 21 15:50:21 2004 From: barry at python.org (Barry Warsaw) Date: Fri May 21 16:22:57 2004 Subject: [Python-Dev] Re: cmp(x,x) In-Reply-To: <200405211607.28390.fred@zope.com> References: <20040521152349.GA6340@vicky.ecs.soton.ac.uk> <20040521153708.02f1fa1d.casey@zope.com> <1085168805.7927.57.camel@localhost> <200405211607.28390.fred@zope.com> Message-ID: <1085169021.7927.67.camel@localhost> On Fri, 2004-05-21 at 16:07, Fred Drake wrote: > No, no, no. "<>" is a misapplication of pointy brackets. Note that there's > nothing in the middle; what's the point of bracketing if there's nothing to > bracket? > > The inequality operator should be spelled "!=", mostly so it's not confused > with the structured-bracketing operation. Okay, Fred's out. He gets to rewrite the Windows installer in XML. most-people-are-wrong-ly y'rs, -Barry From tjreedy at udel.edu Fri May 21 18:15:20 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Fri May 21 18:21:41 2004 Subject: [Python-Dev] Re: SystemError: com_backpatch: offset too large References: <20040521143641.A721316341@front.interia.pl> Message-ID: "Sebastian" wrote in message news:20040521143641.A721316341@front.interia.pl... > I've got a problem with python script. > My script return exceptinon: > > SystemError: com_backpatch: offset too large > > I'm beginer in python programming, and don't known what do with this message Pydev mailing list and the gmane mirror are for discussion of future Python. Questions about using current Python should be directed to comp.lang.python or the main Python mailing list. For this question, you need to include more information: the system you are running on; the version of Python you are running; if short, the code you ran. If the code is long, try to cut it down to get something short that gives the same error. At least say what modules you imported. Also, can you rerun with the same inputs and get the same error in the same place, or is it variable. Post that information in the proper place (not here) and you should get answers or at least more suggestions. Terry J. Reedy From fred at zope.com Fri May 21 16:07:28 2004 From: fred at zope.com (Fred Drake) Date: Fri May 21 21:04:50 2004 Subject: [Python-Dev] Re: cmp(x,x) In-Reply-To: <1085168805.7927.57.camel@localhost> References: <20040521152349.GA6340@vicky.ecs.soton.ac.uk> <20040521153708.02f1fa1d.casey@zope.com> <1085168805.7927.57.camel@localhost> Message-ID: <200405211607.28390.fred@zope.com> On Friday 21 May 2004 03:46 pm, Barry Warsaw wrote: > On Fri, 2004-05-21 at 15:37, Casey Duncan wrote: > > Finally, a use for <>. Barry will be so happy ;^) > > And Just, and Fred. No, no, no. "<>" is a misapplication of pointy brackets. Note that there's nothing in the middle; what's the point of bracketing if there's nothing to bracket? The inequality operator should be spelled "!=", mostly so it's not confused with the structured-bracketing operation. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From greg at cosc.canterbury.ac.nz Sun May 23 20:40:21 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Sun May 23 20:40:43 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040521173654.GA1170@burma.localdomain> Message-ID: <200405240040.i4O0eLAr023799@cosc353.cosc.canterbury.ac.nz> Gustavo Niemeyer : > Is it true that the current list comparison algorithm > goes trough every element doing identity comparison, and > if every element in the first list *is* the element at > the same position in the second list, and the list has > the same size, then lists are considered equal? > If that's true, then I belive there's no reason for > not comparing if the first list *is* the second list at > the top of list_richcompare(), right? What if the list contains a NaN? Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From mcherm at mcherm.com Mon May 24 08:50:07 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Mon May 24 08:50:13 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: <1085403007.40b1ef7f255c1@mcherm.com> I wrote: > Performance is certainly a legitimate concern. Armin replied: > Ok, then the current situation looks like the best one. I > suggest we just > drop a note in the documentation saying that if there are > objects for which > x.__eq__(x) isn't necessarily true, you shouldn't expect them > to be handled in > a fully consistent way. [...] > There is an extra costly indirection until the execution > reaches that point. Armin: Performance is a *legitimate* concern, but not an *overwhelming* one. If we are in danger of having to walk over the entirety of a (possibly large) recusive structure (eg: tuple of tuples) to do a comparison, then that's dangerously slow, and it's worth mucking with the definition of equality to avoid it. But a single indirection is "costly" only to someone capable of reaching your level of close-to-the-machine performance understanding. I say, go with your original plan -- the current behavior is a mess. -- Michael Chermside From mcherm at mcherm.com Mon May 24 09:07:19 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Mon May 24 09:07:24 2004 Subject: [Python-Dev] factory functions in the datetime module (& others) Message-ID: <1085404039.40b1f387f1b5f@mcherm.com> Brian Quinlan writes: > It seems to me that the utility of subclassing certain > library classes would > be increased if the user could set a factory that the base > class would use > when creating new instances. [...] > Any thoughts? Hmm... I guess I'm -0 on the idea... it wouldn't really bother me, but I think it's a feature we might as well leave out. I guess my philosophy on this comes down to a fundamental approach to subclassing basic built-in types. Types like int, float, string, datetime, list and tuple are simple, fundamental types built into Python (or its standard libraries). The usual way to use these is by composition -- create a new type that contains fields which are ints, floats, etc. And that is enough to solve nearly all problems -- in fact, up until Python 2.1, that was pretty much the only choice. But Python wants to go the extra mile to give the programmer flexibility, so we've allowed even these basic types to be subclassed by the developer. The question is just how far we should go in supporting (and encouraging) this use. I feel that 99% of all uses of these basic types should be by composition, and only 1% (or less) by inheritance. So while I am in favor of *permitting* the subclassing, I'd be wary of tweaking other parts of the language to support it. I agree that providing customizable factory functions for these would be very helpful to users of fundamental type subclasses, but I think they are (and ought to be) a fairly small minority. As for HOW to do it... a global factory seems like a recipe for encouraging different packages to conflict with each other -- it only works so long as only ONE module tries to install a subtype (and perhaps not even then). A per-object approach is questionable in terms of performance... and it is not acceptable (to me) to slow the performance of code which uses fundamental types by composition (essentially all Python code) for a feature like this. Just my thoughts on the matter... I certainly agree that subclassing fundamental types is less powerful because there is no way to register some sort of factory function, but I'm not sure it's an important-enough feature to be worth the cost of supporting registrable factory functions. -- Michael Chermside From mcherm at mcherm.com Mon May 24 11:32:22 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Mon May 24 11:32:26 2004 Subject: [Python-Dev] RE: cmp(x,x) Message-ID: <1085412742.40b21586b9536@mcherm.com> [...discussion of Armin's idea of moving the short-circuiting of equality tests comparing an object to itself from richcompare (where it affects all objects) to individual object implementations (like tuple)...] Raymond writes: > Actually, it's a ton of indirection (many steps) not just a single C > lookup. Equality testing is one of the most common and most expensive > ops in Python. Okay, I may be mistaken about the performance impact. Can anyone do actual measurements? Given that equality testing is already quite slow, I'd be surprised if there were a noticable performance difference -- but I'm often surprised by performance behavior... show me. > My main concern is that Armin is about to do backflips (inserting new > code into the details of every object's comparison code) just to > facilitate the otherwise unholy goal of having an non-reflexsive > equality comparison. I think that the REAL goal is consistancy in when user-defined equality comparison overrides (__eq__, __ne__, __cmp__) are invoked and when they're not. I find this program surprising: Python 2.3.4c1 (#52, May 12 2004, 19:37:24) [MSC v.1200 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> class NoisyCompare(int): ... def __cmp__(self, other): ... print 'comparing' ... return int.__cmp__(self, other) ... >>> n = NoisyCompare(3) >>> n == n # CASE 1 comparing True >>> n < n # CASE 2 comparing False >>> cmp(n,n) # CASE 3 0 >>> [n] == [n] # CASE 4 comparing True (Actually, I find it surprising for two reasons... because case 3 does NOT print 'comparing', and because case 4 DOES (and Armin said it wouldn't). Allowing NANs to have wierd behavior is a side effect, probably less important than allowing USERS to define their own classes that misbehave. > My main concern is that Armin is about to do backflips (inserting new > code into the details of every object's comparison code) just to > facilitate the otherwise unholy goal of having an non-reflexsive > equality comparison. Now hang on... why is this in EVERY object's code? I would think that list, tuple, and dict need it because they often contain recursive structures. But for most types, why do it at all? > IMO, it is not unreasonable to insist that equality be reflexsive. That > is somewhat basic to the whole notion of equality and certainly reflects > the assumptions made throughtout the code base. No, I don't think it's unreasonable either. But is it necessary? We should try to have a single, simple rule to explain when user-defined operator-overrides of comparison functions are invoked. If the rule is that they are ALWAYS invoked, and that the short-circuiting of identity comparisons (which is certainly necessary for things like tuple and list) is simply part of the implementation of that type, then the rule is VERY simple. If the rule is that the user-defined function is always invoked on differing objects and MAY be invoked if an object is compared to itself, then that's a lot less clean. Anything else I can think of is probably too complex to consider. By the way, Armin introduced this by saying: > A minor semantic change that creeped in some time ago was an implicit > assumption that any object x should "reasonably" be expected to compare > equal to itself. How long ago was that? -- Michael Chermside From arigo at tunes.org Mon May 24 12:28:31 2004 From: arigo at tunes.org (Armin Rigo) Date: Mon May 24 12:31:40 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <200405240040.i4O0eLAr023799@cosc353.cosc.canterbury.ac.nz> References: <20040521173654.GA1170@burma.localdomain> <200405240040.i4O0eLAr023799@cosc353.cosc.canterbury.ac.nz> Message-ID: <20040524162831.GA20681@vicky.ecs.soton.ac.uk> Hello Greg, On Mon, May 24, 2004 at 12:40:21PM +1200, Greg Ewing wrote: > What if the list contains a NaN? Uh, right. The whole issue with NaNs is muddy. Let's focus on Numeric as a much saner example of why you need 'a==a' to return something else than True. Armin From tim.one at comcast.net Mon May 24 12:38:15 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon May 24 12:38:24 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040524162831.GA20681@vicky.ecs.soton.ac.uk> Message-ID: [Greg Ewing] >> What if the list contains a NaN? [Armin Rigo] > Uh, right. The whole issue with NaNs is muddy. Let's focus on Numeric as > a much saner example of why you need 'a==a' to return something else than > True. Yes. Please . From python at rcn.com Mon May 24 16:57:47 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon May 24 16:59:35 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040524162831.GA20681@vicky.ecs.soton.ac.uk> Message-ID: <007b01c441d1$c4d0b900$c327c797@oemcomputer> > On Mon, May 24, 2004 at 12:40:21PM +1200, Greg Ewing wrote: > > What if the list contains a NaN? > > Uh, right. The whole issue with NaNs is muddy. Let's focus on Numeric as > a > much saner example of why you need 'a==a' to return something else than > True. More specifically, find a valid use case where bool(a==a) returns False (because Py_RichCompareBool() still does a boolean coercion on the result of a.__eq__(b)). For the use case to be compelling, it would need to justify breaking things like: mylist.append(x) assert x in mylist I don't believe that you will find sane examples. Identity-implying-equality is a useful and important invariant. Don't give it up easily. Raymond From bob at redivi.com Mon May 24 18:04:32 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon May 24 18:04:45 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <007b01c441d1$c4d0b900$c327c797@oemcomputer> References: <007b01c441d1$c4d0b900$c327c797@oemcomputer> Message-ID: <55C8D724-ADCE-11D8-BDF7-000A95686CD8@redivi.com> On May 24, 2004, at 4:57 PM, Raymond Hettinger wrote: >> On Mon, May 24, 2004 at 12:40:21PM +1200, Greg Ewing wrote: >>> What if the list contains a NaN? >> >> Uh, right. The whole issue with NaNs is muddy. Let's focus on > Numeric as >> a >> much saner example of why you need 'a==a' to return something else > than >> True. > > More specifically, find a valid use case where bool(a==a) returns False > (because Py_RichCompareBool() still does a boolean coercion on the > result of a.__eq__(b)). > > For the use case to be compelling, it would need to justify breaking > things like: > > mylist.append(x) > assert x in mylist > > I don't believe that you will find sane examples. > Identity-implying-equality is a useful and important invariant. Don't > give it up easily. I think the use case mentioned was something like Numeric, where it may be useful for anArray == anotherArray to return an array of bool, not a single True or False. It would also be useful for defining objects that implement some domain specific language in Python.. for example, something like an AppleScript or SQL bridge where you are basically building an AST, compiling it to some other form, and sending it over the wire. It may be convenient if you could do this without actually parsing the code as a string. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2357 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040524/5f9acbf9/smime.bin From raymond.hettinger at verizon.net Mon May 24 19:13:44 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Mon May 24 19:16:00 2004 Subject: [Python-Dev] Standard Library Tour Part II Message-ID: <001101c441e4$c325cd80$a2b1958d@oemcomputer> Comments are invited on the attached draft for a second overview of the standard library. The first overview of basic modules at http://docs.python.org/tut/node12.html was well received and seemed to get people off to a running start. The primary goal of this follow-up is to reduce the transition time for a programmer to start writing real applications using Python. A second goal is give information to people who work through tutorials to determine the capabilities of various languages. A third goal is provide a jumping-off point for someone learning brand new concepts such as weak references or threading. The current docs merely describe the cockpit controls and assume that you already know how to fly an aircraft. Raymond ========================================================= Brief Tour of Modules in the Standard Libary (part II) This second tour covers more advanced modules that support professional programming needs. Their use rarely arises for small scripts. Output Formatting ----------------- The repr module provides an version of repr() for abbreviated displays of large or deeply nested containers:: >>> import repr >>> repr.repr(set('supercalafraglisticexpallidoches')) "set(['a', 'c', 'd', 'e', 'f', 'g', ...])" The pprint module offers more sophisticated control over printing both built-in and user defined objects in a way that is readable by the interpreter. When the result is longer than one line, the "pretty printer" adds line breaks and indentation to more clearly reveal data structure:: >>> import pprint >>> t = [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', ... 'yellow'], 'blue']]] ... >>> pprint.pprint(t, width=30) [[[['black', 'cyan'], 'white', ['green', 'red']], [['magenta', 'yellow'], 'blue']]] The textwrap module formats paragraphs of text to fit a given screen width:: >>> import textwrap >>> doc = """The wrap() method is just like fill() except that it returns ... a list of strings instead of one big string with newlines to separate ... the wrapped lines.""" ... >>> print textwrap.fill(doc, width=40) The wrap() method is just like fill() except that it returns a list of strings instead of one big string with newlines to separate the wrapped lines. The locale module accesses a database of cultural specific data formats. The grouping attribute of locale's format function provides a direct way of formatting numbers with group separators. >>> import locale >>> locale.setlocale(locale.LC_ALL, 'English_United States.1252') 'English_United States.1252' >>> conv = locale.localeconv() # get a mapping of conventions >>> x = 1234567.8 >>> locale.format("%d", x, grouping=True) '1,234,567' >>> locale.format("%s%.*f", (conv['currency_symbol'], ... conv['int_frac_digits'], x), grouping=True) '$1,234,567.80' Working with Binary Data Record Layouts --------------------------------------- The struct module provides pack() and unpack() functions for working with variable length binary record formats. The following example shows how to loop through header information in a ZIP file (with pack codes "H" and "L" representing two and four byte numbers respectively):: import struct data = open('myfile.zip', 'rb').read() start = 0 for i in range(3): # show the first 3 file headers start += 14 fields = struct.unpack('LLLHH', data[start:start+16]) crc32, comp_size, uncomp_size, filenamesize, extra_size = fields start += 16 filename = data[start:start+filenamesize] start += filenamesize extra = data[start:start+extra_size] print filename, hex(crc32), comp_size, uncomp_size start += extra_size + comp_size # skip to the next header Multi-threading --------------- Threading is a technique for decoupling tasks which are not sequentially dependent. Python threads are driven by the operating system and run in a single process and share memory space in a single interpreter. Threads can be used to improve the responsiveness of applications that accept user input while other tasks run in the background. The following code shows how the high level threading module can run tasks in background while the main program continues to run:: import threading, zipfile class AsyncZip(threading.Thread): def __init__(self, infile, outfile): Thread.__init__(self) self.infile = infile self.outfile = outfile def run(self): f = zipfile.ZipFile(self.outfile, 'w', zipfile.ZIP_DEFLATED) f.write(self.infile) f.close() print 'Finished background zip of: ', self.infile AsyncZip('mydata.txt', 'myarchive.zip').start() print 'The main program continues to run' The principal challenge of multi-thread applications is coordinating threads that share data or other resources. To that end, the threading module provides a number of synchronization primitives including locks, events, condition variables, and semaphores. While those tools are powerful, minor design errors can result in problems that are difficult to reproduce. A simpler and more robust approach to task coordination is concentrating all access to a resource in a single thread and then using the Queue module to feed that thread with requests from other threads. Applications that use Queue objects for inter-thread communication and coordination tend to be easier to design, more readable, and more reliable. Logging ------- The logging module offers a full featured and flexible logging system. At its simplest, log messages are sent to a file or to sys.stderr. Other options include routing messages through email, datagrams, sockets, or to an HTTP Server. Messages are assigned priorities including DEBUG, INFO, WARNING, ERROR, and CRITICAL. Filtering allows messages to be directed based on their priority (for example: email only critcal messages, ignore debug messages, and write everything else to a file): [XXX add example] Weak References --------------- Python does automatic memory management (reference counting for most objects and garbage collection to eliminate cycles). The memory is freed shortly after the last reference to it has been eliminated. This approach works fine for most applications but occasionally there is a need to track objects only as long as they are being used by something else. Unfortunately, just tracking them creates a reference that makes them permanent. The weakref module provides tools tracking objects without creating a reference. When the object is no longer needed, it is automatically removed from a weakref table and a callback is triggered for weakref objects. Typically applications include caching objects that are expensive to create: >>> import weakref, gc >>> class A: def __init__(self, value): self.value = value def __repr__(self): return str(self.value) >>> a = A(10) # create a reference >>> d = weakref.WeakValueDictionary() >>> d['primary'] = a # does not create a reference >>> d['primary'] # fetch the object if it is still alive 10 >>> del a # remove the one reference >>> gc.collect() # make garbage collection run right away 0 >>> d['primary'] = a # the entry was automatically removed Traceback (most recent call last): File "", line 1, in -toplevel- d['primary'] = a # does not create a reference NameError: name 'a' is not defined Tools for working with lists ---------------------------- Many data structure needs can be met with the built-in lists. However, sometimes there is a need for alternative implementations with different performance trades-offs. The array module provides an array() object that is like a list that stores only homongenous data but stores it more compactly. The following example shows an array of numbers stored as two byte unsigned binary numbers (typecode "H") rather than the usual 16 bytes per entry for regular lists of python int objects:: >>> from array import array >>> a = array('H', [4000, 10, 700, 22227]) >>> a[1:3] array('H', [10, 700]) The collections module provides a deque() object that is like a list with faster appends and pops from the left side but slower lookups in the middle. These objects are well suited for implementing queues and breadth first tree searches:: >>> from collections import deque >>> d = deque(["task1", "task2", "task3"]) >>> d.append("task4") >>> print "Handling task,", d.popleft() task1 unsearched = deque([initialnode]) def breadth_first_search(unsearched): node = unsearched.popleft() for m in gen_moves(node): if is_goal(m): return m unsearched.append(m) In addition to alternative list implementations, the library also offers other tools such as the bisect module with functions for manipulating sorted lists:: >>> import bisect >>> scores = [(100, 'perl'), (200, 'tcl'), (400, 'lua'), (500, 'python')] >>> bisect.insort(scores, (300, 'ruby')) >>> scores [(100, 'perl'), (200, 'tcl'), (300, 'ruby'), (400, 'lua'), (500, 'python')] The heapq module provides functions for implementing heaps based on regular lists. The lowest valued entry is always kept at position zero. This is useful for applications which repeatedly access the smallest element but do not want to run a full list sort:: >>> from heapq import heapify, heappop, heappush >>> data = [1, 3, 5, 7, 9, 2, 4, 6, 8, 0] >>> heapify(data) # rerrange the list into heap order >>> heappush(data, -5) # add a new entry >>> [heappop(data) for i in range(3)] # fetch the three smallest entries [-5, 0, 1] [??? Should the tempfile module be in the tour ???] --------------------------------------------------- From python at rcn.com Mon May 24 19:52:29 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon May 24 19:54:19 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <55C8D724-ADCE-11D8-BDF7-000A95686CD8@redivi.com> Message-ID: <000701c441ea$2cd31440$a2b1958d@oemcomputer> [Bob Ippolito] > I think the use case mentioned was something like Numeric, where it may > be useful for anArray == anotherArray to return an array of bool, not a > single True or False The code in question is in Py_RichCompareBool() which *always* just returns True or False. That routine is called by list.__contains__() and many other functions that expect a yes or no answer. The regular rich comparison function, Py_RichCompare() is the same as it always was and can still return arrays of bools, complex numbers, or anything at all. IOW, we're still looking for a use case that warrants removing the identity-implies-equality guarantee out of Py_RichCompareBool(). My argument focused on the invariant assumptions that would be broken if this was done. Second, I don't believe that a sane use case will be found for an object x that returns False for "bool(x==x)" and that returns False for "x in [x]". Stretching ones mind to find such an object seems like an exercise in masochism. Raymond From arigo at tunes.org Mon May 24 20:14:25 2004 From: arigo at tunes.org (Armin Rigo) Date: Mon May 24 20:17:29 2004 Subject: [Python-Dev] Standard Library Tour Part II In-Reply-To: <001101c441e4$c325cd80$a2b1958d@oemcomputer> References: <001101c441e4$c325cd80$a2b1958d@oemcomputer> Message-ID: <20040525001425.GA2600@vicky.ecs.soton.ac.uk> Hello Raymond, On Mon, May 24, 2004 at 07:13:44PM -0400, Raymond Hettinger wrote: > >>> del a # remove the one reference > >>> gc.collect() # make garbage collection run right away > 0 > >>> d['primary'] = a # the entry was automatically removed > > Traceback (most recent call last): > File "", line 1, in -toplevel- > d['primary'] = a # does not create a reference > NameError: name 'a' is not defined Oups, wrong line and hence wrong exception for what you're trying to show. A bientot, Armin From python at rcn.com Mon May 24 20:43:09 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon May 24 20:45:01 2004 Subject: [Python-Dev] Standard Library Tour Part II In-Reply-To: <20040525001425.GA2600@vicky.ecs.soton.ac.uk> Message-ID: <001201c441f1$40d8f2a0$a2b1958d@oemcomputer> [ Armin Rigo ] > > d['primary'] = a # does not create a reference > > NameError: name 'a' is not defined > > Oups, wrong line and hence wrong exception for what you're trying to show. Fixed. Thanks, Raymond From anthony at interlink.com.au Tue May 25 11:27:56 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue May 25 11:29:03 2004 Subject: [Python-Dev] 2.3.4 release this thursday Message-ID: <40B365FC.5040709@interlink.com.au> Thomas is back on deck, so I'm aiming to do the 2.3.4 release this Thursday (AU time). Anthony -- Anthony Baxter It's never too late to have a happy childhood. From niemeyer at conectiva.com Tue May 25 11:30:20 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue May 25 11:29:53 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040524163915.10E18472F5@perninha.conectiva.com.br> References: <20040524162831.GA20681@vicky.ecs.soton.ac.uk> <20040524163915.10E18472F5@perninha.conectiva.com.br> Message-ID: <20040525153020.GB4093@burma.localdomain> > [Greg Ewing] > >> What if the list contains a NaN? > > [Armin Rigo] > > Uh, right. The whole issue with NaNs is muddy. Let's focus on Numeric as > > a much saner example of why you need 'a==a' to return something else than > > True. > > Yes. Please . I belive that either my question was misinterpreted, or I'm missing something. I'm not suggesting to limit 'a==a' to return True/False only. I'm suggesting to introduce a shortcut in the *list* comparison (more specifically, in list_richcompare), since this is what is currently being done, but with a more expensive logic. Going through the function, in a case where v is w and op is either Py_EQ or Py_NE, we have 1) Both objects are lists, go on. 2) Both objects have the same size, go on. 3) Searching for the first index where items are different, using PyObject_RichCompareBool(), won't find anything. Go on. 4) No more items to compare. Check sizes. 5) Sizes match. Check comparison operator. 6) Return either Py_True or Py_False. This whole logic might be replaced by simply: if (v == w && (op == Py_EQ || op == Py_NE)) { res = (op == Py_EQ ? Py_True : Py_False); Py_INCREF(res); return res; } -- Gustavo Niemeyer http://niemeyer.net From niemeyer at conectiva.com Tue May 25 11:40:49 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue May 25 11:40:13 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040525153020.GB4093@burma.localdomain> References: <20040524162831.GA20681@vicky.ecs.soton.ac.uk> <20040524163915.10E18472F5@perninha.conectiva.com.br> <20040525153020.GB4093@burma.localdomain> Message-ID: <20040525154049.GC4093@burma.localdomain> > Going through the function, in a case where v is w and op is > either Py_EQ or Py_NE, we have Looking again, this is true for any operation, not only Py_EQ or Py_NE. So the shortcut might be: if (v == w) { if (op == Py_EQ || op == Py_LE || op == Py_GE) res = Py_True; else res = Py_False; Py_INCREF(res); return res; } -- Gustavo Niemeyer http://niemeyer.net From tim.one at comcast.net Tue May 25 12:05:48 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 25 12:06:08 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040525153020.GB4093@burma.localdomain> Message-ID: [Gustavo Niemeyer] > I belive that either my question was misinterpreted, or I'm > missing something. Probably both, but who cares . > I'm not suggesting to limit 'a==a' to return True/False only. I'm > suggesting to introduce a shortcut in the *list* comparison (more > specifically, in list_richcompare), since this is what is > currently being done, but with a more expensive logic. What does "currently" mean to you? 2.3.3 and HEAD differ here. > Going through the function, in a case where v is w and op is > either Py_EQ or Py_NE, we have These cases aren't interesting on HEAD. Nobody calls list_richcompare directly. If something calls PyObject_RichCompareBool() with Py_EQ or Py_NE, then on HEAD the result is returned instantly, because of this code at the start of PyObject_RichCompareBool(): /* Quick result when objects are the same. Guarantees that identity implies equality. */ if (v == w) { if (op == Py_EQ) return 1; else if (op == Py_NE) return 0; } That code doesn't care whether v and w are lists or ints or Unicode strings or floats (etc). 2.3.3 doesn't have this code. > 1) Both objects are lists, go on. On HEAD, we don't get into list_richcompare. > 2) Both objects have the same size, go on. > 3) Searching for the first index where items are different, using > PyObject_RichCompareBool(), won't find anything. Go on. If v and w were created via, e.g., v = [] v.append(v) w = v then step #3 is a lot more complicated in 2.3.3 (because we recurse into list_richcompare repeatedly then, and a bunch of gone-on-HEAD machinery tries to notice that and guess the right result). > 4) No more items to compare. Check sizes. > 5) Sizes match. Check comparison operator. > 6) Return either Py_True or Py_False. > > This whole logic might be replaced by simply: > > if (v == w && (op == Py_EQ || op == Py_NE)) { > res = (op == Py_EQ ? Py_True : Py_False); > Py_INCREF(res); > return res; > } On HEAD, it has been replaced, but for all object types with one blow. From ark-mlist at att.net Tue May 25 12:07:09 2004 From: ark-mlist at att.net (Andrew Koenig) Date: Tue May 25 12:07:10 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <000701c441ea$2cd31440$a2b1958d@oemcomputer> Message-ID: <006701c44272$554e5b40$6602a8c0@arkdesktop> > IOW, we're still looking for a use case that warrants removing the > identity-implies-equality guarantee out of Py_RichCompareBool(). I can think of two use cases: 1) IEEE-like behavior. Before you say that there is nothing wrong with making floating-point comparison yield true for NaN==NaN, let me remind you of the beast called "Signaling NaN", which should raise an exception on any operation, including comparison. 2) A class might have comparison operations that put trace information in a log file for testing purposes. Such a class might be intended to help in testing searching algorithms, for example. I think it would be useful to be able to know whether x[i] is ever compared with x[i] by putting appropriate tracing comparison operations in the elements of x. From fdrake at acm.org Tue May 25 12:12:43 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue May 25 12:12:53 2004 Subject: [Python-Dev] Re: [Python-checkins] python/dist/src/Doc/tut tut.tex, 1.228, 1.229 In-Reply-To: References: Message-ID: <200405251212.43580.fdrake@acm.org> On Tuesday 25 May 2004 12:08 pm, rhettinger@users.sourceforge.net wrote: > Modified Files: > tut.tex > Log Message: > Follow official recommendation for domain names in examples. Yay! Thanks, Raymond! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jim.jewett at eds.com Tue May 25 12:30:38 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Tue May 25 12:31:05 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: Gustavo Niemeyer: > Looking again, this is true for any operation, not only > Py_EQ or Py_NE. CVS has previously restricted the shortcut to those two, but I agree it should be all or nothing. Even unordered objects normally compare equal to themselves; if "is" does not imply "__eq__", then we are by definition dealing with an exceptional case. (NaN, or SQL NULL, or DUMMY, or ...) Since different exceptional cases will have different rules, it might be reasonable to ask the creator to use a custom comparison function. They will often have to do this anyway. Instead of if (a == b): they can write if (a is not NULL) and (a == b): or if (a == b) and (not isNaN(a)): -jJ From aahz at pythoncraft.com Tue May 25 12:32:52 2004 From: aahz at pythoncraft.com (Aahz) Date: Tue May 25 12:32:55 2004 Subject: [Python-Dev] 2.3.4 release this thursday In-Reply-To: <40B365FC.5040709@interlink.com.au> References: <40B365FC.5040709@interlink.com.au> Message-ID: <20040525163252.GA9862@panix.com> On Wed, May 26, 2004, Anthony Baxter wrote: > > Thomas is back on deck, so I'm aiming to do the 2.3.4 release > this Thursday (AU time). I'm not objecting, precisely, but this weekend is Memorial Day weekend in the USA, so there will be fewer resources available if anything goes wrong. -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "He's Quebecois. He puts mayonnaise on *everything*." --siobhan@virulent.org From niemeyer at conectiva.com Tue May 25 12:50:14 2004 From: niemeyer at conectiva.com (Gustavo Niemeyer) Date: Tue May 25 12:49:38 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: References: <20040525153020.GB4093@burma.localdomain> Message-ID: <20040525165014.GD4093@burma.localdomain> > > I'm not suggesting to limit 'a==a' to return True/False only. I'm > > suggesting to introduce a shortcut in the *list* comparison (more > > specifically, in list_richcompare), since this is what is > > currently being done, but with a more expensive logic. > > What does "currently" mean to you? 2.3.3 and HEAD differ here. I meant HEAD. > > Going through the function, in a case where v is w and op is > > either Py_EQ or Py_NE, we have > > These cases aren't interesting on HEAD. Nobody calls list_richcompare > directly. If something calls PyObject_RichCompareBool() with Py_EQ or > Py_NE, then on HEAD the result is returned instantly, because of this > code at the start of PyObject_RichCompareBool(): I'm wondering what is causing this confusion then: % ./python Python 2.4a0 (#5, May 25 2004, 13:45:26) [GCC 3.3.3 (release)] on linux2 Type "help", "copyright", "credits" or "license" for more information. [...] >>> l = [] >>> l == l Got in list_richcompare()! True >>> if l == l: print "foo" ... Got in list_richcompare()! foo -- Gustavo Niemeyer http://niemeyer.net From tim.one at comcast.net Tue May 25 12:58:07 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 25 12:58:13 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: Message-ID: [Gustavo Niemeyer] >> Looking again, this is true for any operation, not only >> Py_EQ or Py_NE. [Jim J Jewett] > CVS has previously restricted the shortcut to those two, > but I agree it should be all or nothing. It could for lists specifically, but not for objects in general. The rich comparison PEP explicitly promised that, for example, x <= y is not necessarily the same as "x < y or x == y": from x == y, we're not allowed to infer that x <= y, or x >= y, or not x < y, or not x > y. For that matter, the PEP http://www.python.org/peps/pep-0207.html also promised 3 The == and != operators are not assumed to be each other's complement so it's also not kosher to infer x != y is False from that x == y is True. Any shortcuts based on object identity are optimizations, and dubious ones because they violate promises made in that PEP. There's a strong pragmatic ("go fast") case for starting to infer "x == y" and "not x != y" from "x is y", but that doesn't extend to inferring others from "x is y" (PyObject_RichCompareBool gets invoked with Py_EQ and Py_NE all over the place in the core; it's never invoked with Py_GT, Py_GE, or Py_LE; there are a few invocations with Py_LT, mostly involved with sorting). From mcherm at mcherm.com Tue May 25 13:07:31 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Tue May 25 13:07:34 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: <1085504851.40b37d5322d1a@mcherm.com> Raymond writes: > The code in question is in Py_RichCompareBool() which *always* just > returns True or False. That routine is called by list.__contains__() > and many other functions that expect a yes or no answer. > > The regular rich comparison function, Py_RichCompare() is the same as it > always was and can still return arrays of bools, complex numbers, or > anything at all. All right, I took a look at exactly where PyObject_RichCompareBool is called. Here is a COMPLETE list of all uses in 2.3.3 (that's what I had lying around): * In listobject.c, to test containment in a list (ie: "x in [x]"). * In tupleobject.c to perform containment tests on tuples (ie: "x in (a,b,c)"). * In arraymodule.c, in array_contains to test containment in an array. * In listobject.c, in listindex(), listremove(), and listcount() to find the first occurance, delete the first occurance, or count the number of occurances of an object in the list. * In arraymodule.c, in array_index(), array_remove(), and array_count() to find the first occurance, delete the first occurance, or count the number of occurances of an object in the array. * In abstract.c, in _PySequence_IterSearch() to test containment of, find the first occurance of, or count the occurances of some object. * In listobject.c, in list_richcompare(), to skip past identical list elements when comparing lists. * In tupleobject.c, in tuplerichcompare() to skip past identical tuple elements when comparing tuples. * In arraymodule.c, in array_richcompare, to skip past identical array elements when comparing arrays. * In listobject.c, used in sorting if the user does not provide a user-defined comparison function. * In bltinmodule.c, in min_max(), to compare two objects within the min and max functions. * In iterobject.c, to test whether an object is the sentinal object. * In dictobject.c, in lookdict(), to test whether an key in the dict matches the key being looked up. * In dictobject.c, in characterize() ... I'm not quite sure what this is doing. * In dictobject.c when comparing two dicts for equality. * In typeobject.c, in method_is_overloaded, to test whether the methods defined in a class and some subclass are actually the same object. * In ceval.c, for matching keyword arguments to code object varnames in PyEval_EvalCodeEx. * In ceval.c in _PyEval_SliceIndex() to compare a bound to the number 0L. * And finally, it is part of the published API so it could appear anywhere in extension modules... but I would guess that people use PyObject_RichCompare normally and only call PyObject_RichCompareBool if they really want a boolean. What I learned by writing this list is that Raymond is right... being able to assume that "x is x" implies "x == x" is very useful for implementors. And if that assumption is made in PyObject_RichCompareBool() and NOT in PyObject_RichCompare(), then it hardly ever needs to conflict with the desire for user-defined comparison functions to return peculiar things (like Numeric arrays, values that are artifically not equal (like NANs or a poorly-designed user-implemented Max object). Okay, I know this is too much analysis, but I've started now, so I'm going to go through with it and send off this email. Here's my thought: Raymond has convinced me that we want to make this assumption in the interpreter. I want the rules for when a user-defined comparison function (__cmp__, __eq__, or __ne__) is invoked and when it isn't. Maybe we can achieve both. Here's that list above, grouped by function: 1. Test Containment in a sequence, also index(), count(), and remove() on sequences. 2. Comparing sequences or dicts to each other. 3. Sorting lists when no user-defined comparison function is given. 4. in min() and max() 5. checking for sentinal in iteration 6. looking up dict keys 7. characterize() in dictobject (what's this for?) 8. Checking if a method is the same to tell if it's overloaded. 9. Matching keyword arguments to code object varnames in EvalCode 10. Compare a slice bound to 0L. 11. Extension modules that call PyObject_RichCompareBool instead of PyObject_RichCompare. This is the list of all places where RichCompareBool is called instead of RichCompare, and thus of all places where a user-defined comparison function might (surprisingly) not be called. Some are not relevent (eg: #9 compares only strings, a built-in type). For some, I can concoct artifical examples where someone would care (eg: #5: creating a sentinal for iter() that tries to be so VERY clever that it allows the sentinal object itself to occur in the sequence sometimes without stopping the iteration; or #3: trying to understand the behavior of timsort by creating objects that log comparisons rather than by reading the code). But these feel artificial to me. The big ones seem to be #1 and #2. I'd even say that it makes "intuitive sense" to me somehow that containment, index(), count() and remove() all act as if identity implied equality (although I can think of only *slightly* absurd examples where a user might try to alter this behavior. But when comparing sequences or dicts to each other (something people do a LOT), I would intuitively expect that any contained objects with customized comparison methods would have those methods invoked. Okay... I've talked myself into a corner now. Raymond has convinced me that my original idea was misguided, and I've looked closely at the problem, but I don't see an "obvious" solution. I'm tending to think it's best to put the test for identity in PyObject_RichCompareBool, but then how do we explain (in simple terms) when user-defined comparison methods are invoked and when they're not necessarily? Well, even without an answer, I'd better send this email off and get back to my own work. I'm not sure if I've gotten anywhere with this or not. -- Michael Chermside From tim.one at comcast.net Tue May 25 13:28:52 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 25 13:28:59 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: <20040525165014.GD4093@burma.localdomain> Message-ID: [Gustavo Niemeyer] > I'm wondering what is causing this confusion then: > > % ./python > Python 2.4a0 (#5, May 25 2004, 13:45:26) > [GCC 3.3.3 (release)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > [...] > >>> l = [] > >>> l == l > Got in list_richcompare()! > True > >>> if l == l: print "foo" > ... > Got in list_richcompare()! > foo My mistake! And my apologies. PyObject_RichCompare() (not ...Bool()) gets called in this case, and it can't infer anything from object identity alone. The list object may, but it's quite possible that trying to would cost more cycles than it saved. From jim.jewett at eds.com Tue May 25 13:58:52 2004 From: jim.jewett at eds.com (Jewett, Jim J) Date: Tue May 25 14:01:09 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: [Jim J Jewett] >> CVS has previously restricted the shortcut to those two, >> but I agree it should be all or nothing. [Tim Peters] > The rich comparison PEP explicitly promised that, for example, > x <= y is not necessarily the same as "x < y or x == y": > ...it's also not kosher to infer x != y is False from > that x == y is True. With consistent orderings, "x <= y" is strictly weaker than "x < y or x == y"; they differ only when a pair of objects don't know *which* of "<" and "==" is true. If we do not assume consistency, then "is" does not even imply "==", let alone "not !=" > Any shortcuts based on object identity are optimizations, > and dubious ones Agreed. > because they violate promises made in that PEP. PEP 207 http://www.python.org/peps/pep-0207.html said that you can't (generally) reason from one rich compare to another. It said nothing about reasoning from "is" to any of the comparisons directly. So this optimization may violate the spirit of __eq__, but it doesn't violate the letter of the PEP. > There's a strong pragmatic ("go fast") case for starting to > infer "x == y" and "not x != y" from "x is y", but that > doesn't extend to inferring others from "x is y" For normal objects, consistency holds. When consistency does not hold, even Py_EQ and Py_NE are wrong. Doing (only) the other operations correctly won't fix the whole program; it will just make the bugs less obvious (and therefore harder to catch). -jJ From jim.jewett at EDS.COM Tue May 25 15:42:50 2004 From: jim.jewett at EDS.COM (Jewett, Jim J) Date: Tue May 25 15:44:07 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: Michael Chermside: [Where PyObject_RichCompareBool is called, and the is shortcut might be a problem] Most of these operations -- including the container comparison -- are not well-defined if "is" does not imply equality. > I would intuitively expect that any contained objects > with customized comparison methods would have those methods > invoked. Yes, but you couldn't reliably predict the order they would be compared (I *choose* to start with the least significant! Oh yeah? Well I timsort with much galloping!), so inconsistent results are not guaranteed. The real problem is that you won't get the expected side effects. > I'm tending to think it's best to put the test for identity in > PyObject_RichCompareBool, but then how do we explain (in simple > terms) when user-defined comparison methods are invoked and when > they're not necessarily? Say that Python only promises to make the comparisons you explicitly ask for. Implicit comparisons (including cmp, sort, and comparisons between subobjects) may be optimized away. For example: a < b will compare a and b, but: [1, a] < [2, b] or even: [a] < [b] might not. -jJ From python at rcn.com Tue May 25 15:57:28 2004 From: python at rcn.com (Raymond Hettinger) Date: Tue May 25 15:59:26 2004 Subject: [Python-Dev] cmp(x,x) In-Reply-To: Message-ID: <002401c44292$824d9f00$2d24a044@oemcomputer> Forgive me, but perhaps a Letterman style top ten list may be a better way to summarize what I've heard so far. Proposal: Remove 8 lines of code discussed and approved by Raymond, Tim, Guido, and Armin at PyCon. Replicate those lines and push them down in every object known to man. Advantages: 10) Facilitating insane use cases 9) Increasing the size of code base 8) Avoiding that silly concept of code factoring 7) Reducing performance 6) Breaking foolish invariants like "[x]==[x] and x in [x]" 5) Tricking myself into thinking that some sort of consistency has been achieved though some existing calls to Py_RichCompareBool() already have their own identity checks slightly upstream of the call. . . . And the number one reason for reverting good code is: It feels so groovy to call something a hack! Hopefully you find this funny and use it help get things back on course. With so many contrarians the mailing list, it's nearly impossible to get agreement on the simplest things. It's amazing how much support there is for something with no practical use cases. Raymond From python-dev at richardcooper.net Tue May 25 19:02:11 2004 From: python-dev at richardcooper.net (Richard Cooper) Date: Tue May 25 19:02:01 2004 Subject: [Python-Dev] Re: .pth files in current or script directory In-Reply-To: Message-ID: <8E0CCF07-AE9F-11D8-B69E-000A27E3999C@richardcooper.net> From: Richard Cooper Date: Wed May 19, 2004 1:34:12 pm Europe/London To: python-dev@python.org Subject: Re: .pth files in current or script directory Seems this message got lost somewhere. So subscribing and resending. Greg Ewing wrote: > Richard Cooper : > > > When a module is imported, its import search path includes everything it > > does now PLUS the directory that is the top of the module's enclosing > > package structure. > > I don't see how that changes anything, since in order to get > imported in the first place, the module's top-level package has > to already be on the module search path somehow. Sorry, slip of the brain. I meant "When a module is executed ..." Aahz wrote: > What's wrong with PEP 328? Nothing, I think it's great. However it talks about making import statement absolute by default (searching sys.path only) without worrying about how a package gets on sys.path in the first place. Currently a module can't import (absolutely) from it's own package unless: 1) Its package is in one of a few special places (like site-packages) 2) The package is in PYTHONPATH 3) The module does some sys.path tinkering before the import None of these are great solutions. My initial problem was that I picked the PYTHONPATH solution which (as you all know) turns out to be a real pain when working on cvs branches. If the top of the containing package got added to sys.path when a module is executed then importing (absolutely) from your own package would (I think) just work. And given that PEP 328 is going to encourage absolute imports and is going to be changing import semantics anyway, I think this would be a nice feature to have. Rich From daniels at db-x.com Tue May 25 19:26:07 2004 From: daniels at db-x.com (Scott David Daniels) Date: Tue May 25 19:30:23 2004 Subject: [Python-Dev] PEP 320 Message-ID: In : ... Planned features for 2.4 ... PEP 328: Imports: Multi-line and Absolute/Relative. (Possibly the absolute/relative stuff is too controversial and will be pushed off to 2.4.) ... Pushed off to 2.4 is not the best possible description for a feature list for 2.4 :-) -Scott David Daniels Scott.Daniels@Acm.Org From fdrake at acm.org Tue May 25 20:11:29 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Tue May 25 20:11:55 2004 Subject: [Python-Dev] PEP 320 In-Reply-To: References: Message-ID: <200405252011.29777.fdrake@acm.org> On Tuesday 25 May 2004 07:26 pm, Scott David Daniels wrote: > Pushed off to 2.4 is not the best possible description for a feature > list for 2.4 :-) I've fixed this in the PEP sources; the change should be reflected on the site sometime in the next few hours. Thanks! -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From anthony at interlink.com.au Tue May 25 22:08:25 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Tue May 25 22:08:36 2004 Subject: [Python-Dev] 2.3.4 release this thursday In-Reply-To: <20040525163252.GA9862@panix.com> References: <40B365FC.5040709@interlink.com.au> <20040525163252.GA9862@panix.com> Message-ID: <40B3FC19.2070400@interlink.com.au> Aahz wrote: > On Wed, May 26, 2004, Anthony Baxter wrote: > >>Thomas is back on deck, so I'm aiming to do the 2.3.4 release >>this Thursday (AU time). > > > I'm not objecting, precisely, but this weekend is Memorial Day weekend > in the USA, so there will be fewer resources available if anything goes > wrong. There's been zero bugs reported in 2.3.4. The only american involved in the release process is Fred, and he's already cut the doc tarball, so I don't think this is a problem. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From tim.one at comcast.net Tue May 25 22:32:12 2004 From: tim.one at comcast.net (Tim Peters) Date: Tue May 25 22:32:22 2004 Subject: [Python-Dev] 2.3.4 release this thursday In-Reply-To: <40B3FC19.2070400@interlink.com.au> Message-ID: [Anthony] >>> Thomas is back on deck, so I'm aiming to do the 2.3.4 release >>> this Thursday (AU time). [Aahz] >> I'm not objecting, precisely, but this weekend is Memorial Day weekend >> in the USA, so there will be fewer resources available if anything goes >> wrong. [Anthony] > There's been zero bugs reported in 2.3.4. The only american involved > in the release process is Fred, and he's already cut the doc tarball, > so I don't think this is a problem. I agree. If it comes to it, I'll be around all weekend anyway, and I'm every damned bit as American as Fred. In fact, more so, since I avoid eating both cheese and mayonnaise, and am not named after a duck. manifest-destiny-ly y'rs - tim From rwgk at cci.lbl.gov Wed May 26 00:31:03 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Wed May 26 10:13:14 2004 Subject: [Python-Dev] Patch for long-standing bug in socket module Message-ID: <200405260431.i4Q4V32m195543@boa.lbl.gov> On May 22 I submitted a patch to fix this long-standing bug: https://sourceforge.net/tracker/?func=detail&aid=728330&group_id=5470&atid=105470 My hope was that this fix will make it into Python 2.3.4 final but so far nobody is paying attention. Could someone please have a look and let me know if there are any concerns? As it stands Python as distributed is near useless under SGI's IRIX because the socket module doesn't compile. My patch fixes this problem. The danger of introducing undesirable side effects is minimal because all changed are surrounded by very specific #ifdef's. I've tested the patch under three different OS levels with three different compiler versions. Thanks, Ralf From anthony at interlink.com.au Wed May 26 11:39:31 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Wed May 26 11:41:11 2004 Subject: [Python-Dev] Patch for long-standing bug in socket module In-Reply-To: <200405260431.i4Q4V32m195543@boa.lbl.gov> References: <200405260431.i4Q4V32m195543@boa.lbl.gov> Message-ID: <40B4BA33.1010706@interlink.com.au> Ralf W. Grosse-Kunstleve wrote: > On May 22 I submitted a patch to fix this long-standing bug: > > https://sourceforge.net/tracker/?func=detail&aid=728330&group_id=5470&atid=105470 > > My hope was that this fix will make it into Python 2.3.4 final > but so far nobody is paying attention. The patch looks fine, but it won't be in 2.3.4. Between the release candidate and the final release, only critical (usually packaging) bugs are fixed. Anthony -- Anthony Baxter It's never too late to have a happy childhood. From rwgk at cci.lbl.gov Wed May 26 12:02:36 2004 From: rwgk at cci.lbl.gov (Ralf W. Grosse-Kunstleve) Date: Wed May 26 12:02:44 2004 Subject: [Python-Dev] Patch for long-standing bug in socket module Message-ID: <200405261602.i4QG2a4X202099@boa.lbl.gov> In my opinion this is a critical bug. For people in science it is very sad that Python 2.3 has this problem under IRIX. From tjreedy at udel.edu Wed May 26 12:13:25 2004 From: tjreedy at udel.edu (Terry Reedy) Date: Wed May 26 12:13:35 2004 Subject: [Python-Dev] Re: Patch for long-standing bug in socket module References: <200405260431.i4Q4V32m195543@boa.lbl.gov> Message-ID: "Ralf W. Grosse-Kunstleve" wrote in message news:200405260431.i4Q4V32m195543@boa.lbl.gov... > On May 22 I submitted a patch to fix this long-standing bug: > > https://sourceforge.net/tracker/?func=detail&aid=728330&group_id=5470&atid=105470 > > My hope was that this fix will make it into Python 2.3.4 final\\ As AB explained, that was already too late. > Python as distributed is near useless under SGI's IRIX because > the socket module doesn't compile. My patch fixes this problem. > The danger of introducing undesirable side effects is minimal > because all changed are surrounded by very specific #ifdef's. > I've tested the patch under three different OS levels with > three different compiler versions. Perhaps you can make available the 2.3.4 source with this patch and/or your set of IRIX binaries (with the usual no-guarantee license). Then ask for a listing on http://www.python.org/download/download_other.html Terry J. Reedy From mcherm at mcherm.com Wed May 26 13:13:34 2004 From: mcherm at mcherm.com (Michael Chermside) Date: Wed May 26 13:13:39 2004 Subject: [Python-Dev] cmp(x,x) Message-ID: <1085591614.40b4d03e5521d@mcherm.com> Raymond: > Forgive me, but perhaps a Letterman style top ten list may be a better > way to summarize what I've heard so far. *grin* Nice. And in case I wasn't clear before... you're right about what to do with the code, I'm only wondering about how to cleanly and clearly tell users when calls to their user-defined __eq__ might not occur. -- Michael Chermside From guido at python.org Wed May 26 13:19:59 2004 From: guido at python.org (Guido van Rossum) Date: Wed May 26 13:20:12 2004 Subject: [Python-Dev] Patch for long-standing bug in socket module In-Reply-To: Your message of "Wed, 26 May 2004 09:02:36 PDT." <200405261602.i4QG2a4X202099@boa.lbl.gov> References: <200405261602.i4QG2a4X202099@boa.lbl.gov> Message-ID: <200405261720.i4QHJxN04843@guido.python.org> > In my opinion this is a critical bug. For people in science it is > very sad that Python 2.3 has this problem under IRIX. There's always 2.3.5. --Guido van Rossum (home page: http://www.python.org/~guido/) From python at rcn.com Wed May 26 20:29:11 2004 From: python at rcn.com (Raymond Hettinger) Date: Wed May 26 20:31:43 2004 Subject: FW: [Python-Dev] Standard Library Tour Part II Message-ID: <000e01c44381$bade75c0$9cb0958d@oemcomputer> Fred, The current library tour in the tutorial is stored at: http://docs.python.org/tut/node12.html Each of the chapters is consecutively numbered, so if a new chapter is inserted, hard links to a section of the documentation will break. Is there away to have the nodes named rather than numbered. Aahz likes to point directly at the floating point Appendix and is mess up his day when I add to the tutorial. Raymond [Aahz] > I hope that this next round of updates doesn't break links > to the floating point appendix the way the last one did. From barry at python.org Thu May 27 00:14:56 2004 From: barry at python.org (Bwarsaw) Date: Thu May 27 00:16:11 2004 Subject: [Python-Dev] Forum notify Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040527/7ab933cb/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Details.cpl Type: application/octet-stream Size: 0 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040527/7ab933cb/Details.obj From tim.one at comcast.net Thu May 27 01:43:50 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu May 27 01:44:03 2004 Subject: [Python-Dev] Forum notify In-Reply-To: Message-ID: Given that it's Barry claiming to post this particular malware, we have to suspect he really did. OTOH, I've been getting a lot of this crap posted directly to me the past few days, claiming to come from various people on python-dev, and Barry admittedly doesn't usually send email from Kenya (in my experience, he usually sends it from Korea). So who pissed off a script kiddie? takes-one-to-know-one-ly y'rs - tim From anthony at interlink.com.au Thu May 27 02:48:19 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu May 27 02:48:44 2004 Subject: [Python-Dev] pre-announcing 2.4 Message-ID: <40B58F33.4050203@interlink.com.au> So this is the email that I'm planning to send to python-announce and python-list/c.l.py shortly. Anyone have any changes/additions they'd like to see? This will become a standard part of the release process for future releases... """ This is a pre-announcement that the first alpha of Python 2.4 is about a month away. The purpose of this notice is to give people a heads up - if you have a bug that you want to see fixed for 2.4, start looking at it now. Fixes will be accepted all through the release process - although once we get to the first beta, bugs that change behaviour will have a harder path. So, if you have a bug you want to see fixed, what should you do? If it's not logged on SF, log it. If it's logged, consider adding a patch that fixes the problem, or at least a simple test case that demonstrates the bug. If someone else has supplied a fix, see if this fix works for you, and post your results to the bug. If there's a working fix, feel free to add a note asking for the fix to go into CVS. The SF bug tracker for Python has a _lot_ of bugs in it, and it's easy for bugs to be overlooked. On behalf of the python-dev team, thanks! Anthony """ -- Anthony Baxter It's never too late to have a happy childhood. From anthony at interlink.com.au Thu May 27 06:58:10 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu May 27 06:58:49 2004 Subject: [Python-Dev] RELEASED Python 2.3.4 (final) Message-ID: <40B5C9C2.30403@interlink.com.au> -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On behalf of the Python development team and the Python community, I'm happy to announce the release of Python 2.3.4 (final). Python 2.3.4 is a bug-fix release. See the release notes at the website (also available as Misc/NEWS in the source distribution) for details of the bugs squished in this release. For more information on Python 2.3.4, including download links for various platforms, release notes, and known issues, please see: ~ http://www.python.org/2.3.4 Highlights of this new release include: ~ - Bug fixes. According to the release notes, more than 20 bugs ~ have been fixed, including a couple of bugs that could cause ~ Python to crash. These were discovered by Zope3. Highlights of the previous major Python release (2.3) are available from the Python 2.3 page, at ~ http://www.python.org/2.3/highlights.html Enjoy the new release, Anthony Anthony Baxter anthony@python.org Python Release Manager (on behalf of the entire python-dev team) - -- Anthony Baxter -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.3 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFAtcm4Dt3F8mpFyBYRAmgmAJ41U7s98dnQjOOD5XAqqPBJEXvRZwCcCm6x HQQ0vky/iPxSagROZk9Uk8M= =W9rA -----END PGP SIGNATURE----- From mchermside at ingdirect.com Thu May 27 08:02:50 2004 From: mchermside at ingdirect.com (Chermside, Michael) Date: Thu May 27 08:02:54 2004 Subject: [Python-Dev] pre-announcing 2.4 Message-ID: <0CFFADBB825C6249A26FDF11C1772AE1550A1E@ingdexj1.ingdirect.com> > Fixes will be accepted > all through the release process - although once we get to the > first beta, bugs that change behaviour will have a harder path. I'd propose one change. Weaken the above statement. People will take that to mean that THEIR fix will be accepted ANYWHERE in the release process (including the day before the release). I'd suggest something more like this: Fixes are welcomed up through the alpha release, but as we get closer to the release date we will be less and less likely to accept patches that change behavior. Maybe that's too harsh, but I think your original wording is inviting complaints later on. -- Michael Chermside This email may contain confidential or privileged information. If you believe you have received the message in error, please notify the sender and delete the message without copying or disclosing it. From anthony at interlink.com.au Thu May 27 08:18:20 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Thu May 27 08:18:39 2004 Subject: [Python-Dev] pre-announcing 2.4 In-Reply-To: <0CFFADBB825C6249A26FDF11C1772AE1550A1E@ingdexj1.ingdirect.com> References: <0CFFADBB825C6249A26FDF11C1772AE1550A1E@ingdexj1.ingdirect.com> Message-ID: <40B5DC8C.3090600@interlink.com.au> Chermside, Michael wrote: > I'd propose one change. Weaken the above statement. People will > take that to mean that THEIR fix will be accepted ANYWHERE in > the release process (including the day before the release). I'd > suggest something more like this: > > Fixes are welcomed up through the alpha release, but as we > get closer to the release date we will be less and less > likely to accept patches that change behavior. Good point, although this hides the point about the first beta being the cut-off point for behaviour changes. How about Fixes are welcome through the release cycle, although after the first beta fixes that result in a change to behaviour will be much less likely to be accepted. Once I have this message nailed down, I'll produce a similar one for the bug fix releases. -- Anthony Baxter It's never too late to have a happy childhood. From barry at python.org Thu May 27 09:55:34 2004 From: barry at python.org (Bwarsaw) Date: Thu May 27 09:56:51 2004 Subject: [Python-Dev] Re: Hello Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040527/c50c71a9/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Alive_condom.cpl Type: application/octet-stream Size: 0 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040527/c50c71a9/Alive_condom.obj From barry at python.org Thu May 27 10:26:31 2004 From: barry at python.org (Barry Warsaw) Date: Thu May 27 10:27:06 2004 Subject: [Python-Dev] Re: Hello In-Reply-To: References: Message-ID: <1085667991.4051.8.camel@localhost> On Thu, 2004-05-27 at 09:55, Bwarsaw wrote: > > > ______________________________________________________________________ > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: http://mail.python.org/mailman/options/python-dev/barry%40python.org Crap. We're probably seeing the leading edge of a spam technique we've long suspected could happen. I'll be signing my posts from now on.... -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 307 bytes Desc: This is a digitally signed message part Url : http://mail.python.org/pipermail/python-dev/attachments/20040527/228fdc15/attachment.bin From brian at sweetapp.com Thu May 27 11:21:50 2004 From: brian at sweetapp.com (Brian Quinlan) Date: Thu May 27 11:20:47 2004 Subject: [Python-Dev] Re: Hello In-Reply-To: <1085667991.4051.8.camel@localhost> References: <1085667991.4051.8.camel@localhost> Message-ID: <40B6078E.2060700@sweetapp.com> Barry Warsaw wrote: > Crap. We're probably seeing the leading edge of a spam technique we've > long suspected could happen. I'll be signing my posts from now on.... Actually, someone impersonating you has been sending stuff to my personal address for a couple of weeks now. I'm actually interested in the technique: do they harvest all the names on the list and then randomly forge e-mails from one user to another? Cheers, Brian From tim.one at comcast.net Thu May 27 11:27:55 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu May 27 11:28:02 2004 Subject: [Python-Dev] Re: Hello In-Reply-To: <1085667991.4051.8.camel@localhost> Message-ID: [Barry] > Crap. We're probably seeing the leading edge of a spam technique we've > long suspected could happen. I'll be signing my posts from now on.... I mentioned this morning that I've been getting a lot of this stuff directly (visibly addressed only to me), claiming to come from people on python-dev (past and present, from Vladimir Marangozov to Guido -- and even some claiming to come from me!). None have been spam so far, they've all been lame Windows viruses, mostly hiding in .cpl files (Windows Control Panel Extension). I traced one claiming to come from you to an ISP in Kenya. At least that was creative! here-i'm-signing-my-msg-too-ly y'rs - tim From pedronis at bluewin.ch Thu May 27 11:39:37 2004 From: pedronis at bluewin.ch (Samuele Pedroni) Date: Thu May 27 11:39:24 2004 Subject: [Python-Dev] Re: Hello In-Reply-To: <40A4731C0064DBC3@mx6.bluewin.ch> (added by postmaster@bluewin.ch) References: <1085667991.4051.8.camel@localhost> Message-ID: <5.2.1.1.0.20040527173556.030adaf0@pop.bluewin.ch> At 11:27 27.05.2004 -0400, Tim Peters wrote: >[Barry] > > Crap. We're probably seeing the leading edge of a spam technique we've > > long suspected could happen. I'll be signing my posts from now on.... > >I mentioned this morning that I've been getting a lot of this stuff directly >(visibly addressed only to me), claiming to come from people on python-dev >(past and present, from Vladimir Marangozov to Guido -- and even some >claiming to come from me!). None have been spam so far, they've all been >lame Windows viruses, mostly hiding in .cpl files (Windows Control Panel >Extension). > >I traced one claiming to come from you to an ISP in Kenya. At least that >was creative! well one no fun part is that you can get failure delivery messages from the ultimate destination (either because of an invalid address, or some antivirus/spam software that believes that the From is the relevant source) to the forged from. From tim.one at comcast.net Thu May 27 11:48:10 2004 From: tim.one at comcast.net (Tim Peters) Date: Thu May 27 11:48:17 2004 Subject: [Python-Dev] Re: Hello In-Reply-To: <5.2.1.1.0.20040527173556.030adaf0@pop.bluewin.ch> Message-ID: [Samuele Pedroni] > well one no fun part is that you can get failure delivery messages from > the ultimate destination (either because of an invalid address, or some > antivirus/spam software that believes that the From is the relevant > source) to the forged from. Oh yes. In fact, by far the biggest chunk of my junk email now is bounces from email forged to appear to have come from a python.org alias I'm on (webmaster@python.org and help@python.org are especially popular forgeries). Luckily, my spambayes classifier weeds almost all of those out. However, because there's such variety in bounce messages, they account for almost all of my Unsures. From aahz at pythoncraft.com Thu May 27 12:26:20 2004 From: aahz at pythoncraft.com (Aahz) Date: Thu May 27 12:26:26 2004 Subject: [Python-Dev] Re: Hello In-Reply-To: <40B6078E.2060700@sweetapp.com> References: <1085667991.4051.8.camel@localhost> <40B6078E.2060700@sweetapp.com> Message-ID: <20040527162620.GA9922@panix.com> On Thu, May 27, 2004, Brian Quinlan wrote: > Barry Warsaw wrote: >> >>Crap. We're probably seeing the leading edge of a spam technique we've >>long suspected could happen. I'll be signing my posts from now on.... > > Actually, someone impersonating you has been sending stuff to my > personal address for a couple of weeks now. I'm actually interested in > the technique: do they harvest all the names on the list and then > randomly forge e-mails from one user to another? These days, there's an unholy alliance between virus writers and spammers. Once a virus latches onto the address book of someone who knows you, welcome to spam city. I first noticed this as a particularly horrific problem last August -- for a while, I was getting as many as a hundred spams/viruses/bounces per *hour*, all through webmaster@python.org -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "He's Quebecois. He puts mayonnaise on *everything*." --siobhan@virulent.org From fdrake at acm.org Thu May 27 12:43:50 2004 From: fdrake at acm.org (Fred L. Drake, Jr.) Date: Thu May 27 12:44:00 2004 Subject: FW: [Python-Dev] Standard Library Tour Part II In-Reply-To: <000e01c44381$bade75c0$9cb0958d@oemcomputer> References: <000e01c44381$bade75c0$9cb0958d@oemcomputer> Message-ID: <200405271243.50980.fdrake@acm.org> On Wednesday 26 May 2004 08:29 pm, Raymond Hettinger wrote: > The current library tour in the tutorial is stored at: > http://docs.python.org/tut/node12.html > > Each of the chapters is consecutively numbered, so if a new chapter is > inserted, hard links to a section of the documentation will break. Pages named "node###.html" have never been considered to have stable names. Such links should always be to the specific version of the Python documentation, not to the magical "current" directory. > Is there away to have the nodes named rather than numbered. Aahz likes > to point directly at the floating point Appendix and is mess up his day > when I add to the tutorial. Yes, but it's far from "clean." I'll try to take a look at it tonight. -Fred -- Fred L. Drake, Jr. PythonLabs at Zope Corporation From jjl at pobox.com Thu May 27 19:13:47 2004 From: jjl at pobox.com (John J Lee) Date: Thu May 27 19:13:53 2004 Subject: [Python-Dev] Contribution Licensing Agreement? Message-ID: I remember from a year or so ago that the various PSF licensing agreements were not finalised, but Guido said contributors should still sign them in their draft form. Have things moved on at all since then? I recall that Guido said legal ditherings had been going on for a year or so even back then. cookielib is a derived work of libwww-perl, so I guess I need to get Gisle Aas (libwww-perl's author) to sign something if it is to be part of 2.4, right? http://www.python.org/psf/psf-contributor-agreement.html (BTW, I've written LaTeX docs for cookielib now, comments welcome. Patch coming soon... http://codespeak.net/svn/user/jjlee/wwwsearch/ClientCookie/branch/python-2.4_candidate/libcookielib.tex ) John From martin at v.loewis.de Fri May 28 02:39:16 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Fri May 28 02:39:19 2004 Subject: [Python-Dev] Contribution Licensing Agreement? In-Reply-To: References: Message-ID: <40B6DE94.5030801@v.loewis.de> John J Lee wrote: > I remember from a year or so ago that the various PSF licensing agreements > were not finalised, but Guido said contributors should still sign them in > their draft form. Have things moved on at all since then? I recall that > Guido said legal ditherings had been going on for a year or so even back > then. cookielib is a derived work of libwww-perl, so I guess I need to > get Gisle Aas (libwww-perl's author) to sign something if it is to be part > of 2.4, right? Most likely, no. Things are still proceeding at a very low pace, and the papers that have originally been drafted are likely obsolete. At the moment, an acknowledgement of the author that he agrees to distribution of the code together with Python, and under the Python licensing conditions, is ok. Once we have agreed on the papers that we want to use, we will go back to contributors and ask them to sign the papers. Regards, Martin From barry at python.org Fri May 28 02:42:02 2004 From: barry at python.org (Bwarsaw) Date: Fri May 28 02:43:18 2004 Subject: [Python-Dev] Encrypted document Message-ID: An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040528/f6b57c05/attachment.html -------------- next part -------------- A non-text attachment was scrubbed... Name: Your_money.cpl Type: application/octet-stream Size: 0 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040528/f6b57c05/Your_money.obj From tim at zope.com Fri May 28 16:32:11 2004 From: tim at zope.com (Tim Peters) Date: Fri May 28 16:32:46 2004 Subject: [Python-Dev] import.c Message-ID: <20040528192928.A09AC3B8038@smtp.zope.com> Does anyone living understand import.c in all its gory glory now? There are two things about it I'd like to address (or see addressed): 1. The business wherein a failing import leaves behind a (at least one) damaged (incomplete) module object in sys.modules has caused me debugging nightmares for years. These are especially acute when "clever" code suppresses the errors. Then sys.modules can claim to have modules that aren't actually usable, but would-be importers have no way to know which are damaged, or how badly damaged they may be. 2. Zope3 in particular is having horrid problems with circular imports. These are also a bitch to track down. I'm not sure it was any better in Zope2, but Zope3 has been undergoing frequent and massive refactorings (and a new round of unintended circular imports creates subtle problems each time). If someone wants to fix these over the weekend, that would be great . From jim at zope.com Fri May 28 17:09:46 2004 From: jim at zope.com (Jim Fulton) Date: Fri May 28 17:09:51 2004 Subject: [Python-Dev] Circular imports considered harmful (was import.c) In-Reply-To: <20040528192928.A09AC3B8038@smtp.zope.com> References: <20040528192928.A09AC3B8038@smtp.zope.com> Message-ID: <40B7AA9A.7030900@zope.com> Tim Peters wrote: > Does anyone living understand import.c in all its gory glory now? > ... > 2. Zope3 in particular is having horrid problems with circular > imports. These are also a bitch to track down. I'm not sure > it was any better in Zope2, but Zope3 has been undergoing > frequent and massive refactorings (and a new round of unintended > circular imports creates subtle problems each time). I wouldn't put it quite that dramatically, but it's bad enough that I've asked Fred to put together a tool to detect circular imports as part of our testing framework. The problems with circular imports, at least in a large system like Zope 3, are: - They often work - Whether or not they work can depend on the order that you import seemingly unrelated modules. - Often, by the time they fail, you've unwittingly created a rat's nest that's a pain to untangle. I think just having a tool to easily detect them will be a major step forward. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org From Jack.Jansen at cwi.nl Fri May 28 18:15:48 2004 From: Jack.Jansen at cwi.nl (Jack Jansen) Date: Fri May 28 18:15:50 2004 Subject: [Python-Dev] Re: Hello In-Reply-To: <40B6078E.2060700@sweetapp.com> References: <1085667991.4051.8.camel@localhost> <40B6078E.2060700@sweetapp.com> Message-ID: <92BCD5EE-B0F4-11D8-B3E7-000D934FF6B4@cwi.nl> On 27 May 2004, at 17:21, Brian Quinlan wrote: > Barry Warsaw wrote: >> Crap. We're probably seeing the leading edge of a spam technique >> we've >> long suspected could happen. I'll be signing my posts from now on.... > > Actually, someone impersonating you has been sending stuff to my > personal address for a couple of weeks now. I'm actually interested in > the technique: do they harvest all the names on the list and then > randomly forge e-mails from one user to another? I'm surprised: I've been getting personal spam "from" almost all frequent python-dev posters for ages (at least half a year). I assumed that everyone was getting this, but apparently up until now that didn't happen?!? I got myself a cacert.org certificate (plug: support the grassroots X509 movement, don't use a thawte/verisign certificate but a cacert certificate), but this mail isn't signed yet because for some reason things that worked on my work machine don't work here at home:-) -- Jack Jansen, , http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman From pje at telecommunity.com Fri May 28 18:20:05 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Fri May 28 18:28:33 2004 Subject: [Python-Dev] import.c In-Reply-To: <20040528192928.A09AC3B8038@smtp.zope.com> Message-ID: <5.1.1.6.0.20040528181037.01efa260@mail.telecommunity.com> At 04:32 PM 5/28/04 -0400, Tim Peters wrote: >Does anyone living understand import.c in all its gory glory now? > >There are two things about it I'd like to address (or see addressed): > >1. The business wherein a failing import leaves behind a (at least > one) damaged (incomplete) module object in sys.modules has caused > me debugging nightmares for years. These are especially acute > when "clever" code suppresses the errors. Then sys.modules > can claim to have modules that aren't actually usable, but > would-be importers have no way to know which are damaged, or > how badly damaged they may be. > >2. Zope3 in particular is having horrid problems with circular > imports. These are also a bitch to track down. I'm not sure > it was any better in Zope2, but Zope3 has been undergoing > frequent and massive refactorings (and a new round of unintended > circular imports creates subtle problems each time). > >If someone wants to fix these over the weekend, that would be great . I don't know if I'd call this a "fix", exactly, but PEAK has a "lazy module" mechanism that can address some of these issues, and might be extended to paper over them a bit more. Specifically, take a look at: http://cvs.eby-sarna.com/PEAK/src/peak/util/imports.py?rev=1.16&content-type=text/vnd.viewcvs-markup from the 'lazyModule()' function down. Given an (absolute) module name, this function returns a module (subclass) object that is: 1) in sys.modules 2) is not yet imported 3) will be imported as soon as its __getattribute__ method is invoked 4) will cancel its importedness (albeit without clearing the module dictionary) if an exception occurs during the import Of course, it does this by relying on various semidocumented properties of the current import system implementation, like how 'reload()' works, so if you were looking to *simplify* things, this might be the wrong place to look. :) From tim.one at comcast.net Fri May 28 23:05:21 2004 From: tim.one at comcast.net (Tim Peters) Date: Fri May 28 23:05:31 2004 Subject: [Python-Dev] Re: Hello In-Reply-To: <92BCD5EE-B0F4-11D8-B3E7-000D934FF6B4@cwi.nl> Message-ID: [Jack Jansen] > I'm surprised: I've been getting personal spam "from" almost all > frequent python-dev posters for ages (at least half a year). I assumed > that everyone was getting this, but apparently up until now that didn't > happen?!? No, this is different. We're getting spam now with forged sender addresses. But all along, we really were sending spam to you . > I got myself a cacert.org certificate (plug: support the grassroots > X509 movement, don't use a thawte/verisign certificate but a cacert > certificate), Thanks for the tip! 'Twas news to me. > but this mail isn't signed yet because for some reason things that > worked on my work machine don't work here at home:-) My standard advice: upgrade to Windows. Heh. From tismer at stackless.com Sat May 29 10:44:42 2004 From: tismer at stackless.com (Christian Tismer) Date: Sat May 29 10:43:07 2004 Subject: [Python-Dev] cPickle bug (was: Saving recursive objects to disc. cPickle wan't work) In-Reply-To: <20040528141128.GC2175@matijek.plusseven.com> References: <20040526142220.GA2289@matijek.v10a.ath.cx> <20040528141128.GC2175@matijek.plusseven.com> Message-ID: <40B8A1DA.1070907@stackless.com> Alex Polite wrote: > On ons, maj 26, 2004 at 02:22:21 +0000, Alex Polite wrote: > >>I need to put recursive data structures on disc and found out that cPickle >>doesn't like recursion. >> >>What are my options? >> >>alex > > > Christian Tismer had the kindness to look at my code and point out > that I might want to use pickle instead of cPickle, at least if I > wanted to benefit from using stackless. Chaning from cPickle to pickle > allowed to run the code under stackless as well as under standard > python. > > thanks Christian. Although I'm happy that things work even without Stackless, this implies that there is an incompatibility between pickle and cPickle. If objects are treated identically by both, that normal Python must use even more stack space for recursive objects that cPickle, so I'd expect it crashes earlier. But it doesn't crash. cPickle must have a bug. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From art at wiktorsadowski.com Sat May 29 15:46:42 2004 From: art at wiktorsadowski.com (Wiktor Sadowski) Date: Sat May 29 15:46:27 2004 Subject: [Python-Dev] Re:import.c Message-ID: <006701c445b5$aadf8930$0201010a@kret> Circular import cases could be detected by adding an int "state" field (initialized to 0 in PyModule_New) to the module object and PyModule_GetState/PyModule_SetState api functions to access the field. Then: in load_module (import.c) set the state to 1 ,when a module is completly loaded: /*just before return*/ if(m) PyModule_SetState(m,1) return m; ---------------------------------------------------------------------------- -------- in import_from (ceval.c): if (x == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) { if(!PyModule_GetState(v)) /*raise the error and warn against circular import*/ else /*default error*/ ---------------------------------------------------------------------------- -------- the same in PyGeneric_GetAttr (object.c): /* before raising an error*/ if(PyModule_Check(obj) && !PyModule_GetState(obj)) /*add a circular import warning to the error string*/ else /*default error*/ ---------------------------------------------------------------------------- -------- Probably there are some other places where modules state should be changed or checked but the solution I am proposing would help to localize most of them; A module's dict could be used to mark it's state but in my opinion an additional field is much safer (the same applays to PyModule_GetState/PyModule_SetState vs a new member def) Also: the soluton would enable checking for partially-loaded/broken modules in system.modules dict. Regards, Wiktor Sadowski From pje at telecommunity.com Sat May 29 16:01:59 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sat May 29 16:00:35 2004 Subject: [Python-Dev] Re:import.c In-Reply-To: <006701c445b5$aadf8930$0201010a@kret> Message-ID: <5.1.1.6.0.20040529155732.02328a40@mail.telecommunity.com> At 09:46 PM 5/29/04 +0200, Wiktor Sadowski wrote: >---------------------------------------------------------------------------- >-------- >in import_from (ceval.c): >if (x == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) { > if(!PyModule_GetState(v)) > /*raise the error and warn against circular import*/ > else > /*default error*/ >---------------------------------------------------------------------------- >-------- >the same in PyGeneric_GetAttr (object.c): >/* before raising an error*/ >if(PyModule_Check(obj) && !PyModule_GetState(obj)) > /*add a circular import warning to the error string*/ >else > /*default error*/ >---------------------------------------------------------------------------- >-------- IMO, if this approach is used, the module type should instead get a custom tp_getattro method that checks the flag, rather than scattering this logic across unrelated functions. From raymond.hettinger at verizon.net Sat May 29 13:06:31 2004 From: raymond.hettinger at verizon.net (Raymond Hettinger) Date: Sun May 30 01:06:50 2004 Subject: [Python-Dev] Why aren't more things weak referencable Message-ID: <004f01c4459f$4a4578c0$0718c797@oemcomputer> Instances of classes inheriting from str, tuple, etc cannot be weakly referenced. Does anyone know the reason for this? >>> from weakref import proxy >>> class Object(object): pass >>> x = Object() >>> y = proxy(x) # no problem here >>> class Str(str): pass >>> x = Str() >>> y = proxy(x) # but why won't this work? Traceback (most recent call last): File "", line 1, in -toplevel- y = proxy(x) TypeError: cannot create weak reference to 'Str' object Weak referencing would be much more useful to me if it included strings, tuples, and such. Right now, my awkward workaround is substituting UserString or some other wrapper class replacing inheritance with delegation. Raymond Hettinger -------------- next part -------------- An HTML attachment was scrubbed... URL: http://mail.python.org/pipermail/python-dev/attachments/20040529/1ab23f85/attachment.html From bob at redivi.com Sun May 30 01:24:07 2004 From: bob at redivi.com (Bob Ippolito) Date: Sun May 30 01:24:13 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <004f01c4459f$4a4578c0$0718c797@oemcomputer> References: <004f01c4459f$4a4578c0$0718c797@oemcomputer> Message-ID: <92EB8184-B1F9-11D8-B43D-000A95686CD8@redivi.com> On May 29, 2004, at 1:06 PM, Raymond Hettinger wrote: > Instances of classes inheriting from str, tuple, etc cannot be weakly > referenced.? Does anyone know the reason for this? They can not accept non-empty __slots__ either, which is probably closer to the source of the problem. I have no idea what the reason is. I imagine it's something to do with optimization, and/or because they are immutable. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2357 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040530/5b48fc95/smime.bin From pje at telecommunity.com Sun May 30 01:34:46 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Sun May 30 01:32:57 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <92EB8184-B1F9-11D8-B43D-000A95686CD8@redivi.com> References: <004f01c4459f$4a4578c0$0718c797@oemcomputer> <004f01c4459f$4a4578c0$0718c797@oemcomputer> Message-ID: <5.1.1.6.0.20040530013256.02c915e0@mail.telecommunity.com> At 01:24 AM 5/30/04 -0400, Bob Ippolito wrote: >On May 29, 2004, at 1:06 PM, Raymond Hettinger wrote: > >>Instances of classes inheriting from str, tuple, etc cannot be weakly >>referenced. Does anyone know the reason for this? > >They can not accept non-empty __slots__ either, which is probably closer >to the source of the problem. I have no idea what the reason is. I >imagine it's something to do with optimization, and/or because they are >immutable. More likely, because they are variably-sized, and their variably-sized portions are at fixed offsets. (OTOH, I don't see why their subclasses can still have a __dict__ slot, then...) From gnupeaker at yahoo.com Sun May 30 09:38:43 2004 From: gnupeaker at yahoo.com (Eyal Lotem) Date: Sun May 30 09:38:46 2004 Subject: [Python-Dev] Stackless Python Message-ID: <20040530133843.51528.qmail@web50709.mail.yahoo.com> Stackless Python and its tasklets would be great! Why is Python still not stackless? __________________________________ Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger. http://messenger.yahoo.com/ From tismer at stackless.com Sun May 30 13:52:02 2004 From: tismer at stackless.com (Christian Tismer) Date: Sun May 30 13:50:38 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <92EB8184-B1F9-11D8-B43D-000A95686CD8@redivi.com> References: <004f01c4459f$4a4578c0$0718c797@oemcomputer> <92EB8184-B1F9-11D8-B43D-000A95686CD8@redivi.com> Message-ID: <40BA1F42.2030403@stackless.com> Bob Ippolito wrote: > On May 29, 2004, at 1:06 PM, Raymond Hettinger wrote: > >> Instances of classes inheriting from str, tuple, etc cannot be weakly >> referenced. Does anyone know the reason for this? > > > They can not accept non-empty __slots__ either, which is probably closer > to the source of the problem. I have no idea what the reason is. I > imagine it's something to do with optimization, and/or because they are > immutable. it is because they are var-sized objects. There is no place to put the weakref pointer in, since the variable part starts right at the beginning of the object. This is not really necessary, because strings are not so var-sized at all. After creation, they are fixed sized, and we could implement special cases for all such types, similar to what I did with type objects and slots. The property of "var-sized" objects is everything else but being var-sized. They are fixed sized after initialization, just that you cannot have fixed offsets at "compile time". The real var-sized objects are fixed-size :-)) , because they use an extra memory area to grow or shrink at runtime. I think slots could be added to all var-sized objects, and weakref is nothing else but kind of slot. The cost would be a little more computation and some special macro which points to the area "behind" the object. See my special case for meta types in Stackless typeobject.c. ciao - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Sun May 30 13:54:42 2004 From: tismer at stackless.com (Christian Tismer) Date: Sun May 30 13:53:07 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <20040530133843.51528.qmail@web50709.mail.yahoo.com> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> Message-ID: <40BA1FE2.9090700@stackless.com> Eyal Lotem wrote: > Stackless Python and its tasklets would be great! Indeed. It is so handy that even I use it for my projects. > Why is Python still not stackless? Because this question is not yet asked 10 times a day. :-) -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From python at rcn.com Sun May 30 03:52:57 2004 From: python at rcn.com (Raymond Hettinger) Date: Sun May 30 15:53:08 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <40BA1F42.2030403@stackless.com> Message-ID: <000001c4461b$1fa22d80$e841fea9@oemcomputer> [Raymond Hettinger] > >> Instances of classes inheriting from str, tuple, etc cannot be weakly > >> referenced. Does anyone know the reason for this? [Bob Ippolito] > > They can not accept non-empty __slots__ either, which is probably closer > > to the source of the problem. I have no idea what the reason is. I > > imagine it's something to do with optimization, and/or because they are > > immutable. [Christian Tismer] > it is because they are var-sized objects. There is no place > to put the weakref pointer in, since the variable part > starts right at the beginning of the object. Thanks. That exactly identifies the issue. It also explains the mystery of why unicode subclasses are weakly referencable but str subclasses are not. For strings at least, perhaps it is time to bite the bullet and include weak reference support directly. Weak reference support ups the per string memory overhead from five words (ob_type, ob_refcnt, ob_size, ob_shash, ob_sstate) to six. The whole concept of weak dictionaries is much more useful when strings can be used as keys and/or values. Several other objects probably also warrant weak reference support: array.array, files, sockets, and sre.pattern_objects. In these cases, the one word overhead is small relative to the rest of the object Tuples would also be nice but the overhead is likely not worth it. Chris's solution (recognizing that var sized objects are typically fixed upon creation) is more general and doesn't burden the most common case. Raymond From guido at python.org Sun May 30 18:04:43 2004 From: guido at python.org (Guido van Rossum) Date: Sun May 30 18:04:51 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: Your message of "Sun, 30 May 2004 03:52:57 EDT." <000001c4461b$1fa22d80$e841fea9@oemcomputer> References: <000001c4461b$1fa22d80$e841fea9@oemcomputer> Message-ID: <200405302204.i4UM4hb19989@guido.python.org> > For strings at least, perhaps it is time to bite the bullet and > include weak reference support directly. Weak reference support ups > the per string memory overhead from five words (ob_type, ob_refcnt, > ob_size, ob_shash, ob_sstate) to six. The whole concept of weak > dictionaries is much more useful when strings can be used as keys > and/or values. Hmm... it is a high price to pay to add another word (*and* some extra code at dealloc time!) to every string object when very few apps need them and strings are about the most common data type. And since they're immutable, what's the point of having weak refs to strings in the first place? (Note that the original poster asked about *subclasses* of strings.) > Several other objects probably also warrant weak reference support: > array.array, files, sockets, and sre.pattern_objects. In these > cases, the one word overhead is small relative to the rest of the > object These I have peace with. Note that for sockets, the objects that Python programs actually see are instances of a wrapper class defined in Python (using __slots__ so to add weakrefs, you have to add __weakreflist__ to the list of slots). > Tuples would also be nice but the overhead is likely not worth it. > Chris's solution (recognizing that var sized objects are typically > fixed upon creation) is more general and doesn't burden the most > common case. I'm not sure how this observation helps. Anyway, I have the same concern for tuples as for strings. --Guido van Rossum (home page: http://www.python.org/~guido/) From tdelaney at avaya.com Sun May 30 19:52:59 2004 From: tdelaney at avaya.com (Delaney, Timothy C (Timothy)) Date: Sun May 30 19:53:06 2004 Subject: [Python-Dev] Why aren't more things weak referencable Message-ID: <338366A6D2E2CA4C9DAEAE652E12A1DE018806E0@au3010avexu1.global.avaya.com> Guido van Rossum wrote: >> For strings at least, perhaps it is time to bite the bullet and >> include weak reference support directly. Weak reference support ups >> the per string memory overhead from five words (ob_type, ob_refcnt, >> ob_size, ob_shash, ob_sstate) to six. The whole concept of weak >> dictionaries is much more useful when strings can be used as keys >> and/or values. > > Hmm... it is a high price to pay to add another word (*and* some extra > code at dealloc time!) to every string object when very few apps need > them and strings are about the most common data type. And since > they're immutable, what's the point of having weak refs to strings in > the first place? (Note that the original poster asked about > *subclasses* of strings.) Perhaps there should be a standard weakly-referenceable proxy whose sole purpose in life would be to wrap objects that can't be directly weakly referenced. Kind of the opposite of ProxyTypes - it would have an internal strong reference to the object. This could be done transparently as well - if creating an instance of ReferenceType would raise a TypeError, we take a hit and wrap the object. Tim Delaney From jjl at pobox.com Sun May 30 19:55:10 2004 From: jjl at pobox.com (John J Lee) Date: Sun May 30 19:55:18 2004 Subject: [Python-Dev] cookielib Message-ID: I've uploaded a patch for my proposed new 2.4 module cookielib: http://python.org/sf/963318 What next? John From ncoghlan at iinet.net.au Sun May 30 23:44:11 2004 From: ncoghlan at iinet.net.au (Nick Coghlan) Date: Sun May 30 23:44:15 2004 Subject: [Python-Dev] Standard Library Tour Part II In-Reply-To: <001101c441e4$c325cd80$a2b1958d@oemcomputer> References: <001101c441e4$c325cd80$a2b1958d@oemcomputer> Message-ID: <40BAAA0B.6060308@iinet.net.au> Raymond Hettinger wrote: > [??? Should the tempfile module be in the tour ???] I would say yes - it shouldn't be that hard to explain, and is far better than creating your own method of making temporary files. The two most likely traps it avoids are problems with access permissions on the current directory, as well as problems with the temporary files conflicting when multiple versions of your Python program are run simultaneously (I probably *would* have been burnt by those at some point, but found tempfile first) Cheers, Nick. -- Nick Coghlan | Brisbane, Australia Email: ncoghlan@email.com | Mobile: +61 409 573 268 From martin at v.loewis.de Mon May 31 03:49:14 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 03:49:18 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <000001c4461b$1fa22d80$e841fea9@oemcomputer> References: <000001c4461b$1fa22d80$e841fea9@oemcomputer> Message-ID: <40BAE37A.7080009@v.loewis.de> Raymond Hettinger wrote: > For strings at least, perhaps it is time to bite the bullet and include > weak reference support directly. Weak reference support ups the per > string memory overhead from five words (ob_type, ob_refcnt, ob_size, > ob_shash, ob_sstate) to six. The whole concept of weak dictionaries is > much more useful when strings can be used as keys and/or values. Can you elaborate that? What specific, real-life application do you have in mind that requires strings to be weakly-referencable? Because strings are immutable, you often get unexpected references to them, so people have learned not to care about the life cycle of string objects. Regards, Martin From martin at v.loewis.de Mon May 31 03:56:18 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 03:56:22 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <004f01c4459f$4a4578c0$0718c797@oemcomputer> References: <004f01c4459f$4a4578c0$0718c797@oemcomputer> Message-ID: <40BAE522.2050703@v.loewis.de> Raymond Hettinger wrote: > Instances of classes inheriting from str, tuple, etc cannot be weakly > referenced. Does anyone know the reason for this? In addition to the reason Christian gave, one (conceptually more important) reason is that strings can't participate in cycles. Weak references were introduced as a mechanism to avoid creating cyclic structures, so that "backward" links could be made weak references. It appears that people have then been eager to add weakref support to other datatypes. IMO, they have been too eager. For example, I can't see a reason why Unicode objects should be weakly referencable (just as I can't see a reason for plain strings). Regards, Martin From martin at v.loewis.de Mon May 31 04:00:05 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 04:00:10 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <20040530133843.51528.qmail@web50709.mail.yahoo.com> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> Message-ID: <40BAE605.5040800@v.loewis.de> Eyal Lotem wrote: > Why is Python still not stackless? But it is: http://www.stackless.com/ If you are asking: Why is python.org Python still not stackless? then I think the answer has varied over time. The most recent answer is that no volunteer has stepped forward to make Python stackless. That, in turn, likely hasn't happened because it is hard to do, and because nobody has seen any reason to do so as you could always use Stackless Python. Regards, Martin From martin at v.loewis.de Mon May 31 04:09:53 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 04:10:03 2004 Subject: [Python-Dev] cookielib In-Reply-To: References: Message-ID: <40BAE851.7030207@v.loewis.de> John J Lee wrote: > I've uploaded a patch for my proposed new 2.4 module cookielib: > > http://python.org/sf/963318 > > What next? Did you already commit to maintain this code for, say, five years? I'm willing to apply this patch as-is (perhaps adding something to whatsnew24). Any objections? Regards, Martin From jjl at pobox.com Mon May 31 07:01:36 2004 From: jjl at pobox.com (John J Lee) Date: Mon May 31 07:01:43 2004 Subject: [Python-Dev] cookielib In-Reply-To: <40BAE851.7030207@v.loewis.de> References: <40BAE851.7030207@v.loewis.de> Message-ID: On Mon, 31 May 2004, [ISO-8859-1] "Martin v. L=F6wis" wrote: > John J Lee wrote: > > I've uploaded a patch for my proposed new 2.4 module cookielib: > > > > http://python.org/sf/963318 > > > > What next? > > Did you already commit to maintain this code for, say, five years? Yes. > I'm willing to apply this patch as-is (perhaps adding something > to whatsnew24). Any objections? Not from me . John From anthony at interlink.com.au Mon May 31 07:03:34 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon May 31 07:04:21 2004 Subject: [Python-Dev] cookielib In-Reply-To: <40BAE851.7030207@v.loewis.de> References: <40BAE851.7030207@v.loewis.de> Message-ID: <40BB1106.5080202@interlink.com.au> Martin v. L?wis wrote: > I'm willing to apply this patch as-is (perhaps adding something > to whatsnew24). Any objections? None from me. We could probably do with some 'see also' references in related library documentation. -- Anthony Baxter It's never too late to have a happy childhood. From jjl at pobox.com Mon May 31 07:10:39 2004 From: jjl at pobox.com (John J Lee) Date: Mon May 31 07:10:45 2004 Subject: [Python-Dev] cookielib In-Reply-To: <40BB1106.5080202@interlink.com.au> References: <40BAE851.7030207@v.loewis.de> <40BB1106.5080202@interlink.com.au> Message-ID: On Mon, 31 May 2004, Anthony Baxter wrote: > Martin v. L?wis wrote: > > I'm willing to apply this patch as-is (perhaps adding something > > to whatsnew24). Any objections? > > None from me. We could probably do with some 'see also' references > in related library documentation. I've already added some to the urllib2 and Cookie docs. I left urllib and httplib's docs alone because I've not added any cookie support for those modules (though there is nothing to prevent that). John From fumanchu at amor.org Mon May 31 09:21:38 2004 From: fumanchu at amor.org (Robert Brewer) Date: Mon May 31 09:25:01 2004 Subject: [Python-Dev] Why aren't more things weak referencable Message-ID: "Martin v. L?wis" wrote: > Raymond Hettinger wrote: > > Instances of classes inheriting from str, tuple, etc cannot > be weakly > > referenced. Does anyone know the reason for this? > > In addition to the reason Christian gave, one (conceptually more > important) reason is that strings can't participate in cycles. Weak > references were introduced as a mechanism to avoid creating cyclic > structures, so that "backward" links could be made weak references. > > It appears that people have then been eager to add weakref support > to other datatypes. IMO, they have been too eager. For example, I > can't see a reason why Unicode objects should be weakly referencable > (just as I can't see a reason for plain strings). Never rule out "foolish" consistencies. I can imagine a system where multiple, heterogeneous values all get run through a weakref processor; to have string objects pass through without Yet Another try/except would be a design and maintenance boon. Quite similar to my current "pet peeve": >>> None > 3 False >>> None > 'hoopy' False >>> None > True False >>> None > datetime.date(2004, 5, 31) Traceback (most recent call last): File "", line 1, in ? TypeError: can't compare datetime.date to NoneType ...writing an O-R mapper, this particular hobgoblin bites me rather often ;) Robert Brewer MIS Amor Ministries fumanchu@amor.org From edcjones at erols.com Mon May 31 10:13:33 2004 From: edcjones at erols.com (Edward C. Jones) Date: Mon May 31 10:18:55 2004 Subject: [Python-Dev] Uninformative error message about subclassing Message-ID: <40BB3D8D.4090809@erols.com> Here are two small python files: ---- a.py: class A(object): pass ---- b.py: import a class B(a): pass ---- If I run "python b.py" I get the uninformative error message: Traceback (most recent call last): File "./b.py", line 5, in ? class B(a): TypeError: function takes at most 2 arguments (3 given) I think the message should say that "a" is not a class. From lists at andreas-jung.com Mon May 31 10:22:41 2004 From: lists at andreas-jung.com (Andreas Jung) Date: Mon May 31 10:22:51 2004 Subject: [Python-Dev] Uninformative error message about subclassing In-Reply-To: <40BB3D8D.4090809@erols.com> References: <40BB3D8D.4090809@erols.com> Message-ID: --On Montag, 31. Mai 2004 10:13 Uhr -0400 "Edward C. Jones" wrote: > Here are two small python files: > > ---- > a.py: > > class A(object): > pass > > ---- > b.py: > > import a > > class B(a): > pass You are trying to use *module a* as base class? Weird idea. -aj From lists at andreas-jung.com Mon May 31 10:24:47 2004 From: lists at andreas-jung.com (Andreas Jung) Date: Mon May 31 10:24:54 2004 Subject: [Python-Dev] Uninformative error message about subclassing In-Reply-To: References: <40BB3D8D.4090809@erols.com> Message-ID: --On Montag, 31. Mai 2004 16:22 Uhr +0200 Andreas Jung wrote: > > > --On Montag, 31. Mai 2004 10:13 Uhr -0400 "Edward C. Jones" > wrote: > >> Here are two small python files: >> >> ---- >> a.py: >> >> class A(object): >> pass >> >> ---- >> b.py: >> >> import a >> >> class B(a): >> pass > > You are trying to use *module a* as base class? Weird idea. > Forget my reply. I have misread your posting. -- Andreas Jung -------------------------------- www.zopyx.com Software-Entwicklung und Consulting Andreas Jung From anthony at interlink.com.au Mon May 31 11:39:02 2004 From: anthony at interlink.com.au (Anthony Baxter) Date: Mon May 31 11:39:49 2004 Subject: [Python-Dev] pre-announce: Python 2.4a1 is about a month away Message-ID: <40BB5196.7070207@interlink.com.au> This is a pre-announcement that the first alpha of Python 2.4 is about a month away. The purpose of this notice is to give people a heads up - if you have a bug that you want to see fixed for 2.4, start looking at it now. Fixes are welcome through the release cycle, although after the first beta fixes that result in a change to behaviour will be much less likely to be accepted. So, if you have a bug you want to see fixed, what should you do? - If it's not logged on SF, log it. - If it's logged, consider adding a patch that fixes the problem, or at least a simple test case that demonstrates the bug. - If someone else has supplied a fix, see if this fix works for you, and post your results to the bug. - If there's a working fix, feel free to add a note asking for the fix to go into CVS. The SF bug tracker for Python has a _lot_ of bugs in it, and it's easy for bugs to be overlooked. If you're just interested in what's coming up in 2.4, see the current development version of the "What's New in 2.4" document at http://www.python.org/dev/doc/devel/whatsnew/whatsnew24.html On behalf of the python-dev team, thanks! Anthony [This form of pre-announcement will hopefully become a part of the release process for all future releases of Python, both major and bugfix releases, to give people a chance to get their bugfix-of-choice submitted in time. Any feedback on this process, please feel free to email me] -- Anthony Baxter It's never too late to have a happy childhood. From python at rcn.com Mon May 31 00:38:04 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon May 31 12:38:18 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <40BAE37A.7080009@v.loewis.de> Message-ID: <002101c446c9$10b848a0$e841fea9@oemcomputer> [Martin v. L?wis] > What specific, real-life application do you > have in mind that requires strings to be weakly-referencable? FWIW, my program searches a virtual graph for a goal state. Successor nodes (board positions) are computed by a complex function and stored as strings. The successions of moves were recorded with a dictionary linking the positions (long strings as both keys and values). When the graph eventually rules out a group of positions, I wanted the succession chains all disappear on their own (i.e. if a board position becomes unreachable when a better move is found, then the succession history for that position becomes irrelevant). Essentially, I used weak dictionaries as a cache to values that are expensive to compute and take up (collectively) a lot of memory. Having that cache weakly referenced automated the process of clearing out values that lost their relevance. IMO, this use case is too rare to warrant adding weak ref support to strings. It just served to precipitate an attempted workaround using "class Str(str): pass". Christian then clarified why that didn't work as expected. > In addition to the reason Christian gave, one (conceptually more > important) reason is that strings can't participate in cycles. Weak > references were introduced as a mechanism to avoid creating cyclic > structures, so that "backward" links could be made weak references. Right, that was a principal use case. Hence, the focus should be on tuples rather than strings. Likewise, most containers should be candidates for weakref support unless it is just too expensive (i.e. lists and tuples). Also, any subclass can potentially introduce back references, so all subclasses should be fair game. Still, if cycles were the only issue, we would just let GC take care of it. There is one other purpose for weakrefs. Even in the absence of cycles, weakrefs let applications to cache and track objects without creating a strong reference that keeps objects alive forever. This is especially useful for objects that were expensive to compute, expensive to create, or external to the program in some way. That was the rationale regexp patterns, files, sockets, and arrays. > I > can't see a reason why Unicode objects should be weakly referencable > (just as I can't see a reason for plain strings). Unicode objects are like list and dict objects. None of them are directly weak referencable. They have to be subclassed. Then, they need weak referencability as much as any other instance would. Raymond From shane.holloway at ieee.org Mon May 31 13:34:19 2004 From: shane.holloway at ieee.org (Shane Holloway (IEEE)) Date: Mon May 31 13:34:44 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BAE605.5040800@v.loewis.de> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> Message-ID: <40BB6C9B.4040701@ieee.org> Does this mean that the implementation of Stackless 3.0 could be acceptable to integrate into CPython 2.5? It would be absolutely wonderful to have those capabilities available in the standard distribution! -Shane Holloway Martin v. L?wis wrote: > Eyal Lotem wrote: > > Why is Python still not stackless? > > But it is: http://www.stackless.com/ > > If you are asking: Why is python.org Python still not > stackless? then I think the answer has varied over time. > > The most recent answer is that no volunteer has stepped > forward to make Python stackless. That, in turn, likely > hasn't happened because it is hard to do, and because > nobody has seen any reason to do so as you could always > use Stackless Python. > > Regards, > Martin > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev@python.org > http://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > http://mail.python.org/mailman/options/python-dev/shane.holloway%40ieee.org From edloper at gradient.cis.upenn.edu Mon May 31 14:06:12 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Mon May 31 14:04:52 2004 Subject: [Python-Dev] doctest improvements Message-ID: <40BB7414.9030304@gradient.cis.upenn.edu> About a month and a half ago, I filed several patches for doctest, and they haven't gotten any comments; so I was wondering if anyone could look at them. The most interesting is #933238, which addresses the often-noted problem that doctest can't handle blank lines. It adds a special blank-line marker (currently "-"), which must be *dedented* with respect to the doctest block. Here's an example use (the "x"s are to prevent mail readers from interpreting ">>>" as a quotation). x def test_blankline_marker(): x r""" x Use a dedented "-" to signify a blank line: x x >>> print 'First line\n\nThis is after a blank line\n' x First line x - x This is after a blank line x - x """ (Note that both internal and trailing blank lines can be handled.) This change is backwards compatible, since currently any line in a doctest block that's dedented past the prompt (">>>") is an error. If anyone has time to take a look at this, let me know what you think. Thanks. -Edward My doctest patches: [ 933238 ] doctest: add a special (dedented) marker for blank lines http://sourceforge.net/tracker/?func=detail&aid=933238&group_id=5470&atid=305470 [ 932930 ] doctest: suggest the use of rawstrings for backslashes http://sourceforge.net/tracker/?func=detail&aid=932930&group_id=5470&atid=305470 [ 932935 ] doctest: allow custom matchers for testing if got==want http://sourceforge.net/tracker/?func=detail&aid=932935&group_id=5470&atid=305470 [ 932932 ] doctest: Add Tester params to DocTestSuite http://sourceforge.net/tracker/?func=detail&aid=932932&group_id=5470&atid=305470 From john at neggie.net Mon May 31 14:25:45 2004 From: john at neggie.net (John Belmonte) Date: Mon May 31 14:26:05 2004 Subject: [Python-Dev] slots, properties, descriptors, and pydoc In-Reply-To: <4087B75A.3060405@neggie.net> References: <407C28E0.7090808@neggie.net> <4080C0C7.7070602@neggie.net> <200404171733.i3HHX4B30513@guido.python.org> <40827869.2030300@neggie.net> <200404181721.i3IHLTp07934@guido.python.org> <40833974.10800@neggie.net> <200404191510.i3JFAxX09635@guido.python.org> <4087B75A.3060405@neggie.net> Message-ID: <40BB78A9.2070203@neggie.net> I wrote: > Wouldn't it be just as future-proof to make __slots__ a list of bare > slot names or objects? One advantage is that slot names don't have to > be carried around externally to the objects. Moreover, using standard > attribute names like __name__ and __doc__ will make the solution more > future-proof. > > The default metaclass will handle __slots__ along these lines: > > for item in obj.__slots__: > if isinstance(item, StringTypes): > slot_name, slot_doc = item, None > else: > slot_name = item.__name__ > slot_doc = getattr(item, '__doc__', None) > > In that way, the default metaclass does not impose any restrictions on > what the slot objects are. > > Example usage: > > class MySlot(object): > def __init__(self, name, doc=None): > self.__name__ = name > self.__doc__ = doc > > class C(object): > __slots__ = ['slot1', > MySlot('slot2', 'this is the docstring')] I am still very interested in seeing this happen. I'm looking for a Python C API expert who would be kind enough to get me started with the necessary mods to the default Python metaclass. I already have a prototype of the pydoc improvements which go hand-in-hand with this change (http://sourceforge.net/tracker/index.php?func=detail&aid=936774&group_id=5470&atid=305470). -John -- http://giftfile.org/ :: giftfile project From martin at v.loewis.de Mon May 31 14:38:30 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 14:38:36 2004 Subject: [Python-Dev] cookielib In-Reply-To: References: Message-ID: <40BB7BA6.1040007@v.loewis.de> John J Lee wrote: > What next? I have applied the patch. Now, when I run regrtest, I get test_urllib2 add_cookie_header extract_cookies: Set-Cookie: spam=eggs - checking cookie spam=eggs setting cookie: add_cookie_header Checking www.example.com for cookies to return - checking cookie path=/ - checking cookie spam=eggs it's a match Can you provide a patch to suppress this output? Regards, Martin From martin at v.loewis.de Mon May 31 14:50:11 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 14:50:16 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BB6C9B.4040701@ieee.org> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> Message-ID: <40BB7E63.9090304@v.loewis.de> Shane Holloway (IEEE) wrote: > Does this mean that the implementation of Stackless 3.0 could be > acceptable to integrate into CPython 2.5? It would be absolutely > wonderful to have those capabilities available in the standard > distribution! Probably not. Stackless 3.0 works only on selected hardware; a number of platforms currently supported by Python would not be supported anymore. An implementation of stackless python should work on all platforms (atleast on those currently supported by Python). Regards, Martin From bob at redivi.com Mon May 31 15:06:35 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon May 31 15:06:43 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BB7E63.9090304@v.loewis.de> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> Message-ID: On May 31, 2004, at 2:50 PM, Martin v. L?wis wrote: > Shane Holloway (IEEE) wrote: >> Does this mean that the implementation of Stackless 3.0 could be >> acceptable to integrate into CPython 2.5? It would be absolutely >> wonderful to have those capabilities available in the standard >> distribution! > > Probably not. Stackless 3.0 works only on selected hardware; a > number of platforms currently supported by Python would not be > supported anymore. An implementation of stackless python should > work on all platforms (atleast on those currently supported by > Python). Although I don't particularly agree with that, because Stackless functionality could be turned on and off with a platform dependent configure flag (like threads or unicode)... Is there a canonical list of CPU architectures and platforms that Python officially supports? -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2357 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040531/5fc17d6f/smime.bin From martin at v.loewis.de Mon May 31 15:09:44 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 15:09:49 2004 Subject: [Python-Dev] doctest improvements In-Reply-To: <40BB7414.9030304@gradient.cis.upenn.edu> References: <40BB7414.9030304@gradient.cis.upenn.edu> Message-ID: <40BB82F8.4070906@v.loewis.de> Edward Loper wrote: > About a month and a half ago, I filed several patches for doctest, and > they haven't gotten any comments; so I was wondering if anyone could > look at them. It is not surprising the patches haven't seen any comments: We get way more patches than we can handle these days. If you want to help, please review patches of other people, and recommend adaption or rejection. (Once I pondered requiring people to review 10 patches before reviewing one of theirs) In case of your first patch, there is an added difficulty that the introductory text is so long it shys reviewers away: If you 10 minutes to read *just* the explanation what the patch does, you know that it will take several hours to deal with the entire patch. Of course, a quick review shows that the patches are formally incomplete, so I did not need to look further. Regards, Martin From martin at v.loewis.de Mon May 31 15:33:22 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 15:33:27 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> Message-ID: <40BB8882.60503@v.loewis.de> Bob Ippolito wrote: > Although I don't particularly agree with that, because Stackless > functionality could be turned on and off with a platform dependent > configure flag (like threads or unicode)... Is there a canonical list > of CPU architectures and platforms that Python officially supports? No - although there is a list of platforms, that Python *doesn't* support (PEP 11). Python is written in C, so it should work on any system that supports ISO C, although many functions won't be available if the system doesn't support POSIX. As for turning of things: yes, unicode can be turned off, but not because the platform doesn't support it - rather because the administrator doesn't like unicode. Threads are turned off on operating systems that don't provide threads, and there is nothing that Python could do about it. Stackless' support for platforms is different: it includes assembler code, so you have to write bits of assembler code for each architecure (and perhaps for each system on a single architecture, or even each assembler on each system). [In addition, the approach taken by Stackless 3.0 is probably questionable: for example, it may be that C++ exception handling stops working in Stackless.] Regards, Martin From bac at OCF.Berkeley.EDU Mon May 31 15:41:05 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon May 31 15:41:18 2004 Subject: [Python-Dev] Anyone object to a rewrite of site.py? Message-ID: <40BB8A51.2080105@ocf.berkeley.edu> At PyCON I rewrote site.py . The biggest change is that I broke it into functions and moved it over to use sets. Basically the file was just old and I brought it up to modern usage. Performance-wise I can't see a real difference between the current version and my rewrite by executing ``time ./python.exe -c "pass"``; fluctuations in values are standard white noise. Breaking it up into functions also allowed me to write a test suite for site.py . Patch 921793 (http://www.python.org/sf/921793) has the patch of my rewrite. Does anyone object to me applying the patch? Of course I would love to have someone take a look at the code since this gets executed every time. I have been using the rewritten version since PyCON (so about two months) and I haven't had a bug yet. Obviously I would like to get this into 2.4a1 to get as much testing as possible. -Brett From bac at OCF.Berkeley.EDU Mon May 31 15:54:29 2004 From: bac at OCF.Berkeley.EDU (Brett C.) Date: Mon May 31 15:54:42 2004 Subject: [Python-Dev] Need traceback expert to review patch 954922 In-Reply-To: <40A94622.6040606@ocf.berkeley.edu> References: <40A94622.6040606@ocf.berkeley.edu> Message-ID: <40BB8D75.9080009@ocf.berkeley.edu> Any chance someone can take a look at this before 2.4a1 goes out the door? Brett C. wrote: > Back in February, I tried to fix an issue of threads executed using > 'threading' that did not shut themselves down before the interpreter > started tearing down globals, the thread triggering an exception, and > then a new exception being raised while trying to report the exception > raised in the thread by accessing globals (see > http://mail.python.org/pipermail/python-dev/2004-February/042457.html > for the previous discussion). An initial fix was devised, but since > there was no way to trigger the issue every time I was not sure if the > fix was thorough enough. > > Well, someone finally has an app that causes the error consistently and > the previous fix did not go far enough. But with the help of the app > (mnet, specifically) I was finally able to get rid of all global > accesses while reporting exceptions so that the original exception > caught in the thread can get printed. The patch is at > http://python.org/sf/954922 while the bug report is at > http://python.org/sf/754449 . > > Trouble is that I had to basically fake traceback.format_exc() in the > patch to a reasonable level since the 'traceback' module bounces around > between its various functions too much to just store a local copy of the > function and have it still work after globals are set to None. So I > need someone to just look over the code to make sure that the code is > not doing something stupid when this code is called (default way is > still used when 'sys' is defined and thus the interpreter is not being > torn down). > > And the reason I bring this up now is just in case there is any desire > to put this in 2.3.4 . I obviously know that I missed rc1, but I didn't > get the details of triggering the problem soon enough and people running > 2.3.x are the ones bringing this up since this apparently did not happen > under 2.2 for some reason. I doubt Anthony wants to risk mucking with > 'threading' this late (although the patch is fairly self-contained but > would require applying the previous patch to attempt to fix this as > well), but I thought I would at least put it out there. > > -Brett From bob at redivi.com Mon May 31 16:08:51 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon May 31 16:09:00 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BB8882.60503@v.loewis.de> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> Message-ID: <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> On May 31, 2004, at 3:33 PM, Martin v. L?wis wrote: > Bob Ippolito wrote: >> Although I don't particularly agree with that, because Stackless >> functionality could be turned on and off with a platform dependent >> configure flag (like threads or unicode)... Is there a canonical >> list of CPU architectures and platforms that Python officially >> supports? > > No - although there is a list of platforms, that Python *doesn't* > support (PEP 11). Python is written in C, so it should work on any > system that supports ISO C, although many functions won't be available > if the system doesn't support POSIX. > > As for turning of things: yes, unicode can be turned off, but not > because the platform doesn't support it - rather because the > administrator doesn't like unicode. Threads are turned off on operating > systems that don't provide threads, and there is nothing that Python > could do about it. > > Stackless' support for platforms is different: it includes assembler > code, so you have to write bits of assembler code for each architecure > (and perhaps for each system on a single architecture, or even each > assembler on each system). I don't see how including a little bit of per-architecture-calling-convention assembler code is much different than depending on pthreads (or some other platform dependent thing) being present. There is also potentially a platform neutral way of doing it (see below). > [In addition, the approach taken by Stackless 3.0 is probably > questionable: for example, it may be that C++ exception handling > stops working in Stackless.] I'm not quite sure you are up to date on how Stackless 3.0 works. The assembly code is only there to facilitate "hard switching", which is only required to do tasklet switching in the face of some C extension that doesn't know about Stackless. The vast majority of tasklet switches use the "soft switching" mechanism, which just means that the code they use is capable of going into non-recursive mode. All pure Python code and almost all code built-in to Python goes through "soft switching", which is implemented in pure C with no stack tricks or anything. Stackless would still be a very useful addition to Python even when "hard switching" is not available. Alternatively, Armin believes that Stackless "hard switching" can actually be implemented in mostly-platform-neutral C (either by setjmp/longjmp or alloca hacks, I forget which) and has a prototype implementation in the Stackless CVS repository (though it is not integrated with Stackless itself). It might also be possible to coerce something like libffi into doing what we need it to do. I'm not sure how well "hard switching" would play with an operating system that is very protective of its stack, like OpenBSD or future versions of Windows, though. I'm also not sure how it could "break C++ exception handling" in scenarios where it wouldn't be broken anyway. I'd like to see a test of that that passes under regular Python but fails in Stackless for this. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2357 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040531/e89b3b0e/smime.bin From python at rcn.com Mon May 31 04:17:42 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon May 31 16:17:54 2004 Subject: [Python-Dev] Anyone object to a rewrite of site.py? In-Reply-To: <40BB8A51.2080105@ocf.berkeley.edu> Message-ID: <003d01c446e7$bef60380$e841fea9@oemcomputer> > Patch 921793 (http://www.python.org/sf/921793) has the patch of my > rewrite. I haven't looked at the patch yet but am +1 on the concept of refactoring it. The current version is somewhat of a mess. > I have been using the rewritten version since > PyCON (so about two months) and I haven't had a bug yet. I'll run with it for a while and see if I get the same results. Raymond From martin at v.loewis.de Mon May 31 16:56:55 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 16:57:00 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> Message-ID: <40BB9C17.20500@v.loewis.de> Bob Ippolito wrote: > I'm not quite sure you are up to date on how Stackless 3.0 works. The > assembly code is only there to facilitate "hard switching", which is > only required to do tasklet switching in the face of some C extension > that doesn't know about Stackless. Yes, that was my understanding. "Some C extensions" includes Tcl and COM, right? Unless I'm mistaken, it also includes map, filter, and .encode/.decode of unicode objects (or any other C module that calls back into Python, except for tasklet mechanism of stackless itself). In these cases, copying out the stack will break the exception chain that the Visual C++ compiler creates through the FS segment, so that an attempt to throw an exception will cause a program crash, right? > All pure > Python code and almost all code built-in to Python goes through "soft > switching", which is implemented in pure C with no stack tricks or > anything. What is the control flow for, say, the builtin map() function? > Stackless would still be a very useful addition to Python > even when "hard switching" is not available. That is the point I made at the beginning of this thread: Python is currently not stackless because nobody has contributed a patch recently to make it so. A patch that takes recursion out of eval_frame might alone be valuable, but hasn't been contributed. > Alternatively, Armin believes that Stackless "hard switching" can > actually be implemented in mostly-platform-neutral C (either by > setjmp/longjmp or alloca hacks, I forget which) It is setjmp/longjmp. I have implemented a coroutine library on top of setjmp/longjmp myself, so I am well aware of the implications. > I'm also not sure how it could "break C++ exception handling" in > scenarios where it wouldn't be broken anyway. I'd like to see a test of > that that passes under regular Python but fails in Stackless for this. The VC++ exception mechanism puts a linked chain of exception frames on the stack, where, at the beginning of each function, the stack pointer is stored at FS:18 (or some such), and the old value of FS:18 is stored on the stack. Then, when an exception occurs, this linked list is used to traverse the stack. If multiple stacks are used in a single thread, then this linked list gets messed up. Without trying, I think the following call chain might cause a crash if Stackless is compiled with exception handling on: Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> C-func2() -> throw exception() Regards, Martin From bob at redivi.com Mon May 31 17:10:39 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon May 31 17:10:49 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BB9C17.20500@v.loewis.de> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> Message-ID: On May 31, 2004, at 4:56 PM, Martin v. L?wis wrote: > Bob Ippolito wrote: >> I'm not quite sure you are up to date on how Stackless 3.0 works. >> The assembly code is only there to facilitate "hard switching", which >> is only required to do tasklet switching in the face of some C >> extension that doesn't know about Stackless. > > Yes, that was my understanding. "Some C extensions" includes Tcl and > COM, right? Unless I'm mistaken, it also includes map, filter, and > .encode/.decode of unicode objects (or any other C module that calls > back into Python, except for tasklet mechanism of stackless itself). You're definitely correct about Tcl and COM.. COM is however a total non-issue because Win32/x86 is very well supported by Christian ;) I'm believe that map and filter are non-recursive, I am not sure about unicode encode/decode. I don't think it's very common to do a tasklet switch in the middle of a unicode encode or decode, though :) > In these cases, copying out the stack will break the exception chain > that the Visual C++ compiler creates through the FS segment, so that > an attempt to throw an exception will cause a program crash, right? >> I'm also not sure how it could "break C++ exception handling" in >> scenarios where it wouldn't be broken anyway. I'd like to see a test >> of that that passes under regular Python but fails in Stackless for >> this. > > The VC++ exception mechanism puts a linked chain of exception frames > on the stack, where, at the beginning of each function, the stack > pointer is stored at FS:18 (or some such), and the old value of FS:18 > is stored on the stack. Then, when an exception occurs, this linked > list is used to traverse the stack. If multiple stacks are used > in a single thread, then this linked list gets messed up. > > Without trying, I think the following call chain might cause a crash > if Stackless is compiled with exception handling on: > > Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet > Py-func3() -> C-func2() -> throw exception() I don't see how it could cause problems unless this is valid: Py_Func1 -> CPlusPlus_ExceptionCatcher -> Py_Func2 -> CPlusPlus_ExceptionThrower -> throws exception() If that *is* valid, how the heck does Py_Func2 get cleaned up? I have to admit that I'm not very familiar with C++'s implementation of exceptions. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2357 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040531/06835d10/smime.bin From martin at v.loewis.de Mon May 31 17:25:12 2004 From: martin at v.loewis.de (=?ISO-8859-1?Q?=22Martin_v=2E_L=F6wis=22?=) Date: Mon May 31 17:25:16 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> Message-ID: <40BBA2B8.7000907@v.loewis.de> Bob Ippolito wrote: > I'm believe that map and filter are non-recursive Can you point me to the code that demonstrates this? Looking at Stackless' src/Python/bltinmodule.c:builtin_map, I see for (i = 0; ; ++i) { ... value = PyEval_CallObject(func, alist); ... PyList_SetItem(result, i, value) ... } which looks recursive to me. > I am not sure about > unicode encode/decode. I don't think it's very common to do a tasklet > switch in the middle of a unicode encode or decode, though :) I thought Stackless was not only about tasklet switching, but also about running without stack... Wouldn't it perform the "hard" switching if the stack is running too deep? >> Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet >> Py-func3() -> C-func2() -> throw exception() > > > I don't see how it could cause problems unless this is valid: > Py_Func1 -> CPlusPlus_ExceptionCatcher -> Py_Func2 -> > CPlusPlus_ExceptionThrower -> throws exception() It crashes in the process of throwing the exception - not because it finds no exception handler. It crashes because the internal data structures have been corrupted. I should have better made this example: Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet Py-func3() -> Exception-Catcher -> C-func2() -> throw exception() > If that *is* valid, how the heck does Py_Func2 get cleaned up? I have > to admit that I'm not very familiar with C++'s implementation of > exceptions. Throwing exceptions through Python functions is not supported, indeed. However, I was trying to point out that the entire exception handling mechanism gets corrupted, even if there are valid handlers on the stack. Regards, Martin From bob at redivi.com Mon May 31 17:39:01 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon May 31 17:39:09 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BBA2B8.7000907@v.loewis.de> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> <40BBA2B8.7000907@v.loewis.de> Message-ID: On May 31, 2004, at 5:25 PM, Martin v. L?wis wrote: > Bob Ippolito wrote: > >> I'm believe that map and filter are non-recursive > > Can you point me to the code that demonstrates this? Looking at > Stackless' src/Python/bltinmodule.c:builtin_map, I see > > for (i = 0; ; ++i) { > ... > value = PyEval_CallObject(func, alist); > ... > PyList_SetItem(result, i, value) > ... > } > > which looks recursive to me. Ok, then I am mistaken. I thought I had seen a commit message that said it was made non-recursive. Either way, it's possible to write a non-recursive version of map or anything else, it's just a bit of work. >> I am not sure about unicode encode/decode. I don't think it's very >> common to do a tasklet switch in the middle of a unicode encode or >> decode, though :) > > I thought Stackless was not only about tasklet switching, but also > about running without stack... Wouldn't it perform the "hard" switching > if the stack is running too deep? I don't think so, because the C stack should never run too deep, since most code that is going to recurse (from the python-view) will do it non-recursively (from the C-view). >>> Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet >>> Py-func3() -> C-func2() -> throw exception() >> I don't see how it could cause problems unless this is valid: >> Py_Func1 -> CPlusPlus_ExceptionCatcher -> Py_Func2 -> >> CPlusPlus_ExceptionThrower -> throws exception() > > It crashes in the process of throwing the exception - not because > it finds no exception handler. It crashes because the internal > data structures have been corrupted. > > I should have better made this example: > > Py-func1() -> C-func1() -> Py-func2() -> switch to new tasklet > Py-func3() -> Exception-Catcher -> C-func2() -> throw > exception() > >> If that *is* valid, how the heck does Py_Func2 get cleaned up? I >> have to admit that I'm not very familiar with C++'s implementation of >> exceptions. > > Throwing exceptions through Python functions is not supported, indeed. > However, I was trying to point out that the entire exception handling > mechanism gets corrupted, even if there are valid handlers on the > stack. I don't think that the entire exception handling mechanism gets corrupted. Each tasklet should always have a consistent view of the C stack. Anyway, I don't think we need to discuss it any further unless someone writes a failing test... and if that is indeed the case, then the discussion should be taken to the stackless list. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2357 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040531/9544084b/smime.bin From guido at python.org Mon May 31 18:37:26 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 31 18:37:33 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: Your message of "Mon, 31 May 2004 11:34:19 MDT." <40BB6C9B.4040701@ieee.org> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> Message-ID: <200405312237.i4VMbQA21697@guido.python.org> > Does this mean that the implementation of Stackless 3.0 could be > acceptable to integrate into CPython 2.5? It would be absolutely > wonderful to have those capabilities available in the standard > distribution! Let's not get too excited about this possibility. Before something is accepted into the language its semantics must be specified exactly and clearly. Last time this was proposed, the Stackless semantics were extremely murky, with lots of implementation-specific exceptions. Another requirement is that other Python implementations, like Jython and IronPython (Jim Hugunin's C# port) can also support these semantics. Until these conditions are met, Stackless is stuck in the fringe. --Guido van Rossum (home page: http://www.python.org/~guido/) From jjl at pobox.com Mon May 31 19:24:06 2004 From: jjl at pobox.com (John J Lee) Date: Mon May 31 19:24:14 2004 Subject: [Python-Dev] cookielib In-Reply-To: <40BB7BA6.1040007@v.loewis.de> References: <40BB7BA6.1040007@v.loewis.de> Message-ID: On Mon, 31 May 2004, [ISO-8859-1] "Martin v. L=F6wis" wrote: > John J Lee wrote: > > What next? > > I have applied the patch. Now, when I run regrtest, I get [...snip debug output...] > > Can you provide a patch to suppress this output? Whoops, I should never have had a Logger.addHandler() call in there, should I? I've attached a patch to the original SF patch. John From tismer at stackless.com Mon May 31 19:49:34 2004 From: tismer at stackless.com (Christian Tismer) Date: Mon May 31 19:47:59 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <200405302204.i4UM4hb19989@guido.python.org> References: <000001c4461b$1fa22d80$e841fea9@oemcomputer> <200405302204.i4UM4hb19989@guido.python.org> Message-ID: <40BBC48E.6030405@stackless.com> Guido van Rossum wrote: >>For strings at least, perhaps it is time to bite the bullet and >>include weak reference support directly. Weak reference support ups >>the per string memory overhead from five words (ob_type, ob_refcnt, >>ob_size, ob_shash, ob_sstate) to six. The whole concept of weak >>dictionaries is much more useful when strings can be used as keys >>and/or values. > > > Hmm... it is a high price to pay to add another word (*and* some extra > code at dealloc time!) to every string object when very few apps need > them and strings are about the most common data type. And since > they're immutable, what's the point of having weak refs to strings in > the first place? (Note that the original poster asked about > *subclasses* of strings.) Same here. I wouldnot vote to make strings or tuples or any other tiny type weak-reffed in the first place. Instead I would add the possible support to derived types, via the __slot__ mechanism for instance. There is a little coding necessary to make the generic code handle the case of var-sized objects, but this is doable and not very complicated. This may be really needed or not. if we can create the rule "every derived type *can* have weak-refs", this is simpler to memorize than "well, most can, some cannot". cheers -- chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Mon May 31 20:09:08 2004 From: tismer at stackless.com (Christian Tismer) Date: Mon May 31 20:07:36 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BBA2B8.7000907@v.loewis.de> References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> <40BBA2B8.7000907@v.loewis.de> Message-ID: <40BBC924.6000803@stackless.com> Martin v. L?wis wrote: > Bob Ippolito wrote: > >> I'm believe that map and filter are non-recursive They have been. They can be, again, but why. > Can you point me to the code that demonstrates this? Looking at > Stackless' src/Python/bltinmodule.c:builtin_map, I see > > for (i = 0; ; ++i) { > ... > value = PyEval_CallObject(func, alist); > ... > PyList_SetItem(result, i, value) > ... > } > > which looks recursive to me. Yes. trying to do a minimalistic approach, I dropped all non-recursive implementations of non-trivial things (meaning non-tail-recursive things) which were not absolutely necessary. And since we have list comprehensions, map is no longer worth being supported so much. >> I am not sure about unicode encode/decode. I don't think it's very >> common to do a tasklet switch in the middle of a unicode encode or >> decode, though :) > > > I thought Stackless was not only about tasklet switching, but also > about running without stack... Wouldn't it perform the "hard" switching > if the stack is running too deep? Yes, it would, if you allow it to do so. This would happen after a very deep recursion, which you wouldn't be able to do with normal Python at all. ... > It crashes in the process of throwing the exception - not because > it finds no exception handler. It crashes because the internal > data structures have been corrupted. I don't know the C++ exception mechanism. It it walks the whole stack once before taking an action, then this is a problem. If it does proper unwinding of C frames, I could probably play the game and restore things just in time. Anyway, I don't really get the point. 95% of Stackless is doing soft-switched stackless calls. Behavior is completely controllable. We can easily avoid any special C stack operation, by setting a flag that disallows it (easy to implement) or by excluding the hard switching stuff, completely (not an option now, but easy, too). The discussion, as often before, tries to find out why it is *not* possible to include Stackless, instead of finding out the useful parts which could be supported with relatively small impact. This is why Stackless exists, and will continue to exist. For me, this thread is closed -- thanks - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tismer at stackless.com Mon May 31 20:16:52 2004 From: tismer at stackless.com (Christian Tismer) Date: Mon May 31 20:15:25 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: References: <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> <40BBA2B8.7000907@v.loewis.de> Message-ID: <40BBCAF4.7050801@stackless.com> Bob Ippolito wrote: oops, ok, this need some words, still. ... > Ok, then I am mistaken. I thought I had seen a commit message that said > it was made non-recursive. Either way, it's possible to write a > non-recursive version of map or anything else, it's just a bit of work. I had that in 1.0, and I don't plan to do it again, although it is simple, now. ... > I don't think so, because the C stack should never run too deep, since > most code that is going to recurse (from the python-view) will do it > non-recursively (from the C-view). You can create such situations. Until Stackless 3.0, it was possible to do very deep explicit calls to the __call__ wrapper, for instance, and wrappers were not supported. Stackless 3.1 which is about to be announced has full support for all slots and all methods by the machinery. Supported are all tail-recursive calls (like __call__), because this is very cheap to do. But it is possible to implement as many of special method handlers and wrapper handlers as one wishes; I have the infrastructure ready for this. Anyway, for this rare artificial case, I have a check built in that measures stack size instead of recursion counting, and it will save the stack if necessary. Of course this is platform dependent, but my, if we switch this off for non-supported platforms, what do we loose? [C++ exception issues] > I don't think that the entire exception handling mechanism gets > corrupted. Each tasklet should always have a consistent view of the C > stack. Anyway, I don't think we need to discuss it any further unless > someone writes a failing test... and if that is indeed the case, then > the discussion should be taken to the stackless list. Good idea, see you there! cheers - chris -- Christian Tismer :^) Mission Impossible 5oftware : Have a break! Take a ride on Python's Johannes-Niemeyer-Weg 9a : *Starship* http://starship.python.net/ 14109 Berlin : PGP key -> http://wwwkeys.pgp.net/ work +49 30 89 09 53 34 home +49 30 802 86 56 mobile +49 173 24 18 776 PGP 0x57F3BF04 9064 F4E1 D754 C2FF 1619 305B C09C 5A3B 57F3 BF04 whom do you want to sponsor today? http://www.stackless.com/ From tim.one at comcast.net Mon May 31 20:25:40 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon May 31 20:25:48 2004 Subject: [Python-Dev] doctest improvements In-Reply-To: <40BB7414.9030304@gradient.cis.upenn.edu> Message-ID: [Edward Loper] > About a month and a half ago, I filed several patches for doctest, and > they haven't gotten any comments; so I was wondering if anyone could look > at them. Jim Fulton and I intend to do some doctest work for 2.4, so feel free to assign them to me. I can't make time for it immediately, though. > The most interesting is #933238, which addresses the often-noted problem > that doctest can't handle blank lines. It adds a special blank-line > marker (currently "-"), which must be *dedented* with respect to the > doctest block. Fair warning that any gimmick weakening doctest's WYSIWYG guarantee is going to be a hard sell if I'm the reviewer. The business about blank lines is a documented limitation of the design. Limitations can indeed create problems, but removing them can create other problems. Making a doctest reader learn a special "doctest markup language" is A Problem in this case. Offhand, I'd be happier with an almost-self-evident (say) marker than with a cryptic _ "you have no hope of guessing what this means unless you scour the docs" marker. Dedenting is a clever idea, BTW. From pje at telecommunity.com Mon May 31 20:37:09 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon May 31 20:35:41 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BBC924.6000803@stackless.com> References: <40BBA2B8.7000907@v.loewis.de> <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> <40BBA2B8.7000907@v.loewis.de> Message-ID: <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> At 02:09 AM 6/1/04 +0200, Christian Tismer wrote: >Anyway, I don't really get the point. >95% of Stackless is doing soft-switched stackless calls. >Behavior is completely controllable. We can easily avoid >any special C stack operation, by setting a flag that >disallows it (easy to implement) or by excluding the hard >switching stuff, completely (not an option now, but easy, too). If soft-switching is portable (i.e. pure C, no assembly), and is exposed as a library module (so that Jython et al can avoid supporting it), then perhaps a PEP for adding that functionality to mainstream Python would be meaningful. If that gets in, then the other 5% can always sneak in later via feature creep. ;) Or, more importantly (if I understand correctly), it could be separately distributed as an add-on for platforms that can support it. From edloper at gradient.cis.upenn.edu Mon May 31 20:41:44 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Mon May 31 20:41:01 2004 Subject: [Python-Dev] doctest improvements In-Reply-To: <200406010025.i510PjaN021251@gradient.cis.upenn.edu> References: <200406010025.i510PjaN021251@gradient.cis.upenn.edu> Message-ID: <40BBD0C8.206@gradient.cis.upenn.edu> Tim Peters wrote: > [...] Making a doctest > reader learn a special "doctest markup language" is A Problem in this case. > Offhand, I'd be happier with an almost-self-evident (say) > > Perhaps you'd be happier with a different value of BLANKLINE_MARKER, then? E.g. "blank>": def test(): r""" A different marker: >>> print 'a\n\nb' a blank> b """ Although this means that you have to indent the doctest quite far. Perhaps it would be better if I changed the patch so the marker doesn't need to dedented all the way? E.g.: def test(): r""" A different marker: >>> print 'a\n\nb' a b """ Another option to consider might be: def test(): r""" A different marker: >>> print 'a\n\nb' | a | | b """ I.e., the user can optionally preceed each line with a "|" (or some other marker), dedented past the prompt, to mark the extent of the output. I don't think this looks as wysiwyg, though. -Edward From bob at redivi.com Mon May 31 20:49:33 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon May 31 20:49:41 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> References: <40BBA2B8.7000907@v.loewis.de> <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> <40BBA2B8.7000907@v.loewis.de> <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> Message-ID: <8C2776D1-B365-11D8-9388-000A95686CD8@redivi.com> On May 31, 2004, at 8:37 PM, Phillip J. Eby wrote: > At 02:09 AM 6/1/04 +0200, Christian Tismer wrote: >> Anyway, I don't really get the point. >> 95% of Stackless is doing soft-switched stackless calls. >> Behavior is completely controllable. We can easily avoid >> any special C stack operation, by setting a flag that >> disallows it (easy to implement) or by excluding the hard >> switching stuff, completely (not an option now, but easy, too). > > If soft-switching is portable (i.e. pure C, no assembly), and is > exposed as a library module (so that Jython et al can avoid supporting > it), then perhaps a PEP for adding that functionality to mainstream > Python would be meaningful. Soft switching needs to be implemented in a few key places of the interpreter itself or else Stackless would surely have been maintained as an extension module. It is already pure C, no assembly or platform specific code. Supporting the interface from Jython or IronPython should be possible, though at worst case each tasklet might actually be a new thread so it might not be terribly efficient... though it would work. > If that gets in, then the other 5% can always sneak in later via > feature creep. ;) Or, more importantly (if I understand correctly), > it could be separately distributed as an add-on for platforms that can > support it. Hard switching could be an add-on for platforms that support it.. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2357 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040531/4ed2b745/smime.bin From edloper at gradient.cis.upenn.edu Mon May 31 20:51:16 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Mon May 31 20:49:59 2004 Subject: [Python-Dev] anonymous cvs checkout Message-ID: <40BBD304.9000904@gradient.cis.upenn.edu> I'm having trouble checking out an anonymous copy of the Python CVS repository. I tried the instructions from both the developer FAQ and the SF CVS page; and tried checking it out from debian, win2k, and OS X. In all cases, I get an error that looks something like this: cvs server: cannot read file in mydbm_load_file: Input/output error cvs checkout: in directory .: cvs [checkout aborted]: *PANIC* administration files missing A google search for "mydbm_load_file" didn't turn up anything useful. Anyone have an idea what's going wrong? Thanks! -Edward (Here's a copy-paste of my shell session from debian, in case I'm doing something stupid:) > [syse:~/data/programming] export CVSROOT=:pserver:anonymous@cvs.sourceforge.net:/cvsroot/python > [syse:~/data/programming] cvs login > Logging in to :pserver:anonymous@cvs.sourceforge.net:2401/cvsroot/python > CVS password: > [syse:~/data/programming] cvs -z3 co python > cvs server: cannot read file in mydbm_load_file: Input/output error > cvs checkout: cannot read file in mydbm_load_file: Input/output error > cvs checkout: warning: cannot write to history file /cvsroot/python/CVSROOT/history: Read-only file system > cvs checkout: Updating python > cvs checkout: failed to create lock directory for `/cvsroot/python/python' (/cvsroot/python/python/#cvs.lock): Read-only file system > cvs checkout: failed to obtain dir lock in repository `/cvsroot/python/python' > cvs [checkout aborted]: read lock failed - giving up From python at rcn.com Mon May 31 08:50:25 2004 From: python at rcn.com (Raymond Hettinger) Date: Mon May 31 20:50:35 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: <40BBC48E.6030405@stackless.com> Message-ID: <001e01c4470d$d814b840$e841fea9@oemcomputer> > > Hmm... it is a high price to pay to add another word (*and* some extra > > code at dealloc time!) to every string object when very few apps need > > them and strings are about the most common data type. And since > > they're immutable, what's the point of having weak refs to strings in > > the first place? (Note that the original poster asked about > > *subclasses* of strings.) > > Same here. I wouldnot vote to make strings or tuples or any other > tiny type weak-reffed in the first place. > Instead I would add the possible support to derived types, via > the __slot__ mechanism for instance. > There is a little coding necessary to make the generic code > handle the case of var-sized objects, but this is doable > and not very complicated. Right. I think this is all that is needed at this point. That way, the granular types stay granular and the added functionality is available via subclasses if needed. Raymond From lin at centurysoftware.com.my Mon May 31 21:08:51 2004 From: lin at centurysoftware.com.my (Norlin Abd Rahim) Date: Mon May 31 21:10:55 2004 Subject: [Python-Dev] rounding problem Message-ID: <40BBD723.000003.01168@Norlin2> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 494 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040601/257d74ea/attachment.gif From aahz at pythoncraft.com Mon May 31 21:17:55 2004 From: aahz at pythoncraft.com (Aahz) Date: Mon May 31 21:18:00 2004 Subject: [Python-Dev] Uninformative error message about subclassing In-Reply-To: <40BB3D8D.4090809@erols.com> References: <40BB3D8D.4090809@erols.com> Message-ID: <20040601011755.GA20900@panix.com> On Mon, May 31, 2004, Edward C. Jones wrote: > > ---- > a.py: > > class A(object): > pass > > ---- > b.py: > > import a > > class B(a): > pass > ---- > If I run "python b.py" I get the uninformative error message: > > Traceback (most recent call last): > File "./b.py", line 5, in ? > class B(a): > TypeError: function takes at most 2 arguments (3 given) > > I think the message should say that "a" is not a class. Please file a bug report on SF. However, I suspect this will be tricky to solve (if it's even possible), because of the way builtin objects are now part of new-style classes. (That A is a new-style class is irrelevant.) -- Aahz (aahz@pythoncraft.com) <*> http://www.pythoncraft.com/ "as long as we like the same operating system, things are cool." --piranha From gherron at islandtraining.com Mon May 31 21:19:00 2004 From: gherron at islandtraining.com (Gary Herron) Date: Mon May 31 21:19:40 2004 Subject: [Python-Dev] rounding problem In-Reply-To: <40BBD723.000003.01168@Norlin2> References: <40BBD723.000003.01168@Norlin2> Message-ID: <200405311819.01475.gherron@islandtraining.com> On Monday 31 May 2004 06:08 pm, Norlin Abd Rahim wrote: > Hi, > > I'm using python with zope. I used _.round and it seems like it is look > unstable. For example if i have one number, 1130.975 and rounded it to 2 > decimal places, it will be 1130.97 and not 1130.98. But for certain numbers > it will rounded correctly if the last digit is 5. I even used fixedpoint.py > but still i've got the same problem. This applies only to numbers that have > digit 5 after the decimal places that needs to be rounded. > > Any idea why? > > TIA Perhaps the last digit is 5 only because the printing rounded ...4999... or some such up to 5. Unless you KNOW that the number is exactly 1130.975 (in its internal representation) then you must consider this as a possibility. Gary Herron From lin at centurysoftware.com.my Mon May 31 21:22:20 2004 From: lin at centurysoftware.com.my (Norlin Abd Rahim) Date: Mon May 31 21:24:34 2004 Subject: [Python-Dev] rounding problem References: <200405311819.01475.gherron@islandtraining.com> Message-ID: <40BBDA4C.000001.01168@Norlin2> Skipped content of type multipart/alternative-------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: image/gif Size: 494 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040601/2172ac9a/attachment-0001.gif From greg at cosc.canterbury.ac.nz Mon May 31 21:37:57 2004 From: greg at cosc.canterbury.ac.nz (Greg Ewing) Date: Mon May 31 21:38:17 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <40BB8882.60503@v.loewis.de> Message-ID: <200406010137.i511bvnA013748@cosc353.cosc.canterbury.ac.nz> Martin: > [In addition, the approach taken by Stackless 3.0 is probably > questionable: for example, it may be that C++ exception handling > stops working in Stackless.] Indeed. The last time this was discussed, I seem to remember Stackless had trouble with C code that passed pointers to local variables as parameters. Since this is a perfectly legitimate thing to do in C, if Stackless can't handle it, then it's fundamentally flawed. Greg Ewing, Computer Science Dept, +--------------------------------------+ University of Canterbury, | A citizen of NewZealandCorp, a | Christchurch, New Zealand | wholly-owned subsidiary of USA Inc. | greg@cosc.canterbury.ac.nz +--------------------------------------+ From gherron at islandtraining.com Mon May 31 21:39:20 2004 From: gherron at islandtraining.com (Gary Herron) Date: Mon May 31 21:39:33 2004 Subject: [Python-Dev] rounding problem In-Reply-To: <40BBDA4C.000001.01168@Norlin2> References: <200405311819.01475.gherron@islandtraining.com> <40BBDA4C.000001.01168@Norlin2> Message-ID: <200405311839.20514.gherron@islandtraining.com> On Monday 31 May 2004 06:22 pm, Norlin Abd Rahim wrote: > the number actually comes from multiplication between 237.50 and 4.762 > which will get 1130.975. So it suppose to be 1130.98 isnt it if it'll get > rounded to 2 decimal places. But... Due to the wonders and inaccuracies of binary floating point representations on digital computers, 4.762 can not be represented EXACTLY. The closest it can get is 4.7619999999999996, and that's where you're inaccuracy comes from. In the following interactive python session, the values printed out are printed out to their full precision (that's a often misunderstood feature of the python interactive mode). You can see where the floating point inaccuracies are introduced. >>> 237.50 237.5 >>> 4.762 4.7619999999999996 >>> 237.50 * 4.762 1130.9749999999999 >>> Just for clarification .. This is NOT a problem with Python, but is a feature of the binary representation of floating point numbers and can be found on all computers and in all programming languages. Gary Herron From tim.one at comcast.net Mon May 31 21:40:14 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon May 31 21:40:25 2004 Subject: [Python-Dev] rounding problem In-Reply-To: <40BBDA4C.000001.01168@Norlin2> Message-ID: [Norlin Abd Rahim] > the number actually comes from multiplication between 237.50 and 4.762 > which will get 1130.975. Please read the Appendix on floating point issues in the Python Tutorial: http://docs.python.org/tut/node15.html The actual fp product of those two numbers is strictly less than 1130.975. You shouldn't have any such problem with FixedPoint if you're using it correctly, but python-dev isn't the right place for this (so if you want more on this, please use the newsgroup comp.lang.python). From t-meyer at ihug.co.nz Mon May 31 21:41:27 2004 From: t-meyer at ihug.co.nz (Tony Meyer) Date: Mon May 31 21:41:47 2004 Subject: [Python-Dev] rounding problem In-Reply-To: <1ED4ECF91CDED24C8D012BCF2B034F1306966EEF@its-xchg4.massey.ac.nz> Message-ID: <1ED4ECF91CDED24C8D012BCF2B034F13064C0265@its-xchg4.massey.ac.nz> [I think this would be better in python-list, not python-dev] > the number actually comes from multiplication between > 237.50 and 4.762 which will get 1130.975. So it suppose > to be 1130.98 isnt it if it'll get rounded to 2 decimal > places. Gary is correct: >>> 237.50 * 4.762 1130.9749999999999 >>> round(_,2) 1130.97 >>> print 237.50 * 4.762 1130.975 The answer correctly rounds to 1130.97 because the next digit is 4, not 5. When you print it, it rounds to 3 decimal places, so you get the .9749 rounding to .975. This is just floating point arithmetic. You might want to try the Decimal module, or something like that, to avoid this. =Tony Meyer From tim.one at comcast.net Mon May 31 22:00:17 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon May 31 22:00:32 2004 Subject: [Python-Dev] doctest improvements In-Reply-To: <40BBD0C8.206@gradient.cis.upenn.edu> Message-ID: [Edward Loper] > Perhaps you'd be happier with a different value of BLANKLINE_MARKER, > then? E.g. "blank>": > > def test(): > r""" > A different marker: > > >>> print 'a\n\nb' > a > blank> > b > """ > > Although this means that you have to indent the doctest quite far. > Perhaps it would be better if I changed the patch so the marker doesn't > need to dedented all the way? E.g.: > > def test(): > r""" > A different marker: > > >>> print 'a\n\nb' > a > > b > """ That's more what I had in mind, but if the marker is changed to something wordy instead of "a magic character", I don't think the dedent trick is of much value anymore: r""" A different marker: >>> print 'a\n\nb' a b """ Then when the expected output is (exactly) "", doctest would accept "", or a line with nothing other than whitespace, as a match. I can't get upset by that bit of ambiguity. The primary advantage is that the implementation is then trivial, to write, document, and understand. It won't break external tools, and caters to leading, trailing, and embedded blank lines. It needs an option flag to turn it off, akin to the current doctest.DONT_ACCEPT_TRUE_FOR_1 (the only non-wysiwyg gimmick implemented now); it should be enabled by default. > Another option to consider might be: > > def test(): > r""" > A different marker: > > >>> print 'a\n\nb' >> | a >> | >> | b > """ > > I.e., the user can optionally preceed each line with a "|" (or some other > marker), dedented past the prompt, to mark the extent of the output. I > don't think this looks as wysiwyg, though. Nope, it sure doesn't. From pje at telecommunity.com Mon May 31 22:51:39 2004 From: pje at telecommunity.com (Phillip J. Eby) Date: Mon May 31 22:50:26 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <8C2776D1-B365-11D8-9388-000A95686CD8@redivi.com> References: <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> <40BBA2B8.7000907@v.loewis.de> <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> <40BBA2B8.7000907@v.loewis.de> <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> Message-ID: <5.1.1.6.0.20040531222355.02f77ec0@mail.telecommunity.com> At 08:49 PM 5/31/04 -0400, Bob Ippolito wrote: >On May 31, 2004, at 8:37 PM, Phillip J. Eby wrote: > >>At 02:09 AM 6/1/04 +0200, Christian Tismer wrote: >>>Anyway, I don't really get the point. >>>95% of Stackless is doing soft-switched stackless calls. >>>Behavior is completely controllable. We can easily avoid >>>any special C stack operation, by setting a flag that >>>disallows it (easy to implement) or by excluding the hard >>>switching stuff, completely (not an option now, but easy, too). >> >>If soft-switching is portable (i.e. pure C, no assembly), and is exposed >>as a library module (so that Jython et al can avoid supporting it), then >>perhaps a PEP for adding that functionality to mainstream Python would be >>meaningful. > >Soft switching needs to be implemented in a few key places of the >interpreter itself or else Stackless would surely have been maintained as >an extension module. I'm aware of this, which is why I said "exposed" as a library module, not "implemented" as one. :) > It is already pure C, no assembly or platform specific > code. Supporting the interface from Jython or IronPython should be > possible, though at worst case each tasklet might actually be a new > thread so it might not be terribly efficient... though it would work. Yes, I suppose in the simplest case one could implement switching primitives via a simple mutex: acquire it when resuming, release it when yielding. If the PEP described the semantics of tasklets in terms of threads, then a heavyweight implementation could be achieved. However, it seems to me that any Python implementation that supports generators should in principle be able to support co-operative multitasking anyway, so long as there is no C code in the call chain. Since microthreads are basically doable in pure Python 2.2 as long as you write *everything* as a generator and simulate your control stack, it seems you should be able to do this in the Python interpreter. Actually, it seems to me that if there were a way to resume execution of a frame that has thrown a special "task switching" exception (sort of a non-local 'yield' operation), you could implement lightweight threading *without* having to de-stackify Python. Of course, it's possible that the implementation of such a strategy might be just as complex as de-stacking Python. To make it work, I think you'd need a way to tell a nested execution of Python code that it's being called from Python code (as opposed to C), so that it knows where to "resume to". A chain of resumable frames could then be resumed by a special function. Then, upon catching an exception that didn't unwind via any C code, you can call a function to resume the interrupted frame (and its parents). Hm, actually, you'd have to copy the frames in order to be able to resume them, so you'd need to have a designated exception type for resumable exceptions, so as not to waste time copying frames for other kinds of errors. Okay, I think I just convinced myself that it would be better to just make the interpreter core stackless instead of trying to design all that other crud. :) (To clarify, that's not an unqualified +1 to some hypothetical stacklessizing. I'm just saying that if supporting ultra-light threading in Python is desirable, stacklessness seems at first glance like a promising way to do it, compared to trying to deal with unwinding the C stack. And, to top it off, I think you'll end up doing half of what Stackless already does in order to "resume" the interrupted frames anyhow.) From bob at redivi.com Mon May 31 23:00:29 2004 From: bob at redivi.com (Bob Ippolito) Date: Mon May 31 23:00:39 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: <5.1.1.6.0.20040531222355.02f77ec0@mail.telecommunity.com> References: <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> <40BBA2B8.7000907@v.loewis.de> <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> <40BBA2B8.7000907@v.loewis.de> <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> <5.1.1.6.0.20040531222355.02f77ec0@mail.telecommunity.com> Message-ID: On May 31, 2004, at 10:51 PM, Phillip J. Eby wrote: > At 08:49 PM 5/31/04 -0400, Bob Ippolito wrote: >> On May 31, 2004, at 8:37 PM, Phillip J. Eby wrote: >> >>> At 02:09 AM 6/1/04 +0200, Christian Tismer wrote: >>>> Anyway, I don't really get the point. >>>> 95% of Stackless is doing soft-switched stackless calls. >>>> Behavior is completely controllable. We can easily avoid >>>> any special C stack operation, by setting a flag that >>>> disallows it (easy to implement) or by excluding the hard >>>> switching stuff, completely (not an option now, but easy, too). >>> >>> If soft-switching is portable (i.e. pure C, no assembly), and is >>> exposed as a library module (so that Jython et al can avoid >>> supporting it), then perhaps a PEP for adding that functionality to >>> mainstream Python would be meaningful. >> >> Soft switching needs to be implemented in a few key places of the >> interpreter itself or else Stackless would surely have been >> maintained as an extension module. > > I'm aware of this, which is why I said "exposed" as a library module, > not "implemented" as one. :) Oh, it *is* already exposed as a module (named stackless)... but it's *implemented* as a built-in module, like sys or os. -bob -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/pkcs7-signature Size: 2357 bytes Desc: not available Url : http://mail.python.org/pipermail/python-dev/attachments/20040531/eea2a8b0/smime.bin From edloper at gradient.cis.upenn.edu Mon May 31 23:15:16 2004 From: edloper at gradient.cis.upenn.edu (Edward Loper) Date: Mon May 31 23:14:31 2004 Subject: [Python-Dev] doctest improvements In-Reply-To: <200406010203.i5123EaN026869@gradient.cis.upenn.edu> References: <200406010203.i5123EaN026869@gradient.cis.upenn.edu> Message-ID: <40BBF4C4.2090204@gradient.cis.upenn.edu> Tim Peters wrote: > That's more what I had in mind, but if the marker is changed to something > wordy instead of "a magic character", I don't think the dedent trick is of > much value anymore: > > r""" > A different marker: > > >>> print 'a\n\nb' > a > > b > """ > > Then when the expected output is (exactly) "", doctest would > accept "", or a line with nothing other than whitespace, as a > match. I can't get upset by that bit of ambiguity. I had ruled this out because I assumed that any change I made had to maintain backward compatibility. But if this (minor) incompatibility is acceptable, then let me know and I'll write up a patch to implement it. -Edward From guido at python.org Mon May 31 23:15:08 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 31 23:15:18 2004 Subject: [Python-Dev] Why aren't more things weak referencable In-Reply-To: Your message of "Tue, 01 Jun 2004 01:49:34 +0200." <40BBC48E.6030405@stackless.com> References: <000001c4461b$1fa22d80$e841fea9@oemcomputer> <200405302204.i4UM4hb19989@guido.python.org> <40BBC48E.6030405@stackless.com> Message-ID: <200406010315.i513F8E22050@guido.python.org> > Same here. I wouldnot vote to make strings or tuples or any other > tiny type weak-reffed in the first place. > Instead I would add the possible support to derived types, via > the __slot__ mechanism for instance. > There is a little coding necessary to make the generic code > handle the case of var-sized objects, but this is doable > and not very complicated. > > This may be really needed or not. if we can create the rule > "every derived type *can* have weak-refs", this is simpler > to memorize than "well, most can, some cannot". With some (considerable?) effort, slots on var-sized objects can certainly be supported -- the same approach as used for __dict__ should work. I expect it might slow down the normal case a bit, unless you define a new descriptor type just for this. --Guido van Rossum (home page: http://www.python.org/~guido/) From guido at python.org Mon May 31 23:17:57 2004 From: guido at python.org (Guido van Rossum) Date: Mon May 31 23:18:06 2004 Subject: [Python-Dev] Stackless Python In-Reply-To: Your message of "Mon, 31 May 2004 20:37:09 EDT." <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> References: <40BBA2B8.7000907@v.loewis.de> <20040530133843.51528.qmail@web50709.mail.yahoo.com> <40BAE605.5040800@v.loewis.de> <40BB6C9B.4040701@ieee.org> <40BB7E63.9090304@v.loewis.de> <40BB8882.60503@v.loewis.de> <5611D758-B33E-11D8-9388-000A95686CD8@redivi.com> <40BB9C17.20500@v.loewis.de> <40BBA2B8.7000907@v.loewis.de> <5.1.1.6.0.20040531202832.037b7ec0@mail.telecommunity.com> Message-ID: <200406010317.i513Hvv22066@guido.python.org> > If soft-switching is portable (i.e. pure C, no assembly), and is > exposed as a library module (so that Jython et al can avoid > supporting it), then perhaps a PEP for adding that functionality to > mainstream Python would be meaningful. Anything that can be supported by a pure extension module is fair game IMO. But it was my understanding that Stackless requires changes to the VM. And I definitely don't want people to get into a habit of writing code that relies on deep recursion that is only supported by Stackless, only to find that it doesn't work in Jython etc. > If that gets in, then the other 5% can always sneak in later via > feature creep. ;) Or, more importantly (if I understand correctly), > it could be separately distributed as an add-on for platforms that > can support it. Hey, that's what Stackless already does. Why is that approach suddenly not good enough any more? --Guido van Rossum (home page: http://www.python.org/~guido/) From skip at pobox.com Mon May 31 22:01:55 2004 From: skip at pobox.com (Skip Montanaro) Date: Mon May 31 23:31:24 2004 Subject: [Python-Dev] rounding problem In-Reply-To: <40BBDA4C.000001.01168@Norlin2> References: <200405311819.01475.gherron@islandtraining.com> <40BBDA4C.000001.01168@Norlin2> Message-ID: <16571.58259.802091.570024@montanaro.dyndns.org> Norlin> the number actually comes from multiplication between 237.50 and Norlin> 4.762 which will get 1130.975. So it suppose to be 1130.98 isnt Norlin> it if it'll get rounded to 2 decimal places. Nope. 1130.975 is not exactly representable. In fact, it's internal representation is slightly less that 1130.975, so rounding to two decimal places is correctly 1130.97. % python Python 2.4a0 (#25, May 22 2004, 15:16:21) [GCC 3.3 20030304 (Apple Computer, Inc. build 1493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> 1130.975 1130.9749999999999 Skip From tim.one at comcast.net Mon May 31 23:40:38 2004 From: tim.one at comcast.net (Tim Peters) Date: Mon May 31 23:40:45 2004 Subject: [Python-Dev] doctest improvements In-Reply-To: <40BBF4C4.2090204@gradient.cis.upenn.edu> Message-ID: [Tim Peters] >> ... but if the marker is changed to something wordy instead of "a magic >> character", I don't think the dedent trick is of much value anymore: >> >> r""" >> A different marker: >> >> >>> print 'a\n\nb' >> a >> >> b >> """ >> >> Then when the expected output is (exactly) "", doctest would >> accept "", or a line with nothing other than whitespace, as >> a match. I can't get upset by that bit of ambiguity. [Edward Loper] > I had ruled this out because I assumed that any change I made had to > maintain backward compatibility. We broke that last time around, with the gimmick to accept "True" ("False") when "1" ("0") is expected. Such highly limited ambiguities are OK by me by default, when they've got strong pragmatic benefit, and there's a way to turn them off. > But if this (minor) incompatibility is acceptable, It is to me. Anyone object? The consequence is that, by default, a doctest that has as a line of its expected output will pass if it actually produces there, or if it produces an empty line there. There's no effect on doctests that currently contain an empty line as part of their expected output, because no such doctest exits (the rules made it impossible -- and still will). BTW, I liked the dedent with the one-character "_" marker because then it *screams* "something's fishy here", provoking a user to stare at the docs to figure out what that may be. Something like is so close to being self-explanatory that I don't think an extra nudge is necessary (or even helpful -- I'd find it annoying). > then let me know and I'll write up a patch to implement it. Cool! Let's wait to see if someone objects ... OK, time's up. Go for it. Edward, are you familiar with the doctest uses in Zope3? doctest is getting used heavily there, and it's turning out very well so far. Jim has ideas for making it more useful there, and if you'd like I'd be delighted to have you join us in thrashing those ideas out. This week I expect we'll be buried under making various Zope and ZODB releases, but with some luck we'll plunge into the doctest ideas next week. While the invitation is being made to you, it's not necessarily limited to you . From ta-meyer at ihug.co.nz Mon May 31 21:55:17 2004 From: ta-meyer at ihug.co.nz (Tony Meyer) Date: Tue Jun 1 12:57:19 2004 Subject: [Python-Dev] anonymous cvs checkout Message-ID: <1ED4ECF91CDED24C8D012BCF2B034F1304677EE1@its-xchg4.massey.ac.nz> [Opps. Forgot to include the list in my reply. And those that read spambayes-dev will have seen this already. Good grief.] [Edward Loper] > I'm having trouble checking out an anonymous copy of the Python CVS > repository. FWIW, neither a fresh checkout nor an update works here (WinXP), either, although (at least some) other sourceforge projects do. ViewCVS doesn't work, either (shows an empty page), so it's certainly not just you. The SF site status does mention that there's maintenance scheduled (for tomorrow) - maybe that's relevant? Does developer access still work? =Tony Meyer